A brain tumor image segmentation method based on lightweight multi-modal

Abstract

The application belongs to the technical field of image processing, and proposes a brain tumor image segmentation method based on a lightweight multi-modal, the main scheme is: acquiring a multi-modal brain magnetic resonance image dataset, and constructing a multi-modal data fusion feature extraction network based on a binary tree structure, the feature extraction network includes a tree coding layer consistent with the number of modes; constructing a local feature extractor for each mode of brain magnetic resonance image, and performing feature extraction; constructing a multi-modal fusion unit for the feature extraction results of two adjacent modes, and performing inter-modal feature fusion; generating a skip connection parameter for each tree coding layer, and performing up-sampling on the output of all tree coding layers before utilizing the skip connection parameter for splicing and fusion operation; constructing a brain tumor image segmentation network, and obtaining the segmentation result of the brain tumor image; training, verifying and testing the two networks by using the multi-modal brain magnetic resonance image dataset.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a brain tumor image segmentation method based on a lightweight multi-modal. BACKGROUND

[0002] At present, existing brain tumor segmentation networks, such as a segmentation network based on a transformer architecture and a segmentation network based on an nnU-Net structure, need good computing resources (usually more than 8G) in the training and inference processes, and have high deployment costs and hardware costs. If training on high-precision parameters is required, precision pruning training is needed, and the existing network has certain difficulties in actual application, for example, the requirement for computing power increases the difficulty of actual deployment.

[0003] Taking nnU-Net as an example, the core structure of nnU-Net (No New-Network U-Net) is based on the classic U-Net architecture. U-Net is a symmetrical encoder-decoder network composed of a down-sampling path (encoder) and an up-sampling path (decoder).

[0004] Encoder: through a series of convolution and pooling operations, the spatial size of the feature map is gradually reduced, the high-level semantic features of the image are extracted, and information of different scales is captured. For example, in the medical image segmentation task, the encoder continuously extracts the features such as the outline and texture of organs or lesions in the image through convolution operations, and the pooling operation allows the model to focus on the feature distribution in a larger range. A 3x3 convolution kernel is usually used to effectively extract features while keeping the computational amount relatively reasonable.

[0005] Decoder: using up-sampling operations (such as deconvolution, bilinear interpolation, etc.), the spatial size of the feature map is gradually restored, and the feature maps of the corresponding layers in the encoder are connected through a jump connection to fuse low-level spatial information and high-level semantic information, and finally output a segmentation result with the same size as the input image. Taking deconvolution as an example, it can map the low-resolution feature map back to high resolution, and after adding or splicing the feature map of the corresponding layer of the encoder, the model can recover the spatial details while combining the previously learned semantics, so as to better segment the target in the image.

[0006] nnU-Net automatically selects between 2D U-Net, 3D U-Net, or a combination of both depending on the specific task. 2D U-Net is suitable for processing single-slice images, has high computational efficiency, and performs well in scenarios with real-time requirements or small datasets. 3D U-Net can directly process 3D volumetric data, making full use of the spatial context information of the image, but it has a higher computational cost. When segmenting brain tumors, 3D U-Net can more accurately determine the position and shape of the tumor in three-dimensional space because it takes into account the continuity of the tumor between different slices.

[0007] Specifically, for a 3D U-Net segmentation network to achieve accurate and reliable segmentation results, the following parts of processing are generally required:

[0008] Automated preprocessing: Automatic resampling, normalization, and cropping of input data. Medical images of different modalities and resolutions are resampled to a uniform voxel spacing to ensure data consistency. For MRI images, the resolution of images acquired by different devices may vary significantly; resampling ensures the model operates on the same scale when processing data from different sources. Normalization methods (such as Z-score normalization) are used to standardize image pixel values, adjusting the mean and variance of the image to a fixed range. This accelerates model convergence and improves training efficiency. Cropping removes parts of the image irrelevant to the target segmentation region, reducing computation and avoiding interference from irrelevant background information during model training. For example, in liver segmentation tasks, a large number of non-liver regions are cropped from the image.

[0009] Automatic architecture selection: The system automatically selects a suitable U-Net architecture based on the characteristics of the dataset (such as data dimensionality and sample size). For small datasets or scenarios with high computational resource requirements, 2D U-Net may be chosen. This is because small datasets struggle to support training complex 3D models, and 2D U-Net has low computational cost, enabling rapid training and prediction with limited resources. For tasks requiring full utilization of 3D contextual information, 3D U-Net is selected. For example, in the task of segmenting cardiac vessels, 3D U-Net can more accurately segment vascular structures by analyzing the direction and connectivity of blood vessels in three-dimensional space.

[0010] Data augmentation: Various data augmentation techniques, such as random rotation, flipping, scaling, and elastic deformation, are employed to increase the diversity of training data and improve the model's generalization ability. Random rotation can simulate images of organs or lesions from different angles, allowing the model to learn the characteristics of the target from different angles and enhancing its robustness to angle changes. Flipping operations increase the diversity of samples in the horizontal or vertical directions, similar to observing a target from different perspectives. Scaling operations allow the model to adapt to changes in the size of the target. Elastic deformation simulates tissue deformation that may occur in real medical images, allowing the model to better cope with complex deformations in images.

[0011] Ensemble learning: This approach employs multi-model ensemble methods, training multiple different models (such as different U-Net architectures, different training parameters, etc.) and fusing their predictions to improve the accuracy and stability of segmentation. For example, one model might perform better in segmenting large target regions, while another model excels at segmenting small target details. By combining the predictions of multiple models through weighted averaging or voting, the strengths of each model can be fully utilized, reducing the error of a single model and resulting in a more accurate and reliable final segmentation result.

[0012] When using 3D U-Net for medical image segmentation, the following steps are generally followed:

[0013] Step 1: Data Preparation. Organize the medical image data into the format required by nnU-Net, including images and corresponding labels, and store them in designated folders. Simultaneously, define dataset information such as the number of categories and modality information. For example, in a liver tumor segmentation task, liver images and labeled tumor region images should be placed in the prescribed format, with the number of categories clearly defined as liver and tumor, and the modality information as MRI or CT, etc. Accurate and clear data preparation is the foundation for all subsequent steps and directly affects the quality of model training.

[0014] Step 2, Data Preprocessing: The system automatically performs resampling, normalization, and cropping operations on the data to prepare for subsequent training. During resampling, the optimal voxel spacing is adjusted based on dataset analysis to ensure consistency in spatial scale across different samples. During normalization, the mean and variance of the image are calculated, mapping pixel values ​​to a standard range. Cropping removes image edges irrelevant to the segmentation target based on the image's bounding box or other predefined rules. Preprocessed data allows the model to converge faster, reduces training time, and improves the model's adaptability to different datasets.

[0015] Step 3: Network Training: Based on the automatically selected network architecture, train the model using the preprocessed data. Cross-validation is employed, dividing the dataset into training and validation sets. The model is trained multiple times, and its performance is evaluated to select the optimal hyperparameters. During training, commonly used optimizers such as Adam continuously adjust the network parameters based on the model's loss function to minimize the loss. Loss functions typically include Dice Loss or Cross-Entropy Loss, tailored to the characteristics of medical image segmentation tasks, measuring the difference between the model's predictions and the true labels. Cross-validation provides a more comprehensive evaluation of the model's performance on different data subsets, avoiding overfitting and selecting optimal hyperparameters such as learning rate and number of network layers to achieve optimal model performance.

[0016] Step 4, Model Inference: Use the trained model to segment and predict new medical images. During inference, you can choose whether to use Test-Time Augmentation (TTA) to further improve prediction accuracy. TTA transforms the input image by rotating, flipping, etc., and inputs it into the model multiple times for prediction. Then, the prediction results are fused. For example, predicting three cases for a medical image to be segmented: horizontal flip, vertical flip, and no flip, and then averaging the three prediction results can effectively improve the reliability of the segmentation results.

[0017] Step 5: Post-processing: Post-process the model's prediction results, such as removing small connected regions and filling holes, to obtain more reasonable segmentation results. In medical image segmentation, model predictions may produce some isolated small regions or internal holes, which do not conform to actual medical structures. Morphological operations (such as opening operations to remove small regions and closing operations to fill holes) or connected component analysis can be used to correct the prediction results, making the segmentation results more consistent with actual medical conditions and providing more valuable information for clinical diagnosis and treatment.

[0018] In summary, existing traditional brain tumor image segmentation methods rely on cumbersome parameter adjustments and high computational deployment costs to obtain accurate and reliable segmentation results. Summary of the Invention

[0019] The purpose of this invention is to provide a lightweight multimodal brain tumor image segmentation method that can complete network deployment with relatively low computing power and achieve relatively accurate brain tumor image segmentation tasks.

[0020] The technical solution adopted by this invention to solve its technical problem is as follows:

[0021] A lightweight, multimodal brain tumor image segmentation method includes the following steps:

[0022] A multimodal brain MRI image dataset was acquired, and a multimodal data fusion feature extraction network based on a binary tree structure was constructed. The feature extraction network includes a tree coding layer consistent with the number of modalities.

[0023] A local feature extractor is constructed for each modality of brain MRI images, and the local feature extractor is used to extract features from the current modality of brain MRI images;

[0024] A multimodal fusion unit is constructed for the feature extraction results of two adjacent modalities, and the feature extraction results of the two adjacent modalities are fused by the multimodal fusion unit. The output of the current tree coding layer is formed after all intermodal features are fused.

[0025] Skip connection parameters are generated for each tree coding layer, and the outputs of all tree coding layers are upsampled before being connected and fused using the skip connection parameters.

[0026] A brain tumor image segmentation network was constructed, and the segmentation results of brain tumor images were obtained by using this segmentation network after completing the fusion operation.

[0027] We used a multimodal brain MRI image dataset to train, validate, and test a binary tree-based multimodal data fusion feature extraction network and a brain tumor image segmentation network.

[0028] As a further optimization, when the multimodal brain MRI image dataset contains brain MRI images of four modalities, the constructed multimodal data fusion feature extraction network based on a binary tree structure includes four tree coding layers.

[0029] The fourth tree coding layer takes in four modalities of brain MRI images, and after processing by the local feature extractor and the multimodal fusion unit, outputs three images after feature extraction and intermodal feature fusion to the third tree coding layer.

[0030] After being processed by the local feature extractor and the multimodal fusion unit, the third tree coding layer outputs two images that have undergone feature extraction and intermodal feature fusion to the second tree coding layer.

[0031] After being processed by the local feature extractor and the multimodal fusion unit, the second tree coding layer outputs an image that has undergone feature extraction and intermodal feature fusion to the first tree coding layer.

[0032] As a further optimization, the number of channels for each mode in the tree coding layer is consistent, and the number of mode channels in the upper tree coding layer is 1 / 2 times the number of mode channels in the lower tree coding layer.

[0033] As a further optimization, the local feature extractor includes a first convolutional layer, a second convolutional layer, and a newly added convolutional layer;

[0034] The current modality brain MRI image is sequentially passed through the first convolutional layer, the newly added convolutional layer, and the second convolutional layer to complete the feature extraction of the current modality brain MRI image;

[0035] The input and output dimensions of the newly added convolutional layer are the same as those of the first convolutional layer. The input dimension of the third convolutional layer is the same as the output dimension of the newly added convolutional layer, and the output dimension of the third convolutional layer is 1 / 2 times the input dimension.

[0036] As a further optimization, the multimodal fusion unit is a dual-modal fusion unit;

[0037] The dual-modal fusion unit includes a first-channel attention mechanism, a second-channel attention mechanism, a low-rank interaction channel attention mechanism, and a spatial attention mechanism.

[0038] The first channel attention mechanism is used to input the feature extraction result of a modality, and the second channel attention mechanism is used to input the feature extraction result of an adjacent modality.

[0039] As a further optimization, after the feature extraction results of the two adjacent modalities are fused between modal features by the dual-modal fusion unit, a feature fusion result of a single modality is generated, and the size of the feature fusion result of the single modality is consistent with the size of the feature extraction results of the two adjacent modalities.

[0040] The output of the tree coding layer is composed of the feature fusion results of all individual modalities in the current tree coding layer.

[0041] As a further optimization, the formula for the connection and fusion operation using jump connection parameters is as follows:

[0042] ,

[0043] Where A(.) represents the feature fusion function, its input dimension is n*m, n represents the nth modality, m represents the channel dimension of each single modality, and the output dimension of the feature fusion function is m.

[0044] As a further optimization, when using the multimodal fusion unit to perform intermodal feature fusion of the feature extraction results of two adjacent modalities, it includes:

[0045] The channel attention between two adjacent modalities is learned using the first-channel attention mechanism and the second-channel attention mechanism.

[0046] We utilize a low-rank interactive channel attention mechanism, employing two low-rank matrices to further learn the channel attention between two adjacent modalities;

[0047] Spatial attention learning is performed on the learning results of low-rank interactive channel attention mechanism using spatial attention mechanism.

[0048] As a further optimization, the calculation formula for the first channel attention mechanism is as follows:

[0049] ,

[0050] The calculation formula for the second channel attention mechanism is as follows:

[0051] ,

[0052] The calculation formula for the low-rank interaction channel attention mechanism is as follows:

[0053] ,

[0054] The calculation formula for the spatial attention mechanism is as follows:

[0055] ,

[0056] Where a represents the a-th mode, a+1 represents the mode adjacent to mode a, and a≤n-1.

[0057] As a further optimization, after obtaining the segmentation results of brain tumor images, and before training, validating, and testing the binary tree-based multimodal data fusion feature extraction network and the brain tumor image segmentation network using a multimodal brain MRI image dataset, the following steps are also included:

[0058] The multimodal brain MRI image dataset was divided into training, validation, and test sets.

[0059] The multimodal brain MRI images in the training, validation, and test sets are cropped to a uniform size and then filtered to remove all labeled regions.

[0060] Data augmentation is performed on the labeled regions of the training and test sets after cropping and filtering operations.

[0061] The beneficial effects of this invention are as follows: This invention provides a novel multimodal data fusion feature extraction network and a brain tumor image segmentation network based on a binary tree structure. Due to the characteristics of the network structure, it can be rapidly scaled to facilitate users in training with multimodal data. Furthermore, the segmentation results of the tumor region in the final image are not significantly different from those of existing models, which can meet the needs of doctors. At the same time, this invention only requires training on an 8GB graphics card, and its memory requirements are far lower than those of existing model networks. Attached Figure Description

[0062] Figure 1 This is a flowchart of a brain tumor image segmentation method based on lightweight multimodal processing according to Embodiment 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of the network structure of the multimodal data fusion feature extraction network based on a binary tree structure in Embodiment 1 of the present invention;

[0064] Figure 3 This is a schematic diagram of the feature convolutional block composition structure when sampling an existing 3D Unet network in Embodiment 1 of the present invention;

[0065] Figure 4 This is a schematic diagram of the feature convolution block composition structure in Embodiment 1 of the present invention;

[0066] Figure 5 This is a schematic diagram of the structure of a multimodal fusion unit in an embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram of the network structure of the brain tumor image segmentation network in Embodiment 1 of the present invention;

[0068] Figure 7 This is a schematic diagram showing the GPU usage during non-training inference in Embodiment 3 of the present invention;

[0069] Figure 8 This is a schematic diagram showing the graphics card usage during training in Embodiment 3 of the present invention;

[0070] Figure 9 This is a schematic diagram showing the graphics card usage during inference in Embodiment 3 of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0072] Example 1

[0073] This embodiment provides a lightweight, multimodal brain tumor image segmentation method, the flowchart of which can be found in [link to flowchart]. Figure 1 The method includes the following steps:

[0074] S1. Obtain a multimodal brain MRI image dataset and construct a multimodal data fusion feature extraction network based on a binary tree structure. The feature extraction network includes a tree coding layer consistent with the number of modalities.

[0075] S2. Construct a local feature extractor for each modality of brain MRI images, and use the local feature extractor to extract features from the current modality of brain MRI images;

[0076] S3. Construct a multimodal fusion unit for the feature extraction results of two adjacent modalities, and use the multimodal fusion unit to fuse the feature extraction results of the two adjacent modalities. After all the intermodal features are fused, the output of the current tree coding layer is formed.

[0077] S4. Generate skip connection parameters for each tree coding layer, and then perform a connection and fusion operation using the skip connection parameters after upsampling the output of all tree coding layers.

[0078] S5. Construct a brain tumor image segmentation network, and use the segmentation network to obtain the segmentation result of the brain tumor image after completing the fusion operation;

[0079] S6. Using a multimodal brain MRI image dataset, train, validate, and test a binary tree-based multimodal data fusion feature extraction network and a brain tumor image segmentation network.

[0080] In this embodiment, the bras21 dataset is used as an example for detailed explanation.

[0081] To obtain the BRaTS21 dataset, first log in to the Kaggle website (https: / / www.kaggle.com / ) and click the "Sign Up" button to register. After registration, log in to your account, search for "BRaTS 2021 Task 1 Dataset" in the search box, find the corresponding dataset download page, and click the download button to download. The entire dataset includes a training set, a validation set, and a test set.

[0082] Within the training set folder rsna_asnr_miccai_brats2021_training_data, subfolders named after patient IDs contain data for each patient. Each patient subfolder contains four modalities of MRI image files: bras2021_xxxx_flair.nii.gz (T2-FLAIR sequence images), bras2021_xxxx_t1.nii.gz (T1 sequence images), bras2021_xxxx_t1ce.nii.gz (T1 enhanced sequence images), and bras2021_xxxx_t2.nii.gz (T2 sequence images), along with the corresponding segmentation label file bras2021_xxxx_seg.nii.gz.

[0083] In the validation set, each subfolder contains MRI image files for four modalities, but no segmentation label files; only the four image files: bras2021_xxxx_flair.nii.gz, bras2021_xxxx_t1.nii.gz, bras2021_xxxx_t1ce.nii.gz, and brats2021_xxxx_t2.nii.gz.

[0084] The test set data is not publicly available; models must be submitted for testing via a specific method. Its directory structure is similar to the validation set, containing four modalities of MRI images from the patient, but without segmentation labels. All dataset files are stored in NIfTI format (.nii.gz). The four MRI images for each case are all 240×240×155 pixels in size and share segmentation labels. Label information primarily includes enhanced tumor (ET), peritumoral edema, invasive tissue (ED), and necrotic tumor core (NCR). The entire dataset contains 1251 cases, divided into a near 1:4 ratio for the training set (1000 cases) and a validation / evaluation set (test set) (251 cases in total).

[0085] For the bras21 dataset, since it contains image data of four modalities, this embodiment constructs a multimodal data fusion feature extraction network based on a binary tree structure. See [link to relevant documentation]. Figure 2 The number of tree coding layers in this feature extraction network is also four. That is, when the multimodal brain MRI image dataset contains brain MRI images of four modalities, the constructed multimodal data fusion feature extraction network based on binary tree structure includes four tree coding layers.

[0086] The fourth tree coding layer takes in four modalities of brain MRI images, and after processing by the local feature extractor and the multimodal fusion unit, outputs three images after feature extraction and intermodal feature fusion to the third tree coding layer.

[0087] After being processed by the local feature extractor and the multimodal fusion unit, the third tree coding layer outputs two images that have undergone feature extraction and intermodal feature fusion to the second tree coding layer.

[0088] After being processed by the local feature extractor and the multimodal fusion unit, the second tree coding layer outputs an image that has undergone feature extraction and intermodal feature fusion to the first tree coding layer.

[0089] Here, n represents the number of tree coding layers. In the multimodal data fusion feature extraction network based on a binary tree structure, the input of the tree coding layer decreases from 4 to 1 from top to bottom. Furthermore, the number of channels for each modality in the tree coding layer is consistent, and the number of modal channels in the previous tree coding layer is half the number of modal channels in the next tree coding layer.

[0090] In practical applications, see Figure 3 In the original 3D Unet network, while extracting features from the input, the output size is halved. However, training the original 3D Unet network on the brats21 dataset leads to overfitting. Therefore, this embodiment mitigates this by parameter augmentation to adapt to the size of the brats21 dataset. See [link to relevant documentation]. Figure 4 In this embodiment, while expanding to a tree structure, in order to ensure that the network has sufficient parameters to learn, a new layer is added to the feature extraction part, namely, a new convolution. The input size of this layer remains unchanged from the previous layer. Its main purpose is to increase the generalization ability of the model by expanding the network parameters.

[0091] Taking modality 4 as an example, assuming the size of modality 4 is (B, C, H, W, Z), where B is the batch size, C is the number of channels, and (H, W, Z) is the 3D size of the input data. The input of the first convolutional layer is (B, C, H, W, Z), and the output is (B, C, H, W, Z). The input of the newly added convolutional layer is (B, C, H, W, Z), and the output is (B, C, H, W, Z). The input of the second convolutional layer is (B, C, H, W, Z), and the output is (B, 2C, H / 2, W / 2, Z / 2). The entire feature convolutional module acts as a local feature extractor for learning model features, enhancing the network's learning and generalization abilities.

[0092] Therefore, in this embodiment, for each modality of brain MRI image, feature extraction is performed through a feature convolutional block using the multimodal data fusion feature extraction network based on a binary tree structure. The entire feature convolutional module can serve as a local feature extractor for learning model features, enhancing the network's learning and generalization capabilities. Thus, the local feature extractor in this embodiment includes a first convolutional layer, a second convolutional layer, and a newly added convolutional layer.

[0093] The current modality brain MRI image is sequentially passed through the first convolutional layer, the newly added convolutional layer, and the second convolutional layer to complete the feature extraction of the current modality brain MRI image;

[0094] The input and output dimensions of the newly added convolutional layer are the same as those of the first convolutional layer. The input dimension of the third convolutional layer is the same as the output dimension of the newly added convolutional layer, and the output dimension of the third convolutional layer is 1 / 2 times the input dimension.

[0095] In this embodiment, a multimodal fusion unit is designed for multimodal fusion. Since there are four modes, and intermodal feature fusion is required between modes 1 and 2, modes 2 and 3, and modes 3 and 4, the multimodal fusion unit is a bimodal fusion unit. This unit has two modal inputs and outputs a single modality. The bimodal fusion unit in this embodiment can... Figure 4 The output results of the structure are further fused.

[0096] See Figure 5 For a dual-modal fusion unit, it includes a first-channel attention mechanism, a second-channel attention mechanism, a low-rank interaction channel attention mechanism, and a spatial attention mechanism; the first-channel attention mechanism is used to input the feature extraction result of one modality, and the second-channel attention mechanism is used to input the feature extraction result of an adjacent modality.

[0097] Furthermore, after the feature extraction results of the two adjacent modalities are fused between modal features by the dual modal fusion unit, a feature fusion result of a single modality is generated. The size of the feature fusion result of the single modality is consistent with the size of the feature extraction results of the two adjacent modalities. The feature fusion results of all single modalities in the current tree coding layer constitute the output of the tree coding layer.

[0098] Specifically, in the bimodal fusion unit, the two channel attention mechanisms focus on the relationships between high-order feature channels of each modality, emphasizing the learning of channels containing lesion regions. The low-rank interaction channel attention mechanism employs the principle of LORA's fine-tuning of large models, reducing the network's model parameters and further fusing the bimodal results. Finally, the spatial attention mechanism focuses on the relationships between high-order feature pixels, generalizing the learning of lesion edges and structural features. Assuming... Figure 4 The output of the structure is (B, 2C, H / 2, W / 2, Z / 2), while Figure 5 Then there are bimodal features M1=(B,2C,H / 2,W / 2,Z / 2) and M2=(B,2C,H / 2,W / 2,Z / 2).

[0099] Here, the channel attention mechanism uses a weight list of size 2C to store the weight information for each channel. The output of the channel attention mechanism has a size of (B, 2C, H / 2, W / 2, Z / 2). Subsequently, the input size of the low-rank interaction is {(B, 2C, H / 2, W / 2, Z / 2), (B, 2C, H / 2, W / 2, Z / 2)}, and the output is (B, 2C, H / 2, W / 2, Z / 2). It is worth noting that the size of the feature result fused by the spatial attention mechanism is also (B, 2C, H / 2, W / 2, Z / 2). The size of the input remains unchanged in the entire fusion structure; that is, the sizes of modality 1, modality 2, and the fused modality remain unchanged.

[0100] See Figure 6 In the middle modality, to improve generalization learning, the number of channels is doubled compared to the previous layer. The entire network follows the design pattern of the 3D UNet network, incorporating skip connections and upsampling. In the skip connections, this embodiment performs a fusion operation on the outputs of each tree coding layer, as shown in the following formula:

[0101] ,

[0102] Where A(.) represents the feature fusion function, its input dimension is n*m, n represents the nth modality, m represents the channel dimension of each single modality, and the output dimension of the feature fusion function is m.

[0103] In multimodal fusion, directly using the fusion method of formula (1) will directly lead to gaps in network learning, directly affecting the model output results. For example, the network may only learn a portion of the labeled region because the multimodal data comes from different devices and their data distribution is inconsistent. To solve this problem, this embodiment first learns the channel attention of each modality, and then uses two low-rank matrices to form a high-rank matrix to further learn the channel attention of the multimodality. This mode adopts a progressive learning method. Finally, spatial attention is learned from the channel attention learning results. The purpose is to learn the contribution of the corresponding positions of different modalities to the segmentation and enhance the generalization of the network.

[0104] Therefore, in this embodiment, when using the multimodal fusion unit to perform intermodal feature fusion of the feature extraction results of two adjacent modalities, it may include:

[0105] The channel attention between two adjacent modalities is learned using the first-channel attention mechanism and the second-channel attention mechanism.

[0106] We utilize a low-rank interactive channel attention mechanism, employing two low-rank matrices to further learn the channel attention between two adjacent modalities;

[0107] Spatial attention learning is performed on the learning results of low-rank interactive channel attention mechanism using spatial attention mechanism.

[0108] Specifically, the calculation formula for the first channel attention mechanism is as follows:

[0109] ,

[0110] The calculation formula for the second channel attention mechanism is as follows:

[0111] ,

[0112] The calculation formula for the low-rank interaction channel attention mechanism is as follows:

[0113] ,

[0114] The calculation formula for the spatial attention mechanism is as follows:

[0115] ,

[0116] Where a represents the a-th mode, a+1 represents the mode adjacent to mode a, and a≤n-1. The ChannelAttention(.) function adopts the traditional channel attention mechanism, while CrossChannelAttention(.) implements a similar operation to LoRa fine-tuning large model training.

[0117] It should be noted that after obtaining the segmentation results of brain tumor images, and before training, validating, and testing the binary tree-based multimodal data fusion feature extraction network and the brain tumor image segmentation network using a multimodal brain MRI image dataset, the following may also be included:

[0118] The multimodal brain MRI image dataset was divided into training, validation, and test sets.

[0119] The multimodal brain MRI images in the training, validation, and test sets are cropped to a uniform size and then filtered to remove all labeled regions.

[0120] Data augmentation is performed on the labeled regions of the training and test sets after cropping and filtering operations.

[0121] Thus, this embodiment can be achieved through... Figure 6 The segmentation network structure outputs the final segmentation target, and the last layer of the segmentation network will output the probability distribution.

[0122] Example 2

[0123] Building upon Example 1, this example requires preprocessing the brats21 dataset after acquisition to complete the subsequent training process. In this example, preprocessing may include data pruning, data augmentation, and data filtering operations.

[0124] Because the original brain MRI images of various modalities in the brats21 dataset have large file sizes and most pixel values ​​are 0, the data needs to be cropped to a size of 128*128*128 for ease of computation and to reduce computational requirements. In all the labeled datasets, there are four main categories: 0, 1, 2, and 4. 0 represents background, 1 represents the necrotic tumor core, 2 represents edema, and 4 represents enhancing tumors.

[0125] Data augmentation can be performed using the Torchio toolkit. This includes using `tio.ZNormalization` for data standardization, `tio.RandomElasticDeformation` for elastic deformation transformation of 3D medical datasets to simulate the deformation of biological tissues (the parameter `num_control_points` specifies the number of control 3D mesh points, and the parameter `max_displacement` represents the maximum displacement of each control point in the elastic deformation, affecting the severity of the deformation), `tio.RandomBlur` for image blurring (mean and variance parameters are 0 and 1 respectively), `tio.RandomNoise` to add noise and improve the model's generalization ability (mean is 0, variance is 0.001), `tio.RandomGamma` to adjust the brightness of the data (log_gamma parameter is (-0.3, 0.3)), `tio.RandomFlip` for 3D flipping (left and right), `tio.RandomAffine` for affine transformations, including translation, rotation, and distortion, and `tio.RescaleIntensity`. The pixel intensity of the 3D data is normalized to between 0 and 1, and finally the training data of the labeled area is augmented.

[0126] For the training set, some data lacked labels for a specific class, so these unsuitable data points needed to be removed. After preprocessing, the training set size was adjusted to 946 columns. For the test set, aside from cropping to the input size (128*128*128), only tio.ZNormalization was used for data standardization. However, because some data lacked labels for a specific class, these unsuitable data points also needed to be removed. After preprocessing, both the validation and evaluation sets were adjusted to 240 columns.

[0127] After the preprocessing of the brain MRI images for the four modalities described above, the model can be trained. In this embodiment, the training dataset has 946 samples, and the test set (i.e., evaluation set) and validation set have 240 samples each. During training, the batch size is 1. A MultiStepLR learning rate scheduling plan [60, 80] is used, with the first 60 batches trained at a learning rate of 0.001, batches 60-80 at 0.0001, and batches greater than 80 at 0.00001. The Adam optimizer is used for model weight optimization, with a weight decay parameter of 0.00001, and momentum decay and squared policy decay parameters of (0.95, 0.995) controlling the first-order and second-order momentum estimation of the matrix, respectively. To facilitate model performance reproduction, the random factor is 42. The total batch size is 200. Cross-entropy and dice are used during gradient updates. The combined loss functions are cross-entropy loss function for learning label classification, and DICE for calculating the contours of the segmented label and the predicted label. The calculation formula is as follows:

[0128] ,

[0129] Among them, L CE The cross-entropy loss function is used. Since there are many 0 pixel values ​​and few pixel values ​​in the tumor region, the weight parameters of the cross-entropy loss function are [1, 3, 3]. The parameter 'a' represents a balancing parameter used to control the balance between pixel classification and contour calculation in the model. Different optimization directions should be adopted at the beginning, middle, and end of the model training. Therefore, this embodiment introduces epochs (number of training iterations) for further improvement, as shown below:

[0130] ,

[0131] ,

[0132] Where current_epoch is the current iteration number, and epochs is the total number of iterations.

[0133] After training is complete, intermediate data such as optimizer parameters, model parameters, loss function, and current iteration number are saved to facilitate model recovery and continued training.

[0134] Then, model evaluation can be performed. In this embodiment, the Dice metric is used for verification during the model evaluation process. Here, X represents the predicted target region, Y represents the actual target region, and |XnY| is the number of intersection elements between the predicted region and the actual region. |X| and |Y| are the number of elements in the predicted region and the actual region, respectively. The value of Dice is between 0 and 1. The closer it is to 1, the better the model segmentation effect.

[0135] Finally, model inference is performed. During model inference, no other data preprocessing is done besides standardization and cropping to 128*128*128 pixels. In post-processing, the segmentation results need to be filled with holes and have edges blurred to ensure smooth segmentation. Finally, the argmax function is used to convert the multi-channel segmentation predictions to the same size as the label data; essentially, it predicts each pixel on each channel using the probability distribution map output by the model.

[0136] Example 3

[0137] Based on Examples 1 and 2, after preprocessing the dataset, training the model, evaluating the model, and inferring the model, this example deploys and trains the binary tree-based multimodal data fusion feature extraction network and the brain tumor image segmentation network involved in the lightweight multimodal brain tumor image segmentation method in Example 1 on a GPU. The model performance of this example is compared with that of existing models. The specific comparison results are shown in Table 1.

[0138]

[0139] This embodiment only requires an 8GB GPU for training, and the average DICE (excluding edema regions) reaches over 80% within 30 iterations, with the ability to continue optimization and learning. Referring to Table 1, the memory requirements of this embodiment are significantly lower than those of nnU-net v2, Swin UNETR, DiffSegNet, and LKA-Unet networks.

[0140] In this embodiment, the DICE results of the network model were also statistically analyzed. Experiments were conducted on the WT region (the entire lesion region, including the background of category 0, the tumor necrosis region of category 1, the edema region of category 2, and the tumor enhancement region of category 4) and the TC region (the background region of category 0, the tumor necrosis region of category 1, and the tumor enhancement region of category 4) of the brats21 dataset, and the relevant DICE results were calculated.

[0141] Experimental results show that the segmentation performance of this embodiment on TC is significantly higher than that of existing segmentation networks, with improvements of 0.078, 0.061, 0.054, and 0.072 respectively, representing improvements of approximately 7.8%, 6.1%, 5.4%, and 7.2%. Furthermore, the network in this embodiment is trained using two batches with 300 iterations and an initial training learning rate of 0.01.

[0142] Based on Table 1, see [reference] Figure 7 ,Figure 8 and Figure 9 During GPU training and inference, the GPU utilization rate can be obtained by entering the `nvidia-smi` command in the command prompt (cmd). Therefore, the GPU utilization during non-training inference, training, and inference phases of this embodiment can be obtained. It is evident that the network training GPU utilization in this embodiment is 11.425G, far lower than existing multimodal segmentation networks, and the GPU utilization for inference is only 2.083G. It can still run successfully on general consumer-grade GPUs or NVIDIA 1660-level GPUs, possessing broad application and promotion value. In contrast, the GPU utilization used for existing network inference is generally estimated and can be described using empirical values. A higher GPU utilization rate during training results in a higher utilization rate during inference, exhibiting a linear relationship. However, both are greater than the 2.083G in this embodiment.

[0143] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight multimodal brain tumor image segmentation method, characterized in that, Includes the following steps: A multimodal brain MRI image dataset was acquired, and a multimodal data fusion feature extraction network based on a binary tree structure was constructed. The feature extraction network includes a tree coding layer consistent with the number of modalities. When the multimodal brain MRI image dataset contains brain MRI images of four modalities, the constructed multimodal data fusion feature extraction network based on a binary tree structure includes four tree coding layers. The fourth tree coding layer takes in four modalities of brain MRI images, and after processing by the local feature extractor and the multimodal fusion unit, outputs three images after feature extraction and intermodal feature fusion to the third tree coding layer. After being processed by the local feature extractor and the multimodal fusion unit, the third tree coding layer outputs two images that have undergone feature extraction and intermodal feature fusion to the second tree coding layer. After being processed by the local feature extractor and the multimodal fusion unit, the second tree coding layer outputs an image that has undergone feature extraction and intermodal feature fusion to the first tree coding layer. In the multimodal data fusion feature extraction network based on binary tree structure, the input of the tree coding layer decreases from 4 to 1 from top to bottom. The number of channels for each modality in the tree coding layer is the same, and the number of modal channels in the upper tree coding layer is 1 / 2 times the number of modal channels in the lower tree coding layer. A local feature extractor is constructed for each modality of brain MRI images, and the local feature extractor is used to extract features from the current modality of brain MRI images; The local feature extractor includes a first convolutional layer, a second convolutional layer, and a newly added convolutional layer; The current modality brain MRI image is sequentially passed through the first convolutional layer, the newly added convolutional layer, and the second convolutional layer to complete the feature extraction of the current modality brain MRI image; The input and output dimensions of the newly added convolutional layer are the same as those of the first convolutional layer, the input dimension of the third convolutional layer is the same as the output dimension of the newly added convolutional layer, and the output dimension of the third convolutional layer is 1 / 2 times the input dimension; A multimodal fusion unit is constructed for the feature extraction results of two adjacent modalities, and the feature extraction results of the two adjacent modalities are fused by the multimodal fusion unit. The output of the current tree coding layer is formed after all intermodal features are fused. The multimodal fusion unit is a dual-modal fusion unit, which has two modal inputs and finally outputs one modality; The dual-modal fusion unit includes a first-channel attention mechanism, a second-channel attention mechanism, a low-rank interaction channel attention mechanism, and a spatial attention mechanism. The first channel attention mechanism is used to input the feature extraction result of one modality, and the second channel attention mechanism is used to input the feature extraction result of an adjacent modality. After the feature extraction results of the two adjacent modalities are fused between modal features by the dual modal fusion unit, a feature fusion result of a single modality is generated. The size of the feature fusion result of the single modality is the same as the size of the feature extraction results of the two adjacent modalities. The output of the tree coding layer is composed of the feature fusion results of all individual modalities in the current tree coding layer; When using a multimodal fusion unit to perform intermodal feature fusion on the feature extraction results of two adjacent modalities, the following is included: The channel attention between two adjacent modalities is learned using the first-channel attention mechanism and the second-channel attention mechanism. We utilize a low-rank interactive channel attention mechanism, employing two low-rank matrices to further learn the channel attention between two adjacent modalities; Spatial attention learning is performed on the learning results of low-rank interactive channel attention mechanism using spatial attention mechanism. Skip connection parameters are generated for each tree coding layer, and the outputs of all tree coding layers are upsampled before being connected and fused using the skip connection parameters. A brain tumor image segmentation network was constructed, and the segmentation results of brain tumor images were obtained by using this segmentation network after completing the fusion operation. We used a multimodal brain MRI image dataset to train, validate, and test a binary tree-based multimodal data fusion feature extraction network and a brain tumor image segmentation network.

2. The brain tumor image segmentation method based on lightweight multimodal processing according to claim 1, characterized in that, The formula for the connection and fusion operation using jump connection parameters is as follows: ， in, This represents the feature fusion function, whose input dimension is... , where n represents the nth modality, m represents the channel dimension of each single modality, and the output dimension of the feature fusion function is m.

3. The brain tumor image segmentation method based on lightweight multimodal processing according to claim 1, characterized in that, The calculation formula for the first channel attention mechanism is as follows: ， The calculation formula for the second channel attention mechanism is as follows: ， The calculation formula for the low-rank interaction channel attention mechanism is as follows: ， The calculation formula for the spatial attention mechanism is as follows: ， Where a represents the a-th mode, a+1 represents the mode adjacent to mode a, and a≤n-1.

4. The brain tumor image segmentation method based on lightweight multimodal processing according to claim 1, characterized in that, After obtaining the segmentation results of brain tumor images, and before training, validating, and testing the binary tree-based multimodal data fusion feature extraction network and the brain tumor image segmentation network using a multimodal brain MRI image dataset, the following steps are also included: The multimodal brain MRI image dataset was divided into training, validation, and test sets. The multimodal brain MRI images in the training, validation, and test sets are cropped to a uniform size and then filtered to remove all labeled regions. Data augmentation is performed on the labeled regions of the training and test sets after cropping and filtering operations.

Images