Brain tumor segmentation algorithm based on multi-path modal fusion and edge-driven attention

Abstract

The invention discloses a brain tumor segmentation algorithm based on multipath modal fusion and edge-driven attention, and belongs to the technical field of medical image analysis. Comprising the steps of performing two-dimensional slicing and center clipping processing on an original image; decomposing an input image into two independent two-channel features, and extracting preliminary features through a convolutional layer; the decomposed images are respectively input into a Transform-based double-trunk feature learning stream to extract semantic features in a specific mode, and meanwhile, deep integration is carried out through a shared fusion path to mine high-order and low-order detail information of different modes; in the decoding stage, an upper independent path and a lower independent path are used for finely guiding superficial layer edge features through edge driving attention; and meanwhile, the sharing fusion path adopts a multi-scale modal fusion module to be matched with jump connection to carry out multi-feature fusion. The method has remarkable advantages in the aspects of fine-grained edge processing and complex tumor morphological segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention. Background Art

[0002] As an important control center of the human body, the brain controls human physiological movements, cognitive functions and emotional experiences. However, abnormal growth and canceration of abnormal cells in the human brain can cause brain tumors. Glioma, a primary tumor originating from tumor cells inside the brain, can be roughly divided into two categories according to its grade: low-grade glioma (LGG) and high-grade glioma (HGG). High-grade gliomas show cancerous properties, and even if patients receive timely intervention and treatment, the average survival period is only 2 years or even shorter; low-grade gliomas tend to show benign characteristics. Although the average survival period is about 7 years, it is inevitable that all low-grade gliomas will eventually deteriorate into high-grade gliomas and die. In the face of the threat of brain tumors, early diagnosis and accurate location of lesions are crucial to developing effective treatment plans.

[0003] At present, the diagnosis and monitoring of brain tumors mainly rely on medical imaging technology. Among them, magnetic resonance imaging (MRI), as a non-invasive imaging method, can produce high-resolution, non-destructive brain images and is the main technical means for diagnosing and evaluating brain tumors. MRI can provide different imaging sequences (modes) for the same tissue under different imaging techniques, including T1-weighted fluid attenuated inversion recovery (FLAIR), T1-weighted (T1), T1-weighted enhancement (T1c), and T2-weighted (T2). Accurate brain tumor segmentation is a key step in preoperative planning diagnosis, growth rate prediction, and postoperative disease monitoring. However, manual segmentation of brain tumors is a time-consuming and arduous task. Manual annotation by experts is easily interfered by subjective judgment. Due to fatigue or lack of experience, doctors are more likely to make mistakes and inaccurate segmentation, resulting in missed diagnosis and misdiagnosis of certain diseases, which seriously affects the life and health of patients. Therefore, the automatic segmentation scheme of brain tumors has become a critical and widely studied task.

[0004] In recent years, convolutional neural networks (CNNs) have been rapidly applied and developed in the field of medical image segmentation due to their excellent feature extraction capabilities. This technology has also achieved outstanding results in brain tumor segmentation. CNN networks obtain a wider receptive field by stacking convolutional layers and taking advantage of the complementary advantages of translation invariance and shared sliding window strategies, thereby capturing more feature information and global context data. With the introduction of algorithms such as U-Net, Attention U-Net, and U-Net++, end-to-end semantic segmentation models based on U-shaped structures have become the mainstream in the field of brain tumor segmentation. They introduce residual or cyclic structures and combine them with technologies such as multi-scale information interaction and attention mechanisms to make the model have more powerful information representation capabilities and further improve the segmentation effect. For example, Chen et al. introduced a cascade binary classifier in FSENet to segment each tumor sub-region and then erased it to classify other tumor regions. Wang et al. and Yogananda et al. converted the multi-classification problem into three binary segmentation problems and designed three different networks for complete tumors, tumor cores, and enhanced tumors for segmentation, effectively solving class imbalance. In order to deeply explore the hierarchical structure of tumor subregions, Xu et al. used a multi-branch network with a cascaded attention mechanism for each subregion. Ghassemi et al. used the GAN model to generate MR images and trained a deep neural network as a discriminator to perform multi-classification of brain tumors. However, these methods mainly rely on the extraction and processing of local features. When faced with complex and multimodal MRI, it is difficult to fully capture the various semantic information in the image. In recent years, Transformer-based deep learning models have gradually been introduced into medical image segmentation. In particular, when processing images of different modalities, the Transformer's self-attention mechanism can effectively capture global information and long-range dependencies. In the task of brain tumor segmentation, how to effectively fuse the complementary information between different modalities remains a challenge that needs to be solved.

[0005] In addition, edge information also plays a vital role in medical image segmentation. Given the complex morphology and variable tissue characteristics of brain tumors, traditional deep learning methods often have difficulty accurately extracting tumor boundaries, resulting in blurred boundaries or inaccurate segmentation.

[0006] Therefore, the existing brain tumor segmentation algorithms still have certain deficiencies in processing multimodal MRI data, feature fusion and boundary details. In order to overcome these challenges, the present invention proposes a brain tumor segmentation algorithm based on multipath modal fusion and edge-driven attention. The algorithm extracts and fuses multimodal information more accurately through the Transformer-based multipath feature learning flow, multi-scale mode fusion module and edge-driven attention, while retaining detailed edge features during the segmentation process, thereby improving the accuracy and boundary clarity of brain tumor segmentation. The present invention achieves accurate segmentation of brain tumors, providing an innovative and effective technical means for the early diagnosis and treatment of brain tumors. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention, aiming to solve the problems of unclear brain tumor segmentation boundaries, missing complementary features and feature redundancy during multi-modal feature fusion.

[0008] The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention in this scheme includes the following steps:

[0009] S1: Get the original brain tumor image (H×W×D), slice the original 3D data into 2D images (H×W) along the axial plane. Use z-score normalization and statistically determine the center of the cropped image based on the range of irrelevant background areas;

[0010] S2: Decompose the input four-channel image into two independent two-channel features along the channel dimension, and use a 3×3 convolution to set its output channel equal to the number of categories;

[0011] S3: The decomposed images are simultaneously passed into the dual-trunk feature learning stream based on Transformer, and feature representations are extracted from the corresponding brain tumor modalities respectively. At the same time, the modal attributes obtained after each stage in the two independent paths are retained and passed as input to the third shared fusion path based on Transformer to deeply mine the multimodal semantic features and edge features in different modalities of MRI.

[0012] S4: In the decoding stage, the upper and lower independent paths use edge-driven attention in the process of stepwise upsampling to receive the modal features of the corresponding levels in the encoding stage and the prediction information of the higher level in the decoding stage as well as the high-frequency features processed by traditional digital images, so that the shallow edge feature information plays a guiding role, refines the edge details of the brain tumor sub-regions, and obtains a prediction result with clear boundaries;

[0013] S5: The decoding part of the third shared fusion path uses skip connections in conjunction with a multi-scale pattern fusion module to fully fuse multi-scale brain tumor features and upsample using a linear binary interpolation method to obtain high-resolution prediction results, so as to better locate and distinguish brain tumor areas;

[0014] S6: A mixed boundary loss function is used to evaluate the difference between the predicted results and the true labels for the brain tumor segmentation results generated by the three paths.

[0015] The beneficial effect of this scheme is that the brain tumor segmentation algorithm based on multi-path modal fusion and edge-driven attention proposed in the present invention can effectively solve the problems of blurred brain tumor segmentation boundaries, missing complementary features and feature redundancy when multi-modal features are fused in the prior art. Through the multi-path modal fusion strategy, this scheme fully mines the semantic information and edge features in MRI of different modalities. The multi-path Transformer architecture optimizes the fusion of multi-modal features and overcomes the redundancy and lack of complementarity of different modal features. In the decoding stage, edge-driven attention can refine the edge details of the tumor sub-region and obtain clearer segmentation boundaries. At the same time, the multi-scale mode fusion module and jump connection further enhance the multi-modal feature fusion capability, achieve more accurate tumor localization and differentiation, and ensure high-resolution accurate prediction. Finally, the mixed boundary loss function enables the difference between the predicted results and the true labels to be more accurately evaluated, optimizing the training effect of the model. This scheme can provide high-quality automatic segmentation results of brain tumors and provide more reliable auxiliary support for clinical diagnosis and treatment.

[0016] Furthermore, in S1, the original 3D brain tumor image (H×W×D) is obtained and sliced ​​into 2D images (H×W) along the axial plane. The four modality images (except the GT file) are normalized to zero mean and unit standard deviation using z-score normalization, and the center cropped image size is specifically cropped according to the irrelevant background area.

[0017] The beneficial effects are: the slicing operation converts the originally complex 3D data into a simpler 2D image, which helps to reduce memory usage and computational burden, and also provides more accurate local information for subsequent processing. The z-score normalization is used to standardize the images of different modalities to zero mean and unit standard deviation, eliminating the deviation caused by the difference in image intensity between different modalities, so that the model can process the features of each modality more fairly. Center cropping can remove irrelevant background areas, reduce computational complexity, and ensure that the position of the brain tumor area in the image is more concentrated.

[0018] Further, in S2, the original four-channel brain tumor image Specific channel selection and splicing operations are performed along the channel dimension to decompose into two independent features X1 and X 2 After feature reorganization, the feature maps are all 2-channel. 1 Select the second and third channels where feature X 2 Select the first and fourth channels. The formula for channel selection and concatenation of feature maps is as follows:

[0019]

[0020] in Indicates that the specific channel of the selected feature map is c, i represents the specific feature map, when i = 1 When i=2 Concat means channel concatenation along the channel dimension.

[0021] The beneficial effects are: by performing specific channel selection and splicing operations on the original four-channel brain tumor image, each channel is ensured to have higher feature expression ability when input into the network, the key information of each single modality is effectively extracted, the most representative features of each modality are retained, and redundant information interference is avoided. The output channel is adjusted to the number of classifications using 3×3 convolution to ensure that the feature map can adapt to subsequent classification tasks, which improves the pertinence and efficiency of feature representation.

[0022] Furthermore, in S3, the dual-trunk feature learning flow based on Transformer is responsible for extracting feature representations from different brain tumor modality images. 1 From the first set of modality images X 1 Extract feature representation F 1 e , learn the backbone Transformer through the second feature 2 From the second set of modality images X 2 Extracting feature representation

[0023]

[0024] After the feature extraction of each level of the two paths is completed, the modal attributes learned in each stage are retained and

[0025]

[0026] Attributes means to obtain the modal attributes extracted in each stage. Attributes can be high-level semantic information (such as tissue structure, tumor area, etc.) and low-level features (such as edges, textures, etc.) contained in the image. The modal attributes extracted from each stage of the two independent paths are and As the third shared fusion path Transformer 3 The dual input source of , obtains the multimodal feature representation:

[0027]

[0028] The beneficial effect is that the Transformer-based multi-path feature learning flow effectively captures high-level semantic features (brain tumor subregion tissue structure) and low-level visual features (edges, detailed textures, etc.) as well as long-range dependencies in brain tumor images of different modalities through its powerful global modeling capabilities and flexible feature expression.

[0029] The multi-path structure ensures the diversity and comprehensiveness of information by processing features from different modalities or different angles in parallel. Each path focuses on a specific type of feature or the details of a different modality, allowing each path to learn more refined and unique features. By assigning features to different paths, the model can avoid processing all information in one path, thereby reducing the risk of information redundancy and loss. The two independent feature extraction paths effectively extract the key information of each modality and enhance the expressive power of a single modality; the shared fusion path not only enhances the complementarity between different modalities, but also improves the efficiency and accuracy of the model when processing multimodal data, which helps to more accurately segment brain tumor areas.

[0030] Furthermore, in S4, the upper and lower independent paths use edge-driven attention (EDA) to guide the network to focus on the edge features of the lesion area during the step-by-step upsampling process. This module has three input sources, namely: the modal features corresponding to each level in the encoding layer or High-level prediction features generated by each decoder stage or High-frequency features after Laplacian pyramid processing or The specific process includes:

[0031] The decoded prediction features are weighted and decomposed to generate two attention maps: the first is non-salient attention, calculated as The attention score of the background area is combined with the encoded feature matrix or The second is the boundary attention map, which generates boundary predictions through Gaussian filtering and Laplacian operators and combines them with or Then, the high-frequency features pre-processed by the Laplacian pyramid are or Use bilinear interpolation to adjust to or The same spatial size is multiplied to extract high-frequency information segments. To optimize the fusion of multimodal information, the background features are or Boundary Features or High frequency features or The concatenation is performed along the channel dimension, and then element-by-element compression and extraction are performed through the fused convolutional layer. The specific operations are as follows:

[0032]

[0033] The attention mask is introduced in the i-th layer, and the importance of different features is dynamically adjusted by residual connection with the original encoder features, highlighting the tumor area while suppressing irrelevant background information and noise, guiding the network to focus on the edge and shape features of the brain tumor. The features are then passed to the channel depth convolution calibration block to calibrate the features from the channel and spatial dimensions to further capture the feature correlation between the boundary and background areas. The calculation method is as follows:

[0034]

[0035] The beneficial effect is that edge-driven attention accurately guides the network to focus on the edge features of the tumor area by combining the weighted fusion of the encoder, decoder and high-frequency features. By generating non-significant attention maps and boundary attention maps, background noise is suppressed and tumor boundaries are highlighted respectively, ensuring that the tumor shape is accurately captured. At the same time, the original high-frequency boundary features after Laplacian pyramid and Gaussian denoising are used to assist the network in learning edge detail information, and the channel-dimensional features are aligned from space through the channel-depth convolution calibration block.

[0036] Further, in S5, the decoding part of the third shared fusion path adopts a skip connection in conjunction with a multi-scale mode fusion module, and the specific process includes:

[0037] Modality-specific features of two independent path encoders and Separately with the features from the shared fusion path encoder

[0038] The features are multiplied in pairs, and the obtained features are channel-joined and convolved, retaining the original information in the shared fusion path encoding stage while focusing on the tumor region features of the specific modality. In order to further enhance the features in the shared fusion path encoding stage, and and Matrix element-by-element multiplication is also performed, and the calculation formula is as follows:

[0039]

[0040] Through deep fusion interaction between multiple features, the network is ensured to focus on capturing the common features of all modalities. The calculation formula is as follows:

[0041]

[0042] Subsequently, the features at different levels in the encoding stage of the shared fusion path are passed as input features to the cross-attention to achieve multi-level and multi-scale feature interaction, allowing high-level features to guide low-level features, allowing the network to focus on the tumor area, eliminating interference from irrelevant tissues, and generating the final multimodal fusion features.

[0043] The beneficial effects are: the modal attributes learned at each stage of the two independent paths are passed as input to the shared fusion path, and the deep fusion and interaction of multimodal features are realized through the multi-scale modal fusion module. The modal features from the three paths are element-by-element multiplication and splicing of specific modal features, retaining the original information of the shared path. The cross-attention mechanism enables the high-level features in the shared fusion path to guide the low-level features, obtain feature information of different scales, ensure that the network focuses on the tumor area, and exclude interference from irrelevant background. In addition, the splicing operation of the feature map can retain the complementary information between different modalities, thereby enhancing the multimodal learning ability of the model, enabling the network to better understand and fuse the semantic and structural information in different modalities, thereby achieving deep fusion of different modal information and improving the multimodal learning ability of the model.

[0044] Furthermore, in S6, a mixed boundary loss function is used to evaluate the difference between the predicted results and the true labels for the brain tumor segmentation results generated by the three paths.

[0045] The beneficial effect is that the difference between the predicted result and the true label is evaluated by using a mixed boundary loss function, which can comprehensively consider the detailed information of the tumor boundary, especially when dealing with fuzzy or irregular tumor edges, and can better maintain the boundary consistency and accuracy of the segmentation result, thereby reducing false positives and false negatives. This method can provide more accurate and stable results for automatic brain tumor segmentation.

[0046] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 Schematic diagram of the overall network structure of Embodiment 1 of the brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention of the present invention;

[0049] Figure 2 1 is a schematic diagram of edge-driven attention of embodiment 1 of a brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention of the present invention;

[0050] Figure 3 Schematic diagram of a multi-scale mode fusion module of Embodiment 1 of a brain tumor segmentation algorithm based on multi-path mode fusion and edge-driven attention of the present invention;

[0051] Figure 4 It is the brain tumor segmentation target of Embodiment 1 of the brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention of the present invention;

[0052] Figure 5 is a visualization result diagram of an ablation experiment in Example 2 of a brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention of the present invention; DETAILED DESCRIPTION

[0053] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the embodiments of the present invention, and are further described in detail through specific implementation methods.

[0054] Embodiment 1

[0055] The present invention provides a brain tumor segmentation algorithm based on multi-path modal fusion and edge-driven attention, which captures the semantic information and edge features of brain tumors from multiple dimensions and scales, and effectively solves the problems of unclear brain tumor segmentation boundaries when segmenting the three lesion areas of complete tumor (WT), enhancing tumor (ET) and tumor core (TC) in brain tumors, and the lack of complementary characteristics and feature redundancy when multi-modal features are fused.

[0056] Figure 1 This is a schematic diagram of the overall architecture of the present invention. The framework contains two independent trunk paths, each of which extracts specific channel features for a single tumor modality, and introduces a shared modality fusion path to integrate multimodal brain tumor information. The decoding part corresponds to the corresponding level of the encoder, where: the two independent decoding paths use edge-driven attention to refine the tumor edge details. The specific structure of edge-driven attention is Figure 2As shown in the figure, the decoding stage of the shared fusion path receives features from the three encoding paths at each level and uses the multi-scale mode fusion module to deeply fuse the multi-modal and multi-scale information. The specific structure of the multi-scale mode fusion module is as follows: Figure 3 As shown. Finally, after a series of upsampling operations, the decoder is able to generate high-resolution output feature maps.

[0057] The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention includes the following steps:

[0058] S1: Get the original brain tumor image (H×W×D), slice the original 3D data into 2D images (H×W) along the axial plane. Use z-score normalization and statistically determine the center of the cropped image based on the range of irrelevant background areas;

[0059] S2: Decompose the input four-channel image into two independent two-channel features along the channel dimension, and use a 3×3 convolution to set its output channel equal to the number of categories;

[0060] S3: The decomposed images are simultaneously passed into the dual-trunk feature learning stream based on Transformer, and feature representations are extracted from the corresponding brain tumor modalities respectively. At the same time, the modal attributes obtained after each stage in the two independent paths are retained and passed as input to the third shared fusion path based on Transformer to deeply mine the multimodal semantic features and edge features in different modalities of MRI.

[0061] S4: In the decoding stage, the upper and lower independent paths use edge-driven attention in the process of stepwise upsampling to receive the modal features of the corresponding levels in the encoding stage and the prediction information of the higher level in the decoding stage as well as the high-frequency features processed by traditional digital images, so that the shallow edge feature information plays a guiding role, refines the edge details of the brain tumor sub-regions, and obtains a prediction result with clear boundaries;

[0062] S5: The decoding part of the third shared fusion path uses skip connections in conjunction with a multi-scale pattern fusion module to fully fuse multi-scale brain tumor features and upsample using a linear binary interpolation method to obtain high-resolution prediction results, so as to better locate and distinguish brain tumor areas;

[0063] S6: A mixed boundary loss function is used to evaluate the difference between the predicted results and the true labels for the brain tumor segmentation results generated by the three paths.

[0064] In step S2, the original four-channel brain tumor image Specific channel selection and splicing operations are performed along the channel dimension to decompose into two independent features X 1 and X 2After feature reorganization, the feature maps are all 2-channel. 1 Select the second and third channels where feature X 2 Select the first and fourth channels. The formula for channel selection and concatenation of feature maps is as follows:

[0065]

[0066] in Indicates that the specific channel of the selected feature map is c, i represents the specific feature map, when i = 1 When i=2 Concat means channel concatenation along the channel dimension.

[0067] In step S3, the dual-trunk feature learning flow based on Transformer is responsible for extracting feature representations from different brain tumor modality images. 1 From the first set of modality images X 1 Extracting feature representation Learning the backbone Transformer through the second feature 2 From the second set of modality images X 2 Extracting feature representation

[0068]

[0069] After the feature extraction of each level of the two paths is completed, the modal attributes learned in each stage are retained and

[0070]

[0071] Attributes means to obtain the modal attributes extracted in each stage. Attributes can be high-level semantic information contained in the image (such as tissue structure, tumor area, etc.) and low-level features (such as edges, textures, etc.). The modal attributes extracted from each stage of the two independent paths are and As the third shared fusion path Transformer 3 The dual input source of , obtains the multimodal feature representation:

[0072]

[0073] In step S4, the upper and lower independent paths use edge-driven attention to guide the network to focus on the edge features of the lesion area during the step-by-step upsampling process. This module has three input sources, namely: the modal features corresponding to each level in the encoding layer or High-level prediction features generated by each decoder stage or High-frequency features after Laplacian pyramid processing or like Figure 2 As shown, the specific process includes:

[0074] The decoded prediction features are weighted and decomposed to generate two attention maps: the first is non-salient attention, calculated as The attention score of the background area is combined with the encoded feature matrix or The second is the boundary attention map, which generates boundary predictions through Gaussian filtering and Laplacian operators and combines them with or Then, the high-frequency features pre-processed by the Laplacian pyramid are or Use bilinear interpolation to adjust to or The same spatial size is multiplied to extract high-frequency information segments. To optimize the fusion of multimodal information, the background features are or Boundary Features or High frequency features or The concatenation is performed along the channel dimension, and then element-by-element compression and extraction are performed through the fused convolutional layer. The specific operations are as follows:

[0075]

[0076] The attention mask is introduced in the i-th layer, and the importance of different features is dynamically adjusted by residual connection with the original encoder features, highlighting the tumor area while suppressing irrelevant background information and noise, guiding the network to focus on the edge and shape features of the brain tumor. The features are then passed to the channel depth convolution calibration block to calibrate the features from the channel and spatial dimensions to further capture the feature correlation between the boundary and background areas. The calculation method is as follows:

[0077]

[0078]

[0079] In step S5, the decoding part of the third shared fusion path adopts a skip connection in conjunction with a multi-scale mode fusion module, and the specific process includes:

[0080] Modality-specific features of two independent path encoders and Separately with the features from the shared fusion path encoder The features are multiplied in pairs, and the obtained features are channel-joined and convolved, retaining the original information in the shared fusion path encoding stage while focusing on the tumor region features of the specific modality. In order to further enhance the features in the shared fusion path encoding stage, and and Matrix element-by-element multiplication is also performed, and the calculation formula is as follows:

[0081]

[0082] Through deep fusion interaction between multiple features, the network is ensured to focus on capturing the common features of all modalities. The calculation formula is as follows:

[0083]

[0084] Subsequently, the features at different levels in the encoding stage of the shared fusion path are passed as input features to the cross-attention to achieve multi-level and multi-scale feature interaction, allowing high-level features to guide low-level features, allowing the network to focus on the tumor area, eliminating interference from irrelevant tissues, and generating the final multimodal fusion features.

[0085] Embodiment 2

[0086] The present invention selects the BraTS2020 dataset to verify the effective performance of the proposed algorithm. The BraTS2020 dataset contains 369 cases in the training set, 125 cases in the validation set, and 166 cases in the test set. Each case data includes four 3DMRI sequences, namely T1-weighted MRI (T1), T1-weighted MRI enhancement (T1ce), T2-weighted MRI (T2), fluid-attenuated inversion recovery (Flair) and a true value label. All MRI data of different modalities are strictly aligned and co-registered to the same common template, and re-interpolated and sampled to 1mm 3 The unified resolution of the image and skull stripping are achieved. The true value labels annotated by experts are composed of four categories: background, tumor necrosis and non-enhanced area (NCR / NET), peritumoral edema (ED), and tumor enhancement area (ET). Following the evaluation criteria of the BraTs competition, this example uses three nested combinations of label sub-areas to evaluate the accuracy of model prediction, such as Figure 4 As shown, the final segmentation targets are three different types of brain tumor regions: the Whole Tumor (WT: NCR / NET+ED+ET), the Tumor Core (TC: NCR / NET+ET) and the Enhancing Tumor (ET).

[0087] Model training and evaluation were performed on NVIDIA GeForce RTX 4090 GPU. The experimental platform was Windows 11 64-bit operating system with 24G memory. The framework proposed in the present invention was built based on PyTorch. During the training process, Adam with momentum of 0.9 was used as the optimizer. The initial learning rate, weight decay and batch size were set to 3e-4, 1e-4 and 16 respectively. The cosine annealing learning rate scheduler was used to train the model for 500 rounds.

[0088] In order to verify whether the algorithm model is effective. The present invention conducted an ablation experiment on the BraTS2020 dataset, added different modules to the multi-path model in turn, and conducted experiments using the same parameter settings. The specific results are shown in Table 1, where MPSE stands for multi-path semantic extraction, MMFB stands for multi-scale mode fusion module, BDA stands for edge-driven attention, and MB-Loss stands for mixed boundary loss function.

[0089] Table 1 Objective evaluation results of ablation study on BraTS2020 dataset

[0090]

[0091] As can be seen from Table 1, the network architecture using multi-path feature extraction has achieved good segmentation accuracy for the three categories of brain tumor subregions. The Dice coefficients of WT, TC and ET are 95.3%, 94.4% and 90.6%, respectively, and the Hausdorff distances are 2.20mm, 1.59mm and 1.32mm, respectively. After introducing the multi-scale mode fusion module to fuse the features of different modalities, the performance of the model continued to enhance, reaching Dice coefficients of 95.6%, 95.1%, 91.2%, and HD of 2.20mm, 1.55mm and 1.34mm. Subsequently, edge-driven attention was introduced to focus on local and edge detail features to avoid generating blurred images to the greatest extent. The Dice scores were increased to 96.0%, 95.4%, and 91.8%, respectively, and HD was also slightly improved (WT 2.15mm, TC1.53mm, ET 1.34mm). After the introduction of MB-Loss, the model achieved the best performance in all indicators and significantly optimized the accuracy of the tumor boundary. Combined with Figure 5 It can be seen that this ablation experiment verifies the gradual improvement of the model performance by the three modules of MMFB, EDA and MB-Loss, indicating that the synergy between the modules is crucial to improving the segmentation quality and boundary accuracy. The final optimization effect proves the effectiveness of the proposed algorithm.

[0092] Table 2 Objective evaluation results of BraTS2020 dataset and related algorithms

[0093]

[0094] The experimental results show that the proposed algorithm model performs best in the brain tumor segmentation task, especially in the Dice coefficient and Hausdorff distance (HD) indicators. Specifically, the Dice coefficients of the model in the three sub-regions of WT, ET and TC are 96.52%, 95.81% and 92.02%, respectively, and the Hausdorff distances are 2.11mm, 1.41mm and 1.30mm, respectively. The evaluation indicators are significantly better than other methods. In contrast, other models such as UCTransNet and IVD-Net also have high performance in the Dice coefficient (92.37% and 92.53%, respectively), but are slightly inferior in boundary accuracy (HD). Overall, the algorithm model proposed by us significantly improves the comprehensive performance of brain tumor segmentation through more efficient segmentation and boundary preservation capabilities, proving its advantages in this task.

[0095] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention, characterized by: The following steps are involved: S1: Get the original brain tumor image (H×W×D), slice the original 3D data into 2D images (H×W) along the axial plane. Use z-score normalization and statistically determine the center of the cropped image based on the range of irrelevant background areas; S2: Decompose the input four-channel image into two independent two-channel features along the channel dimension, and use a 3×3 convolution to set its output channel equal to the number of categories; S3: The decomposed images are simultaneously passed into the dual-trunk feature learning stream based on Transformer, and feature representations are extracted from the corresponding brain tumor modalities respectively. At the same time, the modal attributes obtained after each stage in the two independent paths are retained and passed as input to the third shared fusion path based on Transformer to deeply mine the multimodal semantic features and edge features in different modalities of MRI. S4: In the decoding stage, the upper and lower independent paths use edge-driven attention in the process of stepwise upsampling to receive the modal features of the corresponding levels in the encoding stage and the prediction information of the higher level in the decoding stage as well as the high-frequency features processed by traditional digital images, so that the shallow edge feature information plays a guiding role, refines the edge details of the brain tumor sub-regions, and obtains a prediction result with clear boundaries; S5: The decoding part of the third shared fusion path uses skip connections in conjunction with a multi-scale pattern fusion module to fully fuse multi-scale brain tumor features and upsample using a linear binary interpolation method to obtain high-resolution prediction results, so as to better locate and distinguish brain tumor areas; S6: A mixed boundary loss function is used to evaluate the difference between the predicted results and the true labels for the brain tumor segmentation results generated by the three paths.

2. The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention according to claim 1 is characterized in that: In step S2, the original four-channel brain tumor image Specific channel selection and splicing operations are performed along the channel dimension, decomposing into two independent features X1 and X2. After feature reorganization, the feature maps are all 2-channel. Feature X1 selects the second and third channels, and feature X2 selects the first and fourth channels. The formula for channel selection and splicing operations on feature maps is as follows: in Indicates that the specific channel of the feature map is selected as c, i represents the specific feature map, when i = 1 When i=2 Concat means channel concatenation along the channel dimension.

3. The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention according to claim 1, characterized in that: In step S3, the dual-trunk feature learning flow based on Transformer is responsible for extracting feature representations from different brain tumor modality images, and extracting feature representations F1 from the first group of modality images X1 through the first feature learning trunk Transformer1 e , extract feature representation from the second set of modal images X2 through the second feature learning backbone Transformer2 After the feature extraction of each level of the two paths is completed, the modal attributes learned in each stage are retained and Attributes means to obtain the modal attributes extracted in each stage. Attributes can be high-level semantic information contained in the image (such as tissue structure, tumor area, etc.) and low-level features (such as edges, textures, etc.). The modal attributes extracted from each stage of the two independent paths are and As the dual input source of the third shared fusion path Transformer3, a multimodal feature representation is obtained:

4. The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention according to claim 1, characterized in that: In step S4, the upper and lower independent paths use edge-driven attention to guide the network to focus on the edge features of the lesion area during the step-by-step upsampling process. This module has three input sources, namely: the modal features corresponding to each level in the encoding layer or High-level prediction features generated by each decoder stage or High-frequency features after Laplacian pyramid processing or The specific process includes: The decoded prediction features are weighted and decomposed to generate two attention maps: the first is non-salient attention, calculated as The attention score of the background area is combined with the encoded feature matrix or The second is the boundary attention map, which generates boundary predictions through Gaussian filtering and Laplacian operators and combines them with or Then, the high-frequency features pre-processed by the Laplacian pyramid are or Use bilinear interpolation to adjust to or The same spatial size is multiplied to extract high-frequency information segments. To optimize the fusion of multimodal information, the background features are or Boundary Features or High frequency features or The concatenation is performed along the channel dimension, and then element-by-element compression and extraction are performed through the fused convolutional layer. The specific operations are as follows: The attention mask is introduced in the i-th layer, and the importance of different features is dynamically adjusted by residual connection with the original encoder features, highlighting the tumor area while suppressing irrelevant background information and noise, guiding the network to focus on the edge and shape features of the brain tumor. The features are then passed to the channel depth convolution calibration block to calibrate the features from the channel and spatial dimensions to further capture the feature correlation between the boundary and background areas. The calculation method is as follows:

5. The brain tumor segmentation algorithm based on multi-path modality fusion and edge-driven attention according to claim 1, characterized in that: In step S5, the decoding part of the third shared fusion path adopts a skip connection in combination with a multi-scale mode fusion module, and the specific process includes: Modality-specific features of two independent path encoders and Separately with the features from the shared fusion path encoder The features are multiplied in pairs, and the obtained features are channel-joined and convolved, retaining the original information in the shared fusion path encoding stage while focusing on the tumor region features of the specific modality. In order to further enhance the features in the shared fusion path encoding stage, and and Matrix element-by-element multiplication is also performed, and the calculation formula is as follows: Through deep fusion interaction between multiple features, the network is ensured to focus on capturing the common features of all modalities. The calculation formula is as follows: Subsequently, the features at different levels in the encoding stage of the shared fusion path are passed as input features to the cross-attention to achieve multi-level and multi-scale feature interaction, allowing high-level features to guide low-level features, allowing the network to focus on the tumor area, eliminating interference from irrelevant tissues, and generating the final multimodal fusion features.

Images