Construction of brain glioma sub-region segmentation model based on multi-modal edge feature fusion

Abstract

The application provides a brain glioma sub-region segmentation model based on multi-modal edge feature fusion, which realizes effective integration of multi-modal features by fusing edge features of a tumor sub-region extracted by a 3D and edge feature interaction and fusion module and an improved trunk network based on FLAIR and T1ce modalities, so as to improve the tumor sub-region segmentation accuracy. Experimental results show that the present method is slightly better than existing methods in terms of Dice and HD95 indexes, and provides an efficient and reliable scheme for brain glioma sub-region segmentation.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis and computer-aided diagnosis, and in particular to the construction of a glioma subregion segmentation model based on multimodal edge feature fusion. Background Technology

[0002] The task of tumor segmentation aims to separate the tumor from normal brain tissue in medical images. It not only provides crucial information such as the tumor's type, location, size, and shape, but also provides vital information for doctors to determine the disease status and develop personalized treatment plans. However, manually segmenting brain tumors is a tedious, time-consuming, and error-prone task. MRI multimodal imaging is currently the mainstream method for the diagnosis and segmentation of gliomas, with T1, T2, FLAIR, and T1ce modalities providing complementary information on anatomical structures, edema areas, and the tumor core, respectively.

[0003] However, clinical practice and existing algorithms still face three major technical challenges in glioma subregion segmentation based on multimodal MRI images, which severely restrict the accuracy and robustness of glioma subregion segmentation: First, current technologies simply stack multimodal information, failing to effectively uncover deep complementary features. Existing deep learning models generally input multimodal MRI images directly in the channel dimension, which may lead to insufficient utilization of multimodal information. Different MRI modalities contain their own unique features, which are often complementary. The current approach of simply stacking multiple modal images fails to delve into the subtle differences and complementary advantages between different modalities. This processing method fails to fully leverage the advantages of each modality, resulting in insufficient precise tumor localization and boundary clarity.

[0004] Second, the ability to capture tumor edges and fine structures is insufficient, and high-frequency details are severely lost. Glioma subregions (such as enhanced tumor core, peritumoral edema, and invasive boundaries) are characterized by irregular shapes, drastic gradient changes, and blurred boundaries. Traditional 3D convolution and U-shaped structures are prone to losing high-frequency edge information during encoding downsampling, and relying solely on deep semantic features cannot accurately depict the fine boundaries of the tumor, resulting in overly smoothed segmentation results and contour deviations, especially with extremely low sensitivity to small-volume invasive regions.

[0005] Third, it is difficult to balance global context and local details, and a multi-scale feature fusion mechanism is lacking. CNN-based models have limited receptive fields and insufficient ability to model long-distance dependencies; while Transformer-based models have global modeling capabilities, they are insufficient in capturing detailed features such as local edges and textures. At the same time, existing skip connections simply concatenate encoded and decoded features without achieving deep interactive fusion of edge features and global semantic features, resulting in segmentation breaks, misclassification, and omissions in complex tumor morphologies and heterogeneous regions.

[0006] In summary, how to integrate multimodal MRI images to improve the accuracy of precise subregion segmentation of gliomas is a key technical challenge that urgently needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to provide a construction of a glioma subregion segmentation model based on multimodal edge feature fusion. Through an optimized model architecture, it can efficiently capture the edge information and complex subregion features of the tumor, showing great advantages in accurate segmentation tasks, enhancing the model's ability to capture detailed information, and improving the ability to process complex tumors.

[0008] To achieve the above objectives, this technical solution provides a method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion, comprising the following steps: S1: Obtain multimodal MRI images labeled with glioma subregions as training datasets, where each set of multimodal MRI images includes T1 modality images, T1 enhanced modality images, T2 modality images, and FLAIR modality images; S2: Construct a glioma subregion segmentation architecture, which includes a 3D edge feature extraction network, a collaborative feature interaction fusion module, an encoder, a bottleneck module, and a decoder. The encoder, bottleneck module, and decoder are cascaded, and there are skip connections between the encoder and decoder. The output features of the encoder at each level are input to the decoder at the corresponding level after passing through the collaborative feature interaction fusion module. The 3D edge feature extraction network includes multi-level cascaded downsampling convolutional blocks. The output features of each downsampling convolutional block are input to the edge feature enhancement module to obtain enhanced features. The enhanced features of the current level and the enhanced features of the previous level are concatenated and input into the edge feature enhancement module to obtain the edge enhanced features of the current level. The edge enhanced features of each level are input into the collaborative feature interaction fusion module at the corresponding level. S3: Input the training dataset into the glioma subregion segmentation architecture to train the glioma subregion segmentation model. FLAIR modal images and T1 enhanced modal images of the same group are input into the 3D edge feature network, and multimodal MRI images of the same group are input into the encoder.

[0009] Secondly, this solution provides a method for glioma subregion segmentation based on multimodal edge feature fusion, including the following steps: S1: Acquire multimodal MRI images, where each set of multimodal MRI images includes T1 modality images, T1-enhanced modality images, T2 modality images, and FLAIR modality images; S2: Input multimodal MRI images into the glioma subregion segmentation model to obtain glioma subregion segmentation results. FLAIR modal images and T1-enhanced modal images of the same group are input into the 3D edge feature network, and multimodal MRI images of the same group are input into the encoder. The glioma subregion segmentation model is constructed according to the aforementioned method for constructing glioma subregion segmentation models.

[0010] Compared with existing technologies, this technical solution has the following characteristics and beneficial effects: This paper presents a glioma subregion segmentation model based on multimodal edge feature fusion. This model integrates tumor subregion edge features extracted using FLAIR and T1ce modalities by a collaborative feature interaction fusion module (3D) and an edge feature extraction network, with the overall full-modal features extracted by the improved backbone network. This integration effectively improves the accuracy of tumor subregion segmentation. Experimental results show that the model slightly outperforms existing methods on both Dice and HD95 metrics, providing an efficient and reliable solution for glioma subregion segmentation. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is an overall architecture diagram of a glioma subregion segmentation model based on multimodal edge feature fusion according to an embodiment of this application.

[0012] Figure 2 This is an architecture diagram of a 3D edge feature extraction network according to an embodiment of this application.

[0013] Figure 3 This is an architecture diagram of the collaborative feature interaction fusion module according to an embodiment of this application.

[0014] Figure 4 This is the architecture diagram of the bottleneck module in this solution.

[0015] Figure 5 The results are from the BraTS2021 dataset.

[0016] Figure 6The results are from the BraTS2019 dataset.

[0017] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0019] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0020] Example 1 Multimodal MRI imaging for gliomas includes four modules: T1 modality, T1-enhanced modality, T2 modality, and FLAIR modality. T1 modality imaging is beneficial for observing anatomical structures, but lesion visualization is not clear enough. T2 modality imaging provides relatively clear lesion visualization, allowing for identification of the entire tumor. FLAIR modality imaging effectively suppresses cerebrospinal fluid signals, highlighting edema (ED) areas, which typically have clear edges or boundaries. T1ce modality imaging enhances the visualization of the tumor core region with contrast agents, providing high-contrast images, particularly suitable for detecting the boundaries of the tumor core. To efficiently fuse multimodal information, especially enhancing the fusion of complementary features between different modalities, this approach provides a method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion, including the following steps: S1: Obtain multimodal MRI images labeled with glioma subregions as training datasets, where each set of multimodal MRI images includes T1 modality images, T1 enhanced modality images, T2 modality images, and FLAIR modality images; S2: Construct a glioma subregion segmentation architecture, which includes a 3D edge feature extraction network, a collaborative feature interaction fusion module, an encoder, a bottleneck module, and a decoder. The encoder, bottleneck module, and decoder are cascaded, and there are skip connections between the encoder and decoder. The output features of the encoder at each level are input to the decoder at the corresponding level after passing through the collaborative feature interaction fusion module. The 3D edge feature extraction network includes multi-level cascaded downsampling convolutional blocks. The output features of each downsampling convolutional block are input to the edge feature enhancement module to obtain enhanced features. The enhanced features of the current level and the enhanced features of the previous level are concatenated and input into the edge feature enhancement module to obtain the edge enhanced features of the current level. The edge enhanced features of each level are input into the collaborative feature interaction fusion module at the corresponding level. S3: Input the training dataset into the glioma subregion segmentation architecture to train the glioma subregion segmentation model. FLAIR modal images and T1 enhanced modal images of the same group are input into the 3D edge feature network, and multimodal MRI images of the same group are input into the encoder.

[0021] It should be noted that, as mentioned earlier, existing deep learning-based brain tumor segmentation methods often simply stack multimodal MRI images as input to semantic segmentation models. This may lead to insufficient utilization of multimodal information. Different modalities contain their own unique features, which are often complementary. If multiple modalities are simply stacked together, the model fails to delve into the subtle differences and complementary advantages between different modalities. This approach fails to fully leverage the strengths of each modality, potentially resulting in insufficient accurate tumor localization and boundary clarity. Therefore, this solution specifically designs a 3D edge feature extraction network. This network extracts rich edge information from the original FLAIR modal images and T1-enhanced modal images, enhancing the fine boundary features of tumor subregions. By combining convolutional operations and edge detection algorithms, high-frequency edge features are extracted to help the network accurately distinguish different tumor subregions.

[0022] The architecture of the 3D edge feature extraction network is as follows: Figure 2 As shown, this 3D edge feature extraction network utilizes multi-level downsampling convolutional blocks and edge feature enhancement modules to efficiently extract the boundary information of tumor subregions from FLAIR modal images and T1 enhanced modal images.

[0023] Specifically, each convolutional block in the 3D edge feature extraction network consists of two sequentially connected 3D edge features. 3 convolutional layers, 1 regularization layer, and 1 2 2. Max pooling layers, with convolutional blocks of different levels used to extract features of different dimensions.

[0024] The edge feature enhancement module in this 3D edge feature extraction network includes 3D Sobel convolution, 3D Laplacian convolution, batch normalization layer, ReLU layer, and 3D convolution. The features input to the edge feature enhancement module are concatenated by 3D Sobel convolution and 3D Laplacian convolution, respectively. The concatenated features are then sequentially input into the batch normalization layer, ReLU layer, and 3D convolution, and concatenated with the output features of the 3D Laplacian convolution. The concatenated features are then input into the 3D convolution to obtain the output features of the edge feature enhancement module.

[0025] The edge feature enhancement module of this scheme uses 3D Sobel convolution and 3D Laplacian convolution for edge feature extraction. The 3D Sobel convolution operator defines 3D convolution kernels in the x, y, and z directions. The input image enters the 3D Sobel convolution to calculate gradients in these three directions, and the square root of the sum of squared gradients is used to obtain the first image edge intensity. Each 3D convolution kernel is a 3×3×3 filter, capable of capturing changes in the image along that direction, as shown below:

[0026]

[0027] ; Where gx, gy, and gz are the gradients of the input image in the x, y, and z directions, respectively.

[0028] The magnitude of the gradient is obtained by calculating the square root of the sum of squared gradients in each direction. The formula for calculating the edge intensity of an image is as follows: .

[0029] Unlike the Sobel operator, the Laplacian operator in 3D Laplacian convolution captures the rate of change in an image and highlights subtle edge features by calculating the second derivative of the image. Specifically, 3D Laplacian convolution defines a Laplacian convolution kernel. The input image is processed by convolution in the 3D Laplacian convolution to obtain the second derivative of the image at each voxel, which is used as the second image edge intensity to detect edges in the image. The convolution operation formula for the Laplacian convolution kernel is as follows: ; Where g is the output gradient image (i.e., edge information), and F(i, j, k) is the input image. It is a Laplacian convolution kernel.

[0030] It should be noted that after activating the gradients of the first image edge intensity obtained from 3D Sobel convolution and the second image edge intensity obtained from 3D Laplacian convolution using the Sigmoid activation function, the calculated edge intensity has a good numerical range. This is then multiplied back into the input image to preserve the spatial features of the original image. Subsequently, the features output from the 3D Sobel convolution and the 3D Laplacian convolution are concatenated and then sequentially input into the batch normalization layer, the ReLU layer, and the 3D convolution, where they are concatenated with the output features of the 3D Laplacian convolution. During the edge feature enhancement process, the output features of one layer are added to the output features of the next layer in a progressive manner to obtain richer edge features.

[0031] It should be noted that the edge enhancement features at each level of the 3D edge feature extraction network in this scheme can provide rich edge information. Therefore, inputting these edge enhancement features into the corresponding level's collaborative feature interaction fusion module can ensure efficient utilization of edge features. It should also be noted that in the step of "concatenating the enhancement features of the current level and the enhancement features of the previous level and inputting them into the edge feature enhancement module to obtain the edge enhancement features of the current level," the enhancement features of the first level are directly used as the edge enhancement features of the first level.

[0032] In some embodiments, the images input to the 3D edge feature network are preprocessed, i.e., the FLAIR modal images and T1 enhanced modal images of the same group are preprocessed. The advantage of doing so is that it ensures that the input data meets the network's uniform input requirements, which helps to accelerate the convergence speed of network training. It also ensures that the data is within a reasonable numerical range in subsequent processing, providing suitable input for edge feature extraction.

[0033] In some embodiments, preprocessing means include any of the following: image normalization, flipping, cropping, intensity shifting, rotation, and filling.

[0034] In some embodiments, the 3D edge feature extraction network includes four cascaded convolutional blocks. Specifically, after an image is input into the 3D edge extraction network, the output features obtained after passing through four downsampled convolutional blocks are: ; in The output features of the first-level downsampling convolutional module, The output features of the second-level downsampling convolutional module, The output features of the third-level downsampling convolutional module, R is the output feature of the fourth-level downsampling convolutional module, H is the image input to the 3D edge feature network, and W is the image height and width.

[0035] As described above, the 3D edge feature extraction network in this scheme successfully preserves details and edge information in images through this multi-level edge enhancement method, playing a crucial role in tumor subregion segmentation tasks. Its refined edge feature extraction and enhancement capabilities enable the model to more accurately identify and segment tumor regions in complex medical images.

[0036] The architecture of the collaborative feature interaction fusion module is as follows: Figure 3 As shown: The collaborative feature interaction fusion module of this scheme introduces a cross-attention mechanism to enhance the model's fusion of different input features. In particular, in the glioma subregion segmentation architecture, it replaces the traditional skip connections and fuses the edge enhancement features of each level in the D edge feature extraction network. Through this design, the model can effectively fuse multi-level features between the encoder and decoder, enhancing its ability to capture details and global information.

[0037] In the step of "inputting the edge enhancement features of each level into the corresponding collaborative feature interaction fusion module", the level of the downsampling convolutional block is the same as the level of the encoder and corresponds one-to-one. The level of the downsampling convolutional block is defined sequentially according to the input order of the features.

[0038] The collaborative feature interaction module of this scheme includes a channel attention module, a cross attention module, a spatial attention module, and a feature interaction module. The overall features output by the encoder and the edge enhancement features of the same level are input into the channel attention module of the collaborative feature interaction module to obtain the overall channel attention and edge channel attention, respectively. The overall channel attention and edge channel attention are then processed by 1... 1. After convolution processing, features are extracted using the ReLU activation function to obtain overall and edge features. These features are then input into the cross-attention module for matrix multiplication to obtain the interactive attention matrix. The cross-attention matrix is ​​standardized using the softmax activation function. Based on the standardized cross-attention matrix, the encoder's output features and edge enhancement features are processed by matrix multiplication to obtain overall and edge weighted features. These features are then input into the spatial attention module to obtain overall and edge spatial weighted features, respectively. Finally, these features are input into the feature interaction module for cross-self-attention mechanism processing to obtain collaborative interactive features.

[0039] Specifically, in the step of "inputting the overall features output by the encoder and the edge enhancement features of the same level into the channel attention module in the collaborative feature interaction module to obtain the overall channel attention and edge channel attention respectively", the overall features are subjected to average pooling and max pooling to obtain the overall channel attention, and the channel attention is then subjected to 1... After convolution, the overall channel attention is extracted using the ReLU activation function. Edge enhancement features are then processed through average pooling and max pooling to obtain edge channel attention. Finally, the channel attention is processed using a 1-step method. 1. After convolution, the edge channel attention is extracted using the ReLU activation function.

[0040] The formulas for calculating overall channel attention and edge channel attention are as follows:

[0041] ; Where A1 is the overall channel attention, A2 is the edge channel attention, avg1 and avg2 are average pooling, max1 and max2 are max pooling, and Conv1 is the average channel attention. 1() is 1 1. Convolution, ReLU() is the ReLU activation function.

[0042] The formula for calculating the interaction attention matrix by inputting the overall extracted features and edge extracted features into the cross-attention module for matrix multiplication is as follows: ; Where A1 is the global channel attention, A2 is the edge channel attention, T represents the inverted matrix, and C... cross It is a cross-attention matrix.

[0043] Regarding the standardization of the cross-attention matrix using the softmax activation function, the calculation formulas for the overall weighted features and edge-weighted features obtained by processing the encoder's output features and edge enhancement features through matrix multiplication based on the standardized cross-attention matrix are as follows: ; ; Where C cross It is a cross-attention matrix, where X1 is the overall feature, X2 is the edge enhancement feature, softmax() is the softmax activation function, W1 is the overall weighted feature, and W2 is the edge weighted feature.

[0044] In the step of "inputting the overall weighted features and edge weighted features into the spatial attention module to obtain the overall spatial weighted features and edge spatial weighted features respectively", the overall weighted features are spatially averaged pooled and max pooled and then stacked. The stacked features are then convolved to obtain the overall intermediate weighted features. The overall intermediate weighted features and the overall features are multiplied together and the result of the multiplication is added to the overall features to obtain the overall spatial weighted features. Similarly, the edge weighted features are spatially averaged pooled and max pooled and then stacked. The stacked features are then convolved to obtain the edge intermediate weighted features. The edge intermediate weighted features and the edge enhancement features are multiplied together and the result of the multiplication is added to the edge enhancement features to obtain the edge spatial weighted features.

[0045] The formulas for calculating the overall spatial weighted features and the marginal spatial weighted features are as follows:

[0046]

[0047] Where Z1 is the overall spatial weighted feature, S1 is the overall intermediate weighted feature, X1 is the overall feature, Z2 is the edge spatial weighted feature, S2 is the edge intermediate weighted feature, and X2 is the edge enhancement feature.

[0048] In the step of "inputting the overall spatial weighted features and edge spatial weighted features into the feature interaction module for cross-self-attention mechanism processing to obtain collaborative interaction features", the overall spatial weighted features and edge spatial weighted features are flattened and segmented respectively to obtain processed overall spatial weighted features and processed edge spatial weighted features. The processed overall spatial weighted features and processed edge spatial weighted features are input into the cross-attention mechanism. The cross-attention mechanism uses the processed overall spatial weighted features as the query and the processed edge spatial weighted features as the key and value to calculate the attention matrix. The attention matrix and the overall spatial weighted features are weighted to obtain the first weighted feature. The attention matrix and the edge spatial weighted features are weighted to obtain the second weighted feature. The average of the first weighted feature and the second weighted feature is taken to obtain the collaborative interaction feature.

[0049] Furthermore, the flattening process converts the height, width, depth, and other dimensions of the input overall spatial weighted features and edge spatial weighted features into one-dimensional vectors. The block processing divides the flattened features into blocks, where each block corresponds to a small region and is mapped to a vector in a low-dimensional space to obtain the processed overall spatial weighted features and the processed edge spatial weighted features.

[0050] Furthermore, in the cross-attention mechanism, the correlation between the processed global spatial weighted features and the processed edge spatial weighted features is learned by calculating the dot product of the query and the key to obtain the attention matrix.

[0051] The calculation formula for the cross-self-attention mechanism is as follows:

[0052]

[0053]

[0054]

[0055] ; Where X1' is the processed overall spatial weighted feature, X2' is the processed marginal spatial weighted feature, Q is the query, K is the key, V is the value, Patch is the block processing, Embedding is the flattening processing, Softmax() is the activation function, O1 is the first weighted feature, O2 is the second weighted feature, and F... fused It is a collaborative interaction feature.

[0056] As described above, the collaborative feature interaction module of this scheme effectively integrates feature information from both the overall and peripheral regions by combining channel attention, spatial attention, and cross-attention mechanisms. This enhances the model's ability to process multi-level features, especially in the glioma subregion segmentation task. This module can perform weighted fusion of features from the overall tumor region and peripheral regions, improving the network's ability to perceive both detailed and global information. This not only enhances the performance of glioma subregion segmentation but also effectively improves the model's robustness and accuracy.

[0057] Furthermore, the bottleneck module in the glioma subregion segmentation architecture of this scheme includes a multi-layered SwinTransformer to improve the global modeling capability of the glioma subregion segmentation model. It should be noted that the SwinTransformer, as a computationally efficient Transformer architecture, utilizes window self-attention (W-MSA) and offset window self-attention (SW-MSA) mechanisms to capture global contextual information. By stacking multiple SwinTransformerBlocks, the model can learn more complex features at different levels.

[0058] Specifically, such as Figure 4As shown, the bottleneck module of this scheme includes multiple cascaded Swing Transformer blocks. The features input into the Swing Transformer block are processed by image segmentation. The features output from the Swing Transformer block are concatenated and then input into the next Swing Transformer block. Each Swing Transformer block includes a window self-attention mechanism, an offset window self-attention mechanism, and a multi-layer perceptual layer.

[0059] In some embodiments, the bottleneck module of this scheme stacks multiple Swing Transformer blocks to learn more complex features at different levels. The output of each Swing Transformer block provides an expression of information at different levels, thus enabling the output features after multiple Swing Transformer blocks to possess abstract and rich semantic information. The output features of each Swing Transformer block are represented as follows: ; Where X L X is the feature input into the Lth Swing Transformer block. L+1 It is the output feature of the Lth SwingTransformer block.

[0060] Specifically, regarding the structure of each Swing Transformer block: Each Swing Transformer block includes a cascaded window self-attention mechanism, a multilayer perceptron layer, an offset window self-attention mechanism, and another multilayer perceptron layer. The features input to the Swing Transformer block are sequentially processed through the window self-attention mechanism, the multilayer perceptron layer, the offset window self-attention mechanism, and the multilayer perceptron layer before being output as features.

[0061] The specific process for the bottleneck module is as follows: First, the feature map of the input Swin Transformer block is divided into blocks using a window of size (4, 4, 4) pixels. These blocks are then used in the self-attention calculation of subsequent Swin Transformer blocks. This process can be represented by the following formula:

[0062] Where X i For the i-th small block, the size is 4×4×4. Each small block will contain part of the information of the feature map. PatchParitition() is the block processing.

[0063] Subsequently, each small block is input into the Swin Transformer Block and subjected to window self-attention mechanism and offset window self-attention mechanism respectively. The window self-attention mechanism calculates self-attention within each window to capture local context information, while the offset window self-attention mechanism increases cross-window information interaction by offsetting the window, thereby enhancing the capture of global information. The window self-attention and offset window self-attention mechanisms are as follows:

[0064]

[0065] Among them, W-MSA() is the window self-attention mechanism, SW-MSA() is the offset window self-attention mechanism, d is the dimension of each head, Q, K and V are the query, key and value matrices respectively, the softmax function is used to calculate attention weights, and then weight the value matrix V, ShiftedK is the offset key matrix, which enhances the interaction between different windows.

[0066] Finally, the features from each window after self-attention computation are reassembled back to their original spatial dimensions. The window merging operation integrates feature information from different windows by aggregating information within each window and returning it to the global space, thereby enhancing the representation of global contextual information.

[0067] Similarly, in image segmentation tasks, especially complex tasks such as tumor subregion segmentation, capturing details and multi-scale features is crucial. To effectively extract features at different scales, the encoder and decoder in this scheme employ a dual-path multi-scale convolution module that combines ordinary convolution and multi-level dilated convolution. This dual-path multi-scale convolution module not only extracts local information through convolution operations but also expands the receptive field through dilated convolution, effectively capturing global contextual information.

[0068] Specifically, each dual-path multi-scale convolutional module includes a 3D convolutional block and a 3D dilated convolutional block. The 3D convolutional block consists of two convolutional layers, a group normalization layer, and a ReLU activation function. The 3D dilated convolutional block inserts holes into the 3D convolutional kernel, and the features output from the 3D convolutional block are added to the features input to the 3D convolutional block before being input into the 3D dilated convolutional block. This scheme expands the receptive field by adding holes to the 3D dilated convolutional block, thereby enhancing the model's ability to capture global information. Specifically, the 3D dilated convolutional block is represented as follows: ; Where d is the dilation rate, which determines the size of the holes between the convolution kernels, X is the feature of the input 3D dilated convolution block, W is the width, and Dliated() is 3D dilated convolution.

[0069] In some embodiments, each dual-path multi-scale convolutional module includes 3D dilated convolutional blocks with different dilation rates, where the dilation rate determines the hole size. Dilated convolution increases the receptive field without increasing computation, enabling the convolutional operation to capture a wider range of contextual information. Generally, a larger dilation rate results in a larger receptive field, thereby helping the network learn dependencies over longer distances.

[0070] Furthermore, each dual-path multi-scale convolutional module includes 3D dilated convolutional blocks with dilation rates of 1, 3, and 5. The features extracted by the convolutional operation at each dilation rate correspond to different ranges of contextual information. By combining dilated convolutions with different dilation rates, the network can capture information simultaneously at multiple scales, thereby enhancing its ability to perceive targets of different shapes.

[0071] Example 2 This solution provides a glioma subregion segmentation model based on multimodal edge feature fusion, constructed according to the construction method of the glioma subregion segmentation model based on multimodal edge feature fusion shown in Embodiment 1. This glioma subregion segmentation model based on multimodal edge feature fusion can segment glioma subregions based on multimodal edge features.

[0072] To characterize the performance of this glioma subregion segmentation model based on multimodal edge feature fusion, our team conducted the following feasibility verification. The "MME-TransBTS model" below represents the glioma subregion segmentation model based on multimodal edge feature fusion in this proposal: 1. Evaluation results on the BraTS2021 dataset In comparative experiments with other state-of-the-art (SOTA) models, the MME-TransBTS model demonstrated superior segmentation performance on various datasets. As shown in Table 1, on the BraTS2021 public dataset, compared to numerous models such as UNet 3D, AttentionUNet, TransBTS, TransUNet, Swin UNETR, UNETR, SwinUNet 3D, nnUNetV2, and SDV-TUNet, MME-TransBTS stood out in key metrics such as Dice and HD95.

[0073] Table 1 presents the quantitative results of all comparative models on the BraTS2021 dataset. The MME-TransBTS model proposed in this study outperforms existing segmentation models in both Dice and HD95 for the ET and WT regions, achieving state-of-the-art (SOTA) segmentation performance. Specifically, the Dice score for ET is the best (84.71%), 0.45% higher than the second-best (nnUNetV2) score (84.26%); the HD95 score for ET is the best (6.83), 0.79 lower than the second-best (SDV-TUNet) score (7.62). For WT, the Dice score is the best (92.11%), 0.33% higher than the second-best (nnUNetV2) score (91.78%); the HD95 score for WT is the best (6.18), 0.4 lower than the second-best (SDV-TUNet) score (6.58). This fully demonstrates the superior performance of MME-TransBTS in high-precision tumor region segmentation. By accurately capturing the subtle differences between the ET and WT regions, MME-TransBTS can effectively improve segmentation accuracy, reduce missed detections and misjudgments of tumor regions, and thus generate more accurate and comprehensive tumor segmentation results.

[0074] The ET region is typically defined by comparing the difference in high-signal areas in the pre- and post-contrast images (T1 and T1ce). The enhanced T1 image (T1ce) more clearly identifies the enhanced areas of the tumor due to the contrast agent injection. Therefore, the ET region is mainly composed of these high-signal areas. By fusing features from multimodal input images and edge features from FLAIR and T1ce modalities, the MME-TransBTS model can combine information from different modalities, significantly improving the segmentation performance of the ET and WT regions, outperforming other baseline models.

[0075] Figure 5 Qualitative results are presented compared to several representative baseline models. Three cases were randomly selected from the test set results. The first row shows that the MME-TransBTS model achieves better segmentation results in the peritumoral edema area. In the second row, compared to all other models, the MME-TransBTS model's results in the enhanced tumor area are closer to the labeled image. In the third row, the MME-TransBTS model achieves more complete and accurate boundaries.

[0076] Table 1 shows the evaluation results on the BraTS2021 dataset. .

[0077] 2. Evaluation results on the BraTS2019 dataset As shown in Table 2, when the experimental dataset is switched to BraTS2019, MME-TransBTS still achieves better results in key metrics such as Dice and HD95 when facing many models such as UNet 3D, Attention UNet, TransBTS, TransUNet, Swin UNETR, UNETR, SwinUNet 3D, nnUNetV2, and SDV-TUNet.

[0078] Table 2 shows the evaluation results on the BraTS2019 dataset. .

[0079] Table 2 presents the quantitative results of all comparative models on the BraTS2019 dataset. The MME-TransBTS model proposed in this study outperforms existing segmentation models in both Dice and HD95 for the ET and WT regions, achieving state-of-the-art (SOTA) segmentation performance. Specifically, the Dice score for ET is the best (76.30%), 0.16% higher than the second-best (nnUNetV2) score (76.14%); the HD95 score for ET is the best (12.47), 0.27 lower than the second-best (nnUNetV2) score (13.74). For WT, the Dice score is the best (89.24%), 0.17% higher than the second-best (nnUNetV2) score (89.07%); the HD95 score for WT is the best (6.84), 0.23 lower than the second-best (UNet 3D) score (7.07). These results further demonstrate the excellent segmentation performance of this model.

[0080] Figure 6 Qualitative results are presented compared to representative baseline models. Three cases were randomly selected from the test set results. The first row shows that the MME-TransBTS model is closer to the true label in terms of overall segmentation. In the second row, the peritumoral edema area is closest to the true label compared to all other models. In the third row, the MME-TransBTS model significantly outperforms other baseline models in segmentation of enhanced tumor areas.

[0081] 3. MME-TransBTS ablation experiment Based on the BraTS2021 dataset, this study further conducted ablation experiments on each module of the model to explore their contribution to the overall segmentation performance. Experimental results show that the MME-TransBTS model continuously improves its performance on the BraTS2021 dataset by gradually adding components.

[0082] As shown in Table 3, when using only the TransBTS model, the Dice values ​​for the ET, TC, and WT regions are 83.07%, 87.71%, and 88.67%, respectively, while the HD95 values ​​are 16.79, 12.89, and 8.29. However, after switching to a TransBTS variant, the Dice values ​​increase to 84.64%, 87.23%, and 91.03%, while the HD95 values ​​decrease to 11.24, 12.77, and 7.28, demonstrating a significant improvement in segmentation performance. This indicates that by improving the encoder-decoder and bottleneck regions of TransBTS, the model's segmentation performance in the ET region is significantly enhanced.

[0083] When 3D EdgeNet is further introduced, the model's Dice in the ET, TC and WT regions increases to 84.56%, 87.57% and 91.97% respectively, while HD95 decreases to 10.51, 11.01 and 6.41, indicating that this module plays an important role in refining tumor segmentation boundaries and significantly optimizes the overall segmentation accuracy.

[0084] Finally, after adding the CIFM module, the performance of the MME-TransBTS model was further improved. The Dice values ​​for the ET, TC, and WT regions reached 84.71%, 89.89%, and 92.11%, respectively, while the HD95 values ​​decreased to 6.83, 6.49, and 6.18. Overall, the model outperformed other baseline models, demonstrating superior segmentation performance. This fully illustrates the contribution of the CIFM feature fusion module to improving the accuracy of detail segmentation and reducing errors.

[0085] In summary, the experimental results of the MME-TransBTS model on different datasets fully demonstrate its superior performance in complex tumor region segmentation tasks. Furthermore, the ablation experiments clarified the importance of each module in the model, providing strong evidence for further optimization and improvement. In clinical applications, it can provide more reliable tumor region identification, offering stronger support for subsequent diagnostic and treatment decisions.

[0086] Table 3 Ablation Experimental Evaluation Data .

[0087] Example 3 This solution provides a glioma subregion segmentation method based on multimodal edge feature fusion, including the following steps: S1: Acquire multimodal MRI images, where each set of multimodal MRI images includes T1 modality images, T1-enhanced modality images, T2 modality images, and FLAIR modality images; S2: Input the multimodal MRI images into the glioma subregion segmentation model of Example 2 to obtain the glioma subregion segmentation results. The FLAIR modal images and T1 enhanced modal images of the same group are input into the 3D edge feature network, and the multimodal MRI images of the same group are input into the encoder.

[0088] Example 4 This embodiment also provides an electronic device, see reference. Figure 7 It includes a memory 404 and a processor 402, the memory 404 storing a computer program and the processor 402 being configured to run the computer program to perform the steps in any of the above embodiments of the method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion.

[0089] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0090] The memory 404 may include a large-capacity memory 404 for data or instructions. The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0091] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the methods for constructing a glioma subregion segmentation model based on multimodal edge feature fusion in the above embodiments.

[0092] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0093] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0094] The input / output device 408 is used to input or output information. In this embodiment, the input information may be multimodal MRI images, etc., and the output information may be the subregional segmentation results of gliomas, etc.

[0095] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program: S1: Obtain multimodal MRI images labeled with glioma subregions as training datasets, where each set of multimodal MRI images includes T1 modality images, T1 enhanced modality images, T2 modality images, and FLAIR modality images; S2: Construct a glioma subregion segmentation architecture, which includes a 3D edge feature extraction network, a collaborative feature interaction fusion module, an encoder, a bottleneck module, and a decoder. The encoder, bottleneck module, and decoder are cascaded, and there are skip connections between the encoder and decoder. The output features of the encoder at each level are input to the decoder at the corresponding level after passing through the collaborative feature interaction fusion module. The 3D edge feature extraction network includes multi-level cascaded downsampling convolutional blocks. The output features of each downsampling convolutional block are input to the edge feature enhancement module to obtain enhanced features. The enhanced features of the current level and the enhanced features of the previous level are concatenated and input into the edge feature enhancement module to obtain the edge enhanced features of the current level. The edge enhanced features of each level are input into the collaborative feature interaction fusion module at the corresponding level. S3: Input the training dataset into the glioma subregion segmentation architecture to train the glioma subregion segmentation model. FLAIR modal images and T1 enhanced modal images of the same group are input into the 3D edge feature network, and multimodal MRI images of the same group are input into the encoder.

[0096] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0097] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0098] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer-executable components configured to perform embodiments when the program is run. One or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted that any block in the logical flow of the figures may represent a program step, or interconnected logical circuitry, blocks and functions, or a combination of program steps and logical circuitry, blocks and functions. The software may be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0099] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion, characterized in that, Includes the following steps: S1: Obtain multimodal MRI images labeled with glioma subregions as training datasets, where each set of multimodal MRI images includes T1 modality images, T1 enhanced modality images, T2 modality images, and FLAIR modality images; S2: Construct a glioma subregion segmentation architecture, which includes a 3D edge feature extraction network, a collaborative feature interaction fusion module, an encoder, a bottleneck module, and a decoder. The encoder, bottleneck module, and decoder are cascaded and connected in a skip connection. The output features of the encoder at each level are input into the decoder at the corresponding level after passing through the collaborative feature interaction fusion module. The 3D edge feature extraction network includes multi-level cascaded downsampling convolutional blocks. The output features of each downsampling convolutional block are input into the edge feature enhancement module to obtain enhanced features. The enhanced features of the current level and the enhanced features of the previous level are concatenated and input into the edge feature enhancement module to obtain the edge enhanced features of the current level. Each level of edge enhancement features is input into the corresponding level's collaborative feature interaction fusion module. The collaborative feature interaction module includes a channel attention module, a cross attention module, a spatial attention module, and a feature interaction module. The overall features output by the encoder and the edge enhancement features of the same level are input into the channel attention module within the collaborative feature interaction module to obtain overall channel attention and edge channel attention, respectively. The overall channel attention and edge channel attention are then processed using a 1...

1. After convolution processing, features are extracted using the ReLU activation function to obtain overall and edge features. These features are then input into the cross-attention module for matrix multiplication to obtain the interaction attention matrix. The cross-attention matrix is ​​standardized using the softmax activation function. Based on the standardized cross-attention matrix, the encoder's output features and edge enhancement features are processed by matrix multiplication to obtain overall and edge weighted features. These features are then input into the spatial attention module to obtain overall and edge spatial weighted features, respectively. Finally, these features are input into the feature interaction module for cross-self-attention mechanism processing to obtain collaborative interaction features. S3: Input the training dataset into the glioma subregion segmentation architecture to train the glioma subregion segmentation model. FLAIR modal images and T1 enhanced modal images from the same group are input into the 3D edge feature extraction network, and multimodal MRI images from the same group are input into the encoder.

2. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 1, characterized in that, The edge feature enhancement module includes 3D Sobel convolution, 3D Laplacian convolution, batch normalization layer, ReLU layer, and 3D convolution. The features input to the edge feature enhancement module are concatenated by 3D Sobel convolution and 3D Laplacian convolution. The concatenated features are then sequentially input into the batch normalization layer, ReLU layer, and 3D convolution, and concatenated with the output features of the 3D Laplacian convolution. The concatenated features are then input into the 3D convolution to obtain the output features of the edge feature enhancement module.

3. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 2, characterized in that, The 3D Sobel convolution operator defines 3D convolution kernels in the x, y, and z directions. The input image enters the 3D Sobel convolution to calculate gradients in the three directions, and the square root of the sum of squared gradients is used to obtain the first image edge intensity. The 3D Laplacian convolution defines a Laplacian convolution kernel. The input image enters the 3D Laplacian convolution to perform convolution processing, and the second derivative of the image on each voxel is used as the second image edge intensity. The first image edge intensity obtained from the 3D Sobel convolution and the second image edge intensity obtained from the 3D Laplacian convolution are activated by the Sigmoid activation function and then concatenated. The concatenated features are then input into the 3D convolution to obtain the output features of the edge feature enhancement module.

4. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 1, characterized in that, The overall weighted features are spatially averaged and max-pooled and then stacked. The stacked features are then convolved to obtain the overall intermediate weighted features. The overall intermediate weighted features and the overall features are multiplied together, and the result is added to the overall features to obtain the overall spatial weighted features. Similarly, the edge weighted features are spatially averaged and max-pooled and then stacked. The stacked features are then convolved to obtain the edge intermediate weighted features. The edge intermediate weighted features and the edge enhancement features are multiplied together, and the result is added to the edge enhancement features to obtain the edge spatial weighted features.

5. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 1, characterized in that, The overall spatial weighted features and edge spatial weighted features are flattened and segmented to obtain processed overall spatial weighted features and processed edge spatial weighted features, respectively. The processed overall spatial weighted features and processed edge spatial weighted features are then input into a cross-attention mechanism. The cross-attention mechanism uses the processed overall spatial weighted features as the query and the processed edge spatial weighted features as the key and value to calculate the attention matrix. The attention matrix and the overall spatial weighted features are weighted to obtain the first weighted feature, and the attention matrix and the edge spatial weighted features are weighted to obtain the second weighted feature. The average of the first weighted feature and the second weighted feature is taken to obtain the collaborative interaction feature.

6. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 1, characterized in that, The bottleneck module consists of multiple cascaded Swing Transformer blocks. The features input to the Swing Transformer block are processed by image segmentation. The features output from the Swing Transformer block are concatenated and then input into the next Swing Transformer block. Each Swing Transformer block includes a window self-attention mechanism, an offset window self-attention mechanism, and a multilayer perceptron layer.

7. The method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion according to claim 1, characterized in that, The encoder and decoder employ a dual-path multi-scale convolutional module that combines ordinary convolution and multi-level dilated convolution. Each dual-path multi-scale convolutional module includes a 3D convolutional block and a 3D dilated convolutional block. The 3D convolutional block consists of two convolutional layers, a group normalization layer, and a ReLU activation function. The 3D dilated convolutional block inserts holes in the 3D convolutional kernel, and the features output by the 3D convolutional block are added to the features input to the 3D convolutional block before being input into the 3D dilated convolutional block.

8. A method for glioma subregion segmentation based on multimodal edge feature fusion, characterized in that, Includes the following steps: S1: Acquire multimodal MRI images, where each set of multimodal MRI images includes T1 modality images, T1-enhanced modality images, T2 modality images, and FLAIR modality images; S2: Input multimodal MRI images into the glioma subregion segmentation model to obtain glioma subregion segmentation results. FLAIR modal images and T1-enhanced modal images of the same group are input into the 3D edge feature extraction network, and multimodal MRI images of the same group are input into the encoder. The glioma subregion segmentation model is constructed using the method described in any one of claims 1 to 7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the method for constructing a glioma subregion segmentation model based on multimodal edge feature fusion as described in any one of claims 1 to 7.

Images