Dual-branch ramsay gating diagram aggregation multi-modal meningioma segmentation method

Abstract

The application discloses a double-branch Raio gating graph aggregation multi-modal meningioma segmentation method, belongs to the cross technical field of computer vision and medical image processing, is used for meningioma segmentation, and comprises the following steps: preparing a data set, constructing a deep neural network model, and performing neural network training; inputting the data set into the trained deep neural network model; first inputting a double-branch multi-modal input module to output a multi-modal fusion feature tensor; then inputting the multi-modal fusion feature tensor into a hierarchical Swin Transformer encoder to output three-dimensional segmentation masks of enhanced tumors, tumor cores and whole tumor regions of meningioma. Through double-branch multi-modal input fusion, hierarchical Swin Transformer coding, edge perception modulation and Laplace gating graph aggregation, high-precision three-dimensional segmentation of enhanced tumors, tumor cores and whole tumor regions of meningioma is realized.

Description

Technical Field

[0001] This invention discloses a multimodal meningioma segmentation method using a dual-branch Lagrange gating image aggregation, belonging to the interdisciplinary field of computer vision and medical image processing. Background Technology

[0002] First, existing feature fusion schemes suffer from misalignment of multimodal heterogeneous features and insufficient semantic complementarity. Accurate segmentation of meningiomas highly relies on complementary information from multimodal MRI data; however, many existing fusion techniques primarily employ simple channel stitching at the input level. Direct stitching ignores the significant differences in semantic feature hierarchy and spatial distribution among different modalities, causing the model to fail to accurately focus on specific lesion-sensitive modalities, often resulting in feature misalignment. The challenge in addressing this issue lies in designing a differentiated attention mechanism that can adaptively filter sensitive modalities in the channel dimension while accurately focusing on the lesion region in the spatial dimension, thereby achieving semantic alignment and effective fusion of heterogeneous features.

[0003] Secondly, existing CNN and Transformer architectures suffer from edge information loss and blurred tumor boundaries due to downsampling operations. The convolution stride operation in CNNs and the image patch segmentation process in Transformers are essentially downsampling processes. This inevitably disrupts the local spatial structure of the image, leading to the irreversible loss of high-frequency edge spatial information. This is a critical flaw for meningiomas, which already possess blurred boundaries; even with the blurred boundaries of meningiomas affected by skull interference, the segmentation results remain insufficiently precise. Existing hybrid architectures typically rely on simple cascading strategies, failing to fundamentally address the problem of edge information entropy loss, resulting in a lack of explicit modeling and recovery of edge residuals during the feature extraction stage.

[0004] Finally, existing voxel-level classification methods lack efficient global topological constraints, leading to fragmentation and isolated noise in the segmentation results. Most existing methods are still based on independent pixel-level classification, lacking sufficient consideration of structural topological continuity. This easily results in frequent internal discontinuities or isolated noise in the segmentation of large-volume tumors. Although the academic community has attempted to introduce conditional random fields or graph convolutional networks to capture structural relationships in non-Euclidean spaces, constructing extensive graph structures typically incurs a huge computational burden. Therefore, the current challenge lies in how to explicitly constrain topological consistency within the feature space while maintaining computational efficiency, ensuring the anatomical continuity and integrity of the segmentation results. Summary of the Invention

[0005] The purpose of this invention is to provide a bi-branch Lagrange gating graph aggregation multimodal meningioma segmentation method to solve the problems in the prior art, such as misalignment of multimodal heterogeneous features, loss of edge information due to downsampling, and segmentation breakage due to lack of topological constraints.

[0006] A multimodal segmentation method for meningiomas using a dual-branch Lagrange gating map is included, comprising: Prepare the dataset, build a deep neural network model, and train the neural network. The deep neural network model includes a dual-branch multimodal input module and a hierarchical Swing Transformer encoder. After training, the dataset is input into the trained deep neural network model. First, it is input into the dual-branch multimodal input module, which outputs a multimodal fusion feature tensor. Then, the multimodal fusion feature tensor is input into the hierarchical SwinTransformer encoder, which outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

[0007] The hierarchical Swin Transformer encoder includes an image block partitioning layer, a linear embedding layer, four Swin Transformer blocks, five UnetR Block layers, five edge-aware feature modulation modules, a Laplacian gating graph aggregation module, and five decoders. The Swin Transformer block consists of a Block layer and a Merge layer.

[0008] Three-dimensional multimodal MRI images of the patient to be segmented were acquired, including four modal data: T1-weighted image, T1-contrast enhancement image (T1Ce), T2-weighted image, and T2-fluid attenuation inversion recovery image (FLAIR). The data were normalized and used as input to a deep neural network model.

[0009] The normalized T1, T1Ce, T2 and FLAIR are input into the dual-branch multimodal input module, and the multimodal fusion feature tensor is output.

[0010] The dual-branch multimodal input module includes two parallel branches: the normalized T1 and T1Ce are input into the first branch, and the normalized T2 and FLAIR are input into the second branch. The first branch includes two sub-branches. The first sub-branch includes an average pooling layer, a linear layer, a ReLU function, another linear layer, and a sigmoid function in sequence. The second sub-branch multiplies the input and output of the first branch after recalibration and outputs the result of the first branch. The second branch includes two sub-branches. The first sub-branch includes parallel average pooling and max pooling layers, connected layers, convolutional layers, and a sigmoid function. The second sub-branch multiplies the input of the second branch with the output of the first sub-branch element by element to output the result of the second branch. The results of the first and second branches are input into the Concat layer and concatenated along the channel dimension to output a multimodal fusion feature tensor.

[0011] The multimodal fusion feature tensor is input into a hierarchical Swin Transformer encoder, and then splits into two branches. The first branch is input into the first Unetr Block layer, and then input into the first decoder through a skip connection. The second branch is input to the image patch segmentation layer and the linear embedding layer in sequence, and the results are input to the first SwinTransformer block and the first edge-aware feature modulation module, respectively. The result of the first edge-aware feature modulation module is input into the second UnetR Block layer, and then input into the second decoder via a skip connection; The output of the first Swing Transformer block is divided into two branches. The first branch is input to the second edge-aware feature modulation module and the third UnetR Block layer in sequence, and then input to the third decoder through a skip connection; the second branch is input to the second Swing Transformer block. The output of the second Swing Transformer block is divided into two branches. The first branch is input to the third edge-aware feature modulation module and the fourth Unetr Block layer in sequence, and then input to the fourth decoder through a skip connection. The second branch is input to the third Swing Transformer block. The output of the third Swing Transformer block is divided into two branches. The first branch is input to the fourth edge-aware feature modulation module in sequence, and then input to the fifth decoder through a skip connection. The second branch is input to the fourth Swing Transformer block. The results of the fourth layer Swing Transformer block are sequentially input into the fifth edge-aware feature modulation module, the fifth UnetR Block layer, and the Laplacian gating graph aggregation module, and the results are input into the fifth decoder via skip connections.

[0012] The fifth decoder receives the results from the fourth edge-aware feature modulation module and the Laplacian gating graph aggregation module, fuses them, and inputs them into the fourth decoder. The fourth decoder receives the results from the fourth Unetr Block layer and the fifth decoder, fuses them, and then inputs them into the third decoder. The third decoder receives the results from the third UnetR Block layer and the fourth decoder, fuses them, and then inputs them into the second decoder. The second decoder receives the results from the second UnetR Block layer and the third decoder, fuses them, and then inputs them into the first decoder. The first decoder receives the results from the first UnetR Block layer and the second decoder, and outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

[0013] The Laplacian-gated graph aggregation module receives the output of the previous neural network layer and is divided into four branches. The first branch is input to the convolutional layer and the Sigmoid function in sequence, the second branch is input to the convolutional layer and the Laplacian kernel in sequence, the third branch is input to the convolutional layer, and the fourth branch is directly input to the output of the previous neural network layer. The results of the second and third branches are added element by element, and the sum is multiplied element by element by the result of the first branch. The result is then fed into the convolutional layer and the Dropout function in sequence. The result is multiplied element by element by the gamma value, and then added element by element by the output of the previous neural network layer. The result is fed into the LN layer and output as the result of the Laplacian gating graph aggregation module.

[0014] The edge-aware feature modulation module receives the output of the previous neural network layer and divides it into two branches. The first branch first inputs the convolutional layer, and then splits into five sub-branches. The first sub-branch retains the output of the convolutional layer, the second sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, the third sub-branch retains the output of the convolutional layer, the fourth sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, and the fifth sub-branch retains the output of the convolutional layer. Subtract the results of the first sub-branch from the results of the second sub-branch element by element, and subtract the results of the fourth sub-branch from the results of the fifth sub-branch element by element; concatenate the two results of element-wise subtraction with the result of the third sub-branch, and then input them sequentially into the spatial convolutional layer, the BN layer, the ReLU function, the spatial convolutional layer, the BN layer, and the ReLU function; the second branch directly inputs the output result of the previous neural network layer; The results of the first branch and the second branch are added element by element to output the result of the edge-aware feature modulation module.

[0015] The Blocks layer receives the output of the previous neural network layer and inputs it sequentially into the LinearNorm layer and the W-MSA layer. The output of the W-MSA layer is added to the output of the previous neural network layer to form the first residual connection. The first residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the first residual connection result element by element to perform a second residual connection. The second residual join result is input into the LinearNorm layer and the SW-MSA layer in sequence. The output result of the SW-MSA layer is then added to the second residual join result element by element to perform a third residual join. The third residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the third residual connection result element by element to perform a fourth residual connection, and the output result of the Blocks layer is output.

[0016] Compared to existing technologies, this invention offers the following advantages: Based on channel and spatial attention mechanisms, it operates as an adaptive feature filter. In the channel dimension, it recalibrates the T1 / T1Ce modality to enhance lesion information, and in the spatial dimension, it focuses on significant tumor regions (such as edema boundaries) in the T2 / FLAIR modality, thereby achieving semantic alignment and effective fusion of heterogeneous features before encoding. Utilizing multi-scale differential operators, it extracts and explicitly amplifies high-frequency residual signals in the feature map through pooling operations. This forces the decoder to prioritize boundary regions with strong gradient changes during feature reconstruction, effectively solving the boundary ambiguity problem at the adhesion between meningiomas and the dura mater and improving the detection sensitivity of small lesions. A Laplacian-gated graph aggregation module is designed at the deepest layer of the network. This module uses discrete Laplacian operators to construct a voxel-level adjacency grid in the feature space to capture local curvature changes and adaptively filters high-frequency noise through a learnable gating mechanism. This effectively imposes global topological constraints without introducing the computational overhead of a large graph network, ensuring the anatomical continuity and integrity of the final segmentation result. This invention effectively solves key clinical technical problems in multimodal meningioma segmentation, such as blurred boundaries, missed detection of small lesions, and insufficient cross-modal fusion. Attached Figure Description

[0017] Figure 1 This is a diagram of the deep neural network architecture of the present invention; Figure 2 This is a diagram of the Blocks layer architecture of this invention; Figure 3 This is the architecture diagram of the first branch of the dual-branch multimodal input module; Figure 4 This is the architecture diagram of the second branch of the dual-branch multimodal input module; Figure 5 This is the architecture diagram of the Laplace gated graph aggregation module; Figure 6 This is an architecture diagram of the edge-aware feature modulation module; Figure 7This is the UnetR Block layer architecture of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] A multimodal segmentation method for meningiomas using a dual-branch Lagrange gating map is included, comprising: Prepare the dataset, build a deep neural network model, and train the neural network. The deep neural network model includes a dual-branch multimodal input module and a hierarchical Swing Transformer encoder. After training, the dataset is input into the trained deep neural network model. First, it is input into the dual-branch multimodal input module, which outputs a multimodal fusion feature tensor. Then, the multimodal fusion feature tensor is input into the hierarchical SwinTransformer encoder, which outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

[0020] The hierarchical Swin Transformer encoder includes an image block partitioning layer, a linear embedding layer, four Swin Transformer blocks, five UnetR Block layers, five edge-aware feature modulation modules, a Laplacian gating graph aggregation module, and five decoders. The Swin Transformer block consists of a Block layer and a Merge layer.

[0021] Three-dimensional multimodal MRI images of the patient to be segmented were acquired, including four modal data: T1-weighted image, T1-contrast enhancement image (T1Ce), T2-weighted image, and T2-fluid attenuation inversion recovery image (FLAIR). The data were normalized and used as input to a deep neural network model.

[0022] The normalized T1, T1Ce, T2 and FLAIR are input into the dual-branch multimodal input module, and the multimodal fusion feature tensor is output.

[0023] The dual-branch multimodal input module includes two parallel branches: the normalized T1 and T1Ce are input into the first branch, and the normalized T2 and FLAIR are input into the second branch. The first branch includes two sub-branches. The first sub-branch includes an average pooling layer, a linear layer, a ReLU function, another linear layer, and a sigmoid function in sequence. The second sub-branch multiplies the input and output of the first branch after recalibration and outputs the result of the first branch. The second branch includes two sub-branches. The first sub-branch includes parallel average pooling and max pooling layers, connected layers, convolutional layers, and a sigmoid function. The second sub-branch multiplies the input of the second branch with the output of the first sub-branch element by element to output the result of the second branch. The results of the first and second branches are input into the Concat layer and concatenated along the channel dimension to output a multimodal fusion feature tensor.

[0024] The multimodal fusion feature tensor is input into a hierarchical Swin Transformer encoder, and then splits into two branches. The first branch is input into the first Unetr Block layer, and then input into the first decoder through a skip connection. The second branch is input to the image patch segmentation layer and the linear embedding layer in sequence, and the results are input to the first SwinTransformer block and the first edge-aware feature modulation module, respectively. The result of the first edge-aware feature modulation module is input into the second UnetR Block layer, and then input into the second decoder via a skip connection; The output of the first Swing Transformer block is divided into two branches. The first branch is input to the second edge-aware feature modulation module and the third UnetR Block layer in sequence, and then input to the third decoder through a skip connection; the second branch is input to the second Swing Transformer block. The output of the second Swing Transformer block is divided into two branches. The first branch is input to the third edge-aware feature modulation module and the fourth Unetr Block layer in sequence, and then input to the fourth decoder through a skip connection. The second branch is input to the third Swing Transformer block. The output of the third Swing Transformer block is divided into two branches. The first branch is input to the fourth edge-aware feature modulation module in sequence, and then input to the fifth decoder through a skip connection. The second branch is input to the fourth Swing Transformer block. The results of the fourth layer Swing Transformer block are sequentially input into the fifth edge-aware feature modulation module, the fifth UnetR Block layer, and the Laplacian gating graph aggregation module, and the results are input into the fifth decoder via skip connections.

[0025] The fifth decoder receives the results from the fourth edge-aware feature modulation module and the Laplacian gating graph aggregation module, fuses them, and inputs them into the fourth decoder. The fourth decoder receives the results from the fourth Unetr Block layer and the fifth decoder, fuses them, and then inputs them into the third decoder. The third decoder receives the results from the third UnetR Block layer and the fourth decoder, fuses them, and then inputs them into the second decoder. The second decoder receives the results from the second UnetR Block layer and the third decoder, fuses them, and then inputs them into the first decoder. The first decoder receives the results from the first UnetR Block layer and the second decoder, and outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

[0026] The Laplacian-gated graph aggregation module receives the output of the previous neural network layer and is divided into four branches. The first branch is input to the convolutional layer and the Sigmoid function in sequence, the second branch is input to the convolutional layer and the Laplacian kernel in sequence, the third branch is input to the convolutional layer, and the fourth branch is directly input to the output of the previous neural network layer. The results of the second and third branches are added element by element, and the sum is multiplied element by element by the result of the first branch. The result is then fed into the convolutional layer and the Dropout function in sequence. The result is multiplied element by element by the gamma value, and then added element by element by the output of the previous neural network layer. The result is fed into the LN layer and output as the result of the Laplacian gating graph aggregation module.

[0027] The edge-aware feature modulation module receives the output of the previous neural network layer and divides it into two branches. The first branch first inputs the convolutional layer, and then splits into five sub-branches. The first sub-branch retains the output of the convolutional layer, the second sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, the third sub-branch retains the output of the convolutional layer, the fourth sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, and the fifth sub-branch retains the output of the convolutional layer. Subtract the results of the first sub-branch from the results of the second sub-branch element by element, and subtract the results of the fourth sub-branch from the results of the fifth sub-branch element by element; concatenate the two results of element-wise subtraction with the result of the third sub-branch, and then input them sequentially into the spatial convolutional layer, the BN layer, the ReLU function, the spatial convolutional layer, the BN layer, and the ReLU function; the second branch directly inputs the output result of the previous neural network layer; The results of the first branch and the second branch are added element by element to output the result of the edge-aware feature modulation module.

[0028] The Blocks layer receives the output of the previous neural network layer and inputs it sequentially into the LinearNorm layer and the W-MSA layer. The output of the W-MSA layer is added to the output of the previous neural network layer to form the first residual connection. The first residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the first residual connection result element by element to perform a second residual connection. The second residual join result is input into the LinearNorm layer and the SW-MSA layer in sequence. The output result of the SW-MSA layer is then added to the second residual join result element by element to perform a third residual join. The third residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the third residual connection result element by element to perform a fourth residual connection, and the output result of the Blocks layer is output.

[0029] The deep neural network architecture of this invention is as follows: Figure 1As shown, four modal data, namely T1 weighted image (T1), T1 contrast enhancement image (T1Ce), T2 weighted image (T2), and T2 fluid attenuation inversion recovery image (FLAIR), are input into the first branch and the second branch respectively. The first branch processes T1 and T1Ce, including a channel attention mechanism to enhance lesion information, while the second branch processes T2 and FLAIR, including a spatial attention mechanism to focus on the tumor edema area. The first and second branches are fused. On one hand, the fused data is input to the first Unetr Block layer, then through a skip connection to the first decoder. On the other hand, the fused features are divided into image blocks, creating 3D blocks, and mapped to the embedding space using linear embedding. The linearly embedded data is then input to the first edge-aware feature modulation module, then to the second Unetr Block layer, then through a skip connection to the second decoder, and then through a Blocks layer before merging. The merged data is then input to the second edge-aware feature modulation module, then to the third Unetr Block layer, then through a skip connection to the third decoder, and then through a Blocks layer before merging. The merged data is then input to the third edge-aware feature modulation module, then to the fourth Unetr Block layer, then through a skip connection to the fourth decoder, and then through a Blocks layer before merging. The merged data is then input to the fourth edge-aware feature modulation module, then through a skip connection to the fifth decoder, and then through a Blocks layer before merging. The merged data is then input to the fifth edge-aware feature modulation module, the fifth Unetr Block layer, then to the Laplacian-gated graph aggregation module, and finally to the fifth decoder. The data flow of the five decoders is sequentially from the fifth decoder to the fourth decoder, the third decoder, the second decoder, and the first decoder. The first decoder ultimately outputs a 3D segmentation mask of the enhanced tumor, the tumor core, and the entire tumor region of the meningioma. The Blocks layer, along with merging, forms the core stage of the SwinTransformer encoder, used for extracting multi-scale global contextual features and downsampling. Edge-aware feature modulation is set at skip connections to extract and fuse high-frequency residual edge signals. Laplacian-gated graph aggregation is set at the deepest layer (bottleneck layer) of the encoder, using the Laplacian operator to extract local curvature changes and apply topological constraints. The UnetRBlock layer, along with the decoder, performs feature smoothing and stepwise upsampling reconstruction.

[0030] like Figure 2As shown, the internal structure of the Blocks layer includes a Multilayer Perceptron (MLP), LinearNorm, Window-based Multi-head Self-Attention (W-MSA), and Shift-Window-based Multi-head Self-Attention (SW-MSA) modules. The architecture of the Blocks layer in this invention is as follows: Figure 1 As shown, the input data is fed into the LinearNorm layer, then into the W-MSA layer, and residual linking is performed between the input data and the W-MSA layer data. The residually linked data is then fed into the LinearNorm layer, then into the MLP layer, and residual linking is performed between the residually linked data and the MLP layer data. This process is repeated three times: the input data is fed into the LinearNorm layer, then into the W-MSA layer, and residual linking is performed between the residually linked data and the W-MSA layer data. Finally, the residually linked data is fed into the LinearNorm layer, then into the MLP layer, and residual linking is performed between the residually linked data and the MLP layer data. The final processed result is then output.

[0031] The first branch architecture is as follows: Figure 3 As shown, this branch, used to process T1 and T1Ce modes, includes average pooling (compressing spatial information into a global descriptor), a Linear layer (fully connected layer), and a ReLU / Sigmoid activation function (used to learn non-linear dependencies between channels and generate channel weights). Finally, recalibrated multiplication (S) is used to multiply the weights with the original features to adaptively enhance the lesion feature channels of the tumor core. The processing order is: average pooling, Linear layer, ReLU function, Linear layer again, and Sigmoid function, followed by recalibrated multiplication.

[0032] The second branch architecture is as follows: Figure 4 As shown, this is used to process T2 and FLAIR modalities. The input features are concatenated (C) after undergoing channel-level average pooling and max pooling operations. Then, a spatial saliency map is generated using a large receptive field convolution (7×7×7) and a sigmoid activation function. Finally, element-wise multiplication (X) modulates the original input features to accurately locate irregular regions with blurred boundaries, such as edema lesions. The processing order is: average pooling and max pooling, concatenation, convolution (7×7×7), and the sigmoid function, followed by element-wise multiplication.

[0033] The architecture of the Laplace gated graph aggregation module of this invention is as follows: Figure 5As shown, after receiving input data, the input features are processed through three parallel 1×1×1 convolutions, mapping them to a topological branch, an identity branch, and a gated branch, respectively. The topological branch is a fixed discrete Laplacian operator with a center weight set to 1 and a neighboring weight set to -0.125, specifically used to extract topological features such as local curvature changes between voxels. The gated branch generates a gated weight map of 0 to 1 using the Sigmoid function and performs element-wise multiplication with the topological features (i.e., gated modulation) to suppress the propagation of high-frequency noise. The identity branch and the topological branch are added element-wise to fuse the extracted topological features with the identity branch features, as well as to fuse the residual connections at the end of the module. The element-wise addition result is multiplied element-wise with the gated branch. The multiplication result is input into a 1×1×1 convolution, and then into Dropout for feature regularization. The regularization result is multiplied element-wise with the gamma value, which is a dynamically learnable scaling factor. The factor is used to adaptively adjust the strength of the residual signal during training to stabilize the optimization process; then the input data is added element-wise with the result of element-wise multiplication, and the result is input into the LN layer for linear normalization of the features, and finally the deep semantic features after topological enhancement are output.

[0034] The edge-aware feature modulation module architecture of this invention is as follows: Figure 6 As shown, the input data is received, and the input features are first linearly projected through a 1×1×1 convolution to reduce computational redundancy and generate basic features for edge extraction; then multi-scale average pooling is performed, including average pooling 2×2×2 and average pooling 4×4×4. Multi-scale average pooling is used as a low-pass filter to smooth the features. Following the unsampling (upsampling operation) operation, features at different scales are restored to their original resolution. The average pooling 2×2×2 and the convolutional data are subtracted element-wise, and the average pooling 4×4×4 and the convolutional data are also subtracted element-wise. The results of these two subtractions are concatenated with the convolutional data, stitching the basic projection features and the extracted multi-scale high-frequency residual edge features along the channel dimension. Then, a dilated convolution (d=1 / d=2) is input, and the stitched features are sequentially passed through 3×3×3 concatenated dilated convolutional layers with dilation rates of 1 and 2, respectively. Batch normalization (BN) and ReLU activation functions are used to fuse contextual information from multiple receptive fields while maintaining resolution. Finally, the fused features are added element-wise to the original input, resulting in a modulated feature infused with multi-scale edge-aware information.

[0035] The Unetr Block layer architecture of this invention is as follows: Figure 7As shown, the module receives feature representations from the corresponding stage (or EFM module) and then inputs them into two stacked 3×3×3 convolutional layers for deep feature extraction, smoothing, and spatial feature mapping. Each convolutional operation is followed by instance normalization and ReLU activation functions to accelerate network convergence and introduce non-linear expressive power. The original input features of the module are directly added element-wise to the output of the second ReLU layer through residual connections. This design aims to maintain smooth gradient flow and provide a structurally consistent feature representation for the subsequent decoder.

[0036] In this embodiment of the invention, a publicly labeled training set is used for re-segmentation and experimental verification. All cases are re-segmented uniformly based on the training set, with 85% used for training, 10% for verification, and 5% for testing.

[0037] The dataset annotations mainly include enhancing tumor, non-enhancing tumor core (NETC), and surrounding non-enhancing FLAIR hyperintensity (SNFH). 1000 cases were used as the training set and publicly labeled.

[0038] The model proposed in this invention is implemented based on the PyTorch framework and the MONAI library, and trained on a workstation equipped with an NVIDIA GeForce RTX 4090D GPU. During the training phase, considering the memory usage of 3D volumetric data, the batch size was set to 1, and iterative training was performed for 100 epochs. To optimize the network weights, the AdamW optimizer was used, with an initial learning rate of 0.0001 and a weight decay of 1×10⁻⁵. The learning rate adjustment strategy employed was cosine annealing, causing the learning rate to gradually decay during training.

[0039] In terms of data preprocessing, all input images first undergo minimum-to-maximum normalization, mapping their intensity values ​​to... The interval. To enhance the robustness of the model and prevent overfitting, this invention implements a rigorous online data augmentation strategy. Specifically, the input image is randomly cropped to... Voxel patching. In addition, the present invention performs random mirror flipping in the axial, coronal and sagittal planes with a probability of 0.5, and introduces a variety of intensity transformation techniques including random scaling (scale 0.7 to 1.0), Gaussian noise (standard deviation 0 to 0.33), Gaussian sharpening and random contrast adjustment (γ range 0.65 to 1.5).

[0040] During the inference phase, this invention employs sliding window inference and test-time enhancement techniques, performing eight different flipping combinations on the input volume data in the axial, coronal, and sagittal directions. The final segmentation result is generated by averaging the probability maps of these eight inferences, thus obtaining a smoother and more accurate prediction boundary.

[0041] This invention employs three metrics commonly used in medical image segmentation tasks: Dice similarity coefficient (Dice), mean intersection over union (MIoU), and 95% Hausdorff distance (HD95).

[0042] The method of this invention achieved optimal performance on most metrics, demonstrating its robustness in treating the complex heterogeneous structures of meningiomas. Specifically, the method achieved average Dice scores of 96.07%, 94.57%, and 95.73% in the ET, WT, and TC regions, respectively, with an mIoU of 91.32% and HD95 scores of 2.27 mm, 3.52 mm, and 2.67 mm, respectively.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting meningiomas using a dual-branch Lagrange gating map and multimodal approach, characterized in that: include: Prepare the dataset, build a deep neural network model, and train the neural network. The deep neural network model includes a dual-branch multimodal input module and a hierarchical Swing Transformer encoder. After training, the dataset is input into the trained deep neural network model. First, it is input into the dual-branch multimodal input module, which outputs a multimodal fusion feature tensor. Then, the multimodal fusion feature tensor is input into the hierarchical SwinTransformer encoder, which outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

2. The method for segmenting meningiomas using a dual-branch Lagrange gating map as described in claim 1, characterized in that, The hierarchical Swin Transformer encoder includes an image block partitioning layer, a linear embedding layer, four Swin Transformer blocks, five UnetR Block layers, five edge-aware feature modulation modules, a Laplacian gating graph aggregation module, and five decoders. The Swin Transformer block consists of a Block layer and a Merge layer.

3. The method for segmenting meningiomas using a dual-branch Lagrange gating map with multimodal visualization according to claim 2, characterized in that, Three-dimensional multimodal MRI images of the patient to be segmented were acquired, including four modal data: T1-weighted image, T1-contrast enhancement image (T1Ce), T2-weighted image, and T2-fluid attenuation inversion recovery image (FLAIR). The data were normalized and used as input to a deep neural network model.

4. The method for segmenting meningiomas using a dual-branch Lagrange gating map aggregation as described in claim 3, characterized in that, The normalized T1, T1Ce, T2 and FLAIR are input into the dual-branch multimodal input module, and the multimodal fusion feature tensor is output.

5. The method for segmenting meningiomas using a dual-branch Lagrange gating map aggregation according to claim 4, characterized in that, The dual-branch multimodal input module includes two parallel branches: the normalized T1 and T1Ce are input into the first branch, and the normalized T2 and FLAIR are input into the second branch. The first branch includes two sub-branches. The first sub-branch includes an average pooling layer, a linear layer, a ReLU function, another linear layer, and a sigmoid function in sequence. The second sub-branch multiplies the input and output of the first branch after recalibration and outputs the result of the first branch. The second branch includes two sub-branches. The first sub-branch includes parallel average pooling and max pooling layers, connected layers, convolutional layers, and a sigmoid function. The second sub-branch multiplies the input of the second branch with the output of the first sub-branch element by element to output the result of the second branch. The results of the first and second branches are input into the Concat layer and concatenated along the channel dimension to output a multimodal fusion feature tensor.

6. The method for segmenting meningiomas using a dual-branch Lagrange gating map aggregation according to claim 5, characterized in that, The multimodal fusion feature tensor is input into a hierarchical Swin Transformer encoder, and then splits into two branches. The first branch is input into the first Unetr Block layer, and then input into the first decoder through a skip connection. The second branch is input to the image patch segmentation layer and the linear embedding layer in sequence, and the results are input to the first SwinTransformer block and the first edge-aware feature modulation module, respectively. The result of the first edge-aware feature modulation module is input into the second UnetR Block layer, and then input into the second decoder via a skip connection; The output of the first Swing Transformer block is divided into two branches. The first branch is input to the second edge-aware feature modulation module and the third UnetR Block layer in sequence, and then input to the third decoder through a skip connection; the second branch is input to the second Swing Transformer block. The output of the second Swing Transformer block is divided into two branches. The first branch is input to the third edge-aware feature modulation module and the fourth Unetr Block layer in sequence, and then input to the fourth decoder through a skip connection. The second branch is input to the third Swing Transformer block. The output of the third Swing Transformer block is divided into two branches. The first branch is input to the fourth edge-aware feature modulation module in sequence, and then input to the fifth decoder through a skip connection. The second branch is input to the fourth Swing Transformer block. The results of the fourth layer Swing Transformer block are sequentially input into the fifth edge-aware feature modulation module, the fifth UnetR Block layer, and the Laplacian gating graph aggregation module, and the results are input into the fifth decoder via skip connections.

7. The method for segmenting meningiomas using a dual-branch Lagrange gating map as described in claim 6, characterized in that, The fifth decoder receives the results from the fourth edge-aware feature modulation module and the Laplacian gating graph aggregation module, fuses them, and inputs them into the fourth decoder. The fourth decoder receives the results from the fourth Unetr Block layer and the fifth decoder, fuses them, and then inputs them into the third decoder. The third decoder receives the results from the third UnetR Block layer and the fourth decoder, fuses them, and then inputs them into the second decoder. The second decoder receives the results from the second UnetR Block layer and the third decoder, fuses them, and then inputs them into the first decoder. The first decoder receives the results from the first UnetR Block layer and the second decoder, and outputs a three-dimensional segmentation mask of the enhanced tumor, tumor core, and the entire tumor region of the meningioma.

8. The method for segmenting meningiomas using a dual-branch Lagrange gating map aggregation according to claim 7, characterized in that, The Laplacian-gated graph aggregation module receives the output of the previous neural network layer and is divided into four branches. The first branch is input to the convolutional layer and the Sigmoid function in sequence, the second branch is input to the convolutional layer and the Laplacian kernel in sequence, the third branch is input to the convolutional layer, and the fourth branch is directly input to the output of the previous neural network layer. The results of the second and third branches are added element by element, and the sum is multiplied element by element by the result of the first branch. The result is then fed into the convolutional layer and the Dropout function in sequence. The result is multiplied element by element by the gamma value, and then added element by element by the output of the previous neural network layer. The result is fed into the LN layer and output as the result of the Laplacian gating graph aggregation module.

9. The method for segmenting meningiomas using a dual-branch Lagrange gating map aggregation according to claim 8, characterized in that, The edge-aware feature modulation module receives the output of the previous neural network layer and divides it into two branches. The first branch first inputs the convolutional layer, and then splits into five sub-branches. The first sub-branch retains the output of the convolutional layer, the second sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, the third sub-branch retains the output of the convolutional layer, the fourth sub-branch performs average pooling and upsampling on the output of the convolutional layer in sequence, and the fifth sub-branch retains the output of the convolutional layer. Subtract the results of the first sub-branch from the results of the second sub-branch element by element, and subtract the results of the fourth sub-branch from the results of the fifth sub-branch element by element; concatenate the two results of element-wise subtraction with the result of the third sub-branch, and then input them sequentially into the spatial convolutional layer, the BN layer, the ReLU function, the spatial convolutional layer, the BN layer, and the ReLU function; the second branch directly inputs the output result of the previous neural network layer; The results of the first branch and the second branch are added element by element to output the result of the edge-aware feature modulation module.

10. The method for segmenting meningiomas using a dual-branch Lagrange gating map as described in claim 9, characterized in that, The Blocks layer receives the output of the previous neural network layer and inputs it sequentially into the LinearNorm layer and the W-MSA layer. The output of the W-MSA layer is added to the output of the previous neural network layer to form the first residual connection. The first residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the first residual connection result element by element to perform a second residual connection. The second residual join result is input into the LinearNorm layer and the SW-MSA layer in sequence. The output result of the SW-MSA layer is then added to the second residual join result element by element to perform a third residual join. The third residual connection result is input into the LinearNorm layer and the MLP layer in sequence. The output result of the MLP layer is added to the third residual connection result element by element to perform a fourth residual connection, and the output result of the Blocks layer is output.

Images