Three-dimensional brain tumor segmentation model based on deformable feature aggregation
By using deformable feature aggregation and spatial perception technology in brain tumor segmentation model, the problems of multimodal information difference and insufficient spatial resolution in 3D MRI are solved, and accurate segmentation and spatial distribution analysis of brain tumors are realized.
Patent Information
- Application Number
- CN202410995548.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
AI Technical Summary
Brain tumor segmentation has problems of poor spatial resolution, noise and multimodal information differences in 3D MRI images, making it difficult to achieve accurate three-dimensional volume information capture and tumor spatial structure analysis.
Using a three-dimensional brain tumor segmentation model based on deformable feature aggregation, multimodal features are aligned and fusion through a modal space-aware dual-stream encoder, combining large-nuclear convolutional attention and spatial channel attention to capture global spatial information, and using multiple Scale convolution attention decoder restores complex spatial relationships.
The full fusion of multimodal information and the capture of global spatial information are achieved, and the precise segmentation effect and spatial distribution perception ability of brain tumors are improved.
Smart Images

Figure CN118967712A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image segmentation, and in particular to a three-dimensional brain tumor segmentation model based on deformable feature aggregation. Background Art
[0002] Brain tumors are space-occupying lesions that originate in the nervous system and metastasize to the brain. Accurate and reproducible tumor quantification and morphology are essential for the diagnosis, treatment planning, and monitoring of tumor treatment response. 3D MRI (Magnetic Resonance Imaging) is a three-dimensional brain image acquired based on magnetic resonance imaging technology, which can provide more comprehensive and accurate brain structure information. Segmenting brain tumors from magnetic resonance images is a key step in brain tumor treatment, enabling clinicians to determine important parameters such as the location, extent, and type of the tumor. As an important topic in the field of medical image segmentation, brain tumor segmentation is of great significance to clinical practice and has a wide range of applications in neurological quantitative analysis, surgical planning, functional imaging, etc. Although 3D MRI can accurately describe the brain tissue structure, brain tumor segmentation remains a challenging task due to problems such as poor spatial resolution, the presence of noise, and data mixed with acquisition artifacts.
[0003] On the one hand, in order to obtain accurate brain tumor segmentation results to monitor the spatial distribution of tumors and analyze disease progression, effective complementary MRI modalities are usually required. However, different modalities have different characteristics, and their sensitivity and contribution to the segmented area are also different. It is necessary to consider the differences in multimodal information and make full use of the relationship between modalities.
[0004] On the other hand, due to the three-dimensional properties of MRI data, spatial information is easily lost during image information learning and feature extraction, and the spatial structure of the tumor and the relationship between adjacent tissues cannot be fully obtained, making it difficult to fully capture the three-dimensional volume information, which may reduce the performance of lesion segmentation. Summary of the invention
[0005] To solve the above technical problems, the present invention provides a three-dimensional brain tumor segmentation model based on deformable feature aggregation. The modal space-aware dual-stream encoder is used to learn the multimodal information and voxel information of the three-dimensional brain tumor magnetic resonance image. The modal space-aware dual-stream encoder consists of a deformable feature aggregation module branch and a spatial perception branch. The deformable feature aggregation module is used to realize multimodal feature alignment and fusion, and the spatial perception branch is used to learn the global spatial information based on three-dimensional voxels. The dual-branch features are processed by the interactive fusion module to achieve feature fusion while maintaining feature consistency, and then input into the multi-scale convolutional attention decoder to capture complex spatial relationships and achieve feature recovery.
[0006] The technical solution adopted by the present invention to solve the technical problem is: a three-dimensional brain tumor segmentation model based on deformable feature aggregation, comprising the following steps:
[0007] S1, data preprocessing:
[0008] S2, multimodal feature alignment and fusion;
[0009] S3, expands the receptive field and captures a wide range of spatial representations, enhancing the global spatial features based on 3D voxels;
[0010] S4, uses the association-driven fusion module to process the association features of different levels and branches, and adopts dense connections to form a continuous memory mechanism;
[0011] S5, the output features of the association-driven fusion module are input into the multi-scale convolutional attention module and the multi-scale convolutional attention decoder of the large core group gated attention cascade to capture complex spatial relationships. The multi-scale convolutional attention performs deep convolution at different levels and performs channel mixing operations between groups. The large core group gated attention refines the features fused with the jump connection through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the features of the next stage.
[0012] Preferably, in step S1, the model training and test data sets are divided into training set, validation set and test set according to 7:2:1, and each case contains three-dimensional MRI scans of four different modalities: T1, T1Gd, T2 and T2-FLAIR;
[0013] During training, data augmentation techniques such as random rotation and scaling, elastic deformation, Gaussian noise, additional brightness enhancement, and gamma scaling are dynamically applied to improve generalization ability;
[0014] Crop the non-zero voxels in brain tumor MRI;
[0015] Voxel normalization was performed by calculating the mean and standard deviation; the image sequence was randomly horizontally flipped in the axial, coronal, and sagittal planes with a probability of 50% to increase the model's robustness to changes in different directions and viewing angles.
[0016] Preferably, step S2 utilizes the deformable feature aggregation module branch in the modal space-aware dual-stream encoder to perform cross-modal multi-layer fusion, utilizes differentiable transformation operations to align multi-modal features, and alleviates the spatial misalignment between different modalities; designs a mapping function to model the relationship between global and local modalities, fuses the aligned features voxel by voxel, and realizes multi-modal feature alignment and fusion.
[0017] Preferably, in step S2.1, the multimodal data set after data preprocessing is input as a three-dimensional volume B, and the image X is transformed into i∈R H×W×D×3 The dimension is converted to C dimension to get the embedded image X i '∈R H×W×D×C , normalize the embedded image F by layer normalization i ∈R H×W×D×C ,Then the embedded image is fed into the modality space-aware two-stream encoder;
[0018] Step S2.2, the modal space-aware dual-stream encoder consists of a deformable feature aggregation module branch and a spatial perception branch;
[0019] The deformable feature aggregation module branch includes feature deformation and feature aggregation operations. Feature deformation is used to align multimodal features and reduce the misalignment between modalities. A 1×1×1 three-dimensional convolution is used to obtain an effective feature description E. i ∈R H×W×D×C , using E i Estimate the deformation field between the modes and then transform the motion features to match the fixed features; the deformation function T is convolved by 3×3×3 convolution θ (D1, D2, D3, D4) = φ for modeling, where θ represents the convolution parameter and φ∈R H×W×D×4 Represents the deformation field between different modes; for F i ∈R H×W×D×C For each pixel position p in , φ(p) represents the offset along the x-axis, y-axis and z-axis, and the transformed position p' can be obtained by p'=p+φ(p); the differentiable and deformable transformation operation of p is defined according to the spatial transformation network:
[0020]
[0021] Here, the symbol · is a transformation operator including shift and interpolation, and δ(p') represents the neighboring voxels of p'. i φ(p) and D i ·φ(p) expects to learn F i and D i Match with the appropriate φ;
[0022] Step S2.3, after feature deformation to align multimodal features, feature aggregation generates a response map based on the local and global inter-modal relationships; the correspondence between the modalities is modeled using a mapping function, which is built on a pyramid block with four parallel layers including a global average pooling layer and a 5×5×5 convolutional layer for extracting global correspondences, and using 1×1×1 and 3×3×3 convolutional layers to capture local inter-modal information; in the case of aligned multimodal features, the response map W is calculated. i ∈R H×W×D×C , modulation and aggregation F i and F i·φ(p); The overall cross-modal feature aggregation is calculated as:
[0023]
[0024] Among them, F Aggr ∈R H×W×D×C is the aggregation feature, Expressed as dense element-wise multiplication.
[0025] Preferably, step S3 utilizes the spatial channel perception branch integrated with large kernel convolutional attention, deformable layer and spatial channel attention to flexibly expand the receptive field and capture a wide range of spatial representations.
[0026] Preferably, step S3.1, the same image F as the deformable feature aggregation branch i ∈R H×W×D×C Input, feature map size and resolution conversion is achieved through downsampling convolution, and the spatial perception module with large kernel convolution attention and spatial attention output parallel calculation processes long-range dependencies and captures global spatial context information;
[0027] Compared to the standard attention mechanism which is limited to a fixed-size region, the large kernel convolutional attention considers large regions within the input image in a flexible and adaptive way, achieving a wider receptive field through deformable convolutions;
[0028] The output of the spatial perception module is as follows:
[0029] F S = comb(F i ,α s LKA(F i )+β s SA(F i ))
[0030] Among them, F i ∈R H×W×D×C represents the input feature map, F S represents outputs with the same dimensions, α s and β s Represents the learnable parameters of each channel, controlling the combined weight of the large kernel convolutional attention and spatial attention mechanisms; SA represents the spatial self-attention mechanism, and LKA represents the large kernel convolutional attention implementation as follows:
[0031]
[0032] Among them, the input tensor F i ∈R H×W×D×CPerform point-by-point convolution operations and reduce the dimension through the activation function Gaussian error linear unit; then, use deep convolution and deep dilated convolution sequence to process the transformed tensor, extract multi-scale features while retaining spatial information, and integrate deformable deep convolution to achieve adaptive sampling of features; the spatial attention mechanism is implemented as follows:
[0033]
[0034] Among them, σ represents the Softmax function, Q, K g and V g They represent query, projection key and projection space value layers respectively. The size of each layer is d. Linear normalization and linear layer are used to enhance the performance of the spatial attention module. Finally, the final output is obtained through the Comb function as shown in the following formula;
[0035] Comb(F i ,A)=C+PW-Conv(DW-Conv3(C))
[0036] in, DW-Conv3 is a 3×3×3 convolution.
[0037] Preferably, in step S3.2, the channel perception module enhances the model's ability to capture complex channel relationships and selectively focuses on relevant features while suppressing noise across spatial positions; integrating large kernel convolutional attention and channel attention to simultaneously process inter-channel relationships and long-range dependencies, and the channel perception output is as follows:
[0038] F S = comb(F i ,λ c LKA(F i ')+γ c CA(F i '))
[0039] Among them, F i '∈R H×W×D×C represents the spatial perception output feature map, λ c and γ c represents the learnable parameters of each channel layer, which are the combined weights of large kernel convolution attention and channel attention; CA represents the channel attention mechanism, which captures the interdependence between feature channels by using dot product operations in the channel dimension between the channel value layer and the channel attention map. The output is calculated as follows:
[0040]
[0041] Preferably, in step S4.1, firstly, the deformable feature aggregation branch is output And the spatial channel perception branch output The initial refined features are obtained by inputting the correlation space feature module composed of local correlation attention and global spatial attention; the stacked correlation space feature module is used to further enhance the initial refined features through the hierarchical features extracted in the early stage; express and of and, and Respectively represent the features extracted from the lth layer; G l represents the output of the associated spatial feature module at layer l; the output of the associated spatial feature module at layer l can be expressed as:
[0042]
[0043] in is the feature enhanced by the lth layer;
[0044] Step S4.2, local correlation attention adopts channel attention in local areas by using maximum pooling and average pooling with different kernel sizes, and selects different attention for each local area in the channel dimension; by using average pooling and maximum pooling operations to operate on four scales and strides, a multilayer perceptron with one hidden layer is a component of the shared network at each scale; the nearest interpolation method is used to keep the attention weight of the local area unchanged; the attention map of each scale is interpolated and fused separately by the Sigmoid activation function;
[0045] In step S4.3, the global spatial attention uses spatial attention with four levels of different types of kernels to capture global spatial features at different levels, and uses a pyramid structure to perform multi-scale analysis on the input; the channel information of the input features is aggregated using maximum pooling and average pooling.
[0046] Preferably, in step S5.1, first, the output of the last stage of the encoder is input into an efficient multi-scale convolutional attention module through the association-driven fusion module, and the multi-scale convolutional attention module is composed of channel attention, spatial attention and multi-scale deep convolution blocks. Multi-scale deep convolution is used to capture multi-scale and multi-resolution contexts, and channel mixing operations are used to merge channel connections. In the spatial attention operation, the relevant spatial features are enhanced after determining the feature focus position, the attention weight is calculated by the activation function, and then the weight is fed to the input through the Hadamard product;
[0047] Step S5.2, during decoding, use efficient upconvolution blocks to gradually upsample the current stage features to match the dimension and resolution of the feature map of the next skip connection; first, use upsampling with a scale factor of 2 to expand the feature map; then, enhance the enlarged feature map by applying 3×3×3 depth convolution, BN and ReLU activation functions; finally, use 1×1×1 convolution to reduce the channel to match the next stage, realizing feature transfer of different resolutions;
[0048] In step S5.3, the feature map is gradually combined with the attention coefficient using large kernel group gated attention, and higher activation of relevant features and suppression of irrelevant features are achieved through coefficient learning; this process uses gating signals derived from high-level features to control the information flow at different stages, and processes features from skip connections and efficient up-convolution blocks by applying separate 3×3×3 group convolutions; then, for the multi-category corresponding channels of the brain tumor segmentation target, the segmentation head is used to generate segmentation outputs from the refined feature maps of the four stages of the decoder.
[0049] Advantages of the present invention:
[0050] This patent is a three-dimensional brain tumor segmentation model based on deformable feature aggregation. First, brain tumor images are difficult to segment due to poor spatial resolution, noise, and data mixed with acquisition artifacts. Secondly, in order to obtain accurate brain tumor segmentation results to facilitate monitoring of the spatial distribution of tumors and analysis of disease progression, effective complementary MRI modalities are usually required. However, different modalities have different characteristics, and their sensitivity and contribution to the segmented area are also different. It is necessary to consider the differences in multimodal information and make full use of the relationship between modalities. Finally, since MRI data has three-dimensional properties, it is easy to cause the loss of spatial information when performing image information learning and feature extraction, and it is impossible to fully obtain the spatial structure of the tumor and the relationship between adjacent tissues, making it difficult to achieve complete capture of three-dimensional volume information, which may reduce the performance of lesion segmentation. In order to solve these problems, this patent proposes a three-dimensional brain tumor segmentation model based on deformable feature aggregation. Using a modal space-aware dual-stream encoder composed of a deformable feature aggregation module branch and a spatial channel perception branch, the multimodal information and voxel information of brain tumor 3D MRI are jointly learned. Through the deformable feature aggregation module, cross-modal multi-layer fusion is performed, and the multi-modal features are aligned using differentiable transformation operations to alleviate the spatial misalignment between different modalities. Then, the mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multi-modal feature alignment and fusion. The spatial channel perception branch integrated with large kernel convolution attention, deformable layer and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations, and learn global spatial information based on three-dimensional voxels. In order to fully fuse the multi-modal information and global spatial information extracted by the deformable aggregation branch and the spatial channel perception branch, the association-driven fusion module is used in the final stage of the encoder to process the association features of different levels and branches, and dense connections are used to form a continuous memory mechanism to improve the effective spatial feature transfer and achieve feature fusion while maintaining feature consistency. Then, the multi-scale convolutional attention module and the multi-scale convolutional attention decoder of the large core group gated attention cascade are input to capture complex spatial relationships and realize feature recovery. The multi-scale convolutional attention performs deep convolution at different levels and performs channel mixing operations between groups. The large core group gated attention refines the features fused with jump connections through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the next stage features, realizing effective global spatial feature decoding and recovery. The three-dimensional brain tumor segmentation model based on deformable feature aggregation can not only fully integrate multimodal information, but also capture global spatial information in combination with the three-dimensional attributes of the data. It can use 3D MRI to obtain accurate segmentation results to realize the positioning and spatial distribution perception of brain tumors. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only three of the drawings of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0052] Figure 1 3D encoder-decoder network based on modality space perception and multi-scale convolutional attention;
[0053] Figure 2 Deformable feature aggregation module;
[0054] Figure 3 Association driver fusion module. DETAILED DESCRIPTION
[0055] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments are only used to explain the present invention and do not limit the protection scope of the present invention.
[0056] Example
[0057] This patent proposes a 3D Brain Tumour Segmentation Model Based on Deformable Feature Aggregation, which mainly includes three modules: a modal space-aware dual-stream encoder, an interactive fusion module, and a multi-scale convolutional attention decoder.
[0058] The modality-space-aware dual-stream encoder consists of a deformable feature aggregation module branch and a spatial channel-aware branch, which jointly learns the multimodal information and voxel information of brain tumor 3D MRI. The deformable feature aggregation module is used to perform cross-modality multi-layer fusion, and the multimodal features are aligned using differentiable transformation operations to alleviate the spatial misalignment between different modalities.
[0059] The mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multimodal feature alignment and fusion. The spatial channel perception branch integrated with large kernel convolution attention, deformable layer and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations, and learn global spatial information based on three-dimensional voxels. The association-driven fusion module processes the associated features of different levels and branches, achieving multimodal and global spatial feature fusion while maintaining feature consistency.
[0060] The multi-scale convolutional attention decoder of the cascade of multi-scale convolutional blocks and channel spatial attention is then input to capture complex spatial relationships and achieve feature recovery. The multi-scale convolutional blocks perform deep convolutions at different levels and perform channel mixing operations between groups, and are gradually upsampled through efficient upconvolution to match the dimension and resolution of the feature map of the next stage.
[0061] The steps of this embodiment include the following:
[0062] Step S1, data preprocessing: The model training and testing datasets are from the BraTS2019, BraTS2020, and BraTS2021 datasets in the BraTS challenge, which are divided into training set, validation set, and test set according to 7:2:1. Each case contains three-dimensional MRI scans of four different modalities: T1, T1Gd, T2, and T2-FLAIR. During the training process, data augmentation techniques such as random rotation and scaling, elastic deformation, Gaussian noise, additional brightness enhancement, and gamma scaling are dynamically applied to improve generalization ability. In order to reduce the amount of calculation, a volume cropping strategy is adopted to crop non-zero voxels in brain tumor MRI. Since the pixel intensity in brain tumor MRI is qualitative, voxel normalization is performed by calculating the mean and standard deviation. The three-dimensional image volume in the dataset is randomly cropped from 240×240×155 to 128×128×128 voxels. The image sequences are randomly flipped horizontally in the axial, coronal and sagittal planes with a probability of 50% to increase the robustness of the model to changes in different directions and perspectives.
[0063] Step S2: In order to solve the problem that there are differences in multimodal information and the relationship between modalities cannot be fully utilized, the deformable feature aggregation module branch in the modal space-aware dual-stream encoder is used to perform cross-modal multi-layer fusion, and the multimodal features are aligned using differentiable transformation operations to alleviate the spatial misalignment between different modalities. Then, a mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multimodal feature alignment and fusion.
[0064] Step S2.1: First, the pre-processed multimodal dataset is input as a three-dimensional volume B, and the image X is transformed into i ∈R H×W×D×3 The dimension is converted to C dimension to get the embedded image X i '∈R H×W×D×C , normalize the embedded image F by layer normalization i ∈R H×W×D×C , and then the embedded image is fed into the modality space-aware two-stream encoder.
[0065] Step S2.2, the modal space-aware dual-stream encoder consists of a deformable feature aggregation module branch and a spatial perception branch. The deformable feature aggregation module branch includes feature deformation and feature aggregation operations, which align multi-modal features and reduce the misalignment between modalities through feature deformation. Use 1×1×1 three-dimensional convolution to obtain effective feature description E i ∈R H×W×D×C , in order to preserve information features and reduce computational costs. i The deformation field between the modalities is estimated, and then the motion features are transformed to match the fixed features. The deformation function T is convolved by a 3×3×3 convolution. θ (D1, D2, D3, D4) = φ for modeling, where θ represents the convolution parameter and φ∈R H×W×D×4 Represents the deformation field between different modes. i ∈R H×W×D×C For each pixel position p in , φ(p) represents the offset along the x-axis, y-axis and z-axis, and the transformed position p' can be obtained by p'=p+φ(p). The differentiable and deformable transformation operation of p is defined according to the spatial transformation network:
[0066]
[0067] Here, the symbol · is a transformation operator including shift and interpolation, and δ(p') represents the neighboring voxels of p'. i φ(p) and D i ·φ(p) expects to learn F i and D i Match with the appropriate φ.
[0068] Step S2.3, after feature deformation to align multimodal features, feature aggregation generates a response map based on the local and global inter-modal relationships. The correspondence between modalities is modeled using a mapping function built on a pyramid block with four parallel layers including a global average pooling layer and a 5×5×5 convolutional layer for extracting global correspondences, and using 1×1×1 and 3×3×3 convolutional layers to capture local inter-modal information. In the case of aligned multimodal features, the response map W is calculated. i ∈R H×W×D×C , modulation and aggregation F i and F i ·φ(p). As shown in formula (2), the overall cross-modal feature aggregation is calculated as:
[0069]
[0070] Among them, F Aggr ∈R H×W×D×C is the aggregation feature, Expressed as dense element-wise multiplication.
[0071] In step S3, in order to fully learn the spatial position relationship and voxel information of the three-dimensional body, the spatial channel perception branch integrated with large kernel convolution attention, deformable layer and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations, thereby enhancing the global spatial features based on three-dimensional voxels.
[0072] Step S3.1, firstly transform the same image F as the deformable feature aggregation branch i ∈R H×W×D×C Input, feature map size and resolution conversion is achieved through downsampling convolution, and the spatial perception module that uses large kernel convolution attention and spatial attention output to process long-range dependencies and capture global spatial context information. Compared with the standard attention mechanism limited to fixed-size areas, large kernel convolution attention considers large areas within the input image in a flexible and adaptive way, and achieves a wider receptive field through deformable convolution. The output of the spatial perception module is as follows:
[0073] F S = comb(F i ,α s LKA(F i )+β s SA(F i ))
[0074] Among them, F i ∈R H×W×D×C represents the input feature map, F S represents outputs with the same dimensions, α s and β s Represents the learnable parameters of each channel, controlling the combined weight of the large kernel convolutional attention and spatial attention mechanisms. SA represents the spatial self-attention mechanism, and LKA represents the large kernel convolutional attention. The implementation is as follows:
[0075]
[0076] Among them, the input tensor F i ∈R H×W×D×C Perform point-by-point convolution operations and reduce the dimension through the activation function Gaussian error linear unit (GELU). Then, use the deep convolution (DW-Conv) and deep dilated convolution (DWD-Conv) sequence to process the transformed tensor, extract multi-scale features while retaining spatial information, and integrate deformable deep convolution (DDW-Conv) to achieve adaptive sampling of features. The spatial attention mechanism is implemented as follows:
[0077]
[0078] Among them, σ represents the Softmax function, Q, K g and V gRepresent the query, projection key and projection space value layers respectively. The size of each layer is d. Through linear normalization and linear layers, the performance of the spatial attention module is enhanced. Finally, as shown in the formula, the final output is obtained through the Comb function.
[0079] Comb(F i ,A)=C+PW-Conv(DW-Conv3(C))
[0080] in, DW-Conv3 is a 3×3×3 convolution.
[0081] Step S3.2, the channel perception module enhances the model's ability to capture complex channel relationships and selectively focuses on relevant features while suppressing noise across spatial locations. The large kernel convolutional attention and channel attention are integrated to simultaneously process inter-channel relationships and long-range dependencies. The channel perception output is as follows:
[0082] F S = comb(F i ,λ c LKA(F i ')+γ c CA(F i '))
[0083] Among them, F i '∈R H×W×D×C represents the spatial perception output feature map, λ c and γ c Represents the learnable parameters of each channel layer, which are the combined weights of large kernel convolution attention and channel attention. CA represents the channel attention mechanism, which captures the interdependence between feature channels by using dot product operations in the channel dimension between the channel value layer and the channel attention map. The output is calculated as follows:
[0084]
[0085] In step S4, in order to fully integrate the multimodal information and global spatial information extracted by the deformable aggregation branch and the spatial channel perception branch, the association-driven fusion module is used in the final stage of the encoder to process the association features of different levels and branches, and a dense connection is used to form a continuous memory mechanism to improve the effective spatial feature transfer and achieve feature fusion while maintaining feature consistency.
[0086] Step S4.1, first output the deformable feature aggregation branch And the spatial channel perception branch output The input is a correlation space feature module composed of local correlation attention and global spatial attention to obtain the initial refined features. Considering that features at different levels contain different information, a stacked correlation space feature module is used to further enhance the initial refined features through the hierarchical features extracted in the early stage. express and of and, and Respectively represent the features extracted from the lth layer. l represents the output of the associated spatial feature module at layer l. The output of the associated spatial feature module at layer l can be expressed as:
[0087]
[0088] in is the feature enhanced by the lth layer.
[0089] Step S4.2, local correlation attention adopts channel attention in local regions by using max pooling and average pooling with different kernel sizes to maintain local correlation feature similarity and select different attention for each local region in the channel dimension. By using average pooling and max pooling operations at four scales and strides, a multilayer perceptron with one hidden layer is a component of the shared network at each scale. In order to fuse the local channel attention maps of the four scales, the nearest interpolation method is used to keep the attention weights of the local regions unchanged. The attention maps of each scale are interpolated and fused separately by the Sigmoid activation function.
[0090] In step S4.3, global spatial attention uses spatial attention with four levels of different types of kernels to capture global spatial features at different levels. In order to obtain a more refined spatial attention map, a pyramid structure is used to perform multi-scale parsing of the input. The channel information of the input features is aggregated using maximum pooling and average pooling.
[0091] Step S5, in order to achieve effective global spatial feature decoding and recovery, the output features of the association-driven fusion module are input into the multi-scale convolutional attention module and the multi-scale convolutional attention decoder of the large core group gated attention cascade to capture complex spatial relationships. The multi-scale convolutional attention performs deep convolution at different levels and performs channel mixing operations between groups. The large core group gated attention refines the features fused with the jump connection through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the features of the next stage.
[0092] Step S5.1, first input the output of the last stage of the encoder through the association-driven fusion module into the efficient multi-scale convolutional attention module. The multi-scale convolutional attention module consists of channel attention, spatial attention and multi-scale deep convolution blocks. Multi-scale deep convolution is used to capture multi-scale and multi-resolution contexts, and channel mixing operations are used to merge channel connections. In the spatial attention operation, the relevant spatial features are enhanced after determining the feature focus position, the attention weight is calculated through the activation function, and then the weight is fed to the input through the Hadamard product.
[0093] Step S5.2, during decoding, use efficient upconvolution blocks to gradually upsample the current stage features to match the dimension and resolution of the feature map of the next skip connection. First, use upsampling with a scale factor of 2 to expand the feature map. Then, enhance the enlarged feature map by applying 3×3×3 depth convolution, BN, and ReLU activation functions. Finally, use 1×1×1 convolution to reduce the channel to match the next stage, realizing feature transfer of different resolutions.
[0094] Step S5.3, using large kernel group gated attention to gradually combine feature maps with attention coefficients, achieving higher activation of relevant features and suppression of irrelevant features through coefficient learning. This process uses gating signals derived from high-level features to control the flow of information at different stages, and processes features from skip connections and efficient up-convolution blocks by applying separate 3×3×3 group convolutions. Then, for the brain tumor segmentation target multi-category corresponding channels, the segmentation head is used to produce segmentation outputs from the refined feature maps of the four stages of the decoder.
[0095] The beneficial effects of this patent are:
[0096] This patent is a three-dimensional brain tumor segmentation model based on deformable feature aggregation. First, brain tumor images are difficult to segment due to poor spatial resolution, noise, and data mixed with acquisition artifacts. Secondly, in order to obtain accurate brain tumor segmentation results to facilitate monitoring of the spatial distribution of tumors and analysis of disease progression, effective complementary MRI modalities are usually required. However, different modalities have different characteristics, and their sensitivity and contribution to the segmented area are also different. It is necessary to consider the differences in multimodal information and make full use of the relationship between modalities. Finally, since MRI data has three-dimensional properties, it is easy to cause the loss of spatial information when performing image information learning and feature extraction, and it is impossible to fully obtain the spatial structure of the tumor and the relationship between adjacent tissues, making it difficult to achieve complete capture of three-dimensional volume information, which may reduce the performance of lesion segmentation. In order to solve these problems, this patent proposes a three-dimensional brain tumor segmentation model based on deformable feature aggregation. Using a modal space-aware dual-stream encoder composed of a deformable feature aggregation module branch and a spatial channel perception branch, the multimodal information and voxel information of brain tumor 3D MRI are jointly learned. Through the deformable feature aggregation module, cross-modal multi-layer fusion is performed, and the multi-modal features are aligned using differentiable transformation operations to alleviate the spatial misalignment between different modalities. Then, the mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multi-modal feature alignment and fusion. The spatial channel perception branch integrated with large kernel convolution attention, deformable layer and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations, and learn global spatial information based on three-dimensional voxels. In order to fully fuse the multi-modal information and global spatial information extracted by the deformable aggregation branch and the spatial channel perception branch, the association-driven fusion module is used in the final stage of the encoder to process the association features of different levels and branches, and dense connections are used to form a continuous memory mechanism to improve the effective spatial feature transfer and achieve feature fusion while maintaining feature consistency. Then, the multi-scale convolutional attention module and the multi-scale convolutional attention decoder of the large core group gated attention cascade are input to capture complex spatial relationships and realize feature recovery. The multi-scale convolutional attention performs deep convolution at different levels and performs channel mixing operations between groups. The large core group gated attention refines the features fused with jump connections through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the next stage features, realizing effective global spatial feature decoding and recovery. The three-dimensional brain tumor segmentation model based on deformable feature aggregation can not only fully integrate multimodal information, but also capture global spatial information in combination with the three-dimensional attributes of the data. It can use 3D MRI to obtain accurate segmentation results to realize the positioning and spatial distribution perception of brain tumors.
[0097] Among them, each step is supplemented as follows:
[0098] 1. In view of the fact that there are differences in multimodal information and the relationship between modalities cannot be fully utilized, the deformable feature aggregation module branch in the modal space-aware dual-stream encoder is used to perform cross-modal multi-layer fusion, and the multimodal features are aligned using differentiable transformation operations to alleviate the spatial misalignment between different modalities. Then, a mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multimodal feature alignment and fusion.
[0099] 2. The spatial channel perception branch integrated with large kernel convolutional attention, deformable layers and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations, learning global spatial information based on three-dimensional voxels.
[0100] 3. In order to fully integrate the multimodal information and global spatial information extracted by the deformable aggregation branch and the spatial channel perception branch, the association-driven fusion module is used in the final stage of the encoder to process the association features of different levels and branches, and a dense connection is used to form a continuous memory mechanism to improve the effective spatial feature transfer and achieve feature fusion while maintaining feature consistency.
[0101] 4. A multi-scale convolutional attention decoder that uses a cascade of multi-scale convolutional attention modules and large-core group gated attention captures complex spatial relationships. Multi-scale convolutional attention performs deep convolutions at different levels and performs channel mixing operations between groups. The large-core group gated attention refines the features fused with jump connections through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the features of the next stage, thereby achieving effective global spatial feature decoding and recovery.
[0102] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A 3D brain tumor segmentation model based on deformable feature aggregation, characterized in that: The following steps are involved: S1, data preprocessing: S2, multimodal feature alignment and fusion; S3, expands the receptive field and captures a wide range of spatial representations, enhancing the global spatial features based on 3D voxels; S4, uses the association-driven fusion module to process the association features of different levels and branches, and adopts dense connections to form a continuous memory mechanism; S5, the output features of the association-driven fusion module are input into the multi-scale convolutional attention module and the multi-scale convolutional attention decoder of the large core group gated attention cascade to capture complex spatial relationships. The multi-scale convolutional attention performs deep convolution at different levels and performs channel mixing operations between groups. The large core group gated attention refines the features fused with the jump connection through the gated attention mechanism, and uses efficient up-convolution to gradually upsample to match the dimension and resolution of the features of the next stage.
2. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S1, the model training and test data sets are divided into training set, validation set and test set according to 7:2:1, and each case contains three-dimensional MRI scans of four different modalities: T1, T1Gd, T2 and T2-FLAIR; During training, data augmentation techniques such as random rotation and scaling, elastic deformation, Gaussian noise, additional brightness enhancement, and gamma scaling are dynamically applied to improve generalization ability; Crop the non-zero voxels in brain tumor MRI; Voxel normalization was performed by calculating the mean and standard deviation; the image sequence was randomly horizontally flipped in the axial, coronal, and sagittal planes with a probability of 50% to increase the model's robustness to changes in different directions and viewing angles.
3. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: In step S2, the deformable feature aggregation module branch in the modal space-aware dual-stream encoder is used to perform cross-modal multi-layer fusion, and the multi-modal features are aligningly operated using differentiable transformation operations to alleviate the spatial misalignment between different modalities. The mapping function is designed to model the relationship between global and local modalities, and the aligned features are fused voxel by voxel to achieve multi-modal feature alignment and fusion.
4. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S2.1: Input the pre-processed multimodal dataset as a three-dimensional volume B, and transform the image X into i ∈R H×W×D×3 The dimension is converted to C dimension to get the embedded image X i '∈R H×W×D×C , normalize the embedded image F by layer normalization i ∈R H×W×D×C ,Then the embedded image is fed into the modality space-aware two-stream encoder; Step S2.2, the modal space-aware dual-stream encoder consists of a deformable feature aggregation module branch and a spatial perception branch; The deformable feature aggregation module branch includes feature deformation and feature aggregation operations, which aligns multimodal features and reduces the misalignment between modalities through feature deformation; Use 1×1×1 3D convolution to obtain effective feature description E i ∈R H×W×D×C , using E i Estimate the deformation field between the modes and then transform the motion features to match the fixed features; The deformation function T is transformed by 3×3×3 convolution θ (D1, D2, D3, D4) = φ for modeling, where θ represents the convolution parameter and φ∈R H×W×D×4 Represents the deformation field between different modes; for F i ∈R H×W×D×C For each pixel position p in , φ(p) represents the offset along the x-axis, y-axis and z-axis, and the transformed position p' can be obtained by p'=p+φ(p); the differentiable and deformable transformation operation of p is defined according to the spatial transformation network: Here, the symbol · is a transformation operator including shift and interpolation, and δ(p') represents the neighboring voxels of p'. i φ(p) and D i ·φ(p) expects to learn F i and D i Match with the appropriate φ; Step S2.3, after feature deformation to align multimodal features, feature aggregation generates a response map based on the local and global inter-modal relationships; the correspondence between the modalities is modeled using a mapping function, which is built on a pyramid block with four parallel layers including a global average pooling layer and a 5×5×5 convolutional layer for extracting global correspondences, and using 1×1×1 and 3×3×3 convolutional layers to capture local inter-modal information; in the case of aligned multimodal features, the response map W is calculated. i ∈R H×W×D×C , modulation and aggregation F i and F i ·φ(p); The overall cross-modal feature aggregation is calculated as: Among them, F Aggr ∈R H×W×D×C is the aggregation feature, Expressed as dense element-wise multiplication.
5. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: In step S3, the spatial channel perception branch integrated with large kernel convolutional attention, deformable layer and spatial channel attention is used to flexibly expand the receptive field and capture a wide range of spatial representations.
6. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S3.1, the same image F as the deformable feature aggregation branch i ∈R H×W×D×C Input, feature map size and resolution conversion is achieved through downsampling convolution, and the spatial perception module with large kernel convolution attention and spatial attention output parallel calculation processes long-range dependencies and captures global spatial context information; Compared to the standard attention mechanism which is limited to a fixed-size region, the large kernel convolutional attention considers large regions within the input image in a flexible and adaptive way, achieving a wider receptive field through deformable convolutions; The output of the spatial perception module is as follows: F S =comb(F i ,a s LKA(F i )+b s SA(F i )) Among them, F i ∈R H×W×D×C represents the input feature map, F S represents outputs with the same dimensions, α s and β s Represents the learnable parameters of each channel, controlling the combined weight of the large kernel convolutional attention and spatial attention mechanisms; SA represents the spatial self-attention mechanism, and LKA represents the large kernel convolutional attention implementation as follows: Among them, the input tensor F i ∈R H×W×D×C Perform point-by-point convolution operations and reduce the dimension through the activation function Gaussian error linear unit; then, use deep convolution and deep dilated convolution sequence to process the transformed tensor, extract multi-scale features while retaining spatial information, and integrate deformable deep convolution to achieve adaptive sampling of features; the spatial attention mechanism is implemented as follows: Among them, σ represents the Softmax function, Q, K g and V g They represent query, projection key and projection space value layers respectively. The size of each layer is d. Linear normalization and linear layer are used to enhance the performance of the spatial attention module. Finally, the final output is obtained through the Comb function as shown in the following formula; Comb(F i ,A)=C+PW-Conv(DW-Conv3(C)) in, DW-Conv3 is a 3×3×3 convolution.
7. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S3.2: The channel perception module enhances the model’s ability to capture complex channel relationships and selectively focuses on relevant features while suppressing noise across spatial locations. It integrates large kernel convolutional attention and channel attention to simultaneously process inter-channel relationships and long-range dependencies. The channel perception output is as follows: F S =comb(F i ,l c LKA(F′ i )+c c CA(F′ i )) Among them, F′ i ∈R H×W×D×C represents the spatial perception output feature map, λ c and γ c represents the learnable parameters of each channel layer, which are the combined weights of large kernel convolution attention and channel attention; CA represents the channel attention mechanism, which captures the interdependence between feature channels by using dot product operations in the channel dimension between the channel value layer and the channel attention map. The output is calculated as follows:
8. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S4.1, firstly aggregate the deformable feature branch output F i Aggr And the spatial channel perception branch output The initial refined features are obtained by inputting the correlation space feature module composed of local correlation attention and global spatial attention; the stacked correlation space feature module is used to further enhance the initial refined features through the hierarchical features extracted in the early stage; express and of and, and Respectively represent the features extracted from the lth layer; G l represents the output of the associated spatial feature module at layer l; the output of the associated spatial feature module at layer l can be expressed as: in is the feature enhanced by the lth layer; Step S4.2, local correlation attention adopts channel attention in local areas by using maximum pooling and average pooling with different kernel sizes, and selects different attention for each local area in the channel dimension; by using average pooling and maximum pooling operations to operate on four scales and strides, a multilayer perceptron with one hidden layer is a component of the shared network at each scale; the nearest interpolation method is used to keep the attention weight of the local area unchanged; the attention map of each scale is interpolated and fused separately by the Sigmoid activation function; In step S4.3, the global spatial attention uses spatial attention with four levels of different types of kernels to capture global spatial features at different levels, and uses a pyramid structure to perform multi-scale analysis on the input; the channel information of the input features is aggregated using maximum pooling and average pooling.
9. The three-dimensional brain tumor segmentation model based on deformable feature aggregation according to claim 1, characterized in that: Step S5.1, firstly, the output of the last stage of the encoder is input into the efficient multi-scale convolutional attention module through the association-driven fusion module. The multi-scale convolutional attention module consists of channel attention, spatial attention and multi-scale deep convolution blocks. Multi-scale deep convolution is used to capture multi-scale and multi-resolution contexts, and channel mixing operation is used to merge channel connections. In the spatial attention operation, the relevant spatial features are enhanced after determining the feature focus position, the attention weight is calculated through the activation function, and then the weight is fed to the input through the Hadamard product. Step S5.2, during decoding, use efficient upconvolution blocks to gradually upsample the current stage features to match the dimension and resolution of the feature map of the next skip connection; first, use upsampling with a scale factor of 2 to expand the feature map; then, enhance the enlarged feature map by applying 3×3×3 depth convolution, BN and ReLU activation functions; finally, use 1×1×1 convolution to reduce the channel to match the next stage, realizing feature transfer of different resolutions; Step S5.3, using large core group gated attention to gradually combine feature maps with attention coefficients, and achieve higher activation of relevant features and suppression of irrelevant features through coefficient learning; The process uses gating signals derived from high-level features to control the flow of information at different stages, processing features from skip connections and efficient upconvolution blocks by applying separate 3×3×3 groups of convolutions; then, a segmentation head is used to produce segmentation outputs from the refined feature maps of the four stages of the decoder for the multi-category corresponding channels of the brain tumor segmentation target.
Citation Information
Cited By
Polyp image segmentation method based on double-branch feature progressive fusion network
CN119360031A
Method for reducing multi-modal characteristic quantity of large model based on multi-level coding
CN119559477A
Method for establishing brain glioma hierarchical hybrid network based on multi-view feature fusion
CN119693698A
Method for establishing brain glioma grading hybrid network based on multi-view feature fusion
CN119693698B
Remote sensing target detection method and device, equipment and storage medium
CN119785220A
