MRI (Magnetic Resonance Imaging) brain tumor image segmentation method based on multi-scale adaptive prototype aggregation

Abstract

The invention discloses an MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation, and relates to the technical field of medical image analysis, and the method comprises the steps: obtaining a to-be-segmented brain tumor MRI image, and carrying out the preprocessing of the brain tumor MRI image; and inputting the preprocessed brain tumor MRI image into the constructed multi-scale adaptive prototype aggregated brain tumor segmentation network. The network is based on a U-Net architecture, prototypes of all categories are aggregated by introducing a prototype aggregation module into a bottleneck layer and a second layer to a fourth layer of a decoder, and the prototypes perform weighted aggregation on all features belonging to the corresponding categories according to prediction of each feature, so that the context of the sensing prototypes is captured; the tumor features are extracted from different space and scale backgrounds by introducing double multi-scale aggregation modules into jump connection. According to the method, multi-scale space and semantic information are effectively utilized, different types of input specific prototypes are extracted and aggregated at the same time, and brain tumor MRI image segmentation with higher precision is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation. Background Art

[0002] Brain tumors seriously endanger human life and health, so early detection and early treatment can greatly increase the cure rate for brain tumor patients. Magnetic resonance (MR) images are one of the most commonly used means of detecting brain tumors, which can produce tumor images of different modalities and provide doctors with a basis for diagnosis. However, due to the huge differences in the location, size, shape and anatomical structure of lesions among patients, even experienced physicians find it difficult to accurately manually segment these lesions in MRI images. In addition, due to differences in subjective and professional levels among different physicians, there are large differences in manual segmentation of MRI, which is not conducive to the analysis of patients' conditions. Therefore, the development of a high-precision, automated MRI segmentation algorithm is of great significance for providing accurate quantitative analysis results and assisting doctors in making decisions.

[0003] In recent years, medical image segmentation methods based on deep learning have become the most popular methods, especially the U-shaped network represented by U-Net. Due to its simple U-shaped structure and jump connection design, U-Net only needs a small amount of data to achieve ideal segmentation performance. However, U-Net is limited to a single MRI slice and it is difficult to capture spatial context information. Some researchers have proposed to integrate multi-scale context modeling into U-Net, and gradually compensate for the lost spatial information by aggregating context information of different scales. Although the proposed method can effectively improve the segmentation accuracy, it only focuses on aggregating context information from the overall and local aspects of the image, ignoring the meaning of voxel representation between the same categories. Therefore, they do not make full use of semantic information, and they also face the same problem, that is, when voxels capture too much context information of other categories, the network will misjudge these pixels as other categories. Summary of the invention

[0004] In view of the problem that the existing network does not fully utilize the multi-scale information in the context of each image and the differences between different categories, the present invention discloses an MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation. The method is based on a brain tumor segmentation network that effectively utilizes multi-scale spatial and semantic information, and simultaneously extracts and aggregates input specific prototypes of different categories to achieve more accurate tumor segmentation. A dual multi-scale aggregation module is introduced in the jump connection to extract tumor features from different spatial and scale backgrounds. The prototype aggregation module is integrated into the bottleneck layer and the decoder to aggregate prototypes of all categories, where the prototype is weighted and aggregated by all features belonging to the corresponding category according to the prediction of each feature, including the following steps:

[0005] Collect multimodal MR images of brain tumors to form a brain tumor image dataset, which is used as the input dataset of the network;

[0006] Perform image preprocessing on the dataset and process the dataset into the format required by the network model;

[0007] Construct a brain tumor MRI segmentation network model based on multi-scale adaptive prototype aggregation;

[0008] Training the brain tumor MRI segmentation network model and saving weights;

[0009] The brain tumor MRI segmentation network model is loaded, the network model is tested to obtain the brain tumor segmentation result, and the segmentation result is post-processed to obtain the final segmentation result.

[0010] Furthermore, suppose that the brain tumor image dataset includes MR images of four modalities, and the dataset is annotated into four types: background and healthy part, gangrene part, edema part and enhanced tumor part, and the useless background information in the image is trimmed;

[0011] The Z-score method is used to normalize each modality image to reduce the difference in contrast between different modalities. The formula is:

[0012]

[0013] Where z is the input MR image, z' is the normalized MR image, μ is the input MR image mean, and δ is the input MR image standard deviation;

[0014] Create a new channel and one-hot encode the foreground voxels to distinguish between background voxels and voxels with values ​​close to 0 after normalization;

[0015] The input data set is divided into training set and test set according to a certain ratio, and the training set is enhanced using a variety of data enhancement methods, including random scaling, random flipping, Gaussian noise, Gaussian blur and random brightness methods.

[0016] Furthermore, the brain tumor MRI segmentation network model is based on a 4-layer original U-Net encoder structure, and also includes a prototype aggregation module and a dual multi-scale aggregation module, wherein each layer of the encoder uses a 2-layer convolution block to extract brain tumor feature information and a layer of maximum pooling layer for downsampling, and the prototype aggregation module is set in the bottleneck layer and the 2nd to 4th layers of the decoder to classify each voxel in the upsampling process, and the dual multi-scale aggregation module is set in the lateral connection to fuse different spatial information of different scales into the same size, and reconstruct the mixed feature map into the original size, and use a deep supervision strategy to enhance the model's ability to guide the aggregation of voxels of the same category.

[0017] Furthermore, the following approach is used to construct the prototype aggregation module:

[0018] Calculate the features and probability distribution of brain tumor images: Let the brain tumor image be X∈R C×D×H×W , the size of the category probability distribution P is (N, C), the brain tumor image X is divided into different category regions based on the category probability distribution P, and each category region generates a matrix of size C×D×H×W based on the argmax operation, C cp ={X [*,i,j,k] = argmax(P [*,i,j,k] =cx},

[0019] Aggregate the voxels of each category area and assign the corresponding composite vector: Aggregate the category represented by each voxel in the image to obtain the composite vector: where X' cp For category c p The prototype vector of , mean is the mean operation, To perform exponential operation on the probability value, ∑e Pcp It is the denominator in the softmax normalization, which is used for normalization when calculating the weighted average;

[0020] Distribute the composite vectors into a new matrix so that each category has a corresponding composite vector:

[0021] X PA,[*,i,j,k] =X' cp if atgmax(P [*,i,j,k] )=c p

[0022] Where: X PA,[*,i,j,k] Represents the matrix X PA The eigenvector of position (i, j, k) in X' cp Indicates category cp The prototype vector, argmax(P [*,i,j,k]) represents the function argmax returning the probability vector P [*,i,j,k] The index of the maximum value in P [*,j,j,k] Represented as a probability vector at position (i,j,k).

[0023] Furthermore, the dual multi-scale aggregation module integrates multi-scale information of the brain tumor image at the spatial and semantic levels to enhance the perception of brain tumor features. The dual multi-scale aggregation module includes a scale-aware aggregation module and a semantic fusion module. The scale-aware aggregation module is used to extract and reconstruct multi-scale spatial features of the brain tumor image. The semantic fusion module merges feature maps of different scales and adjusts them to the size required by the decoder.

[0024] When building the scale-aware aggregation module: Input brain tumor feature map Use multiple convolutions with different kernel sizes to extract spatial features from different receptive fields, starting with a 3×3×3 convolution kernel, expressed as: Where: X' i Represented as output feature matrix or vector, Cat represents channel concatenation operation, Represented as a depth-wise separable convolution operation;

[0025] Through point-wise convolution cross-group fusion, the sub-feature map X containing different sensory fields is reconstructed im , X' im =LeakyReLU(GN(PWConv(x im )))) to ensure that the features of each sub-channel complement and integrate each other, X' im It is represented as the output feature matrix, PWConv is represented as point-by-point convolution, GN is represented as group normalization, and LeakyReLU is represented as the ReLU activation function with leakage;

[0026] When building the semantic fusion module: Integrate semantic information of different scales into U-Net, all containing 1-4 layers of different semantic insights are uniformly sampled to match the same dimension G m =Cat(Down(X' 1m ),Down(X' 2m ),(X' 3m ),Up(X' 4m )), where Cat represents the connection operation, Down represents downsampling, Up represents upsampling, and X' im Represented as feature maps to be connected;

[0027] x' containing different semantic features im Fusion G' through pointwise convolution m=LeakyReLU(GN(PWConv(G m ))), where: G' m represents the output features, PWConv represents point-wise convolution, GN represents group normalization, and LeakyReLU represents the ReLU activation function with leakage;

[0028] G' m ∈R n×32×32×32 Connect to the feature map Y and sample the feature map to the shape Y required by the decoder i , shape Y i With encoder output X i Fusion, get the final output Y' of the semantic fusion module i .

[0029] Furthermore, when training the brain tumor MRI segmentation network model:

[0030] The network hyperparameters were set, the Adam optimizer was used to optimize the network model structure, the cosine annealing strategy was used to adjust the dynamic learning rate, the automatic mixed precision was used to reduce the amount of calculation, the optimal training weights were saved, and during the verification process, the prediction results were post-processed using test time enhancement and voxel clipping. The training set was input into the multi-scale adaptive prototype aggregation model for training, the Focal loss function and the Dice loss function were fused as the loss function of the network model, and the three lesion areas of the entire tumor, tumor core, and enhanced tumor were taken as segmentation targets.

[0031] The present invention discloses an MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation. The method is based on the U-Net architecture. By introducing a prototype aggregation module into the bottleneck layer and the second to fourth layers of the decoder, prototypes of all categories are aggregated, wherein the prototype weights and aggregates all features belonging to the corresponding category according to the prediction of each feature, thereby capturing the context of the perceptual prototype; by introducing a dual multi-scale aggregation module in the jump connection, tumor features are extracted from different spatial and scale backgrounds. The present invention effectively utilizes multi-scale space and semantic information to simultaneously extract and aggregate input specific prototypes of different categories, thereby achieving higher-precision brain tumor MRI image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a flow chart of the method disclosed in the present invention;

[0034] Figure 2 A network structure diagram of the method disclosed in the present invention;

[0035] Figure 3 A schematic diagram of a prototype aggregation module structure of the method disclosed in the present invention;

[0036] Figure 4 This is a structural diagram of the dual multi-scale aggregation module of the method disclosed in the present invention. DETAILED DESCRIPTION

[0037] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0038] The present invention provides a brain tumor MRI segmentation method based on multi-scale adaptive prototype aggregation. A prototype aggregation module and a dual multi-scale aggregation module are proposed and added to the 3DU-Net architecture. The multi-scale spatial and semantic information are effectively utilized, and different categories of input specific prototypes are extracted and aggregated at the same time to achieve more accurate tumor segmentation.

[0039] Figure 1 This is a flow chart of the method of the present invention. The first step is image preprocessing to convert the data set into the input required by the network. The second step is to build an adaptive prototype multi-scale network model structure. Then the data set is trained and the weights are saved. Finally, the training weights are used to predict the test set to obtain the segmentation results. The accuracy of brain tumor segmentation is further improved through post-processing methods.

[0040] Figure 2 This is the network model structure diagram of the dual multi-scale aggregation module proposed in the present invention. Based on the U-Net architecture, the prototype aggregation module is integrated into the bottleneck layer and the second to fourth layers of the decoder to aggregate all categories of prototypes.

[0041] The specific structure of the prototype aggregation module is as follows Figure 3 As shown. The prototype processes the multimodal MR images of brain tumors into the input format of the network according to each S1: data preprocessing. The specific method is as follows:

[0042] S11: Brain tumor image dataset BraTS2021 dataset has four modal MR images, each with an image size of 240mm×240mm×155mm, which are manually annotated by experts, including background and healthy parts with label 0, gangrene with label 1, edema with label 2, and enhanced tumor with label 4. Since there is a lot of redundant background information in the image, the useless background information is trimmed first.

[0043] S12: The Z-score method is used to normalize the images of each modality to solve the problem of contrast differences between different modalities. The formula is:

[0044]

[0045] Where z is the input MR image, z' is the normalized MR image, μ is the input MR image mean, and δ is the input MR image standard deviation.

[0046] S13: In order to distinguish background voxels from voxels whose normalized values ​​are close to 0, an additional new channel is created to perform one-hot encoding on foreground voxels. During the training phase, the image size of the input network is 5×128×128×128.

[0047] S2 constructs a model structure for brain tumor MRI segmentation based on multi-scale adaptive prototype aggregation, specifically in the following way:

[0048] S21: Build a prototype aggregation module:

[0049] like Figure 3 As shown, in the prototype aggregation module, given the input X∈R C×D×H×W , first use the category probability distribution represented by the classification head M, input X, that is, P, as shown below:

[0050] P=M(x)

[0051] The size of P is C P ×D×H×W,M is implemented by a 1×1×1 convolutional layer. X and P can be used to represent different category regions X cp and P cp ,as follows:

[0052] X cp ={X [*,i,j,k]|argmax(P[*,i,j,k]) =c p}

[0053]

[0054] Among them, c p ∈Cp is the category label, X ck For size N cp ×C matrix, P cp For size N cp ×1 matrix, N cp is the number of voxels belonging to category cp.

[0055] In order to aggregate the prototypes according to the category represented by each voxel, the representation is as follows:

[0056]

[0057] Where: X' cp For Category cp The prototype vector of , mean is the mean operation, here it refers to the weighted average, e Pcp,[n,*] To perform exponential operation on the probability value, it is usually used for normalization in the softmax function, ∑e Pcp It is the denominator in the softmax normalization and is used for normalization when calculating the weighted average.

[0058] After computing all class representations, the prototypes are assigned to a new matrix according to the class label of the corresponding element:

[0059] X PA,[*,i,j,k] =X' cp if argmax(P [*,i,j,k] )=c p

[0060] Where: X PA,[*,i,j,k] Represents the matrix X PA The feature vector at position (i, j, k) in the image is represented by the symbol *, which represents all dimensions of the feature vector (i.e., the number of channels). cp It represents the category c p The prototype vector, argmax(P [*,i,j,k] ) represents the function argmax returning the probability vector P [*,i,j,k] The index of the maximum value in P [*,i,j,k] Represented as a probability vector at position (i,j,k).

[0061] In this way, the prototype aggregation module can extract and aggregate input-specific prototypes of different categories to achieve more accurate tumor segmentation.

[0062] S22: Construct dual multi-scale aggregation module:

[0063] In semantic segmentation tasks, multi-scale features effectively improve segmentation accuracy, especially brain tumor segmentation. However, most studies only use different convolution kernels or feature map fusion to capture multi-scale information, and fail to fully integrate this information at the spatial and semantic levels. Inspired by the work of researchers on scale-aware aggregation, this study incorporates the scale-aware aggregation method into the lateral connection of U-Net to extract and reconstruct multi-scale spatial features. At the same time, a semantic fusion module is introduced to merge feature maps of different scales and adjust them to the size required by the decoder. By combining the scale-aware aggregation module and the semantic fusion module, a dual multi-scale aggregation module is formed to achieve multi-scale information fusion at the spatial and semantic levels, and enhance the perception of multi-scale information of brain tumors.

[0064] S221: Building a scale-aware aggregation module

[0065] The scale-aware aggregation module uses multiple convolutions with different kernel sizes to extract spatial features from different receptive fields, and expands the receptive field through larger convolution kernels, thereby enhancing the long-range modeling capability. Input feature map Where i∈4 represents the output of the i-th layer decoder, which is divided into n sub-channels, each channel using separable convolutions of different depths. The process starts with a 3×3×3 convolution kernel, and then the kernel size of each sub-channel increases by 2. This process can be expressed as:

[0066]

[0067] Among them, X in Represents the i-th layer input feature map X i ,X i Divided into n sub-channels along the channel dimension, k n ∈{3,5,…,2n+3} represents the size of the convolution kernel of the nth layer. Cat represents the connection operation, and DW represents the depth convolution of different convolution kernels.

[0068] Then, in order to enhance the different sub-channels x n The information interaction between them reconstructs x containing different sensory fields. n , where a channel is selected from each sub-channel to construct a new sub-feature map x im , where m = C i / n,C i For X i The number of channels, x im Each channel in contains features of different granularity, which are fused across groups through point-wise convolution. All features are fused across groups, as shown below:

[0069] X im =LeakyReLU(GN(PWConv(x im )))

[0070] Among them, GN represents GroupNorm, PWConv represents point-wise convolution, its kernel size is 1, the step size is 1, and the padding is 0.

[0071] S222: Building a semantic fusion module

[0072] Integrating semantic information of different scales into U-Net amplifies the multi-scale interactions. All layers containing 1-4 layers of different semantic insights is uniformly sampled to match the same dimension. It is set to 32×32×32 to minimize the computational load. Therefore, x' contains different semantic features. im Further fusion through point-wise convolution is expressed as:

[0073] G m =Cat(Down(X 1m ),Down(X 2m ),(X 3m ),Up(X 4m ))

[0074] G' m =LeakyReLU(GN(PWConv(G m )))

[0075] Among them, Down represents maximum pooling, Up represents trilinear interpolation, and Gm means merging the mth sub-feature maps of different layers into one feature map after sampling.

[0076] Finally, m(G' m ∈R n×32×32×32 ) is concatenated into the feature map Y, which is then downsampled to the shape Y required by the decoder i . Compare it with the encoder output X i Fusion, get the final output Y' of the semantic fusion module i , which can be expressed as:

[0077] Y=Cat(G'1,G'2,…,G' m )

[0078] Y i =Sampling(LeakyReLU(GN(PWConV(Y)))

[0079] Y' i =LeakyReLU(GN(PWConv(X i ,Y i )))

[0080] Among them, Sampling refers to the process of adjusting the feature map to the required target size by upsampling or downsampling.

[0081] The dual multi-scale aggregation model combines the scale-aware aggregation module and the semantic fusion module technology to enhance the network's efficient aggregation ability of multi-scale spatial and semantic information, thereby enabling more accurate extraction of brain tumor features.

[0082] S23: Construct a multi-scale adaptive prototype aggregation module. The overall model architecture is as follows Figure 2As shown here, the widely used 3DU-Net structure is used as the baseline, and the innovative dual multi-scale aggregation module and prototype aggregation module are further introduced. This improvement enables the model to effectively integrate spatial and semantic information of different scales and distinguish voxels of each category through prototype aggregation, thereby significantly improving the segmentation performance of the network.

[0083] In the specific implementation, given the input of the model, the size is 5×H×W×D, and the encoder structure of the 4-layer original U-Net is initially adopted. Each layer of the encoder includes 2 layers of convolution blocks for extracting brain tumor feature information and a layer of maximum pooling for downsampling, where each layer of convolution blocks includes a 3D convolution, GroupNorm and LeakyReLu.

[0084] To more accurately restore semantic information to high-resolution images, prototype aggregation blocks are used in the bottleneck layer and layers 2-4 of the decoder to refine the classification of each voxel during upsampling, where trilinear interpolation is used for upsampling. Since the large spatial dimension of the feature map in the first decoder layer has a large impact on the computational speed, only convolution blocks are used in the initial decoder layers. In the prototype aggregation module, feature extraction starts with two layers of convolution blocks, similar in structure to the convolution blocks in the encoder. Subsequently, a 1×1×1 convolution kernel serves as the guided classifier head of the prototype aggregation module to help accurately classify each voxel into its respective category.

[0085] A dual multi-scale aggregation module is inserted into the skip connection to fuse the spatial information of different scales into a scale of 32×32×32, and then the fused feature map is reconstructed into the original dimension. Finally, a deep supervision strategy is used to combine the classification results guided by the aggregation of the prototype aggregation module with the classification results after aggregation, and the deep loss is calculated to enhance the network's ability to effectively guide the aggregation of voxels of the same category.

[0086] S3: Train the network and save the network model, specifically in the following way:

[0087] S31: Set network hyperparameters, use Adam optimizer, use cosine annealing to dynamically adjust the learning rate and use automatic mixed precision for strategy training, and finally save the optimal training weights.

[0088] S32: The training set is input into the multi-scale background brain tumor model of adaptive prototype aggregation for training, the Focal loss function and the Dice loss function are fused as the loss function of the network model, and the three lesion areas of the entire tumor, tumor core, and enhanced tumor are taken as segmentation targets.

[0089] S4: Load the network model, test the test set to obtain the brain tumor segmentation result, and perform post-processing to obtain the final segmentation result. The specific method is as follows:

[0090] S41: Load the training weights into the network and predict the segmentation results of all tumors, tumor cores, and enhanced tumors in the test set.

[0091] S42: Through the test time augmentation (TTA) method, the input brain tumor is flipped along the x-axis, y-axis, z-axis, xy-axis, xz-axis, yz-axis, and xyz-axis respectively, and the segmentation results are predicted, and the predicted results are averaged.

[0092] S43: Use the voxel clipping method to replace the prediction results with corresponding labels according to a certain probability and obtain the final segmentation results. In the prediction results, the voxels belonging to the entire tumor are replaced with label 0 with a certain probability; the voxels belonging to the tumor core are replaced with label 2 with a certain probability; the voxels belonging to the enhanced tumor are replaced with label 1 with a certain probability. At the same time, the independent enhanced tumor blocks and the overall enhanced tumor blocks whose number of voxels and average probability are less than a certain value are replaced with label 1.

[0093] The prediction of features is weighted aggregation of all features belonging to the corresponding category, thereby capturing the context of the perceptual prototype. A dual multi-scale aggregation module is introduced in the skip connection to extract tumor features from different spatial and scale contexts. The specific structure is as follows Figure 4 shown.

[0094] The specific steps of the method disclosed by the present invention are as follows:

[0095] S1: Data preprocessing, processing brain tumor multimodal MR images into the input format of the network. The specific method is as follows:

[0096] S11: Brain tumor image dataset BraTS2021 dataset has four modal MR images, each with an image size of 240mm×240mm×155mm, which are manually annotated by experts, including background and healthy parts with label 0, gangrene with label 1, edema with label 2, and enhanced tumor with label 4. Since there is a lot of redundant background information in the image, the useless background information is trimmed first.

[0097] S12: The Z-score method is used to normalize the images of each modality to solve the problem of contrast differences between different modalities. The formula is:

[0098]

[0099] Where z is the input MR image, z' is the normalized MR image, μ is the input MR image mean, and δ is the input MR image standard deviation.

[0100] S13: In order to distinguish background voxels from voxels whose normalized values ​​are close to 0, an additional new channel is created to perform one-hot encoding on foreground voxels. During the training phase, the image size of the input network is 5×128×128×128.

[0101] S2 constructs a model structure for brain tumor MRI segmentation based on multi-scale adaptive prototype aggregation, specifically in the following way:

[0102] S21: Build a prototype aggregation module:

[0103] like Figure 3 As shown, in the prototype aggregation module, given the input X∈R C×D×H×W , first use the category probability distribution represented by the classification head M, input X, that is, P, as shown below:

[0104] P=M(x)

[0105] The size of P is C P ×D×H×W,M is implemented by a 1×1×1 convolutional layer. X and P can be used to represent different category regions X cp and P cp ,as follows:

[0106] X cp ={X [*,i,j,k] |argmax(P [*,i,j,k] )=c p}

[0107]

[0108] Among them, cp∈Cp is the category label, X ck For size N cp ×C matrix, P cp For size N cp ×1 matrix, N cp is the number of voxels belonging to category cp.

[0109] In order to aggregate the prototypes according to the category represented by each voxel, the representation is as follows:

[0110]

[0111] Where: X' cp For category c p The prototype vector of , mean is the mean operation, here it refers to the weighted average, e Pcp,[n , *] To perform exponential operation on the probability value, it is usually used for normalization in the softmax function, ∑e PcpIt is the denominator in the softmax normalization and is used for normalization when calculating the weighted average.

[0112] After computing all class representations, the prototypes are assigned to a new matrix according to the class label of the corresponding element:

[0113] X PA,[*,i,j,k] =X' cp if argmax(P [*,i,j,k] )=c p

[0114] Where: X PA,[*,i,j,k] Represents the matrix X PA The feature vector at position (i, j, k) in the image is represented by the symbol *, which represents all dimensions of the feature vector (i.e., the number of channels). cp It represents the category c p The prototype vector, argmax(P [*,i,j,k] ) represents the function argmax returning the probability vector P [*,i,j,k] The index of the maximum value in P [*,i,j,k] Represented as a probability vector at position (i,j,k).

[0115] In this way, the prototype aggregation module can extract and aggregate input-specific prototypes of different categories to achieve more accurate tumor segmentation.

[0116] S22: Construct dual multi-scale aggregation module:

[0117] In semantic segmentation tasks, multi-scale features effectively improve segmentation accuracy, especially brain tumor segmentation. However, most studies only use different convolution kernels or feature map fusion to capture multi-scale information, and fail to fully integrate this information at the spatial and semantic levels. Inspired by the work of researchers on scale-aware aggregation, this study incorporates the scale-aware aggregation method into the lateral connection of U-Net to extract and reconstruct multi-scale spatial features. At the same time, a semantic fusion module is introduced to merge feature maps of different scales and adjust them to the size required by the decoder. By combining the scale-aware aggregation module and the semantic fusion module, a dual multi-scale aggregation module is formed to achieve multi-scale information fusion at the spatial and semantic levels, and enhance the perception of multi-scale information of brain tumors.

[0118] S221: Building a scale-aware aggregation module

[0119] The scale-aware aggregation module uses multiple convolutions with different kernel sizes to extract spatial features from different receptive fields, and expands the receptive field through larger convolution kernels, thereby enhancing the long-range modeling capability. Input feature map Where i∈4 represents the output of the i-th layer decoder, which is divided into n sub-channels, each channel using separable convolutions of different depths. The process starts with a 3×3×3 convolution kernel, and then the kernel size of each sub-channel increases by 2. This process can be expressed as:

[0120]

[0121] Among them, X in Represents the i-th layer input feature map X i ,X i Divided into n sub-channels along the channel dimension, k n ∈{3,5,…,2n+3} represents the size of the convolution kernel of the nth layer. Cat represents the connection operation, and DW represents the depth convolution of different convolution kernels.

[0122] Then, in order to enhance the different sub-channels x n The information interaction between them reconstructs x containing different sensory fields. n , where a channel is selected from each sub-channel to construct a new sub-feature map x im , where m = C i / n,C i For X i The number of channels, x im Each channel in contains features of different granularity, which are fused across groups through point-wise convolution. All features are fused across groups, as shown below:

[0123] X im =LeakyTeLU(GN(PWConv(x im )))

[0124] Among them, GN represents GroupNorm, PWConv represents point-wise convolution, its kernel size is 1, the step size is 1, and the padding is 0.

[0125] S222: Building a semantic fusion module

[0126] Integrating semantic information of different scales into U-Net amplifies the multi-scale interactions. All layers containing 1-4 layers of different semantic insights is uniformly sampled to match the same dimension. It is set to 32×32×32 to minimize the computational load. Therefore, x' contains different semantic features. im Further fusion through point-wise convolution is expressed as:

[0127] G m =Cat(Down(X′ 1m ),Down(X′ 2m ),(X′ 3m ),Up(X′4m ))

[0128] G' m =LeakyTeLU(GN(PWConv(G m )))

[0129] Among them, Down represents maximum pooling, Up represents trilinear interpolation, and Gm means merging the mth sub-feature maps of different layers into one feature map after sampling.

[0130] Finally, m(G' m ∈R n×32×32×32 ) is concatenated into the feature map Y, which is then downsampled to the shape Y required by the decoder i . Compare it with the encoder output X i Fusion, get the final output Y' of the semantic fusion module i , which can be expressed as:

[0131] Y=Cat(G'1,G'2,…,G' m )

[0132] Y i =Sampling(LeakyReLU(GN(PWConV(Y)))

[0133] Y' i =LeakyReLU(GN(PWConv(X i ,Y i )))

[0134] Among them, Sampling refers to the process of adjusting the feature map to the required target size by upsampling or downsampling.

[0135] The dual multi-scale aggregation model combines the scale-aware aggregation module and the semantic fusion module technology to enhance the network's efficient aggregation ability of multi-scale spatial and semantic information, thereby enabling more accurate extraction of brain tumor features.

[0136] S23: Construct a multi-scale adaptive prototype aggregation module. The overall model architecture is as follows Figure 2 As shown here, the widely used 3DU-Net structure is used as the baseline, and the innovative dual multi-scale aggregation module and prototype aggregation module are further introduced. This improvement enables the model to effectively integrate spatial and semantic information of different scales and distinguish voxels of each category through prototype aggregation, thereby significantly improving the segmentation performance of the network.

[0137] In the specific implementation, given the input of the model, the size is 5×H×W×D, and the encoder structure of the 4-layer original U-Net is initially adopted. Each layer of the encoder includes 2 layers of convolution blocks for extracting brain tumor feature information and a layer of maximum pooling for downsampling, where each layer of convolution blocks includes a 3D convolution, GroupNorm and LeakyReLu.

[0138] To more accurately restore semantic information to high-resolution images, prototype aggregation blocks are used in the bottleneck layer and layers 2-4 of the decoder to refine the classification of each voxel during upsampling, where trilinear interpolation is used for upsampling. Since the large spatial dimension of the feature map in the first decoder layer has a large impact on the computational speed, only convolution blocks are used in the initial decoder layers. In the prototype aggregation module, feature extraction starts with two layers of convolution blocks, similar in structure to the convolution blocks in the encoder. Subsequently, a 1×1×1 convolution kernel serves as the guided classifier head of the prototype aggregation module to help accurately classify each voxel into its respective category.

[0139] A dual multi-scale aggregation module is inserted into the skip connection to fuse the spatial information of different scales into a scale of 32×32×32, and then the fused feature map is reconstructed into the original dimension. Finally, a deep supervision strategy is used to combine the classification results guided by the aggregation of the prototype aggregation module with the classification results after aggregation, and the deep loss is calculated to enhance the network's ability to effectively guide the aggregation of voxels of the same category.

[0140] S3: Train the network and save the network model, specifically in the following way:

[0141] S31: Set network hyperparameters, use Adam optimizer, use cosine annealing to dynamically adjust the learning rate and use automatic mixed precision for strategy training, and finally save the optimal training weights.

[0142] S32: The training set is input into the multi-scale background brain tumor model of adaptive prototype aggregation for training, the Focal loss function and the Dice loss function are fused as the loss function of the network model, and the three lesion areas of the entire tumor, tumor core, and enhanced tumor are taken as segmentation targets.

[0143] S4: Load the network model, test the test set to obtain the brain tumor segmentation result, and perform post-processing to obtain the final segmentation result. The specific method is as follows:

[0144] S41: Load the training weights into the network and predict the segmentation results of all tumors, tumor cores, and enhanced tumors in the test set.

[0145] S42: Through the test time augmentation (TTA) method, the input brain tumor is flipped along the x-axis, y-axis, z-axis, xy-axis, xz-axis, yz-axis, and xyz-axis respectively, and the segmentation results are predicted, and the predicted results are averaged.

[0146] S43: Use the voxel clipping method to replace the prediction results with corresponding labels according to a certain probability and obtain the final segmentation results. In the prediction results, the voxels belonging to the entire tumor are replaced with label 0 with a certain probability; the voxels belonging to the tumor core are replaced with label 2 with a certain probability; the voxels belonging to the enhanced tumor are replaced with label 1 with a certain probability. At the same time, the independent enhanced tumor blocks and the overall enhanced tumor blocks whose number of voxels and average probability are less than a certain value are replaced with label 1.

[0147] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation, characterized in that: The following steps are involved: Collect multimodal MR images of brain tumors to form a brain tumor image dataset, which is used as the input dataset of the network; Perform image preprocessing on the dataset and process the dataset into the format required by the network model; Construct a brain tumor MRI segmentation network model based on multi-scale adaptive prototype aggregation; Training the brain tumor MRI segmentation network model and saving weights; The brain tumor MRI segmentation network model is loaded, the network model is tested to obtain the brain tumor segmentation result, and the segmentation result is post-processed to obtain the final segmentation result.

2. The MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation according to claim 1, characterized in that: Assume that the brain tumor image dataset includes four modal MR images. The dataset is annotated into four types: background and healthy part, gangrene part, edema part and enhanced tumor part, and the useless background information in the image is trimmed. The Z-score method is used to normalize each modality image to reduce the difference in contrast between different modalities. The formula is: Where z is the input MR image, z' is the normalized MR image, μ is the input MR image mean, and δ is the input MR image standard deviation; Create a new channel and one-hot encode the foreground voxels to distinguish between background voxels and voxels with values ​​close to 0 after normalization; The input data set is divided into training set and test set according to a certain ratio, and the training set is enhanced using a variety of data enhancement methods, including random scaling, random flipping, Gaussian noise, Gaussian blur and random brightness methods.

3. The MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation according to claim 1, characterized in that: The brain tumor MRI segmentation network model is based on a 4-layer original U-Net encoder structure, and also includes a prototype aggregation module and a dual multi-scale aggregation module, wherein each layer of the encoder uses a 2-layer convolution block to extract brain tumor feature information and a layer of maximum pooling layer for downsampling, and the prototype aggregation module is set in the bottleneck layer and the 2nd to 4th layers of the decoder to classify each voxel in the upsampling process, and the dual multi-scale aggregation module is set in the lateral connection to fuse different spatial information of different scales into the same size, and reconstruct the mixed feature map into the original size, and a deep supervision strategy is used to enhance the model's ability to guide the aggregation of voxels of the same category.

4. The MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation according to claim 3, characterized in that: The following approach is used to build the prototype aggregation module: Calculate the features and probability distribution of brain tumor images: Let the brain tumor image be X∈R C×D×H×W , the size of the category probability distribution P is (N, C), the brain tumor image X is divided into different category regions based on the category probability distribution P, and each category region generates a matrix of size C×D×H×W based on the argmax operation, X cp ={X [*,i,j,k] = argmax(P [*,i,j,k] =c p }, Aggregate the voxels of each category area and assign the corresponding composite vector: Aggregate the category represented by each voxel in the image to obtain the composite vector: where X' cp For category c p The prototype vector, mean is the mean operation, e Pcp , [n,*] To perform exponential operation on the probability value, ∑e Pcp It is the denominator in the softmax normalization, which is used for normalization when calculating the weighted average; Distribute the composite vectors into a new matrix so that each category has a corresponding composite vector: X PA,[*,i,j,k] =X' cp if argmax(P [*,i,j,k] )=c p Where: X PA,[*,i,j,k] Represents the matrix X PA The eigenvector of position (i, j, k) in X' cp Indicates that it is of category c p The prototype vector, argmax(P [*,i,j,k] ) represents the function argmax returning the probability vector P [*,i,j,k] The index of the maximum value in P [*,i,j,k] Represented as a probability vector at position (i,j,k).

5. The MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation according to claim 3, characterized in that: The dual multi-scale aggregation module integrates multi-scale information of the brain tumor image at the spatial and semantic levels to enhance the perception of brain tumor features. The dual multi-scale aggregation module includes a scale-aware aggregation module and a semantic fusion module. The scale-aware aggregation module is used to extract and reconstruct multi-scale spatial features of the brain tumor image. The semantic fusion module merges feature maps of different scales and adjusts them to the size required by the decoder. When building the scale-aware aggregation module: Input brain tumor feature map Use multiple convolutions with different kernel sizes to extract spatial features from different receptive fields, starting with a 3×3×3 convolution kernel, expressed as: Where: X' i Represented as output feature matrix or vector, Cat represents channel concatenation operation, Represented as a depth-wise separable convolution operation; Through point-wise convolution cross-group fusion, the sub-feature map X containing different sensory fields is reconstructed im , X' im =LeakyReLU(GN(PWConv(x im )))) to ensure that the features of each sub-channel complement and integrate each other, X' im It is represented as the output feature matrix, PWConv is represented as point-by-point convolution, GN is represented as group normalization, and LeakyReLU is represented as the ReLU activation function with leakage; When building the semantic fusion module: Integrate semantic information of different scales into U-Net, all containing 1-4 layers of different semantic insights are uniformly sampled to match the same dimension G m =Cat(Down(X' 1m ),Down(X' 2m ),(X' 3m ),Up(X' 4m )), where Cat represents the connection operation, Down represents downsampling, Up represents upsampling, and X' im Represented as feature maps to be connected; x' containing different semantic features im Fusion G' through pointwise convolution m =LeakyReLU(GN(PWConv(G m ))), where: G' m represents the output features, PWConv represents point-wise convolution, GN represents group normalization, and LeakyReLU represents the ReLU activation function with leakage; G' m ∈R n×32×32×32 Connect to the feature map Y and sample the feature map to the shape Y required by the decoder i , shape Y i With encoder output X i Fusion, get the final output Y' of the semantic fusion module i .

6. The MRI brain tumor image segmentation method based on multi-scale adaptive prototype aggregation according to claim 3, characterized in that: When training a brain tumor MRI segmentation network model: The network hyperparameters were set, the Adam optimizer was used to optimize the network model structure, the cosine annealing strategy was used to adjust the dynamic learning rate, the automatic mixed precision was used to reduce the amount of calculation, the optimal training weights were saved, and during the verification process, the prediction results were post-processed using test time enhancement and voxel clipping. The training set was input into the multi-scale adaptive prototype aggregation model for training, the Focal loss function and the Dice loss function were fused as the loss function of the network model, and the three lesion areas of the entire tumor, tumor core, and enhanced tumor were taken as segmentation targets.

Images