Brain tumor 3DMRI image segmentation method based on covariance prototype clustering

Abstract

The invention discloses a brain tumor 3DMRI image segmentation method based on covariance prototype clustering, and the method comprises the steps: obtaining a to-be-segmented brain tumor MRI image, and carrying out the preprocessing of the brain tumor MRI image; and inputting the preprocessed brain tumor MRI image into the constructed prototype aggregated brain tumor MRI segmentation network. According to the method, on the basis of a 3DU-Net architecture, a covariance prototype clustering module is introduced into a bottleneck layer of a constructed prototype aggregation brain tumor MRI segmentation network, so that effective aggregation of all types of prototypes is realized. The covariance prototype clustering module generates a more representative prototype vector for each category by capturing statistical correlation between channels, thereby enhancing feature representation. According to the method, the segmentation precision is improved by effectively capturing the dynamic relationship between the categories and the complex features in the categories.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a brain tumor 3D MRI image segmentation method based on covariance prototype clustering. Background Art

[0002] Medical image segmentation is a core technology in medical image analysis. It provides important information for disease diagnosis and treatment by identifying and separating specific structures in images. However, this process faces many challenges, including the complexity of images, the scarcity of labeled data, and the requirements for segmentation accuracy. This patent solves the challenges faced by traditional methods in segmentation tasks by introducing a covariance prototype clustering module. It improves the accuracy and generalization ability of segmentation.

[0003] Prototype aggregation is a pattern recognition method that works by classifying data points into a limited number of prototypes. It is used in brain tumor segmentation due to its advantages in accurately capturing tumor features and improving segmentation accuracy.

[0004] Covariance is a statistical method that can measure the linear relationship between two variables and is widely used in data analysis, image processing, and machine learning. The covariance matrix is ​​used to represent the correlation between features and help understand the data structure.

[0005] Existing brain tumor segmentation methods usually use softmax to calculate weights, or use query vectors as learnable class prototypes for classification. However, this strategy has limitations when dealing with complex image feature distributions and is difficult to fully capture the fine-grained features of the image, resulting in poor performance of the model in accurately segmenting tumor boundaries and extracting key areas. Summary of the invention

[0006] In view of the defects in the prior art, the present invention proposes a brain tumor 3D MRI image segmentation method based on covariance prototype clustering. By introducing a covariance-based prototype clustering module, the challenges faced by traditional methods in segmentation tasks are solved and the accuracy and generalization ability of segmentation are improved.

[0007] In order to achieve the above object, the present invention comprises the following steps:

[0008] Preprocess the brain tumor multimodal MRI image dataset and divide it into training set and test set;

[0009] Construct a brain tumor MRI segmentation network model based on prototype aggregation;

[0010] Set the hyperparameters of the brain tumor MRI segmentation network model, dynamically adjust the learning rate based on cosine annealing to train the model, save the optimal training weights, fuse the Focal loss function and the Dice loss function as the loss function of the network model, and use the three lesion areas of the entire tumor, tumor core, and enhanced tumor as segmentation targets;

[0011] The trained network model was recorded, and the brain tumor MRI segmentation network model was tested using a test set to obtain the segmentation results of the entire tumor, tumor core, and enhanced tumor.

[0012] Furthermore, the brain tumor multimodal MRI image dataset was divided into four modes: background and healthy parts, gangrenous parts, edema parts, and enhanced tumor parts, and useless background information in the images was trimmed; the Z-score method was used to normalize the images of each mode, create new channels and one-hot encode the foreground voxels to distinguish between background voxels and voxels with values ​​close to 0 after normalization, and a variety of data augmentation methods were used to enhance the training set, including random scaling, random flipping, Gaussian noise, Gaussian blur, and random brightness methods.

[0013] Furthermore, when constructing the brain tumor MRI segmentation network model, a covariance-based prototype clustering module is first established. The covariance-based prototype clustering module performs dimensionality reduction processing on the input 3D brain tumor MRI image to obtain a new tensor, and calculates the normalized matrix on the flattened new tensor:

[0014]

[0015] Of which: 1 M×M is an M×M matrix with all 1s, I M is an M×M identity matrix, and the covariance matrix of the feature matrix is ​​calculated using batch matrix multiplication:

[0016]

[0017] The covariance matrix is ​​used to capture the correlation between each feature and extract the covariance information between the input features;

[0018] The numerical stability of the covariance matrix is ​​determined by calculating the normalization factor, the matrix square root of the normalized covariance matrix is ​​calculated, the second-order statistical information of the covariance matrix is ​​extracted by Newton-Schulz iteration, and then the Newton-Schulz complement iteration is performed to update the matrix Y k and W k

[0019] W k =0.5(3I-W k-1 ·Y k-1 )

[0020] Y k =Y k-1 ·W k

[0021] After several iterations, the matrix square root P is finally obtained:

[0022]

[0023] Y interN-1 It is the matrix Y after several iterations. The new feature P is transformed back to the original spatial dimension, and the number of channels is expanded from C' back to C through convolution operation to obtain the final output feature X':

[0024] X'=Conv 1×1 (reshape(P))∈R B×C'×D×H×W

[0025] The argmax operation is used to select the category corresponding to the maximum probability or feature value of each voxel among all categories. Each category area is represented by the corresponding category label cp∈Cp, and the formula is:

[0026] cp i = argmax c X' i,c

[0027] where cp i Indicates the category label to which the i-th voxel or feature point belongs, X' i,c Represents the eigenvalue of the i-th voxel on the c-th category channel to construct the feature matrix according to the category label:

[0028] X cp ={X' i |cp(X') i =cp}

[0029] Feature matrix X cp For size N cp ×C matrix where N cp is the number of voxels belonging to category cp. According to the category represented by each voxel, the aggregate prototype of each category is calculated, which is specifically expressed as follows:

[0030]

[0031] where X' cp is a composite vector of size 1×C, P cp Category C p The probability distribution of , which indicates the probability that each pixel belongs to this category;

[0032] Assign the calculated aggregate prototypes of the categories to the new matrix:

[0033] X PA,[*,i,j,k] =X' cp ifargmax(P [*,i,j,k ])=c p

[0034] Where X PA,[*,i,j,k] is the feature vector at a specific position (i, j, k), P [*,i,j,k] is the probability distribution of all categories at position (i, j, k), aargmax(P [*,i,j,k] ) means returning the category c with the highest probability at that position p , X PA,[*,i,j,k] is the final output of the covariance prototype clustering module.

[0035] Furthermore, the brain tumor MRI segmentation network model includes a 4-layer original 3U-Net encoder structure, wherein each encoder layer has 2 layers of convolution blocks to extract feature information of brain tumor images, a maximum pooling layer is used to downsample the brain tumor images, and the covariance prototype clustering module is introduced in the bottleneck layer to enhance the network's ability to guide the aggregation of voxels of the same category based on the deep supervision strategy.

[0036] Furthermore, when testing the brain tumor MRI segmentation network model: through the test time augmentation method, the input brain tumor image is flipped along the x-axis, y-axis, z-axis, xy-axis, xz-axis, yz-axis, and xyz-axis and the segmentation results are predicted, and the predicted results are averaged.

[0037] Due to the adoption of the above technical solution, the present invention provides a brain tumor 3D MRI image segmentation method based on covariance prototype clustering. This method is based on the 3DU-Net architecture and effectively aggregates all category prototypes by introducing a covariance prototype clustering module in the bottleneck layer of the constructed prototype aggregation brain tumor MRI segmentation network. The covariance prototype clustering module captures the statistical correlation between channels and generates a more representative prototype vector for each category, thereby enhancing feature representation. The present invention improves segmentation accuracy by effectively capturing the dynamic relationship between categories and the complex features within the categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1Flow chart of the brain tumor 3DMRI image segmentation method based on covariance prototype clustering of the present invention

[0040] Figure 2 The framework diagram of the method of the present invention is

[0041] Figure 3 This is a schematic diagram of the covariance-based prototype clustering module framework constructed in the present invention. DETAILED DESCRIPTION

[0042] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0043] like Figure 1 A brain tumor 3D MRI image segmentation method based on covariance prototype clustering is shown, which specifically includes the following steps:

[0044] S1. Process brain tumor multimodal MRI images as input datasets for the network.

[0045] S2. Construct a brain tumor MRI segmentation network structure based on prototype aggregation.

[0046] S3. Train the network structure and save the network model.

[0047] S4. Load the network model, test the network model to obtain the brain tumor segmentation result, and perform post-processing to obtain the final segmentation result.

[0048] Among them, S1 is specifically:

[0049] The brain tumor image dataset includes MRI images of four modalities. The dataset is annotated into four types: background and healthy parts, gangrene parts, edema parts, and enhanced tumor parts. The useless background information in the image is trimmed.

[0050] The Z-score method is used to normalize the images of each modality to reduce the difference in contrast between different modalities. The formula is:

[0051]

[0052] Where z is the input MRI image, z' is the normalized MRI image, μ is the mean value of the input MRI image, and δ is the standard deviation of the input MRI image;

[0053] Create a new channel and one-hot encode the foreground voxels to distinguish between background voxels and voxels with values ​​close to 0 after normalization;

[0054] The input data set is divided into training set and test set according to a certain ratio, and the training set is enhanced using a variety of data enhancement methods, including random scaling, random flipping, Gaussian noise, Gaussian blur and random brightness methods. The brain tumor MRI images are preprocessed and input into the next segmentation network.

[0055] S2 is specifically:

[0056] S21: Construct a covariance-based prototype clustering module:

[0057] like Figure 3 As shown, the module first receives the shape X∈R B×C×D×H×W 3D brain tumor MRI images, where B is the batch size, C is the number of channels, and D, H, and W are depth, height, and width, respectively. In order to reduce computational complexity, a 1×1 convolution operation is used to reduce the number of channels from C to C′, so that only linear transformations can be performed on the channels without changing the spatial dimension. The formula for dimensionality reduction is:

[0058] Z=Conv 1×1 (X)∈R B×C×D×H×W

[0059] The spatial dimension D×H×W of the reduced data Z is flattened to a dimension M, where M=D×H×W. The shape of the new tensor Z1 is:

[0060] Z1=reshape(Z)∈R B×C'×M

[0061] The covariance matrix is ​​calculated on the flattened data Z1 to capture the statistical correlation between channels. Before calculating the covariance matrix, the normalization matrix is ​​constructed first:

[0062]

[0063] Of which: 1 M×M is an M×M matrix with all 1s, I M is an M×M identity matrix. Next, batch matrix multiplication is used to calculate the covariance of the feature matrix:

[0064]

[0065] The covariance matrix F captures the correlation between each feature and extracts the covariance information between the input features. In order to ensure the numerical stability of the covariance matrix, normalization is required and the normalization factor is calculated:

[0066]

[0067] Among them, F ij is the element value of the covariance matrix, Iij is the identity matrix I M The element value at position (i, j) in F is normalized using the normalization factor to obtain the matrix A:

[0068]

[0069] is a normalization factor used to standardize the covariance matrix, and ∈ is a small positive number used to prevent division by zero.

[0070] In order to extract the effective features of the covariance matrix, the square root of the normalized covariance matrix is ​​calculated. The square root operation extracts the second-order statistical information of the covariance matrix through Newton-Schultz iteration, so that the geometric characteristics of the covariance matrix can be more clearly manifested, which is convenient for subsequent feature learning. Initialization is performed first:

[0071] Y0=A·(0.5(3I-A))

[0072] W0=0.5(3I-A)

[0073] Where I is the identity matrix, size dim×dim, and then the Newton-Schur complement iteration is performed to update the matrix Y k and W k :

[0074] W k =0.5(3I-W k-1 ·Y k-1 )

[0075] Y k =Y k-1 ·W k

[0076] After several iterations, the matrix square root P is finally obtained:

[0077]

[0078] where Y interN-1 is the matrix Y after several iterations.

[0079] Finally, the new feature P is transformed back to the original spatial dimension, and the number of channels is expanded from C' back to C through a 1×1 convolution operation to obtain the final output X':

[0080] X'=Conv 1×1 (reshape(P))∈R B×C'×D×H×W

[0081] At this point, the output tensor combines the features processed by covariance to better capture the statistical information of the input data. After the output feature X' is processed by covariance, the argmax operation will select the category corresponding to the maximum probability or feature value of each voxel in all categories. Each category area is represented by the corresponding category label cp∈Cp, and the formula is:

[0082] cp i = argmax c X' i,c

[0083] where cp i Indicates the category label to which the i-th voxel or feature point belongs, X' i,c Represents the eigenvalue of the i-th voxel on the c-th category channel. Next, the feature matrix is ​​constructed according to the category label:

[0084] X cp ={X' i |cp(X') i =cp}

[0085] Feature matrix X cp For size N cp ×C matrix where N cp is the number of voxels belonging to category cp. According to the category represented by each voxel, the aggregate prototype of each category is calculated, which is specifically expressed as follows:

[0086]

[0087] where X' cp is a composite vector of size 1×C, i.e., the aggregate prototype of the category; P cp Category C p The probability distribution of each pixel belongs to this category. p Each of them has an aggregated representative vector, which, through multiple geometric transformations, can more accurately represent the characteristic structure of the category and enhance the representation of the prototype. Next, the calculated category prototypes are assigned to the new matrix:

[0088] X PA,[*,i,j,k] =X' cp if argmax(P [*,i,j,k] )=c p

[0089] Where X PA,[*,i,j,k] Refers to the feature vector at a specific location (i, j, k) (multi-channel representation, usually the feature at each pixel or voxel point). [*,i,j,k]is the probability distribution of all categories at position (i, j, k), with a size of 1×c p . aargmax(P [*,i,j,k] ) means returning the category c with the highest probability at that position p This process ensures that each category has an accurate representative prototype vector that is used to better represent the characteristics of the category. The final output X PA,[*,i,j,k] is the final output of the covariance prototype clustering module.

[0090] The covariance prototype clustering module enhances the expressiveness of features and generates more representative prototype vectors for each category. These prototypes not only contain the dynamic relationship between categories, but also reflect the complex feature associations within each category, thereby achieving more accurate category representation. This enhanced prototype representation greatly improves the segmentation performance of the module in complex scenes, while improving efficiency by simplifying the calculation steps.

[0091] S22 Construction of a brain tumor MRI segmentation network based on prototype aggregation

[0092] The basic structure of the model consists of a 4-layer original 3U-Net encoder structure. Each encoder has 2 layers of convolution blocks to extract brain tumor feature information and a maximum pooling layer for downsampling. A covariance prototype clustering module is introduced in the bottleneck layer, and a deep supervision strategy is adopted to enhance the network's ability to effectively guide the aggregation of voxels of the same category.

[0093] S3 is specifically:

[0094] S31: Set network hyperparameters, use Adam optimizer, use cosine annealing to dynamically adjust the learning rate and use automatic mixed precision for strategy training, and finally save the optimal training weights.

[0095] S32: The training set is input into the multi-scale background brain tumor model of adaptive prototype aggregation for training, the Focal loss function and the Dice loss function are fused as the loss function of the network model, and the three lesion areas of the entire tumor, tumor core, and enhanced tumor are taken as segmentation targets.

[0096] S4 is specifically:

[0097] S4: Load the network model, test the test set to obtain the brain tumor segmentation result, and perform post-processing to obtain the final segmentation result. The specific method is as follows:

[0098] S41: Load the training weights into the network and predict the segmentation results of all tumors, tumor cores, and enhanced tumors in the test set.

[0099] S42: Through the test time augmentation (TTA) method, the input brain tumor is flipped along the x-axis, y-axis, z-axis, xy-axis, xz-axis, yz-axis, and xyz-axis respectively, and the segmentation results are predicted, and the predicted results are averaged.

[0100] S43: Use the voxel clipping method to replace the prediction results with corresponding labels according to a certain probability and obtain the final segmentation results. In the prediction results, the voxels belonging to the entire tumor are replaced with label 0 with a certain probability; the voxels belonging to the tumor core are replaced with label 2 with a certain probability; the voxels belonging to the enhanced tumor are replaced with label 1 with a certain probability. At the same time, the independent enhanced tumor blocks and the overall enhanced tumor blocks whose number of voxels and average probability are less than a certain value are replaced with label 1.

[0101] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A brain tumor 3D MRI image segmentation method based on covariance prototype clustering, characterized in that include: Preprocess the brain tumor multimodal MRI image dataset and divide it into training set and test set; Construct a brain tumor MRI segmentation network model based on prototype aggregation; Set the hyperparameters of the brain tumor MRI segmentation network model, dynamically adjust the learning rate based on cosine annealing to train the model, save the optimal training weights, fuse the Focal loss function and the Dice loss function as the loss function of the network model, and use the three lesion areas of the entire tumor, tumor core, and enhanced tumor as segmentation targets; The trained network model was recorded, and the brain tumor MRI segmentation network model was tested using a test set to obtain the segmentation results of the entire tumor, tumor core, and enhanced tumor.

2. The method for brain tumor 3D MRI image segmentation based on covariance prototype clustering according to claim 1, characterized in that: The brain tumor multimodal MRI image dataset is divided into four modes: background and healthy parts, gangrenous parts, edema parts and enhanced tumor parts, and useless background information in the image is trimmed; the Z-score method is used to normalize the images of each mode, create new channels and one-hot encode the foreground voxels to distinguish between background voxels and voxels with values ​​close to 0 after normalization, and use a variety of data augmentation methods to enhance the training set, including random scaling, random flipping, Gaussian noise, Gaussian blur and random brightness methods.

3. The method for brain tumor 3D MRI image segmentation based on covariance prototype clustering according to claim 1, characterized in that: When constructing the brain tumor MRI segmentation network model, a covariance-based prototype clustering module is first established. The covariance-based prototype clustering module performs dimensionality reduction processing on the input 3D brain tumor MRI image to obtain a new tensor, and calculates the normalized matrix on the flattened new tensor: Of which: 1 M×M is an M×M matrix with all 1s, I M is an M×M identity matrix, and the covariance matrix of the feature matrix is ​​calculated using batch matrix multiplication: The covariance matrix is ​​used to capture the correlation between each feature and extract the covariance information between the input features; The numerical stability of the covariance matrix is ​​determined by calculating the normalization factor, the matrix square root of the normalized covariance matrix is ​​calculated, the second-order statistical information of the covariance matrix is ​​extracted by Newton-Schulz iteration, and then the Newton-Schulz complement iteration is performed to update the matrix Y k and W k <h2 style=";text-align:left;direction:ltr">W<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> =0.5(3I-W<h2 style=";text-align:left;direction:ltr"> k-1 <h2 style=";text-align:left;direction:ltr"> ·Y<h2 style=";text-align:left;direction:ltr"> k-1 <h2 style=";text-align:left;direction:ltr"> ) Y k =Y k-1 ·W k After several iterations, the matrix square root P is finally obtained: Y interN-1 It is the matrix Y after several iterations. The new feature P is transformed back to the original spatial dimension, and the number of channels is expanded from C' back to C through convolution operation to obtain the final output feature X': X'=Conv 1×1 (reshape(P))∈R B×C'×D×H×W The argmax operation is used to select the category corresponding to the maximum probability or feature value of each voxel among all categories. Each category area is represented by the corresponding category label cp∈Cp, and the formula is: cp i =argmax c X′ i,c where cp i Indicates the category label to which the i-th voxel or feature point belongs, X′ i,c Represents the eigenvalue of the i-th voxel on the c-th category channel to construct the feature matrix according to the category label: X cp ={X' i |cp(X') i =cp} Feature matrix X cp For size N cp ×C matrix where N cp is the number of voxels belonging to category cp. According to the category represented by each voxel, the aggregate prototype of each category is calculated, which is specifically expressed as follows: where X' cp is a composite vector of size 1×C, P cp Category C p The probability distribution of , which indicates the probability that each pixel belongs to this category; Assign the calculated aggregate prototypes of the categories to the new matrix: X PA,[*,i,j,k] =X' cp ifargmax(P [*,i,j,k] )=c p Where X PA,[*,i,j,k] is the feature vector at a specific position (i, j, k), P [*,i,j,k ] is the probability distribution of all categories at position (i, j, k), argmax(P [*,i,j,k] ) means returning the category c with the highest probability at that position p , X PA,[*,i,j,k] is the final output of the covariance prototype clustering module.

4. The method for brain tumor 3D MRI image segmentation based on covariance prototype clustering according to claim 3, characterized in that: The brain tumor MRI segmentation network model includes a 4-layer original 3U-Net encoder structure, wherein each encoder layer has 2 layers of convolution blocks to extract feature information of brain tumor images, a maximum pooling layer is used to downsample the brain tumor images, and the covariance prototype clustering module is introduced in the bottleneck layer to enhance the network's ability to guide the aggregation of voxels of the same category based on a deep supervision strategy.

5. The method for brain tumor 3D MRI image segmentation based on covariance prototype clustering according to claim 3, characterized in that: When testing the brain tumor MRI segmentation network model: through the test time augmentation method, the input brain tumor image is flipped along the x-axis, y-axis, z-axis, xy-axis, xz-axis, yz-axis, and xyz-axis and the segmentation results are predicted, and the predicted results are averaged.

Images