A multi-structure segmentation method for brain medical images

Abstract

A multi-structure segmentation method for brain medical images belongs to the field of image processing and computer vision. The self-attention mechanism effectively extracts low-frequency information from the boundaries of brain gray matter structures and establishes global correlations among features. Combining convolution with self-attention more effectively extracts both high- and low-frequency features of images, improving the loss of shallow-layer information during downsampling and achieving more accurate segmentation results. Furthermore, by performing dimensionality reduction and normalization during feature interaction, the convergence speed of the network model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and particularly relates to a multi-structure segmentation method for brain medical images. Background Art

[0002] Medical image segmentation plays a crucial role in medical image analysis and clinical diagnosis. Extracting features from regions of interest (ROIs) in medical images enables pixel-level classification and segmentation of diseased tissues and organs. This helps doctors understand relevant diseases, improves clinical workflow, and plays a crucial role in disease diagnosis and treatment planning. Medical images come in a variety of formats, and different modalities convey information with varying emphasis. Due to differences in imaging equipment and individual characteristics, imaging results can vary. Brain-related diseases are often diagnosed based on the integration of multiple tissues and structures. However, the overall shape of brain images, as well as the shape and size of different brain structures and lesion areas, vary significantly between individuals, and the dividing lines between brain structures can be fuzzy. Therefore, achieving accurate multi-structure segmentation in brain medical images remains challenging.

[0003] With the development of deep learning, deep neural networks have been widely used in the field of computer vision. In particular, fully convolutional neural networks have replaced traditional segmentation methods based on thresholding and clustering in medical image segmentation. Olaf Ronneberger et al. proposed the U-net segmentation framework, which uses an encoder to capture contextual information and a decoder to restore image positioning. It also uses skip connections to effectively utilize global and local information. This method has been widely used in the field of medical image segmentation. In subsequent research, many scholars have improved the network and derived many U-Net variants. Xiang Li et al. proposed a new U-shaped network structure that uses a multi-scale guided attention module to selectively aggregate discriminative features, improving the network's ability to learn and represent discriminative features. Essam A. Rashed et al. improved the network structure and proposed a single-encoder, multi-decoder convolutional neural network for multi-structure brain segmentation. Different decoders are designed based on the texture variations of different anatomical structures to adapt to multi-structure segmentation.

[0004] Currently, most brain medical image segmentation datasets are based on magnetic resonance imaging (MRI). This imaging technique provides excellent visualization of brain tissue structure. However, the pixel values at the boundaries of some gray matter structures are very close, making it difficult to accurately segment the boundaries of brain structures in regions of interest. Furthermore, deep neural networks use numerous convolution and pooling operations, which, as the network deepens, lose much of the shallow spatial information. This results in poor edge segmentation for smaller brain structures with complex edges. Therefore, improving model generalization and accurate boundary segmentation have become hot issues in medical image segmentation. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a method for multi-structure segmentation of brain medical images, comprising the following steps:

[0006] S1. Slice the public brain medical image dataset and select the 2D image of the brain structure to be segmented;

[0007] S2. Preprocess the image to be segmented to obtain training data and randomly divide it into training set and test set;

[0008] S3. Construct a brain multi-structure segmentation network, which includes an encoding part and a decoding part. The encoding part is composed of convolution and multi-head self-attention mechanism, and the decoding part is composed of bilinear interpolation upsampling and convolution;

[0009] S4. Use the divided training data to train the brain multi-structure segmentation model, and randomly divide the training data into a training set and a validation set;

[0010] S5. Input the training set into the network model to obtain the predicted segmentation result. Use the cross entropy loss function to calculate the loss value between the predicted result of the training process and the true label, and adjust the parameters.

[0011] S6. Input the validation set into the network model to obtain the predicted segmentation result, and use the cross entropy loss function to calculate the loss value between the predicted result of the training process and the true label;

[0012] S7. Determine the loss of the current verification process and the loss of the previous round of verification. If the current loss is less than the loss of the previous round, save the model parameters. If the current loss is greater than the loss of the previous round, continue training.

[0013] S8, determine whether the current number of iterations reaches the preset value, if not, return to step S5 for the next training, if it reaches the preset value, the training of the network model is completed;

[0014] S9. After the trained model is obtained in step S8, the test set is input into the trained brain multi-structure model to obtain the corresponding segmentation results and calculate the segmentation index.

[0015] Furthermore, the step S1 includes the following steps: filtering the labels of the original data set to retain the labels of the brain structure of interest, and then performing format conversion on the image data to convert the 3D image into a 2D image.

[0016] Furthermore, the preprocessing described in step S2 is: adjusting the size of the image data to a uniform size, normalizing and binarizing the unified image, and then expanding the data set by translating, rotating, flipping, and noise perturbation the data.

[0017] Furthermore, the encoding part described in step S3 includes high-frequency feature acquisition and low-frequency feature acquisition. The high-frequency features are acquired by a convolutional neural network composed of convolution and pooling, and the low-frequency feature acquisition is composed of multi-head self-attention and feedforward neural networks. There is a feature interaction process between high-frequency features and low-frequency features. The decoding part includes upsampling deep features and dimensional splicing with shallow features.

[0018] Furthermore, high-frequency feature acquisition consists of three consecutive convolutional layers. A ReLU activation function is added after each convolutional layer to increase the nonlinearity of the features. The extracted features are subjected to BN processing to improve the convergence speed of the network, and the features are subjected to maximum pooling operation to extract deep semantic features.

[0019] Furthermore, the low-frequency feature acquisition is to perform a patch embedding operation on the original image, divide the original image into 16 blocks, and perform dimensionality transformation, convert the two-dimensional matrix into a one-dimensional vector, calculate the correlation of global features through Multi Head Self-Attention, perform nonlinear transformation through a feedforward neural network, and map the results of the multi-head attention layer to a higher-dimensional space for feature extraction.

[0020] Furthermore, the feature interaction process is a process of complementing high-frequency and low-frequency features. The global features are obtained through multi-head self-attention and then dimensionalized after mapping through the fully connected layer. Then, they are fused with high-frequency features to enhance the global correlation of the features. Deeper feature extraction is performed through convolution operations. At the same time, the deep semantic features are reduced in dimension and dimensionalized to provide keys and values for multi-head self-attention, retaining the global correlation of the current features and compensating for the information loss caused by the pooled features.

[0021] The beneficial effects of the present invention are:

[0022] The present invention provides a brain multi-structure segmentation method based on the complementary features of convolution and multi-head self-attention mechanism. Most brain structures are composed of gray matter and white matter. In brain medical images, the structural boundary between gray matter and white matter is relatively clear. However, the pixel values of the gray matter structure below the ventricle are very close and difficult to distinguish. Since convolution is very good at extracting local features, it can well segment the adjacent brain structures of gray matter and white matter. Low-frequency signals with similar pixel values can be effectively extracted by using the global correlation modeling of the multi-head self-attention mechanism. In the decoding part, the present invention fuses the features extracted by the two methods to achieve complementary advantages. This improves the accuracy of multi-structure segmentation of brain medical images.

[0023] The present invention provides a multi-structure segmentation method for brain medical images based on the interaction of convolution and self-attention features. The self-attention mechanism can effectively extract low-frequency information at the boundaries of brain gray matter structures and establish global feature correlation. Combining convolution and self-attention more effectively extracts high- and low-frequency features of images, improving the problem of shallow information being lost during the downsampling process and achieving more accurate segmentation results. Furthermore, by performing feature dimensionality reduction and normalization during the feature interaction process, the convergence speed of the network model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the overall flow chart of the multi-structure of the brain of the present invention;

[0025] Figure 2 This is a flowchart of the preprocessing of the brain medical image dataset of the present invention;

[0026] Figure 3 This is a schematic diagram of the brain multi-structure segmentation network structure based on convolution and multi-head self-attention mechanism of the present invention;

[0027] Figure 4 This is a schematic diagram of the high-frequency feature extraction network structure in the encoding stage of the present invention;

[0028] Figure 5 Schematic diagram of the interaction between high- and low-frequency features. DETAILED DESCRIPTION

[0029] A multi-structure segmentation method for brain medical images comprises the following steps:

[0030] S1. Slice the public brain medical image dataset and select the 2D image of the brain structure to be segmented;

[0031] S2. Preprocess the image to be segmented to obtain training data and randomly divide it into training set and test set;

[0032] S3. Construct a brain multi-structure segmentation network, which includes an encoding part and a decoding part. The encoding part is composed of convolution and multi-head self-attention mechanism, and the decoding part is composed of bilinear interpolation upsampling and convolution;

[0033] S4. Use the divided training data to train the brain multi-structure segmentation model, and randomly divide the training data into a training set and a validation set;

[0034] S5. Input the training set into the network model to obtain the predicted segmentation result. Use the cross entropy loss function to calculate the loss value between the predicted result of the training process and the true label, and adjust the parameters.

[0035] S6. Input the validation set into the network model to obtain the predicted segmentation result, and use the cross entropy loss function to calculate the loss value between the predicted result of the training process and the true label;

[0036] S7. Determine the loss of the current verification process and the loss of the previous round of verification. If the current loss is less than the loss of the previous round, save the model parameters. If the current loss is greater than the loss of the previous round, continue training.

[0037] S8, determine whether the current number of iterations reaches the preset value, if not, return to step S5 for the next training, if it reaches the preset value, the training of the network model is completed;

[0038] S9. After the trained model is obtained in step S8, the test set is input into the trained brain multi-structure model to obtain the corresponding segmentation results and calculate the segmentation index.

[0039] Wherein, the step S1 includes the following steps: filtering the labels of the original data set to retain the labels of the brain structure of interest, and then performing format conversion on the image data to convert the 3D image into a 2D image.

[0040] The preprocessing described in step S2 is as follows: adjusting the image data size to a uniform size, normalizing and binarizing the unified image, and then performing transformations such as translation, rotation, flipping, and noise perturbation on the data to expand the data set.

[0041] Among them, the encoding part described in step S3 includes high-frequency feature acquisition and low-frequency feature acquisition. The high-frequency features are acquired by a convolutional neural network composed of convolution and pooling. The low-frequency feature acquisition is composed of multi-head self-attention and feedforward neural networks. There is a feature interaction process between high-frequency features and low-frequency features. The decoding part includes upsampling deep features and dimensional splicing with shallow features.

[0042] Among them, the high-frequency feature acquisition is composed of three consecutive convolutional layers, and a ReLU activation function is added after each convolutional layer to increase the nonlinearity of the feature. The extracted features are subjected to BN processing to improve the convergence speed of the network, and the features are subjected to maximum pooling operation to extract deep semantic features.

[0043] The low-frequency feature acquisition is to perform a patch embedding operation on the original image, divide the original image into 16 blocks, and perform dimensionality transformation, converting the two-dimensional matrix into a one-dimensional vector. The correlation of global features is calculated through Multi Head Self-Attention, and a nonlinear transformation is performed through a feedforward neural network. The results of the multi-head attention layer are mapped to a higher-dimensional space for feature extraction.

[0044] Among them, the feature interaction process is a process of complementing high-frequency and low-frequency features. The global features are obtained through multi-head self-attention, mapped through the fully connected layer, and then dimensionalized. They are then fused with high-frequency features to enhance the global correlation of the features. Deeper feature extraction is performed through convolution operations. At the same time, the deep semantic features are reduced in dimension and transformed into keys and values for multi-head self-attention, retaining the global correlation of the current features to compensate for the information loss caused by the pooled features.

[0045] See Figure 1 , the overall flow chart of multi-structure segmentation of brain medical images. First, the dataset of brain medical images is preprocessed. Due to the relationship between the acquisition equipment and individual differences, the size and position of the dataset vary greatly. Therefore, the shape and center position of the data are adjusted without changing the original image. This step also includes the conversion of the image data format. The processed dataset is randomly divided into a training set and a test set. Then a part of the training set is randomly divided as a validation set to verify the training parameters of the model. In the network model, the input image is used as a high-pass filter to extract the local features and high-level semantic information of the image using a convolutional neural network. The input image is divided into blocks and its position information is retained. The low-frequency features and global correlation of the image are obtained through a multi-head self-attention mechanism, and the higher-dimensional and high-level features of the feature map are fused through a feedforward neural network to provide rich semantic features for the deep network. The high-level semantic features are then upsampled and feature extracted to obtain the predicted segmentation results. Finally, the loss between the predicted result and the true label is calculated by the binary cross entropy loss function, and the calculation formula is:

[0046]

[0047] Where N is the training batch size, y k represents the predicted segmentation image, g k represents the true label.

[0048] The validation set is input into the brain multi-structure segmentation model, and the loss value between the validation set prediction result and the true label is calculated through the loss function. The network model parameters with the minimum loss are saved in the preset number of iterations.

[0049] See Figure 2, is the process of processing brain medical image data sets. The data set used in the present invention is derived from the data set of the MICCAI competition. The data format is 3D data and the data label is a whole-brain label. First, the labels are screened by medical analysis software, the brain structure labels of interest are retained, and the remaining labels are deleted. The 3D image data is then sliced and converted into 2D image data. When the image data is converted, it is easy to cause pixel loss, so the 2D image is subjected to contrast enhancement. However, deep learning usually requires a large amount of data for continued training, because medical data acquisition is very expensive and confidential. In order to avoid overfitting of the network model, the data needs to be expanded. In the present invention, the number of samples in the training set is increased by translation, rotation, flipping and adding disturbance noise to ensure the diversity of the training data.

[0050] See Figure 3 , which is a block diagram of the brain multi-structure segmentation network. The network mainly consists of three parts: high-frequency feature encoding, low-frequency feature encoding and decoder. High-frequency feature extraction mainly extracts the obvious edges, shapes and other features of image features through convolution. First, the original image is mapped to high-dimensional features through the convolution layer to provide input for subsequent feature extraction. High-frequency feature encoding is divided into three stages. The details of each encoder can be found in Figure 4In each stage, after the first 3×3 convolution, the number of feature channels in the input feature map doubles, mapping the features to a higher dimension. The next 3×3 convolution then extracts high-level semantic information from the high-dimensional features. A nonlinear activation function (ReLU()) is performed after each convolution to prevent vanishing and exploding gradients during network training. Since a batch of image data is input during training, some data is distributed in the saturation phase of the activation function after each feature extraction. Therefore, Batch Normalization is used after each feature extraction and before the activation function to more effectively feed the feature values into the nonlinear activation function. This accelerates network training and convergence and prevents overfitting. The low-frequency feature extraction process primarily utilizes the self-attention method used in natural language processing to capture global feature correlations without compressing the image. This is particularly important in medical image segmentation, where gray matter boundaries are unclear. Self-attention can establish correlations between pixels in these regions and surrounding pixels. The main low-frequency feature extraction processes include Patch Embedding, Positional Encoding, Multi-Head Self-Attention, and Multi-Layered Layer (MLP). Since the self-attention mechanism mainly processes one-dimensional sequence types, it is necessary to convert the two-dimensional image into a one-dimensional vector. In order to reduce the complexity of the calculation, the image is divided into blocks. In the present invention, the original image is divided into 16 blocks, and then the image is linearly expanded. The length of each block of the image after expansion is the length of the sequence, and then each block is used as the dimension of the linear projection layer. At the same time, position coding needs to be added to the block coding of the image, and the dimension of the position coding is the same as the dimension of the input sequence embedding. Then initialize the weights and bias of the Query, Key and Value of the self-attention mechanism, and calculate the correlation of the features through the fully connected layer. The calculation formula of this process is,

[0051]

[0052] The softmax() function is used to calculate the correlation between input vectors. It is used to constrain the variance to be stable and stabilize the gradient value during the training process. Finally, the fully connected neural network maps the obtained image features to a higher dimensional operation to extract global features.

[0053] In the brain multi-structure segmentation network, the main method is to fuse high-frequency and low-frequency features to obtain richer semantic features and restore the segmentation prediction results through upsampling. Figure 5, which is the entire process of interaction between high- and low-level features. In the encoding process, the convolution-extracted features contain a large amount of high-frequency information and local features. In order to alleviate the ability to extract low-frequency features, the present invention transfers feature maps between self-attention and convolution. First, the original image is feature extracted, and then the features are reduced in dimension using 1×1 convolution, and these features are applied to the feature dimension of self-attention through Flatten, and the features are added with the key and value of self-attention. However, in order to reduce the computational cost, Layer Normalization is used to normalize the samples. Then the dimension is enlarged and then reduced through the MLP layer, and the one-dimensional vector is converted into a two-dimensional feature through the resape function and added to the feature map extracted by convolution. Even if the feature map has global correlation. In the entire encoding process, the three downsampling operations result in high-level semantic information. At the bottom layer of the network, convolution is used for feature filtering to obtain more effective features for multi-structure segmentation of the brain.

[0054] During the decoding phase of the network, three upsampling operations are performed to recover the segmentation prediction results. During each upsampling, the encoded features and the upsampled decoded features are concatenated, and then convolution is performed to extract the segmented image and filter out other redundant features.

[0055] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed in the present invention, who makes equivalent replacements or changes based on the technical solutions and concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-structure segmentation method for brain medical images, characterized in that: The following steps are involved: S1. Slice the public brain medical image dataset and select the 2D image of the brain structure to be segmented; S2. Preprocess the image to be segmented to obtain training data and randomly divide it into training set and test set; S3. Construct a brain multi-structure segmentation network model, which includes an encoding part and a decoding part. The encoding part is composed of convolution and multi-head self-attention mechanism, and the decoding part is composed of bilinear interpolation upsampling and convolution; The encoding part described in step S3 includes high-frequency feature acquisition and low-frequency feature acquisition. The high-frequency features are acquired by a convolutional neural network composed of convolution and pooling. The low-frequency feature acquisition is composed of multi-head self-attention and feedforward neural networks. There is a feature interaction process between high-frequency features and low-frequency features. The decoding part includes upsampling deep features and dimensional splicing shallow features. The high-frequency feature acquisition consists of three consecutive convolutional layers. A ReLU activation function is added after each convolutional layer to increase the nonlinearity of the feature. The extracted features are subjected to BN processing to improve the convergence speed of the network. The features are subjected to maximum pooling operation to extract deep semantic features. The low-frequency feature acquisition involves performing a patch embedding operation on the original image, dividing the original image into 16 blocks, and performing a dimensionality transformation to convert the two-dimensional matrix into a one-dimensional vector. The correlation of global features is calculated through Multi-Head Self-Attention. A nonlinear transformation is performed through a feedforward neural network, and the results of the multi-head attention layer are mapped to a higher-dimensional space for feature extraction. The feature interaction process is a process in which high-frequency and low-frequency features complement each other. Global features are obtained through multi-head self-attention and then dimensionalized after mapping through the fully connected layer. They are then fused with high-frequency features to enhance the global relevance of the features. Deeper feature extraction is performed through convolution operations. At the same time, deep semantic features are reduced in dimension and dimensionalized to provide keys and values for multi-head self-attention, preserving the global relevance of the current features and compensating for the information loss caused by pooling features. S4. Use the divided training data to train the brain multi-structure segmentation network model, and randomly divide the training data into a training set and a validation set; S5. Input the training set into the brain multi-structure segmentation network model to obtain the predicted segmentation results. Use the cross entropy loss function to calculate the loss value between the predicted results and the true labels during the training process, and adjust the parameters. S6. Input the validation set into the brain multi-structure segmentation network model to obtain the predicted segmentation results, and use the cross entropy loss function to calculate the loss value between the predicted results of the training process and the true labels; S7. Determine the loss of the current verification process and the loss of the previous round of verification. If the current loss is less than the loss of the previous round, save the model parameters. If the current loss is greater than the loss of the previous round, continue training. S8, determining whether the current number of iterations has reached a preset value, if not, returning to step S5 for the next training, if reaching the preset value, completing the training of the brain multi-structure segmentation network model; S9. After the trained model is obtained in step S8, the test set is input into the trained brain multi-structure segmentation network model to obtain the corresponding segmentation results and calculate the segmentation index.

2. The multi-structure segmentation method for brain medical images according to claim 1, wherein: The step S1 includes the following steps: screening the labels of the original data set, retaining the labels of the brain structures of interest, and then performing format conversion on the image data to convert the 3D image into a 2D image.

3. The multi-structure segmentation method for brain medical images according to claim 1, wherein: The preprocessing described in step S2 is: adjusting the size of the image data to a uniform size, normalizing and binarizing the unified image, and then expanding the data set by translating, rotating, flipping, and noise perturbation the data.

Images