A medical image segmentation method based on multi-scale and attention

By adopting a multi-scale and attention U-Net model in medical image segmentation, combining channel and mixed hollow attention module, the problem of excessive computing resources and parameters in the existing technology is solved, and efficient medical image segmentation and Feature extraction improves segmentation accuracy and model stability.

CN114359292BActive Publication Date: 2024-11-26NANCHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111504413.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-11-26
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

When the existing medical image segmentation method deals with goals with large differences in shape and size, there are too many computing resources and model parameters, and the context information is not fully utilized, making it difficult to effectively extract useful features of the channel.

Method used

Using a medical image segmentation method based on multi-scale and attention, the U-Net model containing channel attention module and mixed hollow attention module is constructed, combining short jump connections and batch normalization layers to reduce the amount of parameters and improve the model stability. , realize the extraction and feature representation of multi-scale context information.

Benefits of technology

It improves the accuracy of medical image segmentation and the stability of the model, reduces the consumption of computing resources, enhances feature representation ability, and can effectively perform lesion segmentation with fewer training samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114359292B_ABST
    Figure CN114359292B_ABST
Patent Text Reader

Abstract

The present invention discloses a medical image segmentation method based on multi-scale and attention, comprising the following steps: firstly, the acquired medical image is processed and the data set is divided, then a two-dimensional network model is constructed, and the pre-processed data is sent to the network for training; the model is adjusted to the optimal effect by optimizing the loss function; the medical image to be segmented is sent to the network model with adjusted parameters to obtain the predicted segmentation result, and the model performance is evaluated by comparing it with the real label. In view of the problem of lack of context information in the classic U-shaped network, the present invention adds a plug-and-play, very lightweight channel attention mechanism and a hybrid void attention convolution layer to the network to make full use of context information and explore useful channel features. The segmentation model of the present invention can complete the segmentation of medical images with very few parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to a medical image segmentation method based on multi-scale and attention. Background Art

[0002] Medical images play a vital role in medical treatment and diagnosis. Medical image segmentation is a key issue that determines whether medical images can provide reliable evidence in clinical diagnosis and treatment. It has extremely important research and application value in research fields such as medical research, disease diagnosis, disease analysis, and assisted surgery.

[0003] In recent years, deep learning methods, like other research areas of computer vision, have achieved excellent results in medical image segmentation and are far ahead of traditional image segmentation methods. In 2014, Fully Convolutional Networks (FCN) were proposed by Long et al. from the University of California, Berkeley. They replaced all the fully connected layers of the convolutional neural network with convolutional layers. This method can not only input data of any size, but also achieve pixel-level image segmentation, which further improves the accuracy of segmentation. Based on FCN, researchers have proposed many excellent fully convolutional neural networks, which have achieved very good results in actual image segmentation.

[0004] The most noteworthy is the U-net proposed by Ronneberger et al. This is a very meaningful innovation. The network has been successfully applied to many image segmentation tasks. It is similar to an autoencoder, consisting of a contraction branch and an expansion branch, and supports multi-resolution analysis. In addition, the network also adds jump connections to connect different features of the encoder and decoder to model the multi-scale image-to-image conversion of different resolutions. It can work with few training samples, fully considering the scarcity of medical image segmentation data and the imbalance of categories. So far, many improvements based on U-Net have been proposed. In 2018, Li et al. proposed a hybrid U-net network to segment liver tumor images. This method combines 2D and 3D U-net to extract corresponding features, which can realize automatic liver tumor image segmentation. Despite their good representation capabilities, these architectures still rely on multi-level cascaded CNNs when the segmentation targets show large differences in shape and size. The cascade framework extracts the region of interest (ROI) and makes dense predictions for the specific ROI. Application areas include abdominal pancreatic segmentation, abdominal segmentation such as liver, spleen, stomach, and lung CT nodule detection. However, this method leads to excessive and redundant use of computing resources and model parameters. The attention mechanism can better solve the above problems. The attention mechanism is widely used in text processing, speech recognition, image processing and other fields. Its biggest feature is that it can dynamically allocate the input weights of neurons to achieve selective attention to the most critical part of the information, that is, using a weight generation network to generate the weight vector of the input vector according to the current context information, and then multiplying the input vector by the corresponding weight, and then connecting the current context information as a new input, and finally obtaining the current translation output through the decoder. Oktay et al. proposed a gate structure and applied the attention mechanism to medical image segmentation. The network first extracts feature maps in the jump connection and sends them to the attention gate and then connects them. Its main disadvantage is that the processing steps are performed on two sets of feature maps separately. This operation does not make full use of context information and explore useful channel features. Summary of the invention

[0005] To solve the above problems, the present invention proposes a medical image segmentation method based on multi-scale and attention. The technical solution is as follows:

[0006] Step 1: Obtain medical images, preprocess them, and divide the data set;

[0007] If the acquired medical image is a three-dimensional image, it is resampled to 1mm in the axial direction and cut into two-dimensional slices; if it is a two-dimensional image, this step is not processed; if the acquired medical image is a grayscale image, the histogram equalization algorithm is implemented to enhance the image contrast, making the image clearer and conducive to highlighting the image information; if it is a color image, this step is not processed. After these two steps, the image is normalized so that the mean value of the image pixels sent to the network is 0 and the variance is 1, and online data enhancement is used to increase data diversity, including random rotation and random flipping. In addition, the data set is divided into training set, validation set and test set in a ratio of 8:1:1.

[0008] Step 2: Build a two-dimensional network model and send the preprocessed data into the network for training;

[0009] The two-dimensional network model consists of 1 encoding branch and 1 decoding branch, which are connected through a channel attention module. The encoding branch contains 4 down-sampling residual modules and 1 mixed hole attention module, and the decoding branch contains 4 up-sampling residual modules.

[0010] The downsampling residual module is a small structural block composed of two 3×3 convolutional layers (with the same padding). Residual learning is used for each convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU nonlinear layer. Each small structural block is followed by a maximum pooling layer with a pooling kernel size of 2×2 and a sampling step of 2 for downsampling. In each downsampling process, the number of layers of the feature layer is doubled, and the initial number of channels is 32. Then the hybrid void attention module is connected to summarize the global information and generate the output of the encoding branch.

[0011] The hybrid hole attention module includes three bottleneck hole modules with different hole rates and one channel attention module. The structure of the bottleneck hole module is three convolutions with residual connections, namely one 1×1 convolution, one 3×3 hole convolution, and one 1×1 convolution. The hole rates of the hole convolutions included in the three bottleneck hole modules are 2, 3, and 5.

[0012] By connecting multiple bottleneck hole modules with different expansion rates in series, we can achieve the fusion of receptive field information of different sizes, fully extract multi-scale context information, and effectively reduce the number of parameters. We add residual connections to each bottleneck hole module, which is conducive to network optimization. At the same time, we use the channel attention module mentioned above to perform nonlinear fusion of information from each channel, characterize the relationship between context information of different channels through nonlinear functions, and then assign weights to multi-scale context information, thereby increasing the multi-scale context information learning ability of the hole convolution.

[0013] The channel attention module has three branches K, Q, V. First, the input feature map I∈R C*H*W Reshape I into feature maps K∈R C*(H*W) , feature map Q∈R (H*W)*C , forming branches K and Q at the same time, and then performing matrix multiplication between K and Q, divided by the factor Finally, the Softmax layer is applied to obtain the channel feature map A∈R C*C The branch V forms a feature map V∈R through the feature map I through global average pooling. C*1*1 , after V is reshaped, we get V'∈R C*1*1 , let A and V' perform matrix multiplication and convert the result into R C*1*1 After that, it is multiplied by the scale parameter γ and element-wise summed with I to obtain the final output O∈R C*H*W .

[0014] Each channel corresponds to a specific semantic response. We hope to improve the representation ability of the network by modeling the dependencies between channels, and adjust the features channel by channel so that the network can learn to selectively enhance features containing useful information and suppress useless features through global information. The channel attention module models semantic dependencies to emphasize the target acquisition channel. By exploiting the interdependence between channel maps, we emphasize the size of interdependent feature maps and improve the feature representation of specific semantics.

[0015] The upsampling residual module includes a 2×2 deconvolution layer to implement the upsampling operation. Each upsampling operation will halve the number of feature layers. The deconvolution layer is connected to two 3×3 convolution layers using residual learning. Each convolution layer is followed by a batch normalization layer and a ReLU nonlinear layer.

[0016] In this network structure, we use a large number of batch normalization layers, which increase the stability of the neural network and speed up the neural network training process. The neural network standardizes the input of a certain layer in the network by subtracting the batch mean and dividing by the batch standard deviation. In addition, in some cases, the performance of the model is improved due to the moderate regularization effect.

[0017] Moreover, our network does not use fully connected layers, which ensures that the segmentation results are all predictions based on contextual information, and also reduces the parameters of the network model. In the encoding process, higher-level features are obtained by continuously using a combination of convolutional layers and pooling. With each downsampling process, the receptive field of the convolution kernel increases exponentially. In the decoding process, the underlying feature maps are stacked on the features output by the upsampling layer, and then the high-level and low-level features are learned through continuous convolutional layers. Through such continuous upsampling, the final output image size is the same as the input image size. This feature extraction method combines shallow position information and deep semantic information very well.

[0018] In addition, we added short skip connections, namely residual structures. The residual structure makes the network easier to optimize, speeds up model convergence, and can improve accuracy by increasing depth. The proper combination of long and short connections allows the network to better extract features at different levels, enhance its expressive power, supplement semantic information at high levels, and refine segmentation contours at the bottom level.

[0019] Step 3: Adjust the model to the optimal effect by optimizing the loss function;

[0020] The loss function is the weighted cross entropy loss, and the formula is as follows:

[0021]

[0022] in represents the probability that voxel n belongs to class c (background or lesion), represents the weight of voxel n, represents the true value label of voxel n.

[0023] Step 4: Send the corresponding medical image used to evaluate the model performance into the network model with adjusted parameters to obtain the predicted segmentation results, and compare them with the true labels to evaluate the model performance.

[0024] The evaluation indicators include Dice coefficient (Dice), accuracy (AC), precision (Pre), sensitivity (SE), and specificity (SP). The specific formula is as follows:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] Among them, FP, FN, TP, and TN are false positive, false negative, true positive, and true negative, respectively.

[0031] The beneficial effects of the present invention are:

[0032] The present invention proposes a residual convolutional neural network based on the U-Net model, and adds a plug-and-play, very lightweight channel attention mechanism, which can be better applied to medical image segmentation. Unlike the simple copy and splicing during U-Net jump connection, we insert the channel attention module into the extracted feature map of the encoding path before decoding. Because this block is lightweight, we can apply it to multiple layers in the backbone network to optimize the channel effect of the encoding operation of this layer. In addition, the convolution layer at the bottom of the "U"-shaped network is replaced with a hybrid void attention module, which improves the stability of the model and reduces the number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart for implementing the method of the present invention;

[0034] Figure 2 It is a schematic diagram of a two-dimensional network structure of the present invention;

[0035] Figure 3 is a schematic diagram of a channel attention module of the present invention;

[0036] Figure 4 Schematic diagram of the hybrid hole attention module of the present invention; DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings.

[0038] A multi-scale and attention-based medical image segmentation method, such as Figure 1 As shown, the following steps are included:

[0039] Step 1: Obtain medical images, preprocess them, and divide the data set;

[0040] If the acquired medical image is a three-dimensional image, it is resampled to 1mm in the axial direction and cut into two-dimensional slices; if it is a two-dimensional image, this step is not processed; if the acquired medical image is a grayscale image, the histogram equalization algorithm is implemented to enhance the image contrast, making the image clearer and conducive to highlighting the image information; if it is a color image, this step is not processed. After these two steps, the image is normalized so that the mean value of the image pixels sent to the network is 0 and the variance is 1, and online data enhancement is used to increase data diversity, including random rotation and random flipping. In addition, the data set is divided into training set, validation set and test set in a ratio of 8:1:1.

[0041] Step 2: Build a two-dimensional network model and send the preprocessed data into the network for training;

[0042] like Figure 2 As shown in Figure 1, the two-dimensional network model includes 1 encoding branch and 1 decoding branch, which are connected through a channel attention module. The encoding branch includes 4 downsampling residual modules and 1 mixed void attention module, and the decoding branch includes 4 upsampling residual modules.

[0043] The downsampling residual module is a small structural block composed of two 3×3 convolutional layers (with the same padding). Residual learning is used for each convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU nonlinear layer. Each small structural block is followed by a maximum pooling layer with a pooling kernel size of 2×2 and a sampling step of 2 for downsampling. In each downsampling process, the number of layers of the feature layer is doubled, and the initial number of channels is 32. Then the hybrid void attention module is connected to summarize the global information and generate the output of the encoding branch.

[0044] like Figure 4 As shown, the hybrid hole attention module includes three bottleneck hole modules with different hole rates and one channel attention module. The structure of the bottleneck hole module is three convolutions with residual connections, namely, one 1×1 convolution, one 3×3 hole convolution, and one 1×1 convolution. The hole rates of the hole convolutions included in the three bottleneck hole modules are 2, 3, and 5.

[0045] By connecting multiple bottleneck hole modules with different expansion rates in series, we can achieve the fusion of receptive field information of different sizes, fully extract multi-scale context information, and effectively reduce the number of parameters. We add residual connections to each bottleneck hole module, which is conducive to network optimization. At the same time, we use the channel attention module mentioned above to perform nonlinear fusion of information from each channel, characterize the relationship between context information of different channels through nonlinear functions, and then assign weights to multi-scale context information, thereby increasing the multi-scale context information learning ability of the hole convolution.

[0046] like Figure 3 As shown, the channel attention module has three branches K, Q, V. First, the input feature map I∈R C*H*W Reshape I into feature maps K∈R C*(H*W) , feature map Q∈R (H*W)*C , forming branches K and Q at the same time, and then performing matrix multiplication between K and Q, divided by the factor Finally, the Softmax layer is applied to obtain the channel feature map A∈R C*C .

[0047]

[0048] Among them, a ji It is used to measure the influence of the i-th channel on the j-th channel, and the function f is used to calculate the relationship between i and all j.

[0049] Branch V forms feature map V∈R through global average pooling of feature map I C*1*1 , after V is reshaped, we get V'∈R C*1*1 , let A and V' perform matrix multiplication and convert the result into R C*1*1 After that, it is multiplied by the scale parameter γ and element-wise summed with I to obtain the final output O∈R C*H*W .

[0050]

[0051] g represents global average pooling, where k = 1, 2, ..., c, I = [i 1 ,i 2 ,…,i c ].

[0052]

[0053] γ gradually learns the weights starting from 0. ω θ represents the weight of 1×1 convolution, b θ Represents deviation.

[0054] Each channel corresponds to a specific semantic response. We hope to improve the representation ability of the network by modeling the dependencies between channels, and adjust the features channel by channel so that the network can learn to selectively enhance features containing useful information and suppress useless features through global information. The channel attention module models semantic dependencies to emphasize the target acquisition channel. By exploiting the interdependence between channel maps, we emphasize the size of interdependent feature maps and improve the feature representation of specific semantics.

[0055] The upsampling residual module includes a 2×2 deconvolution layer to implement the upsampling operation. Each upsampling operation will halve the number of feature layers. The deconvolution layer is connected to two 3×3 convolution layers using residual learning. Each convolution layer is followed by a batch normalization layer and a ReLU nonlinear layer.

[0056] In this network structure, we use a large number of batch normalization layers, which increase the stability of the neural network and speed up the neural network training process. The neural network standardizes the input of a certain layer in the network by subtracting the batch mean and dividing by the batch standard deviation. In addition, in some cases, the performance of the model is improved due to the moderate regularization effect.

[0057] Moreover, our network does not use fully connected layers, which ensures that the segmentation results are all predictions based on contextual information, and also reduces the parameters of the network model. In the encoding process, higher-level features are obtained by continuously using a combination of convolutional layers and pooling. With each downsampling process, the receptive field of the convolution kernel increases exponentially. In the decoding process, the underlying feature maps are stacked on the features output by the upsampling layer, and then the high-level and low-level features are learned through continuous convolutional layers. Through such continuous upsampling, the final output image size is the same as the input image size. This feature extraction method combines shallow position information and deep semantic information very well.

[0058] In addition, we added short skip connections, namely residual structures. The residual structure makes the network easier to optimize, speeds up model convergence, and can improve accuracy by increasing depth. The proper combination of long and short connections allows the network to better extract features at different levels, enhance its expressive power, supplement semantic information at high levels, and refine segmentation contours at the bottom level.

[0059] Step 3: Adjust the model to the optimal effect by optimizing the loss function;

[0060] The loss function is the weighted cross entropy loss, and the formula is as follows:

[0061]

[0062] in represents the probability that voxel n belongs to class c (background or lesion), represents the weight of voxel n, represents the true value label of voxel n.

[0063] Step 4: Send the corresponding medical image used to evaluate the model performance into the network model with adjusted parameters to obtain the predicted segmentation results, and compare them with the true labels to evaluate the model performance.

[0064] The evaluation indicators include Dice coefficient (Dice), accuracy (AC), precision (Pre), sensitivity (SE), and specificity (SP). The specific formula is as follows:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Among them, FP, FN, TP, and TN are false positive, false negative, true positive, and true negative, respectively.

[0071] The above description is only a preferred embodiment of the present invention. The technicians may make corresponding changes according to the actual situation. The present invention is not limited to the above embodiment.

Claims

1. A medical image segmentation method based on multi-scale and attention, It is characterized in that The following steps are involved: Step 1: Obtain medical images, preprocess them, and divide the data set; Step 2: Build a two-dimensional network model and send the preprocessed data into the network for training; Step 3: Adjust the model to the optimal effect by optimizing the loss function; Step 4: Send the corresponding medical image used to evaluate the model performance into the network model with adjusted parameters to obtain the predicted segmentation results and compare them with the true labels; In the step 2, the two-dimensional network model includes one encoding branch and one decoding branch, and the encoding branch and the decoding branch are connected through a channel attention module; The encoding branch includes 4 down-sampling residual modules and 1 mixed hole attention module, and the decoding branch includes 4 up-sampling residual modules; The downsampling residual module is a small structural block consisting of two 3×3 convolutional layers. Residual learning is used for each convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU nonlinear layer. Each small structural block is followed by a maximum pooling layer with a pooling kernel size of 2×2 and a sampling step of 2 for downsampling. In each downsampling process, the number of feature layers is doubled, and the initial number of channels is 32; then the hybrid hole attention module is connected to summarize the global information and generate the output of the encoding branch; The hybrid hole attention module includes three bottleneck hole modules with different hole rates and one channel attention module; The upsampling residual module includes a 2×2 deconvolution layer to implement the upsampling operation. Each upsampling operation will halve the number of feature layers. The deconvolution layer is connected to two 3×3 convolution layers using residual learning. Each convolution layer is followed by a batch normalization layer and a ReLU nonlinear layer.

2. The medical image segmentation method based on multi-scale and attention according to claim 1, It is characterized in that In the step 1, if the acquired medical image is a three-dimensional image, it is resampled to 1 mm in the axial direction and cut into two-dimensional slices; if it is a two-dimensional image, this step is not processed; if the acquired medical image is a grayscale image, a histogram equalization algorithm is implemented to enhance the image contrast; if it is a color image, this step is not processed; after these two steps, the image is normalized so that the mean value of the pixels of the image sent to the network is 0 and the variance is 1, and online data enhancement is used to increase data diversity, including random rotation and random flipping; in addition, the data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

3. The medical image segmentation method based on multi-scale and attention according to claim 1, It is characterized in that The channel attention module has three branches K, Q, V. First, the input feature map I∈R C * H * W Reshape I into feature maps K∈R C*(H*W) , feature map Q∈R (H*W)*C , forming branches K and Q at the same time, and then performing matrix multiplication between K and Q, divided by the factor Finally, the Softmax layer is applied to obtain the channel feature map A∈R C * C ; Branch V forms feature map V∈R through global average pooling of feature map I C*1*1 , after V is reshaped, we get V'∈R C*1*1 , let A and V' perform matrix multiplication and convert the result into R C*1*1 After that, it is multiplied by the scale parameter γ and element-wise summed with I to obtain the final output O∈R C*H*W .

4. The medical image segmentation method based on multi-scale and attention according to claim 1, It is characterized in that The structure of the bottleneck hole module is 3 convolutions with residual connections, namely 1 1×1 convolution, 1 3×3 hole convolution, and 1 1×1 convolution. The hole rates of the hole convolutions contained in the 3 bottleneck hole modules are 2, 3, and 5.

5. The medical image segmentation method based on multi-scale and attention according to claim 1, It is characterized in that In step 3, the loss function is a weighted cross entropy loss, and the formula is as follows: in represents the probability that voxel n belongs to class c, represents the weight of voxel n, represents the true value label of voxel n.

6. The medical image segmentation method based on multi-scale and attention according to claim 1, It is characterized in that In step 4, the evaluation indicators include Dice coefficient Dice, accuracy AC, precision Pre, sensitivity SE, and specificity SP. The specific formula is as follows: Among them, FP, FN, TP, and TN are false positive, false negative, true positive, and true negative, respectively.