Liver ct image segmentation method based on global self-attention and multi-scale feature fusion
By combining the ResNeXt network with the global self-attention mechanism, the problems of blurred boundaries and shape changes in liver CT image segmentation are solved, achieving a more accurate liver segmentation effect.
Patent Information
- Application Number
- CN202211064580.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-31
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2042-08-31
AI Technical Summary
Existing technologies in liver CT image segmentation have problems such as low internal contrast of the liver organ, blurred boundaries with adjacent organs, and large shape changes, which make segmentation difficult. Traditional methods are sensitive to noise and have limited machine learning feature selection. Deep learning methods do not fully utilize multi-scale features and self-attention, resulting in poor segmentation results.
The ResNeXt network with grouped convolution is used to extract multi-scale features, and the global self-attention mechanism is combined to capture the relationship between features. The features are fused through the improved residual convolution block, and the multi-level residual attention module is used to highlight important semantic features. The loss function is designed to improve the segmentation accuracy.
The precision and accuracy of liver CT image segmentation are improved, the problems of blurred boundaries and shape changes are solved, and a higher segmentation effect is achieved.
Smart Images
Figure CN115457051B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application designs a liver CT image segmentation method based on global self-attention and multi-scale feature fusion, and belongs to the technical field of medical image processing. BACKGROUND
[0002] Liver cancer is one of the fastest growing cancers in terms of morbidity and mortality worldwide. Computed tomography (CT) is a commonly used method for tumor diagnosis in clinics, which is mainly due to the fact that CT imaging technology can usually avoid the problem of organ image overlap in other imaging technologies, and is more conducive to tumor identification. Liver segmentation is a key step in the clinical diagnosis and analysis of interventional liver cancer, and accurate liver segmentation results can greatly improve the efficiency of doctors in reading CT images, so as to develop a diagnosis and treatment plan as soon as possible.
[0003] With the continuous growth of CT images, the CT scan data of a case usually accompanies several hundred CT slices, and there are problems such as subjective interference, inconsistent standards, complex process, time-consuming and laborious, and non-reproducibility in analyzing each slice by artificial method. Therefore, it is of great value to accurately and automatically segment the liver organ in abdominal CT images compared with manual segmentation. The main difficulties of liver segmentation at present mainly lie in that the contrast inside the liver organ is low, the intensity difference between the liver and other adjacent organs is small, the boundary of adjacent organs is blurred, and the shape changes greatly, which makes liver segmentation difficult. Therefore, liver organ segmentation based on CT images is a challenging task.
[0004] Liver automatic segmentation is mainly solved by the following three methods: 1) traditional image segmentation method: using shallow features such as gray level, texture, etc. to complete the segmentation task. However, this also leads to a relatively high sensitivity of traditional methods to noise pixels, and it is difficult for traditional image segmentation methods to make good use of deeper image features. 2) Machine learning method: analyze data patterns from large-scale data. However, most machine learning algorithms need to be carefully designed with artificial image features, and the expression of features and the final segmentation results are also subject to the selection method of features. 3) Deep learning method: without additional intermediate process, more abstract features can be extracted, and the selection method of features can be adjusted according to the results, greatly improving the accuracy. The existing deep learning method usually has better segmentation results than the traditional image processing method, but it still has shortcomings for liver and liver tumor segmentation. The consideration of the related characteristics such as the fuzzy boundary and the variable position of the liver and liver tumor in the CT image is lacking. When down-sampling, many features extracted by the deep learning method have little or no effect on the segmentation result. These features have not been weakened, and they are expressed equally with the key features for segmentation, which is not conducive to the segmentation result. And the traditional U-Net jump link method will cause semantic gap and cause feature mismatching problem, and the part of the multi-scale model method does not fully consider the correlation between features, thereby affecting the performance of the segmentation model. SUMMARY
[0005] In order to solve the above-mentioned problems, the present application provides a liver CT image segmentation method based on global self-attention and multi-scale feature fusion. The present application selects ResNeXt using grouping convolution as an image feature extraction network to obtain more image features without increasing the calculation time. For the problem of fuzzy boundary of liver organ, the multi-scale architecture is used to extract and fuse different scale features. And because there is a certain relationship between the liver organ and other organs in the CT image, the self-attention mechanism is introduced to capture the relationship between the extracted features. Finally, the residual convolution block of the improved attention method is used to fuse these features, so that these features can be better expressed, thereby obtaining better liver segmentation results.
[0006] The technical scheme of the present application is: a liver CT image segmentation method based on global self-attention and multi-scale feature fusion, the specific steps of the method are as follows:
[0007] Step 1, image preprocessing: the CT images in the LiTS dataset are processed according to the HU value range to increase the contrast, and random flipping and other methods are used to expand the dataset.
[0008] Step2, obtaining the same dimension feature and multi-scale feature: after the pre-processing operation in Step1, the ResNeXt convolutional neural network is used to extract the image feature, and the uniform dimension convolutional feature and the multi-scale feature based on the convolutional feature are obtained through linear transformation.
[0009] Step3, obtaining the global self-attention fusion feature: the multi-scale feature obtained in Step2 is obtained through the global self-attention module (Non-Local) to obtain the self-attention fusion feature containing global information, so as to capture the relationship between the target feature and the surrounding feature.
[0010] Step4, the self-attention fusion feature obtained in Step3 is extracted through the improved convolutional module, and the role of important semantic features in the channel dimension is highlighted, and finally the segmentation result is obtained through upsampling.
[0011] Further, the specific steps of Step1 are as follows:
[0012] Step1.1 The CT images in the LiTS dataset are processed according to the HU value range corresponding to the liver organ to increase the contrast; the CT value in the range of-130HU to 230HU, i.e. the window width is 360HU and the window level is 50HU, is processed, and the processed CT image is normalized.
[0013] Step1.2 Data augmentation adopts the way of random horizontal flip, vertical flip, scaling and cropping for data enhancement; after random expansion, the data is divided, of which 82% is used as the training set, and the remaining 18% is used as the test set; the training set is further divided into training data and validation data according to the ratio of 8:2, which is used for model training and model selection.
[0014] Further, the specific steps of Step2 are as follows:
[0015] Step2.1 After image preprocessing, the first five layers of ResNeXt-101 network are used as feature extraction layers, and the convolution in each ResNeXt block is divided into 32 paths, and the intermediate channel dimension of each path is 4, and different paths are equivalent to different feature subspaces, which are used to extract different semantic features, and the relationship learned by different path convolution kernels is more sparse, reducing the risk of overfitting.
[0016] Step2.2 The channel dimension of the output result of Layer 1-4 in the ResNeXt network structure is unified to 64 through linear transformation, and the feature map size is upsampled to be consistent with Layer 1. The four features are spliced, and after being compressed to 64 through a 1x1 convolution, the multi-scale feature is obtained, and the feature channel number and feature map size are consistent with the feature dimension after processing of Layer 1-4.
[0017] Further, the specific steps of Step3 are:
[0018] Step3.1 There is a certain relationship between different organs in the abdominal CT image. Obtaining this relationship can improve the liver organ segmentation effect. Inspired by the idea of calculating the relevance of the current position and other positions in the image in the non-local mean algorithm, the multi-scale features obtained in Step2 are linearly mapped three times to obtain Key, Query, and Value embedding space features. Linear mapping is implemented using 1x1 convolution.
[0019] Step3.2 Calculate the similarity of features Key and Query. The function for calculating the relevance is obtained according to the Gaussian function selected in the non-local mean, and the calculation formula is: where x i is the i-th position of the input feature map, and j represents all positions that may be related to i. The calculated similarity is weighted to the Value to obtain the self-attention feature.
[0020] Step3.3 The self-attention feature passes through the Softmax layer to obtain the output of the self-attention weight, so as to integrate the learned long-distance dependency relationship into the output feature. The overall calculation formula is as follows:
[0021]
[0022] where C(x) is the Softmax normalization function, function g linearly maps the representation of position j, and 1x1 convolution is usually used to realize it, and function f calculates the relevance of the i-th position and the j-th position.
[0023] Further, the specific steps of Step4 are:
[0024] Step4.1 The fusion feature containing multi-scale information and self-attention relationship extracted by Step3 is further extracted by an improved convolution module. The multi-scale self-attention fusion feature passes through a 1x1 convolution to map the feature channels to a specified dimension, and then passes through a 1x1 convolution and a 3x3 convolution and obtains the sum of the features of the two.
[0025] Step4.2 Use the attention module acting on the channel dimension (Channel Attention, CA) to re-label the feature channels, and use a residual path to fuse the original feature and the channel attention feature to obtain the output feature of the residual module. The specific calculation is shown in the formula:
[0026] Y MRA (X)=Y CA (WL X+W E X)+X
[0027] wherein, Y MRA (X) represents a multi-level residual attention convolution operation, X represents an input feature. W L is a 1x1 convolution matrix, which is used for linear mapping of the original input, equivalent to a residual path. W E is a 3x3 convolution matrix, which is used for feature extraction of the input feature, Y CA represents a channel attention operation.
[0028] Step4.3 The features extracted in Step4.3 are obtained by using the multi-path parallel idea of ensemble learning to obtain four groups of segmentation outputs. The average of the four groups of outputs is calculated as the final output result.
[0029] Further description of the application, in Step1 and Step4:
[0030] 1) Data preprocessing method:
[0031] The original CT image contains a large range of CT values, and the overall performance is poor in contrast. The gray difference between each organ in the image is small, which is difficult to distinguish. On medical CT images, the HU value range corresponding to the target organ is usually used for processing to increase the contrast. The liver part is usually processed with a window width of 150 and a window level of 30, but there is a gray difference between the liver organ and the liver tumor. If the HU value processing method of the liver is used, it will inevitably lead to the loss of gray scale in part of the liver tumor area, resulting in the loss of important information and causing poor training effect. In view of this problem, the application analyzes the histogram HU value distribution to obtain the CT value range of-130HU to 230HU, that is, the window width is 360HU and the window level is 50HU. After normalization operation, the processed CT image is obtained. The image after processing retains the information of the target area to the greatest extent while enhancing the contrast between organs, which is more conducive to the training of the model. The image before processing is as shown in Figure 3 (a), and the image after processing is as shown in Figure 3 (b).
[0032] 2) Loss function design:
[0033] In view of the situation that the positive and negative samples in the data set are not balanced, the weighted combination of binary cross entropy (Binary cross entropy, BCE) and dice loss (Dice Loss, DL) is used as the training loss. The calculation formula of the loss function L is as follows:
[0034]
[0035]
[0036]
[0037] wherein y represents the true segmentation map value, is the model predicted segmentation map value, and omega is the weight setting of two kinds of losses as 0.5, and epsilon is the smoothing item set as 1.0 to avoid the denominator as 0.
[0038] The beneficial effects of the present application are:
[0039] 1. The liver CT image segmentation method based on global self-attention and multi-scale feature fusion of the present application, according to the characteristics of the liver segmentation task, multi-scale strategy is selected to extract diverse features, and the multi-scale features extracted by different network layers are used to introduce multi-scale spatial information, so as to solve the problem of fuzzy boundary in liver segmentation. The global self-attention mechanism is used to build the relationship between the image features corresponding to different semantic categories, so that the model can better capture the correlation between the corresponding semantic features of the liver and other organs, and solve the problem of large shape change of the liver.
[0040] 2. Each channel dimension corresponds to a class of semantic information, which is mapped to a class of image features in the original image. However, we hope that the image features corresponding to the liver organ should have a higher importance, and obviously the same channel weight is not conducive to the expression of key features. In view of this problem, the present application designs a multi-level residual attention convolution MRA module to highlight the role of important semantic features in the channel dimension.
[0041] In summary, the liver CT image segmentation method based on global self-attention and multi-scale feature fusion, first uses the ResNeXt convolutional neural network to obtain multi-scale features in the abdominal CT image, then uses the global self-attention module to capture the spatial position relationship, and combines the multi-level residual attention module to highlight the role of important semantic features in the channel dimension; finally, the accuracy of liver image segmentation is improved. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 Structure diagram of the liver CT image segmentation method based on global self-attention and multi-scale feature fusion;
[0043] Figure 2 The schematic diagram based on the global self-attention module in the present application;
[0044] Figure 3 The contrast chart before and after preprocessing of the present application; wherein (a) is the image before processing; (b) is the image after processing;
[0045] Figure 4The visualization comparison chart of the segmentation result of the application; wherein, (a) CT picture; (b) reference segmentation standard; (c) original model segmentation result; (d) segmentation result of model added with self-attention module; (e) segmentation result of model added with improved convolution module. DETAILED DESCRIPTION
[0046] Embodiment 1: As shown in the figure, the specific steps of a liver CT image segmentation method based on global self-attention and multi-scale feature fusion are as follows: Figures 1-4
[0047] Step 1, image preprocessing: the CT images in the LiTS dataset are processed according to the HU value range to increase the contrast, and random flipping and other methods are used to expand the dataset.
[0048] Further, the specific steps of Step 1 are as follows:
[0049] Step 1.1, the CT images in the LiTS dataset are processed according to the HU value range corresponding to the liver organ to increase the contrast. The CT values in the range of-130HU to 230HU, i.e. window width 360HU, window level 50HU, are processed, and the processed CT images are then normalized.
[0050] Step 1.2, data augmentation is performed by random horizontal flipping, vertical flipping, scaling and cropping. After random expansion, the data is divided, of which 82% is used as the training set, and the remaining 18% is used as the test set. The training set is further divided into training data and validation data in the ratio of 8:2 for model training and model selection.
[0051] Step 2, obtain the same dimension feature and multi-scale feature: after the preprocessing operation in Step 1, the ResNeXt convolutional neural network is used to extract the image features, and the unified dimension convolutional features and multi-scale features based on the convolutional features are obtained through linear transformation.
[0052] Further, the specific steps of Step 2 are as follows:
[0053] Step 2.1, after image preprocessing, the first five layers of ResNeXt-101 network are used as feature extraction layers, and the convolution in each ResNeXt block is divided into 32 paths, and the intermediate channel dimension of each path is 4. Different paths are equivalent to different feature subspaces, which are used to extract different semantic features.
[0054] Step2.2 The channel dimension of the output result of Layer 1-4 in the ResNeXt network structure is unified to 64 through linear transformation, and the feature map size is up-sampled to be consistent with Layer 1. The four features are spliced, and after being compressed to 64 through a 1x1 convolution, multi-scale features are obtained, and the feature channel number and feature map size are consistent with the feature dimension processed by Layer 1-4.
[0055] Step3, obtain global self-attention fusion features: with the multi-scale features obtained in Step2, global self-attention fusion features containing global information are obtained through a global self-attention module (Non-Local) to capture the relationship between the target features and the surrounding features.
[0056] Further, the specific steps of Step3 are as follows:
[0057] Step3.1 There is a certain relationship between different organs in the abdominal CT image, and obtaining this relationship can improve the liver organ segmentation effect. Inspired by the non-local mean algorithm, the Non-Local global self-attention is selected, and the multi-scale features obtained in Step2 are linearly mapped three times to obtain Key, Query and Value embedding space features. Linear mapping is realized by using a 1x1 convolution, and the comparison of different attention methods for multi-scale fusion features is shown in Table 1.
[0058] Table 1 Comparison of different attention methods for multi-scale fusion features
[0059]
[0060]
[0061] Step3.2 The similarity of the features Key and Query is calculated, and the function for calculating the correlation is obtained according to the Gaussian function selected by the non-local mean, and the calculation formula is: Where x i is the i-th position of the input feature map, and j represents all positions that may be related to i. The similarity calculated is weighted to the Value to obtain the self-attention feature.
[0062] Step3.3 The self-attention feature passes through a Softmax layer to obtain the output of the self-attention weight, so that the long-distance dependent relationship learned is integrated into the output feature; the overall calculation formula is as follows:
[0063]
[0064] Where C(x) is a Softmax normalization function, function g linearly maps the representation of input j position, usually realized by 1x1 convolution, and function f calculates the correlation between the input i position and j position.
[0065] Step4, the self-attention fusion features obtained in Step3 are extracted by an improved convolution module to highlight the role of important semantic features in the channel dimension, and finally up-sampling is performed to obtain a segmentation result.
[0066] Further, the specific steps of Step4 are as follows:
[0067] Step4.1 extracts the fusion features containing multi-scale information and self-attention relationship from Step3, and further extracts the information in the fusion features by an improved convolution module. The multi-scale self-attention fusion features are mapped to a specified dimension by a 1x1 convolution, and then a 1x1 convolution and a 3x3 convolution are performed and the features of the two are added.
[0068] Step4.2 uses a channel attention module (CA) acting on the channel dimension to re-label the feature channels, and uses a residual path to fuse the original features and the channel attention features to obtain the output features of the residual module; the specific calculation is shown in the formula:
[0069] Y MRA (X)=Y CA (W L X+W E X)+X
[0070] Where Y MRA (X) represents a multi-level residual attention convolution operation, and X represents an input feature. W L is a 1x1 convolution matrix for linearly mapping the original input, which is equivalent to a residual path. W E is a 3x3 convolution matrix for feature extraction of the input feature, and Y CA represents a channel attention operation.
[0071] Step4.3 extracts four groups of segmentation outputs using the multi-path parallel idea of ensemble learning, and calculates the average of the four groups of outputs as the final output result.
[0072] The hardware environment used in the experiments of the present application is configured as Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHZ, GPU NVDIA TITAN XP hardware platform, the operating system is Ubuntu 18.04.1, and the software platform includes GPU parallel computing architecture CUDA and Pytorch deep learning framework based on Python programming language. In this paper, the Adam optimizer is used, the learning rate change strategy adopts the CosineAnnealing strategy, the initial learning rate is 0.001, the minimum value is 0.00001, and it is reset once every 30 rounds. The total number of training rounds is 80 rounds, and the batch size is 4.
[0073] Table 2 is the experimental comparison result of the present application with the method in the field of medical image segmentation on the data set LiTS, which includes the classic segmentation algorithms such as UNet, FCN, and the like, DAF, MsAUNet and the like algorithms using similar multi-scale feature fusion means of the present application, and the well-known algorithms H-DenseUNet, Multiple UNet and the like algorithms in the field of liver segmentation. The liver segmentation result of the method of the present application is much higher than that of the classic segmentation algorithm FCN, and the average Dice value of the overlapping area of the segmentation result and the true segmentation reaches 96.4%, which is 4.3% higher than that of the classic model UNet. Since the present application has achieved similar results to 3D methods in the case of using 2D model, it shows that the present method has strong competitiveness in the task of liver segmentation.
[0074] Table 2 and the existing method
[0075]
[0076] Figure 4 is the experimental image segmentation result of the present application on the data set LiTS, wherein (a) is a CT picture; (b) is a reference segmentation standard; (c) is a segmentation result of an original model; (d) is a segmentation result of a model with added self-attention module; (e) is a segmentation result of a model with added improved convolution module; it can be seen that after adding the self-attention mechanism, the false prediction of the area other than the liver in the prediction can be relieved to a certain extent, and part of the false positive prediction result is eliminated, further proving the role of the self-attention mechanism in the task of liver segmentation. Further adding the MRA module perfected by the channel attention mechanism, through the enhancement or inhibition of the semantic features in the channel dimension, most of the false positive prediction is successfully eliminated, and the segmentation edge is closer to the true segmentation result.
[0077] The specific embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.
Claims
1. A liver CT image segmentation method based on global self-attention and multi-scale feature fusion, characterized in that, The specific operation steps of the method are as follows: Step 1, image preprocessing: the CT images in the LiTS dataset are processed according to the HU value range to increase the contrast, and the dataset is expanded; Step 2, obtain the same dimension feature and multi-scale feature: after the preprocessing operation in Step 1, the ResNeXt convolutional neural network is used to extract the image features, and the unified dimension convolutional features and multi-scale features based on the convolutional features are obtained through linear transformation; Step 3, obtain the global self-attention fusion feature: With the multi-scale features obtained in Step 2, the global self-attention fusion feature containing global information is obtained through the global self-attention module Non-Local, so as to capture the relationship between the target feature and the surrounding features; Step 4, the self-attention fusion feature obtained in Step 3 is extracted through the improved convolution module, highlighting the role of important semantic features in the channel dimension, and finally up-sampling to obtain the segmentation result; The specific steps of Step 2 are as follows: Step 2.1, after image preprocessing, the first five layers of ResNeXt-101 network are used as feature extraction layers, and the convolution in each ResNeXt block is divided into 32 paths, and the intermediate channel dimension of each path is 4. Different paths are equivalent to different feature subspaces, which are used to extract different semantic features. At the same time, the relationship learned by different path convolution kernels is more sparse, which reduces the risk of overfitting; Step 2.2, the channel dimension of the output results of Layer 1-4 in the ResNeXt network structure is unified to 64 through linear transformation, and the feature map size is up-sampled to be consistent with Layer 1; the four features are spliced, and after being compressed to 64 through a 1x1 convolution, the multi-scale feature is obtained. The feature channel number and feature map size are consistent with the feature dimension of Layer 1-4 after processing; The specific steps of Step 3 are as follows: Step 3.1, there is a certain relationship between different organs in abdominal CT images. Obtaining this relationship can improve the liver organ segmentation effect; inspired by the idea of calculating the relevance between the current position and other positions in the image in the non-local mean algorithm, starting from the multi-scale features obtained in Step 2, three linear mappings are performed to obtain Key, Query, and Value embedding space features, and linear mapping is realized by 1x1 convolution; Step3.2 calculates the similarity of the feature Key and Query, and the function of correlation is obtained according to the non-local mean selected Gaussian function, and the calculation formula is: Where x i is the i-th position of the input feature map, j represents all positions that may be related to i, and the similarity obtained by calculation is weighted to the Value to obtain the self-attention feature; Step 3.3, the self-attention feature passes through the Softmax layer to obtain the output of the self-attention weight, so as to integrate the learned long-distance dependence relationship into the output feature, and the overall calculation formula is as follows: Where C(x) is the Softmax standardization function, the function g linearly maps the representation of the input j position, usually realized by 1x1 convolution, and the function f calculates the relevance between the input i position and the j position.
2. The liver CT image segmentation method based on global self-attention and multi-scale feature fusion according to claim 1, characterized in that, The specific steps of Step 1 are as follows: Step1.1 The CT images in the LiTS dataset are processed according to the HU value range corresponding to the liver organ to increase the contrast; the CT values in the range of-130HU to 230HU, i.e. the window width is 360HU and the window level is 50HU, are processed, and the processed CT images are normalized; Step1.2 Data augmentation is performed by random horizontal flipping, vertical flipping, scaling, and cropping; after random expansion, the data is divided, 82% of which is used as a training set, and the remaining 18% is used as a test set, the training set is further divided into training data and validation data according to the ratio of 8:2, which is used for model training and model selection.
3. The liver CT image segmentation method based on global self-attention and multi-scale feature fusion according to claim 1, characterized in that, The specific steps of Step4 are: Step4.1 The fusion features containing multi-scale information and self-attention relationship extracted from Step3 are further extracted by an improved convolution module; the multi-scale self-attention fusion features pass through a 1x1 convolution to map the feature channels to a specified dimension, then pass through a 1x1 convolution and a 3x3 convolution and get the sum of the features of the two; Step4.2 The channel attention module acting on the channel dimension is used to re-label the feature channels, and a residual path is used to fuse the original features and the channel attention features to obtain the output features of the residual module, and the specific calculation is as shown in the formula: Y MRA (X) = Y CA (W L X + W E X) + X wherein Y MRA (X) represents a multi-stage residual attention convolution operation, X represents an input feature, W L is a 1×1 convolution matrix, used for linear mapping of the original input, equivalent to a residual path; W E is a 3×3 convolution matrix, used for feature extraction on the input feature, Y CA represents a channel attention operation; Step4.3 The extracted features use the multi-path parallel idea of ensemble learning to obtain four groups of segmentation outputs, and the average of the four groups of outputs is taken as the final output result.