A pancreatic image segmentation method based on multi-scale feature selective fusion

Abstract

A pancreatic image segmentation method with multi-scale feature selective fusion was proposed. A two-stage coarse-to-fine pancreatic segmentation framework was constructed. First, coarse segmentation and segmentation optimization were performed on the preprocessed whole image to obtain the coarse segmentation result, the bounding box of the pancreatic segmentation area was determined, the preprocessed image was cropped, and then fine segmentation and segmentation optimization were performed on the cropped image to obtain the fine segmentation result of the pancreatic image. The SE mechanism and Inception idea were introduced into the feature fusion at different levels, and a multi-scale feature selective fusion module MSSFM was constructed. Based on the Unet model, the encoding part used residual convolution blocks to make the network deeper and prevent model degradation, thereby improving the learning of effective features. The decoding part used the multi-scale feature selective fusion module to fuse multi-scale features from bottom to top, thereby increasing the network's adaptability to multi-scale targets and focusing on the target area, thereby improving the accuracy of model segmentation.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method for performing semantic segmentation on a pancreatic image after selecting spatial features and scale features. Background Art

[0002] The pancreas is one of the most important organs in the human body, with both endocrine and exocrine functions. It is located deep in the abdominal cavity, with an insidious onset. It is also small in size and highly deformable, which poses a great challenge to the diagnosis of pancreatic diseases. High-precision pancreatic image segmentation is a prerequisite for quantitative analysis of the pancreas and disease diagnosis with the help of a computer-aided diagnosis system. Since manual segmentation is inefficient, cumbersome, and error-prone, the automatic segmentation of pancreatic images has a very important research value and significance. At present, the automatic segmentation of pancreatic CT images is still a very challenging task. The main factors are: 1) The pancreas accounts for a small proportion of the entire image, has a low contrast with the surrounding tissues, has blurred boundaries, and is easily affected by the background; 2) The pancreas has a high degree of anatomical deformability in shape and size. These factors have caused great difficulties for the automatic and accurate segmentation of the pancreas.

[0003] With the development of deep neural networks and their wide application in medical image semantic segmentation, a large number of deep learning-based pancreatic segmentation methods have been proposed and have made effective progress. Most of the existing deep learning methods are based on 2D FCNs or 3D FCNs to achieve single or multi-stage segmentation. Whether it is a method based on 2D FCNs or 3D FCNs, the performance of segmentation depends on the learning ability of the model. However, the models of most methods have large feature channels, large number of model parameters, and a large amount of feature redundant learning and computing resource consumption.

[0004] Traditional codec semantic segmentation networks (such as FCN and Unet) use Add or Concat operations in the decoder to fuse low- and high-level features of different scales, which can retain more spatial detail features and semantic features while restoring the feature size. In particular, the Unet model uses layer-by-layer multi-scale feature fusion to obtain more refined results and has good performance in medical image segmentation. However, for highly deformable targets of different sizes, shapes, and positions (such as the pancreas), the Unet segmentation performance has great limitations. One of the important reasons is that the feature fusion of different levels of Unet is consistent for all input images, while this type of target has different adaptability to features of different receptive fields and different levels. In addition, the learning ability of the Unet network is limited by the size of the training data set and the size of the network. Summary of the invention

[0005] In order to improve the performance of pancreatic segmentation and reduce the consumption of computing resources, the present invention provides a pancreatic image segmentation method of multi-scale feature selective fusion to improve the accuracy of model segmentation, constructs a multi-scale feature selective fusion network, utilizes the consistency and difference of spatial attention between multi-scale features, assigns different scale weights and the same spatial weight to features of different scales, so as to indicate the importance of features of different scales for input samples and focus on the same region of interest.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A pancreatic image segmentation method with multi-scale feature selective fusion is proposed. A two-stage pancreatic segmentation framework from coarse to fine is constructed. First, coarse segmentation and segmentation optimization are performed on the preprocessed whole image to obtain the coarse segmentation result, the bounding box of the pancreatic segmentation area is determined, the preprocessed image is cropped, and then fine segmentation and segmentation optimization are performed on the cropped image to obtain the fine segmentation result of the pancreatic image.

[0008] The segmentation optimization method of the coarse segmentation and the fine segmentation comprises the following steps:

[0009] 1) The SE mechanism and Inception idea are introduced into feature fusion at different levels, and a multi-scale feature selective fusion module MSSFM is constructed;

[0010] MSSFM uses multi-scale feature information to generate a spatial position weight and a scale feature weight of the corresponding scale feature. For given features at different levels, we first use a convolution operation with a convolution kernel size of 1 and an upsampling operation to unify the feature size and feature dimension to C×H×W. In addition, in order to increase the adaptability of the model to multiple scale features, we use convolution operations with convolution kernel sizes of 3 and 5 to obtain multiple features of different scales F. X3 , F X5 , F Y3 , F Y5 , the feature dimension is C / r×H×W, r is the dimension reduction coefficient, and then the consistency and difference of multi-scale feature space decision are used to selectively fuse the multi-scale features. The fused features are expressed as:

[0011]

[0012] W X3,X5,Y3,Y5 ∈4×1 =σ(Conv 1*1 (δ(Conv 1*1 (g(S)))) (2)

[0013] W S =Max(S) (3)

[0014] S = Concat(S X3 ,S X5 ,S Y3 ,S Y5 ) (4)

[0015] S X3 =σ(Conv 1*1 F X3 )

[0016] S X5 =σ(Conv 1*1 F X5 ) (5)

[0017] S Y3 =σ(Conv 1*1 F X3 )

[0018] S Y5 =σ(Conv 1*1 F X3 )

[0019] Where W X3 , W X5 , W Y3 , W Y5 The scale information weights corresponding to the features of different scales are the spatial position weights of the features of different scales. Concat represents the feature stacking of the feature dimension. σ represents sigmoid activation, δ represents Relu activation, and Conv 1*1 represents a convolution operation with a convolution kernel size of 1*1, and g is a global pooling operation used to compress the feature space;

[0020] 2) For pancreatic image data, thresholding was first performed, with the CT value range of [-100, 240] HU to highlight the pancreatic area, and then normalized to [0, 1]. Four-fold cross validation was used, with three samples taken as training sets in each round and the remaining one as the test set, for a total of four rounds, during which each patient was evaluated once.

[0021] During the model training process, random flipping, rotation, translation, shearing, and scaling operations are used to augment the training data to increase the number of samples.

[0022] Furthermore, during the model training phase, the learning rate was set to 0.0001, and the Adam optimizer was selected. A total of 20 epochs were trained, and the learning rate was reduced by 10 times every 10 epochs. During this period, a batch size of 4 samples was used for gradient update, and the loss function used the general Dice loss function:

[0023]

[0024] The morphological closing operation and maximum connected domain processing are used to optimize the segmentation results of the three-dimensional volume data, ensure the three-dimensional connectivity of each slice, smooth the boundary information, and remove the influence of outliers; the Ball kernel is used for morphological operations.

[0025] The present invention selectively fuses multi-scale features to make the network adapt to target structures of different shapes and sizes, improve the generalization ability of the model and the accuracy of segmentation, and alleviate the challenges brought by the high deformability and background interference of the pancreas to a certain extent. Limited by the video memory, a method based on 2D FCNs is used to verify the effectiveness of the model, but only the output in the axial direction is used as the prediction result. In order to combine 3D context information, eliminate outliers and fill the holes between and within slices, morphological closing operations and maximum connected domains are used for post-segmentation processing optimization to ensure the connectivity and smoothness of the three-dimensional segmentation results. Based on the Unet model, the encoding part of the present invention uses residual convolution blocks to make the network deeper and prevent model degradation, thereby improving the learning of effective features; the decoding part uses a selective fusion module of multi-scale features to fuse multi-scale features from bottom to top, increase the adaptability of the network to multi-scale targets and focus on the target area, and improve the accuracy of model segmentation.

[0026] The beneficial effects of the present invention are mainly manifested in:

[0027] 1) In order to improve the performance of pancreas segmentation and reduce the consumption of computing resources, a multi-scale feature selective fusion module is proposed to increase the adaptability of the network to multi-scale targets, improve the generalization ability of the network, and also reduce the learning of redundant features.

[0028] 2) In order to solve the problem of missing intermediate layers caused by 2D slice segmentation, the 3D segmentation results are post-processed and optimized using morphological closing operations and maximum connected domains to ensure the integrity and connectivity of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall framework for pancreatic image segmentation in the present invention.

[0030] Figure 2 It is the entire network structure of the model, where the curved arrows represent upsampling, the hollow arrows represent downsampling, and PixelMax represents the maximum value of all feature maps point by point.

[0031] Figure 3 It is the structural diagram of the residual convolution block RCB.

[0032] Figure 4 It is the structural diagram of the multi-scale feature selective fusion module MSSFM. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings.

[0034] Reference Figure 1 to Figure 4 , a pancreatic image segmentation method with multi-scale feature selective fusion, introduces the SE mechanism and Inception idea into feature fusion at different levels, and constructs a multi-scale feature selective fusion module MSSFM (Multi scale Selctive Fusion Module) to increase the network's adaptability to highly deformable targets.

[0035] The structure of MSSFM is as follows Figure 4 As shown in the figure, MSSFM mainly uses multi-scale feature information to generate a spatial position weight and a scale feature weight corresponding to the scale feature. These weights can be regarded as feature selectors. For different input images, they can adaptively select features of different scales for fusion decisions, increase the flexibility of the network, adapt to highly deformed segmentation targets, and improve the performance of network segmentation. For given different levels of features, firstly, the convolution operation with a convolution kernel size of 1 and the upsampling operation are used to unify the size of the feature and the dimension of the feature to C×H×W. In addition, in order to increase the adaptability of the model to multiple scale features, convolution operations with convolution kernel sizes of 3 and 5 are used to obtain multiple different scale features F. X3 , F X5 , F Y3 , F Y5 , the feature dimension is C / r×H×W (r is used to reduce the dimension and the number of parameters, and r=2 is used). Then, the consistency and difference of multi-scale feature space decision are used to selectively fuse the multi-scale features. The fused features can be expressed as:

[0036]

[0037] W X3,X5,Y3,Y5 ∈4×1 =σ(Conv 1*1 (δ(Conv 1*1 (g(S)))) (2)

[0038] W S =Max(S) (3)

[0039] S = Concat(S X3 ,S X5 ,S Y3 ,S Y5 ) (4)

[0040] S X3 =σ(Conv 1*1 F X3 )

[0041] S X5 =σ(Conv 1*1 F X5 ) (5)

[0042] S Y3 =σ(Conv 1*1 F X3 )

[0043] S Y5 =σ(Conv 1*1 F X3 )

[0044] Where W X3 , W X5 , W Y3 , W Y5 Corresponding to the scale information weights of different scale features, Concat represents the feature stacking of feature dimensions. σ represents sigmoid activation, δ represents Relu activation, Conv represents the spatial position weights of different scale features. 1*1 It represents a convolution operation with a convolution kernel size of 1*1, and g is a global pooling operation used for feature space compression.

[0045] The data set used in the present invention is the NIH pancreas data set, which contains 82 contrast-enhanced CT data. The voxel size of each data set is 512×512×C (C[181,466]). The true labels of the data set include background (label 0) and pancreas (label 1). In order to effectively evaluate the method in this paper, the data set is not subjected to any cropping operation. First, the data is thresholded, and the CT value range is [-100,240]HU, the pancreatic area is highlighted, and then normalized to [0,1]. According to most cross-validation strategies, a four-fold cross-validation is adopted. First, the 82 patients are randomly divided into four parts [21,21,20,20]. In each round, three parts are taken as training sets, and the remaining one part is taken as test set. A total of four rounds are performed, during which each patient will be evaluated once.

[0046] During the model training process, in order to alleviate the overfitting of the model, random flipping, rotation, translation, shearing, and scaling operations are used to augment the training data to increase the number of samples and improve the generalization ability of the model.

[0047] The code of this method is implemented based on the keras framework, and the model is trained on RTX 2080 (11G). During the training phase, the learning rate is set to 0.0001, and the Adam optimizer is selected. A total of 20 epochs are trained, and the learning rate is reduced by 10 times every 10 epochs. During this period, a batch size of 4 samples is used for gradient update, and the loss function uses the general Dice loss function:

[0048]

[0049] Since the strategy of implementing 3D volume data segmentation based on 2D model slice segmentation ignores the connectivity of adjacent slices, there will be a gap in the middle of the segmentation result. The present invention uses morphological closing operation and maximum connected domain processing to optimize the segmentation result of 3D volume data, ensure the 3D connectivity of each slice and smooth the boundary information, and remove the influence of outliers. The present invention uses Ball kernel (radius=3) for morphological operation, which can ensure the 3D connectivity of most data.

[0050] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A pancreatic image segmentation method with multi-scale feature selective fusion, characterized in that: A two-stage coarse-to-fine pancreatic segmentation framework is constructed. First, coarse segmentation and segmentation optimization are performed on the preprocessed whole image to obtain the coarse segmentation result, the bounding box of the pancreatic segmentation area is determined, the preprocessed image is cropped, and then fine segmentation and segmentation optimization are performed on the cropped image to obtain the fine segmentation result of the pancreatic image. The segmentation optimization method of the coarse segmentation and the fine segmentation comprises the following steps: 1) The SE mechanism and Inception idea are introduced into feature fusion at different levels, and a multi-scale feature selective fusion module MSSFM is constructed; MSSFM uses multi-scale feature information to generate a spatial position weight and a scale feature weight of the corresponding scale feature. For given features at different levels, we first use a convolution operation with a convolution kernel size of 1 and an upsampling operation to unify the feature size and feature dimension to C×H×W. In addition, in order to increase the adaptability of the model to multiple scale features, we use convolution operations with convolution kernel sizes of 3 and 5 to obtain multiple features of different scales F. X3 , F X5 , F Y3 , F Y5 , the feature dimension is C / r×H×W, r is the dimension reduction coefficient, and then the consistency and difference of multi-scale feature space decision are used to selectively fuse the multi-scale features. The fused features are expressed as: W X3,X5,Y3,Y5 ∈4×1 =σ(Conv 1*1 (δ(Conv 1*1 (g(S)))) (2) W S =Max(S) (3) S=Concat(S X3 ,S X5 ,S Y3 ,S Y5 ) (4) S X3 =σ(Conv 1*1 F X3 ) S X5 =σ(Conv 1*1 F X5 ) (5) S Y3 =σ(Conv 1*1 F X3 ) S Y5 =σ(Conv 1*1 F X3 ) Where W X3 , W X5 , W Y3 , W Y5 The scale information weights corresponding to the features of different scales are the spatial position weights of the features of different scales. Concat represents the feature stacking of the feature dimension. σ represents sigmoid activation, δ represents Relu activation, and Conv 1*1 represents a convolution operation with a convolution kernel size of 1*1, and g is a global pooling operation used to compress the feature space; 2) For pancreatic image data, thresholding was first performed, with the CT value range of [-100, 240] HU to highlight the pancreatic area, and then normalized to [0, 1]. Four-fold cross validation was used, with three samples taken as training sets in each round and the remaining one as the test set, for a total of four rounds, during which each patient was evaluated once. During the model training process, random flipping, rotation, translation, shearing, and scaling operations are used to augment the training data to increase the number of samples.

2. The pancreatic image segmentation method with multi-scale feature selective fusion as claimed in claim 1, characterized in that: During the model training phase, the learning rate was set to 0.0001, and the Adam optimizer was selected. A total of 20 epochs were trained, and the learning rate was reduced by 10 times every 10 epochs. During this period, a batch size of 4 samples was used for gradient update, and the loss function used was the Dice loss function: The morphological closing operation and maximum connected domain processing are used to optimize the segmentation results of the three-dimensional volume data, ensure the three-dimensional connectivity of each slice, smooth the boundary information, and remove the influence of outliers; the Ball kernel is used for morphological operations.

Images