An image processing method for liver lesion segmentation in non-aligned multi-phase CT

Abstract

The present invention discloses an image processing method for liver lesion segmentation of non-aligned multi-phase CT. First, a 3D region alignment module is used to align multiple phases of CT to obtain multi-phase features, and then a residual stereo attention module is used to extract multi-phase stereo features from the multi-phase features, thereby realizing the combination of multi-phase context and stereo context. Specifically, first, CNN is used to extract features for the target layer CT and its multi-phase stereo context; then, the multi-phase context is aligned using the 3D region alignment module to generate multi-phase features; then, the multi-phase stereo features are extracted from the multi-phase features using the residual stereo attention module; finally, the multi-phase stereo features are restored to the final segmentation using a decoder. The present invention extracts features with both multi-phase information and stereo information from non-aligned multi-phase CT to improve the accuracy of liver lesion segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation and deep learning, and in particular relates to an image processing method for liver lesion segmentation of non-aligned multi-phase CT. Background Art

[0002] Multi-phase context and stereo context are crucial for liver lesion segmentation based on CT images. Liver lesions often need to be enhanced with contrast agents to clearly and accurately determine the boundaries and types of lesions. Before contrast agent injection, the abdomen is scanned to produce a sequence of CT images, which is called the plain phase. After contrast agent injection, the contrast agent will flow through veins and arteries with the blood in two specific time periods, and scans are performed in these two time periods to produce two sequences of CT images: venous phase (arterial phase) and arterial phase (portal phase). In actual operation, due to patient movement, breathing, visceral movement and some technical reasons, the image content in the three sequences is not aligned at the pixel level. This makes it difficult to use multi-phase context and stereo context.

[0003] For the combination of multi-period contexts, the current simpler method is to directly splice multi-period images at the input stage or to splice their features after feature extraction. For example, (C.Sun, S.Guo, H.Zhang, et al., "Automatic segmentation of liver tumors from multiphase contrast-enhanced CT images based on FCNs," in Artificial intelligence in medicine.2017, pp.58-66.), (F.Ouhmich, V.Agnus, V.Noblet, et al., "Liver tissue segmen-tation in multiphaseCT scans using cascaded convolutional neu-ral networks," in Int J CARS14.2019, pp.1275–1284.Available: https: / / doi.org / 10.1007 / s11548-019-01989-z), (R.Hasegawa, Y.Iwamoto, L.Lin, et al., "Automatic Segmentation of Liver Tumor inMultiphase CT Images by Mask R-CNN,” in 2020IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech). 2020, pp. 231-234. Available: https: / / doi.org / 10.1007 / s11548-019-01989-z), (J. Dolz, et al., "HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation," in IEEE Transactions on Medical Imaging, vol. 38, no. 5, pp. 1116-1126, May 2019, doi: 10.1109 / TMI.2018.2878669.), etc. However, due to the problem of misalignment of multi-phase CT, this simple stitching method is difficult to capture effective multi-phase information.

[0004] Another method is to use the attention mechanism to automatically filter out information from one phase of features that is helpful for segmenting another phase. Such as (Y.Xu, M.Cai, L.Lin et al., "PA-ResSeg: A phase attention residual network for liver tumor segmentation from multiphase CT images." in Med.Phys., 48(7), May 2021, p.3752-3766), (X.Jiang et al., "Multi-phase and Multi-level Selective Feature Fusion for Automated Pancreas Segmentation from CT Images," in Proc.Int.Conf.Med.Image Comput.Comput.Assist.Intervent.(MICCAI).Lima, Peru: Springer, 2020, pp.460-469.) etc. This type of method combines the information of two phases in a linear way and generates an attention map, and filters out useful cross-phase information through the attention map. However, when this type of linear transformation method is used on non-aligned CT, some multi-phase context information will still be lost.

[0005] At the same time, the above model uses 3D convolution to extract stereo information. On the one hand, 3D convolution is limited by the perception field of view, and on the other hand, it does not fully utilize the global information of CT. Summary of the invention

[0006] The object of the present invention is to provide an image processing method for liver lesion segmentation in non-aligned multi-phase CT in view of the deficiencies in the prior art.

[0007] The object of the present invention is achieved through the following technical solution: an image processing method for liver lesion segmentation of non-aligned multi-phase CT, comprising the following steps:

[0008] (1) CT images to be predicted and its sweep period context X p , arterial phase context X a and venous phase context X v Perform preprocessing to make it meet the input requirements of the network;

[0009] (2) Use the trained liver lesion segmentation model to predict the preprocessed CT images.

[0010] Furthermore, the step (2) includes the following sub-steps:

[0011] (2.1) Use a convolutional neural network (CNN) to extract the CT image to be predicted High-level features and low-level features and the high-level features of the context corresponding to multiple periods, the high-level features of the flat sweep period f p 、Advanced features of arterial phase a and advanced features of the venous phase v ;

[0012] (2.2) Based on the high-level features f of multiple periods obtained in step (1.1) p 、f a and f v , use the 3D region alignment module to align the high-level features of multiple periods and generate multi-period features f m ;

[0013] (2.3) Using the residual stereo attention module, the multi-period feature f generated in step (2.2) m In the process of extracting multi-phase stereo features, mv ;

[0014] (2.4) The multi-phase stereo features extracted in step (2.3) are mv , and the low-level features in step (2.1) Combine them, use the decoder to combine them, and enlarge them to the original image size to produce the segmentation result.

[0015] Further, the step (2.2) includes the following sub-steps:

[0016] (2.2.1) For f a For each feature point in , we use non-local attention to calculate the alignment relationship between the feature point and the pixels in the corresponding position within a limited range in the flat-scanning period feature, and use this alignment relationship to generate f a With f p The intertemporal characteristics of m,p→a , and f is generated in the same way a With f v The intertemporal characteristics of m,v→a ;

[0017] (2.2.2) Combine the inter-period features f in step (2.2.1) m,p→a and f m,v→a , and the high-level features f of multiple periods in step (2.2) a , generating multi-period features f with multi-period context information m .

[0018] Furthermore, the step (2.3) includes the following sub-steps:

[0019] (2.3.1) Using the multi-period feature f in step (2.2.2) m , using global inter-layer attention, calculate the global inter-layer feature f I ;

[0020] (2.3.2) Using the global inter-layer feature f in step (2.3.1) I , using spatial attention, calculate the spatial feature f S ;

[0021] (2.3.3) Using residual link, f′ I and the spatial feature f in step (2.3.2) S Combined, we get the feature f that has both multi-period context information and stereo context information mv ; Among them, f I ′ is the global inter-layer feature f I and The normalized result of the residual and .

[0022] The beneficial effects of the present invention are as follows: the present invention aligns the multi-phase contexts and fuses them to generate multi-phase features, and further extracts stereoscopic features from the multi-phase features. The present invention mitigates the impact of misalignment between multi-phase CTs through a 3D region alignment module. The present invention makes up for the defect that the models on the market cannot output all sequence segmentation results and cannot be applied to non-aligned multi-phase CT. The present invention can take into account both 3D information and multi-phase information at the same time, greatly improving the segmentation and classification performance of liver lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an example image of non-aligned multi-phase CT;

[0024] Figure 2 It is a schematic diagram of the overall structure of the liver lesion segmentation model;

[0025] Figure 3 It is a schematic diagram of the 3D region alignment module;

[0026] Figure 4 is a schematic diagram of the residual stereo module;

[0027] Figure 5 It is a comparison chart of the effects of the present invention and other existing models. DETAILED DESCRIPTION

[0028] The present invention provides an image processing method for liver lesion segmentation of non-aligned multi-phase CT, and proposes a 3D region alignment module and a residual stereo attention module. Figure 1As shown in the figure, since the feature points in a certain period are only aligned with the neighboring feature points in a limited range of corresponding positions in other periods, the 3D region alignment module uses non-local attention to calculate the alignment relationship of each feature point and use this relationship to generate cross-period features. Finally, the 3D region alignment module integrates the cross-period features and the original features of the CT image to generate multi-period features; the residual stereo attention module extracts stereo features based on the multi-period features. The residual stereo attention module first uses global pooling to calculate the global inter-layer features, and then uses the global inter-layer features to enhance the ability of spatial attention to extract spatial features. It includes the following steps:

[0029] (1) CT images to be predicted and its sweep period context X p ∈R N×W×H , Arterial Phase Context X a ∈R N×W×H and venous phase context X v ∈R N×W×H , preprocessing to meet the input requirements of the network. C, N, W, H represent the channel, length, width and height of the input CT context.

[0030] Will X p , X a and X v The HU value is clipped to the range of [low, high] and normalized using the following formula:

[0031]

[0032] Among them, X can be X p , X a or X v In the embodiment of the present invention, low = -55, high = 155, and N = 9. The size of the CT image is generally (512, 512).

[0033] (2) Construct and train a liver lesion segmentation model to segment the CT images preprocessed in step (1) Make predictions and get the segmentation results. Figure 2 As shown, the liver lesion segmentation model includes a 3D region alignment module, a residual stereo attention module and a decoder; and includes the following steps:

[0034] (2.1) Use a convolutional neural network (CNN) to extract the preprocessed CT image High-level features and low-level features and the high-level features of the context corresponding to multiple periods, the high-level features of the flat sweep period f p、Advanced features of arterial phase a and advanced features of the venous phase v .

[0035] Since the input image of the pre-trained convolutional neural network (CNN) used as a feature extractor is 3-channel, that is, the input matrix specification is (3, W, H), therefore, when predicting the i-th CT image in CT, the i-1th and i+1th CT images of CT will be used and spliced ​​together with the i-th CT image to form a (3, W, H) matrix. In the present invention, in order to enable the model to capture richer context information, when predicting the i-th CT image, the Z consecutive CT images nearby will be used as input auxiliary prediction, and N can be 1, 3, 5, 7, 9... (odd number). Since the input of the present invention is multi-period medical images, the same operation as above is taken for all sequences. Therefore, the context X sent to the feature extractor p , X a and X v The input specification is (N, 3, W, H). All contexts X p , X a and X v After passing through the feature extractor, we get a context feature f of size (N, C, W / 16, H / 16) p 、f a With f v . CT image of the target layer after preprocessing After the feature extractor, a target feature with a size of (C, W / 16, H / 16) is obtained. And the corresponding low-level features The size is (C / 4, W / 4, H / 4). The formula is:

[0036]

[0037] f p =CNN(X p )

[0038] y a =CNN(X a )

[0039] f v =CNN(X v )

[0040] (2.2) Based on the high-level features f of multiple periods obtained in step (1.1) p 、f a and f v , use the 3D region alignment module (3DLocal Fusion) to align the high-level features of multiple periods and generate multi-period features fm ;like Figure 3 As shown, the following steps are included:

[0041] (2.2.1) For f a For each feature point in , we use non-local attention to calculate the alignment relationship between the feature point and the pixels in the corresponding position within a limited range in the flat-scanning period feature, and use this alignment relationship to generate f a With f p The intertemporal characteristics of m,p→a , and f is generated in the same way a With f v The intertemporal characteristics of m,v→a .

[0042] Assume that the size of the finite range is n=3, f a One of the feature points is Its characteristic during the flat sweep period is f p The neighbor feature points within a limited range of the corresponding position are The feature point Limited range feature points corresponding to the flat sweep period Alignment relationship The calculation method is:

[0043]

[0044] in, represents matrix multiplication, and softmax represents the normalized exponential function. The intertemporal characteristics The calculation method is:

[0045]

[0046] The same operation is applied to f a After each feature point on the surface, the cross-period feature f′ of the plain scan period and the arterial period is obtained. m，p→a ∈R N×C×H×W , and then f′ m，p→a With f p Add together to get the inter-period feature f m，p→a ∈R N×C×H×W .

[0047] Similarly, we can get f m，v→a ∈R N×C×H×W :

[0048]

[0049]

[0050] in, for In the arterial phase, the characteristics v Neighbor feature points within a limited range of the corresponding position, for and The same operation is applied to f a After each feature point on the m，v→a ∈R N ×C×H×W , and then f′ m，v→a With f v Add together to get the inter-period feature f m，v→a ∈R N×C×H×W .

[0051] (2.2.2) Combine the inter-period features f in step (2.2.1) m,p→a and f m，v→a , and the high-level features f in step (2.2) a , generating multi-period features f with multi-period context information m .

[0052] f m,p→a 、f m,v→a and f a Concatenate them and use a 2D convolutional network conv2d with a kernel of 1×1 to reduce the dimension. The calculation method is:

[0053] f m =conv2d(concat(f m，p→a , f m,v→a , f a );W m )

[0054] Where Wm is a learnable parameter.

[0055] (2.3) Using the Residual Volumetric Attention module, the multi-period feature f generated in step (2.2.2) m In the process of extracting multi-phase stereo features, mv ;like Figure 4 As shown, the following steps are included:

[0056] (2.3.1) Using the multi-period feature f in step (2.2.2) m , using global inter-layer attention, calculate the global inter-layer feature f I .

[0057] First, we use a 2D convolutional layer conv2d with a kernel of 1×1, and f m Generate a new mapping, calculated as follows:

[0058]

[0059]

[0060]

[0061] in, and is a learnable parameter.

[0062] Then, a global pooling operation is used to extract Q I and K I The global features of are calculated as follows:

[0063]

[0064]

[0065] Among them, GAP is global average pooling. and The sizes of are (1, C) and (N, C) respectively.

[0066] In order to utilize the positional relationship between layers, the present invention uses sine and cosine distances (GitHub-tatp22 / multidim-positional-encoding: An implementation of 1D, 2D, and 3Dpositional encoding in Pytorch and TensorFlow) to generate a 1D position encoding p1∈R N×C The inter-layer attention weight α I ∈R N×1 The calculation method is as follows:

[0067]

[0068] Among them, k represents In X a location.

[0069] Finally, the global inter-layer feature f I ∈R C×H×W The calculation method is:

[0070]

[0071] Among them, the convblock consists of a 2D convolutional network with a kernel of 1×1, a batch normalization layer, and a ReLU layer.

[0072] (2.3.2) Using the global inter-layer feature f obtained in step (2.3.1) I , using spatial attention, calculate the spatial feature f S .

[0073] Similar to (2.3.1), we first use a 2D convolutional layer conv2d with a kernel of 1×1 to generate f′ I and f m The mapping is calculated as follows:

[0074]

[0075]

[0076]

[0077]

[0078] in, and is a learnable parameter, BN is a batch normalization layer, and f I ′ represents the global inter-layer feature f I and The normalized result of the residual and .

[0079] In order to utilize the spatial position relationship, the present invention generates a 2D position encoding p2∈R using sine and cosine distances C×H×W The weight of spatial attention is α S ∈R N×HW×HW The calculation method is as follows:

[0080]

[0081] Spatial feature f S The calculation method is as follows:

[0082]

[0083] (2.3.3) Using the residual link, f in step (2.3.2) I ′, and the spatial feature f in step (2.3.2) S Combined with each other, we can obtain a multi-period stereo feature f that has both multi-period context information and stereo context information. mv .

[0084] f mv =BN(f′ I +f S )

[0085] (2.4) Use the decoder to convert the multi-period stereo feature f extracted in step (2.3.3) mv , and the low-level features in step (2.1) Combined and enlarged to the original image size (W, H), the segmentation result is generated. The decoder can directly use the decoder of Deeplab V3 plus (LC Chen, Y. Zhu, G. Papandreou et al. "Encoder-decoder with atrous separable convolution for semantic image segmentation," in Proc. Int. Conf. Euro. Conf. on Comput. Vision. (ECCV). 2018, pp. 801-818.).

[0086] The embodiment of the present invention is implemented on a machine equipped with an Intel Core i7-3770 central processor, an NVidia 3090 graphics processor and 128GB of memory, and the following results are obtained: Figure 5 The experimental results are shown in Figure 1. Among them, a, b, c and d are one layer of four cases. Compared with nn-UNet and MMNet, the present invention can give more complete segmentation results and more accurate tumor categories. In summary, the present invention fully considers the context between multiple phases, and not only gives a more complete segmentation result, but also gives the correct category.

Claims

1. An image processing method for liver lesion segmentation in non-aligned multi-phase CT, characterized in that: The steps include: (1) CT images to be predicted and its sweep period context X p , arterial phase context X a and venous phase context X b Perform preprocessing to make it meet the input requirements of the network; (2) using the trained liver lesion segmentation model to predict the preprocessed CT image; the liver lesion segmentation model includes a 3D region alignment module, a residual stereo attention module and a decoder; (2.1) Use a convolutional neural network (CNN) to extract the CT image to be predicted High-level features and low-level features and the high-level features of the context corresponding to multiple periods, the high-level features of the flat sweep period f p 、Advanced features of arterial phase a and advanced features of the venous phase v ; (2.2) Based on the high-level features f of multiple periods obtained in step (1.1) p 、f a and f b , use the 3D region alignment module to align the high-level features of multiple periods and generate multi-period features f m ; ( 2.3) Using the residual stereo attention module, the multi-period features f generated in step (2.2) m In the process of extracting multi-phase stereo features, mv ; (2.4) The multi-phase stereo features extracted in step (2.3) are mv , and the low-level features in step (2.1) Combine them, use the decoder to combine them, and enlarge them to the original image size to produce the segmentation result.

2. The image processing method for liver lesion segmentation in non-aligned multi-phase CT according to claim 1, characterized in that: The step (2.2) includes the following sub-steps: (2.2.1) For f a For each feature point in , we use non-local attention to calculate the alignment relationship between the feature point and the pixels in the corresponding position within a limited range in the flat-scanning period feature, and use this alignment relationship to generate f a With f p The intertemporal characteristics of m,p→a , and f is generated in the same way a With f v The intertemporal characteristics of m,v→a ; (2.2.2) Combine the inter-period features f in step (2.2.1) m,p→a and f m,v→a , and the high-level features f of multiple periods in step (2.2) a , generating multi-period features f with multi-period context information m .

3. The image processing method for liver lesion segmentation in non-aligned multi-phase CT according to claim 1, characterized in that: The step (2.3) includes the following sub-steps: (2.3.1) Using the multi-period feature f in step (2.2.2) m , using global inter-layer attention, calculate the global inter-layer feature f I ; (2.3.2) Using the global inter-layer feature f in step (2.3.1) I , using spatial attention, calculate the spatial feature f S ; (2.3.3) Using residual link, f I ′ and the spatial feature f in step (2.3.2) S Combined, we get the feature f that has both multi-period context information and stereo context information mv ; Among them, f I ′ is the global inter-layer feature f I and The normalized result of the residual and .

Images