Knee MRI bone structure segmentation method based on 2D-3D feature fusion

Abstract

The present invention discloses a method for segmenting the bone structure of knee MRI based on 2D-3D feature fusion. The present invention first calculates the maximum intensity projection image (MIP) of the sagittal direction of MR data, thereby constructing a high-precision convolutional encoding and decoding neural network architecture for automatic segmentation of the knee joint: 1) a 2D bypass network for extracting global features based on MIP; 2) a 3D backbone network for extracting local detail features based on MR, and 3) a feature fusion module for 2D global information and 3D local detail information. In particular, the global feature, as position information, will be fused with the local detail network at each resolution of the encoding path to increase the contextual information of the local network and improve the segmentation accuracy. This method has been verified on a public data set, and the average dice similarity coefficients of the femur, femoral cartilage, tibia, and tibial cartilage are as high as 97.78%, 84.83%, 97.93%, and 84.80%, respectively, and the segmentation performance is significantly better than other methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of Magnetic Resonance Imaging (MRI) knee joint bone segmentation, and specifically relates to a knee joint MRI bone structure segmentation method based on 2D-3D feature fusion. Background Art

[0002] Automatic segmentation of the knee bone structure (tibia, tibial cartilage, femur, femoral cartilage) is an important task when diagnosing knee diseases based on MRI images. However, general segmentation methods are extremely challenging when accurately segmenting joint structures because they cannot take into account both large hard bones and small cartilage. Currently, the main problems with knee bone segmentation are: 1. The bone boundaries of the knee bone structure in MRI images are extremely irregular, the gaps at the joints are narrow, and the cartilage wear state is complex. 2. Due to the limitation of current GPU video memory, the three-dimensional (3D) convolutional encoder-decoder network (CED) cannot be trained on the entire image, and the width of the 3D CED network based on block training cannot take into account both the spatial context information and the network's expression ability, while the 2D CED network loses a lot of three-dimensional spatial information.

[0003] In practical applications, in order to solve the problems of blurred bone boundaries, irregular morphology and current hardware limitations of the knee bone structure in MR images, domestic and foreign scholars have proposed a variety of methods based on deep learning and deep learning plus anatomical priors, which have improved the segmentation performance of the knee bone structure to a certain extent. Deep learning is a widely used machine learning method. This method extracts potential features from the target data by constructing a reasonable convolutional neural network to accurately characterize the distribution and characteristics of such data. In recent years, deep learning methods have shown excellent ability to handle complex pattern recognition problems in different fields. [1-3] Compared with traditional machine learning methods, deep learning methods have obvious advantages. They can automatically extract rotation-invariant, translation-invariant, and robust features in images. This feature enables them to effectively cope with the complex conditions of the knee bone structure in MRI images and ensure accurate segmentation of the structure. Summary of the invention

[0004] The purpose of the present invention is to solve two problems existing in the segmentation of knee joint bone structure based on MR images: First, the bone structure of the knee joint has extremely irregular bone boundaries, narrow gaps at the joint joints, and complex cartilage wear. Second, due to the limitation of the current GPU video memory, the 3D convolutional codec network (CED) cannot be trained on the entire image, and the width of the 3D CED network based on block training cannot take into account both the spatial context information and the network's expression ability, while the 2D CED network loses a large amount of three-dimensional spatial information. In order to be able to segment the knee joint bone structure efficiently and accurately at the same time, the present invention proposes a new end-to-end segmentation method for knee joint magnetic resonance images based on a segmentation network based on a 2D-3D feature hierarchical fusion based on a convolutional codec structure. In view of the problem of low segmentation accuracy of the previous deep learning method 3D-Unet, the new method uses a 2D network to extract the global positioning features of the maximum density projection image of the volume data to enrich the global context information missing from the 3D network to improve the segmentation accuracy of the bone structure. In addition, this method of supplementing global information can effectively deal with the problem of a large number of semantic missing caused by the small input blocks of the 3D network due to insufficient GPU video memory. In order to successfully fuse 2D global context information and 3D local detail features, the present invention proposes a new feature fusion module, so that the global context information and local detail features can maintain the same size and relatively consistent features, and can be successfully fused, effectively improving the segmentation accuracy.

[0005] The technical solution of the present invention is specifically described as follows.

[0006] A method for segmenting the bone structure of knee MRI based on 2D-3D feature fusion is proposed. The maximum intensity projection image (MIP) of 3D volume data in the sagittal direction is calculated and generated, and a segmentation network based on 2D-3D feature hierarchical fusion is constructed to segment multiple bone structures on the knee MR image. The input of the segmentation network based on 2D-3D feature hierarchical fusion is the 3D MR image of the knee joint, and the output is the segmentation results of the corresponding femur, femoral cartilage, tibia, and tibial cartilage.

[0007] The segmentation network based on 2D-3D feature hierarchical fusion includes a local 3D convolutional encoding and decoding segmentation network, a global 2D convolutional encoding and decoding positioning network and a 2D-3D feature fusion module. The local 3D convolutional encoding and decoding segmentation network is referred to as the 3D-CED network, and the global 2D convolutional encoding and decoding positioning network is referred to as the 2D-CED network. Among them:

[0008] The 3D-CED network and the 2D-CED network use 3D volume data and maximum density projection images as input for feature extraction and encoding, respectively. The 3D-CED network and the 2D-CED network have completely corresponding encoding in the sagittal position. Both the 3D-CED network and the 2D-CED network use encoder structures and the same network structure, including 4-level encoding modules and 4-level decoding modules respectively. Each level of encoding module includes a composite module of two convolutional layers, a Relu activation layer, a group normalization layer, and a maximum pooling module. Each level of decoding module includes two convolutional modules and an upsampling module. In addition to the bottom-level 4th-level encoding module, the features extracted by each level of encoding module are used as the input of the feature fusion module for context information fusion, and the fusion is input into the corresponding decoder module.

[0009] The feature fusion module fuses the features of the encoder of the 3D-CED network with those of the encoder of the 2D-CED network layer by layer, and inputs them into the decoder of the 3D-CED network layer by layer, so that when the 3D-CED network focuses on the detailed features of the corresponding local position, the 2D-CED network provides the global context information required by the 3D-CED network.

[0010] In the present invention, in the decoding module, the convolution kernel of the up-sampling module in the last level of decoding module is 1×1, and the convolution kernel of the up-sampling module in the other three levels of decoding modules is 3×3×3.

[0011] In the present invention, the fusion mode of the feature fusion module is as follows:

[0012] Assume that the maximum density projection global context feature map f is obtained from the 2D-CED network G , with a size of B×C×W×H, obtain the global context feature map f of the maximum density projection from the 3D-CED network L , size is B×C×W c ×H c ×D c , B is the batch size, C is the number of channels, W and H are f G Width and height, W c , H c , D c f L The width, height and depth of the 2D-CED network and the 3D-CED network have the same B and C. For the convenience of discussion, G and f L The dimensions are recorded as W×H and W c ×H c ×D c , before entering the encoders at all levels of the 3D-CED network, randomly crop I to obtain I cThe corresponding three-dimensional index position recorded at the time is i, j, k. After passing through the first-level encoder, the new three-dimensional index position is,

[0013]

[0014]

[0015]

[0016] Among them, (H K ,W K ,D K ) is the convolution kernel size of the convolution layer, i.e. (3,3,3), and S is the stride of the maximum pooling layer, 2;

[0017] Using the calculated i n , j n Crop the features of the area corresponding to the local semantic information feature map output by the 3D-CED network encoder from the global context feature map Size (W c *H c ), the feature of this region is a 2D feature, and the stacking method of this feature is f L Each sagittal position of generates the corresponding context feature, namely:

[0018]

[0019] here, The size is (W c *H c *D c ), a global context feature map with the same size as the 3D feature map is obtained, and the two feature maps are added in the form of linear superposition to form the encoded image overall context information and local semantic information.

[0020]

[0021] Among them, ω is an adjustable parameter, is the 3D feature reconstructed by the 2D-CED encoder, f L It is the three-dimensional feature extracted by the 3D-CED encoder; the encoded image overall context information and local semantic information f are sent to the decoder network through skip connections to obtain better segmentation performance.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention proposes a novel end-to-end knee bone structure segmentation network, which directly outputs the segmentation result corresponding to the input image without the need for additional anatomical prior information or post-processing steps;

[0024] (2) The present invention can simultaneously and efficiently segment multiple bone structures on the knee joint MR image, including the femur, femoral cartilage, tibia, and tibial cartilage, thereby improving practicality;

[0025] (3) The segmentation accuracy of the present invention is high, the main bone structure is close to the manual marking of experts, and the cartilage structure is greatly improved compared with previous methods.

[0026] (4) The network architecture of the present invention is less related to specific tasks and can be used for the segmentation of other three-dimensional medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is an illustration of the overall framework of the invention method.

[0028] Figure 2 Illustration of the feature fusion module structure.

[0029] Figure 3 The segmentation visualization results of the method of the present invention. (a) sagittal MR image of the knee joint (b) gold standard (c) segmentation result (d) 3D rendering of the segmentation result. DETAILED DESCRIPTION

[0030] The overall structure of the method of the present invention is as follows Figure 1 As shown, unlike previous deep learning-based methods, in order to efficiently and accurately segment the knee bone structure, we proposed a new segmentation network architecture based on 2D-3D feature hierarchical fusion.

[0031] 1. Segmentation network based on 2D-3D feature hierarchical fusion

[0032] Previously, deep learning-based segmentation of knee bone structure was mainly based on 2D convolutional neural networks, 2D or 3D network hybrid cascades, and neural network segmentation methods combined with anatomical prior information. The new network is a segmentation network based on the hierarchical fusion of features from 2D and 3D networks. The detailed structure is as follows: Figure 1 As shown in the figure, the input of the network is the 3D MR image of the knee joint, and the output is the corresponding segmentation results of the femur, femoral cartilage, tibia, and tibial cartilage. The network is mainly composed of three parts, namely the local 3D convolutional encoder-decoder segmentation network, the global 2D convolutional encoder-decoder positioning network, and the 2D-3D feature fusion module, which are described as follows:

[0033] (1) The local 3D convolutional encoding and decoding segmentation network, i.e., the main segmentation network, includes 4 levels of encoding modules and 4 levels of decoding modules. Each level of encoding module includes two convolution kernels of 3 3The convolutional layer, Relu activation layer, group normalization layer composite module and a maximum pooling module. The number of convolutional feature channels N corresponding to the encoding module are {{16, 32}, {32, 64}, {64, 128}, {128, 256}} respectively. Each level of decoding module includes two convolutional modules with a convolution kernel of 3×3×3 and an upsampling module. The number of convolutional feature channels N corresponding to the decoding module are {{256, 128}, {128, 64}, {64, 32}, {32, 16}} respectively. The last decoding module is different from the previous ones. The upsampling module is replaced with a 1×1 convolutional layer to compress the number of channels to output the predicted segmentation map.

[0034] In particular, except for the bottom-level 4th-level encoding module, the features extracted by each level of the encoding module will be used as the input of the feature fusion module for contextual information fusion, and then input into the corresponding decoder module to supplement the global context information missing in the block-based training 3D-CED model.

[0035] (2) Global 2D convolutional encoder-decoder localization network, i.e., global information extraction network. To ensure feature consistency, the 2D network uses the same structure as the 3D network. We use the 2D-CED network to extract and localize global contextual information of the knee bone structure in the sagittal plane. Compared with taking any single sagittal section, the maximum density projection in the sagittal direction is an image with less overall contextual information loss of the three-dimensional knee MRI image, and the network can capture potential three-dimensional spatial information features from the maximum density projection image.

[0036] (3) Feature fusion module: The features of the 3D network and the 2D network encoder are fused layer by layer through the feature fusion module and then input into the 3D-CED decoder layer by layer to complete the global context information missing from the 3D-CED encoder.

[0037] 2. Combining global positioning features and local detail features to improve segmentation accuracy

[0038] Previous deep learning-based methods had low segmentation accuracy because cartilage is a thin structure and cartilage imaging requires a high spatial resolution, which results in thinner layers and larger image volumes of knee MRI images. Due to the current GPU memory limitations, 3D convolutional encoding and decoding networks cannot be trained on the entire image. The width and depth of the 3D CED network based on block training are limited by the block size and cannot take into account both spatial context information and the network's expressive power. The 2D CED network can only learn information on a single slice, losing a lot of three-dimensional spatial information. [4-5]. In order to overcome these difficulties, first, we calculate and generate the maximum density projection image of the 3D volume data in the sagittal direction, and input the maximum density projection image and 3D volume data into the 2D global network and the 3D local detail network for feature calculation respectively. Secondly, both the 2D and 3D networks adopt a codec structure, and the 2D and 3D networks are completely encoded in the sagittal position, so that when the 3D network focuses on the corresponding local position detail features, the 2D network provides the global context information required by the 3D network. Third, after each level of the corresponding encoder, the 2D-3D feature fusion module will fuse the input 2D global context information and 3D local detail features, and input the fusion result into the decoding path, assisting the decoder to obtain rich context information, effectively improving the segmentation accuracy of the knee bone structure.

[0039] 3. Feature fusion method

[0040] Figure 2 This is a diagram illustrating the structure of the feature fusion module.

[0041] Assume that the maximum density projection global context feature map f is obtained from the 2D-CED network G , with a size of B×C×W×H, obtain the global context feature map f of the maximum density projection from the 3D-CED network L , size is B×C×W c ×H c ×D c , B is the batch size, C is the number of channels, W and H are f G Width and height, W c ×H c ×D c f L The width, height and depth of the 2D network and the 3D network have the same B and C. For the convenience of discussion, G and f L The dimensions are recorded as W×H and W c ×H c ×D c Before entering the 3D network encoder, randomly crop I to obtain I c The corresponding three-dimensional index position recorded at the time is i, j, k. After passing through the first-level encoder, the new three-dimensional index position is,

[0042]

[0043]

[0044]

[0045] Among them, (H K ,W K ,DK ) is the convolution kernel size of the convolution layer, that is, (3,3,3), and S is the stride 2 of the maximum pooling layer.

[0046] Using the calculated i n , j n Crop the features of the area corresponding to the local semantic information feature map output by the 3D network encoder from the global context feature map Size (W c *H c ). At this time, the regional feature is a 2D feature, and the stacking method is f L Each sagittal position of generates the corresponding context feature, namely:

[0047]

[0048] here, The size is (W c *H c *D c ), and obtain a global context feature map with the same size as the 3D feature map. Adding the two feature maps in the form of linear superposition constitutes the encoded image's overall context information and local semantic information.

[0049]

[0050] Among them, ω is an adjustable parameter, is the 3D feature reconstructed by the 2D-CED encoder, f L It is the three-dimensional feature extracted by the 3D-CED encoder. The encoded image overall context information and local semantic information f are fed into the decoder network through skip connections to obtain better segmentation performance.

[0051] Figure 3 The segmentation visualization results of the method of the present invention are shown in Figure 1. The structures in green, yellow, brown, and blue represent the femur, femoral cartilage, tibia, and tibial cartilage, respectively. (a) shows the sagittal MR image of the knee joint, (b) shows the gold standard of the data, (c) shows the segmentation result of the bone structure of the image by the method of the present invention, and (d) is the 3D rendering of the segmentation result.

[0052] The performance comparison results of the proposed method and other methods on the same data set are shown in Table 1. In Table 1, the evaluation index is the same as that in the literature. [6] , ASD represents average surface distance, RSD represents root mean square surface distance, VD represents volume data difference, and VOE represents volume data overlap error (mean + variance). As can be seen from Table 1, the score of our method in knee bone structure segmentation is significantly improved, especially in VOE and VD of cartilage structure, which are effectively improved to varying degrees.

[0053] Table 1 Performance comparison results

[0054]

[0055] This method was verified on a public dataset, and the average dice similarity coefficient (DSC) of femur, femoral cartilage, tibia, and tibial cartilage were as high as 97.78%, 84.83%, 97.93%, and 84.80%, respectively. The segmentation performance was significantly better than other methods on the same dataset.

[0056] Our experimental results on the same public dataset show that our method achieves the best segmentation accuracy under the Dice coefficient and challenge evaluation indicators.

[0057] References

[0058] [1] J.Long, E.Shelhamer, and T.Darrell, "Fully convolutional networks forsemantic segmentation," inProceedings of the IEEE conference on computervision and pattern recognition, 2015, pp.3431-3440.

[0059] [2] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," in 2015 Medical Image Computing and Computer-Assisted Intervention, Cham, N. Navab, J.

[0060] Hornegger, WM Wells, and AF Rangi, Eds., 2015: Springer International Publishing, pp. 234-241.

[0061] [3] A.Abdulkadir,S.S.Lienkamp,T.Brox,and O.Ronneberger,"3D U-Net:Learning DenseVolumetric Segmentation from Sparse Annotation,"in 2016Medical Image Computing and Computer-AssistedIntervention,Cham,S.Ourselin,L.Joskowicz,M.R.Sabuncu,G.Unal,and W.Wells,Eds.,2016:SpringerInternationalPublishing,pp.424-432.

[0062] [4]A.Raj,S.Vishwanathan,B.Ajani,K.Krishnan,and H.Agarwal,"Automaticknee cartilage segmentationusing fully volumetric convolutional neuralnetworks for evaluation of osteoarthritis,"in 2018 IEEE 15thInternationalSymposium on Biomedical Imaging,4-7 April 2018,pp.851-854.

[0063] [5]Z.Zhou,G.Zhao,R.Kijowski,and F.Liu,"Deep convolutional neuralnetwork for segmentation of knee jointanatomy,"(in eng),Magn Reson Med,vol.80,no.6,pp.2759-2770,Dec 2018.

[0064] [6]T.Heimann,B.J.Morrison,M.A.Styner,M.Niethammer,and S.Warfield,"Segmentation of knee images:agrand challenge,"in Proc.MICCAI Workshop onMedical Image Analysis for the Clinic,2010,pp.207-214.

[0065] [7]G.Vincent,C.Wolstenholme,I.Scott,and M.Bowes,"Fully automaticsegmentation of the knee joint usingactive appearance models,"Medical ImageAnalysis for the Clinic:A Grand Challenge,vol.1,p.224,2010.

[0066] [8]F.Liu,Z.Zhou,H.Jang,A.Samsonov,G.Zhao,and R.Kijowski,"Deepconvolutional neural network and 3Ddeformable approach for tissuesegmentation in musculoskeletal magnetic resonance imaging,"Magneticresonancein medicine,vol.79,no.4,pp.2379-2391,2018.

[0067] [9]E.B.Dam,M.Lillholm,J.Marques,and M.Nielsen,"Automatic segmentationof high-and low-field kneeMRIs using knee image quantification with data fromthe osteoarthritis initiative,"Journal of Medical imaging,

[0068] vol.2,no.2,p.024001,2015.

[0069]

[10] F.Ambellan,A.Tack,M.Ehlke,and S.Zachow,"Automated segmentation ofknee bone and cartilagecombining statistical shape knowledge andconvolutional neural networks:Data from the OsteoarthritisInitiative,"MedicalImage Analysis,vol.52,pp.109-118,2019.

Claims

1. A knee joint MRI bone structure segmentation method based on 2D-3D feature fusion, characterized in that: The maximum intensity projection image MIP of the 3D volume data in the sagittal direction is calculated, and a segmentation network based on 2D-3D feature hierarchical fusion is constructed to segment multiple bone structures on the knee joint MR image; the input of the segmentation network based on 2D-3D feature hierarchical fusion is the 3D MR image of the knee joint, and the output is the corresponding segmentation results of the femur, femoral cartilage, tibia, and tibial cartilage; The segmentation network based on 2D-3D feature hierarchical fusion includes a local 3D convolutional encoder-decoder segmentation network, a global 2D convolutional encoder-decoder positioning network and a 2D-3D feature fusion module; the local 3D convolutional encoder-decoder segmentation network is referred to as the 3D-CED network, and the global 2D convolutional encoder-decoder positioning network is referred to as the 2D-CED network; where: The 3D-CED network and the 2D-CED network use 3D volume data and maximum density projection images as input for feature extraction and encoding, respectively. The 3D-CED network and the 2D-CED network have completely corresponding encoding in the sagittal position. Both the 3D-CED network and the 2D-CED network use encoder structures and the same network structure, including 4-level encoding modules and 4-level decoding modules respectively. Each level of encoding module includes a composite module of two convolutional layers, a Relu activation layer, a group normalization layer, and a maximum pooling module. Each level of decoding module includes two convolutional modules and an upsampling module. Except for the bottom-level 4th-level encoding module, the features extracted by each level of encoding module are used as the input of the 2D-3D feature fusion module for context information fusion, and then input into the corresponding decoder module after fusion. The 2D-3D feature fusion module fuses the features of the encoder of the 3D-CED network with those of the encoder of the 2D-CED network layer by layer, and inputs them into the decoder of the 3D-CED network layer by layer, so that when the 3D-CED network focuses on the detailed features of the corresponding local position, the 2D-CED network provides the global context information required by the 3D-CED network.

2. The knee joint MRI bone structure segmentation method according to claim 1, characterized in that: In the decoding module, the convolution kernel of the upsampling module in the last level decoding module is 1×1, and the convolution kernel of the upsampling modules in the other three levels of decoding modules is 3×3×3.

3. The knee joint MRI bone structure segmentation method according to claim 1, characterized in that: The fusion method of the 2D-3D feature fusion module is as follows: Assume that the maximum density projection global context feature map f is obtained from the 2D-CED network G , with a size of B×C×W×H, obtain the global context feature map f of the maximum density projection from the 3D-CED network L , size is B×C×W c ×H c ×D c , B is the batch size, C is the number of channels, W and H are f G Width and height, W c , H c , D c f L The width, height and depth of the 2D-CED network and the 3D-CED network have the same B and C. For the convenience of discussion, G and f L The dimensions are recorded as W×H and W c ×H c ×D c Before entering the encoders of each level of the 3D-CED network, the knee MR image I is randomly cropped to obtain I c The corresponding three-dimensional index position recorded at the time is i, j, k. After passing through the first-level encoder, the new three-dimensional index position is, Among them, (H K ,W K ,D K ) is the convolution kernel size of the convolution layer, i.e. (3,3,3), and S is the stride of the maximum pooling layer, 2; Using the calculated i n , j n Crop the features of the area corresponding to the local semantic information feature map output by the 3D-CED network encoder from the global context feature map Size (W c *H c ), the feature of this region is a 2D feature, and the stacking method of this feature is f L Each sagittal position of generates the corresponding context feature, namely: here, The size is (W c *H c *D c ), a global context feature map with the same size as the 3D feature map is obtained, and the two feature maps are added in the form of linear superposition to form the encoded image overall context information and local semantic information. Among them, ω is an adjustable parameter, is the 3D feature reconstructed by the 2D-CED encoder, f L It is the three-dimensional feature extracted by the 3D-CED encoder; the encoded image overall context information and local semantic information f are sent to the decoder network through skip connections to obtain better segmentation performance.

Images