A brain MR image segmentation method based on high-order neural network
By combining high-order convolution with traditional convolution, the high-order features and prior knowledge of anatomical structures of brain MR images are learned, which solves the problems of low segmentation accuracy and slow speed of deep networks and achieves accurate and fast brain MR image segmentation.
Patent Information
- Application Number
- CN202211271582.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-18
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2042-10-18
AI Technical Summary
In existing technologies, deep network segmentation accuracy is not high, multi-map segmentation methods are slow, features are difficult to define manually, and traditional convolution operations are difficult to describe the complex relationships between voxels in brain MR images, resulting in suboptimal brain MR image segmentation.
We employ high-order convolution operations to establish the correlation between voxels within the local receptive field of convolution, combine traditional convolution operations to learn low-order mean features and high-order prior knowledge of brain anatomy, adaptively fuse feature representations through a channel attention model, and use high-order neural networks to segment brain MR images.
It achieves accurate and rapid brain MR image segmentation, provides quantitative analysis indicators, and improves segmentation accuracy and speed.
Smart Images
Figure CN115526894B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of images, and particularly relates to a brain MR image segmentation method based on a high-order neural network. BACKGROUND
[0002] In recent years, deep learning methods have achieved great success in medical image segmentation and computer-aided diagnosis. Deep learning networks can automatically learn the context information of images through convolution operations, thereby obtaining more optimal image feature representations. Among numerous neural networks, end-to-end-based neural networks are widely used in image segmentation. For example, the U-Net structure is widely used in image segmentation and mainly consists of two parts of encoding and decoding. The encoding part is responsible for learning high-level feature representations of images, and the decoding part is used for restoring images to the original resolution. In addition, there is a skip connection between the encoding part and the decoding part of the U-Net for fusing high-level texture features and high-resolution spatial features into the U-Net network. The use of the skip connection can effectively improve the image segmentation result. However, due to the very complex anatomical structure of the human brain, the traditional convolution operation is difficult to describe the complex relationship between the voxels of the brain MR image, and can only depict the low-order brain MR image texture features, thereby resulting in a suboptimal result of the brain MR image segmentation task. Therefore, there is an urgent need for a brain MR image segmentation method capable of obtaining accurate brain segmentation results. SUMMARY
[0003] The application provides a brain MR image segmentation method based on a high-order neural network to solve the problems of low segmentation precision of deep networks, slow speed of multi-atlas segmentation methods and difficulty in manually defining features in the prior art.
[0004] The application provides a brain MR image segmentation method based on a high-order neural network, which specifically comprises the following steps:
[0005] (1) For a pre-acquired brain MR image, a high-order convolution operation is used to establish the correlation between voxels in a convolution local receptive field and extract high-order features of the image;
[0006] (2) A traditional convolution operation is used to learn low-order mean features of the local receptive field and obtain texture feature representations of the image;
[0007] (3) A high-order convolution operation is used to learn high-order brain anatomical structure prior knowledge provided by atlas images, and the input is a plurality of atlas image label maps registered to the space of the image to be segmented;
[0008] (4) A traditional convolution operation is used to learn brain anatomical structure prior knowledge provided by atlas images, and the input is a plurality of atlas image label maps registered to the space of the image to be segmented;
[0009] (5) The channel fusion weight is obtained by the channel attention model, the feature representation learned by steps (1) to (4) and the prior knowledge are fused to obtain a better brain image feature representation;
[0010] (6) The brain magnetic resonance image is segmented by the residual high-order neural network using the brain image feature representation fused in step (5).
[0011] Further, the high-order convolution in step (1) is defined as:
[0012]
[0013] wherein w k and f i represent the kth parameter of the convolution kernel in the receptive field and the ith feature on the corresponding feature map, represent the jth high-order feature output of the corresponding high-order neural network output.
[0014] Further, the traditional convolution in step (2) is defined as:
[0015]
[0016] wherein w k and f i represent the kth parameter of the convolution kernel in the receptive field and the ith feature on the corresponding feature map, represent the jth low-order feature output of the corresponding high-order neural network output.
[0017] Further, the step (5) is implemented as follows:
[0018] The high-order neural network has two inputs, i.e. a target brain MR image and multiple label images of an atlas image; for the target image, two 3*3*3 traditional convolutional layers are used to extract low-order context information of the target image; meanwhile, a 3*3*3 high-order convolutional layer is used to extract high-order pair-dependent features; then, a 3*3*3 traditional convolutional layer is used to learn context information of the extracted high-order pair-dependent features; a set of atlas label images are used to learn anatomical prior knowledge to guide the segmentation process; the two 3*3*3 traditional convolutional layers are used to learn anatomical prior features from the set of atlas label images; meanwhile, a 3*3*3 high-order convolutional layer and a 3*3*3 traditional convolutional layer are used to learn high-order anatomical prior features for brain ROI segmentation; after each convolutional layer, batch normalization and a ReLU nonlinear mapping unit are used; finally, a channel attention feature fusion block is introduced to adaptively fuse the features; specifically, first, the learned features are concatenated in the channel dimension, Global pooling operation is performed, then the dimension is compressed to one quarter through a fully connected operation, then a ReLU nonlinear mapping unit is used, then the dimension is restored to the original dimension through a fully connected operation, finally, a Sigmoid nonlinear mapping unit is used to obtain a weight vector; the weight vector is multiplied with the concatenated original features in the channel to obtain the output features used as the input of the next layer of neural network.
[0019] Further, the step (6) is implemented as follows:
[0020] First, a 2*2*2 max-pooling operation with a step size of 2 is used to down-sample the fused multi-order features; the remaining high-order neural network is composed of a high-order feature learning path and a low-order feature learning path, the high-order feature learning path is used to model the pair-dependent relationship of voxels within a local receptive field, and the low-order feature learning path is used to learn high-level context features of the image; each encoding path block in the high-order feature learning path is composed of a high-order convolutional block and a traditional convolutional layer, wherein the high-order convolutional block is used to learn high-order pair-dependent features between voxels within a local receptive field, and both the high-order and traditional convolutional operations use a 3*3*3 convolution kernel; each encoding path block in the low-order feature learning path is composed of two traditional convolutional layers, and batch normalization and ReLU nonlinear operation are further performed after each convolutional layer; then, a channel attention fusion block is used to fuse the feature maps extracted by the high-order feature learning path and the low-order feature learning path; finally, a 2*2*2 max-pooling operation with a step size of 2 is performed to down-sample the fused feature maps;
[0021] In the decoding path of the high-order feature learning path, each decoding path block is composed of a deconvolution layer, a high-order convolution block and a traditional convolution layer; the deconvolution layer with a 2x2x2 kernel is used to restore the low-resolution feature map to a high-resolution feature map, and at the same time, the feature map derived by the deconvolution layer is combined with the corresponding high-resolution feature map in the encoding path by using a skip connection; then, the output feature map is input to a 3x3x3 high-order convolution layer and a 3x3x3 traditional convolution layer; in the decoding path of the low-order feature learning path, each block is composed of a 2x2x2 deconvolution layer and two 3x3x3 traditional convolution layers; similarly, the feature map derived by the deconvolution layer is combined with the corresponding high-resolution feature map in the encoding path by using a skip connection; batch normalization and ReLU are further performed after each convolution layer; in addition, a channel attention module after each decoding path block is used to fuse the up-sampled high-order feature map and the advanced context feature map into the network for brain region of interest segmentation.
[0022] Finally, a 1x1x1 convolution layer with a softmax nonlinear unit is used to generate a segmentation probability map, and then a maximum a posteriori probability is used to obtain a final segmentation image.
[0023] Further, the structure of the high-order convolution in step (1) is as follows:
[0024] The input features are divided into G groups, and the proposed high-order convolution operation is performed on each group respectively, the learned high-order pair-dependent feature maps are connected, and the channel number of the feature map is changed by using a 1x1x1 convolution operation.
[0025] Beneficial effects: Compared with the prior art, the beneficial effects of the present application: the present application uses high-order convolution to learn the high-order relationship between image voxels, and uses anatomical structure prior information to obtain accurate and rapid brain MR image segmentation results; brain MR image segmentation is an important step in brain image analysis, which can provide quantitative indicators for subsequent analysis. BRIEF DESCRIPTION OF DRAWINGS
[0026] Figure 1 is a high-order neural network structure diagram proposed by the present application;
[0027] Figure 2 is a high-order convolution structure diagram proposed by the present application;
[0028] Figure 3 The hippocampus segmentation result surface distance reconstruction map of the brain segmentation method based on the high-order neural network;
[0029] Figure 4 The segmentation result of the brain MR image based on the brain segmentation method based on the high-order neural network. DETAILED DESCRIPTION
[0030] The application will be described in further detail below with reference to the drawings.
[0031] Brain MR image segmentation is an important step of brain structure analysis, and the medical anatomical structure of the brain is very complex, and the traditional convolution can only depict the low-order texture features of the brain MR image, in order to represent the relationship between the voxels of the brain MR image, the application provides a brain MR image segmentation method based on a high-order neural network, for a pre-acquired brain MR image, a high-order convolution operation is used to establish the correlation between the voxels in the local receptive field of the convolution, and high-order features of the image are extracted; a traditional convolution operation is used to learn the low-order mean features of the local receptive field, and the texture features of the image are obtained; a high-order convolution operation is used to learn the high-order anatomical structure prior knowledge of the atlas image, and the input is a plurality of atlas image label maps registered to the space of the image to be segmented; a traditional convolution operation is used to learn the anatomical structure prior knowledge of the atlas image, and the input is a plurality of atlas image label maps registered to the space of the image to be segmented; a channel attention model is used to adaptively obtain the weight of channel fusion, the learned feature representation and prior knowledge are fused, and a better brain image feature representation is obtained; through the remaining high-order neural network, the fused brain image feature representation is used to segment the brain magnetic resonance image.
[0032] The high-order feature representation between the voxels of the image is learned through a high-order convolution operation, the input is an MR image to be segmented, and the used high-order convolution is defined as follows:
[0033]
[0034] Wherein, w k And f i Respectively represent the kth parameter of the convolution kernel in the receptive field and the ith feature on the corresponding feature map, Represent the jth high-order feature output of the corresponding high-order neural network output.
[0035] The low-order feature representation of the image is learned through a traditional convolution operation, the input is a brain MR image to be segmented, and the used traditional convolution is defined as follows:
[0036]
[0037] Wherein, w k And f i Respectively represent the kth parameter of the convolution kernel in the receptive field and the ith feature on the corresponding feature map, Represent the jth low-order feature output of the corresponding high-order neural network output.
[0038] respectively, and prior knowledge, and then fuse the learned feature representations and prior knowledge using a channel attention model, as shown in Figure 1 The high-order neural network proposed by the present application has two inputs, i.e. a target brain MR image and a plurality of label images of an atlas image; for the target image, two 3x3x3 traditional convolutional layers are used to extract low-order context information of the target image. At the same time, a 3x3x3 high-order convolutional layer is used to extract high-order pair-dependent features; then, a 3x3x3 traditional convolutional layer is used to learn the context information of the extracted high-order pair-dependent features; a set of atlas label images is used to learn anatomical prior knowledge to guide the segmentation process. In particular, two 3x3x3 traditional convolutional layers are used to learn anatomical prior features from a set of atlas label images. At the same time, a 3x3x3 high-order convolutional layer and a 3x3x3 traditional convolution are used to learn high-order anatomical prior features for brain ROI segmentation. Each convolutional layer is followed by batch normalization and a ReLU nonlinear mapping unit. In order to better fuse these learned features, a channel attention feature fusion block is introduced to adaptively fuse these features. Specifically, first, the learned features are concatenated in the channel dimension, Global pooling is performed, then the dimension is compressed to one quarter using a fully connected operation, then a ReLU nonlinear mapping unit is used, then the dimension is restored to the original dimension using a fully connected operation, and finally a Sigmoid nonlinear mapping unit is used to obtain a weight vector; the weight vector is multiplied by the concatenated original features in the channel to obtain the output feature used as the input of the next layer of neural network. Therefore, the low-order context features of the image, the high-order pair-dependent features and the anatomical prior knowledge are adaptively fused together for subsequent brain MR image segmentation process.
[0039] Through the remaining high-order neural network, the fused brain image feature representation is used to segment the brain magnetic resonance image. First, a 2x2x2 max-pooling operation with a step size of 2 is used to downsample the fused multi-order features. The remaining high-order neural network consists of two paths, i.e. 1) a high-order feature learning path for modeling the pair-dependent relationship of voxels within a local receptive field, and 2) a low-order feature learning path for learning high-level context features of the image.
[0040] Specifically, each encoding path block in the high-order feature learning path consists of a high-order convolution block and a traditional convolution layer. The high-order convolution block is used to learn the high-order pairwise dependency features between voxels within a local receptive field. Both the high-order and traditional convolution operations use 3x3x3 convolution kernels. Each encoding path block in the low-order feature learning path consists of two traditional convolution layers. Batch normalization and ReLU nonlinear operation are further performed after each convolution layer. Then, a channel attention fusion block is employed to fuse the feature maps extracted by the high-order feature learning path and the low-order feature learning path. Finally, a 2x2x2 max-pooling operation with a stride of 2 is performed to down-sample the fused feature maps.
[0041] In the decoding path of the high-order feature learning path, each block consists of a deconvolution layer, a high-order convolution block and a traditional convolution layer. The deconvolution layer with a 2x2x2 kernel is used to restore the low-resolution feature maps to high-resolution feature maps. Meanwhile, the feature maps derived from the deconvolution layer are combined with the corresponding high-resolution feature maps in the encoding path by using the skip connection. Then, the output feature maps are input to a 3x3x3 high-order convolution layer and a 3x3x3 traditional convolution layer. In the decoding path of the low-order feature learning path, each block consists of a 2x2x2 deconvolution layer and two 3x3x3 traditional convolution layers. Again, the feature maps derived from the deconvolution layer are combined with the corresponding high-resolution feature maps in the encoding path by using the skip connection. Batch normalization and ReLU are further performed after each convolution layer. In addition, a channel attention module after each block is used to fuse the up-sampled high-order feature maps and high-level context feature maps into the network for brain region of interest segmentation.
[0042] Finally, a 1x1x1 convolution layer with a softmax nonlinear unit is used to generate a segmentation probability map, and then the maximum a posteriori probability is used to obtain the final segmentation image.
[0043] As Figure 2 shown in the high-order convolution structure diagram, the specific structure is as follows:
[0044] The input features are first divided into G groups. Then, the proposed high-order convolution operation is performed on each group respectively. Finally, the learned high-order pairwise dependency feature maps are concatenated. In order to further improve the flexibility of the proposed model, 1x1x1 convolution operation is used to change the number of channels of the feature maps.
[0045] The present application is verified on the international standard dataset ADNI and LONI-LPBA40 dataset. As Figure 3As shown in the figure, the surface distance reconstruction map of the hippocampus segmentation result of the ADNI data set obtained by the application is shown.The application achieves high precision in hippocampus segmentation, and the results show that the average surface distance of the hippocampus segmentation of the application reaches 0.295±0.045, reaching the current optimal result.
[0046] As Figure 4 As shown in the figure, the segmentation result on the brain MR image of the LONI-LPBA40 data set obtained by the application is shown.The results on the LONI-LPBA40 data set show that the Dice index of the application can reach 0.8167;at the same time, the speed of the application in segmenting one MR image reaches the order of seconds.
[0047] The application uses the brain MR image segmentation method and system based on high-order neural network to segment the brain MR image, uses the high-order feature of the input MR image and atlas image obtained by the proposed high-order convolution learning, and combines the low-order feature of the traditional convolution learning to obtain accurate brain MR image segmentation results.
[0048] The above embodiments are only specific embodiments of the application, and cannot limit the protection scope of the application, and any modification made according to the technical idea of the application on the basis of the technical scheme falls within the protection scope of the application.
Claims
1. A brain MR image segmentation method based on a high-order neural network, characterized in that, The method comprises the following steps: (1) for the pre-acquired brain MR image, the correlation between the voxels in the convolution local receptive field is established by high-order convolution operation, and the high-order features of the image are extracted; (2) the low-order mean feature of the local receptive field is learned by traditional convolution operation, and the texture feature representation of the image is obtained; (3) the high-order brain anatomical structure prior knowledge provided by the atlas image is learned by high-order convolution operation, and the input is multiple atlas image label maps registered to the space of the image to be segmented; (4) the brain anatomical structure prior knowledge provided by the atlas image is learned by traditional convolution operation, and the input is multiple atlas image label maps registered to the space of the image to be segmented; (5) the channel fusion weight is adaptively obtained by the channel attention model, the feature representation and prior knowledge learned in steps (1) to (4) are fused, and the brain image feature representation is obtained; (6) the brain magnetic resonance image is segmented by the remaining high-order neural network using the brain image feature representation fused in step (5); The high-order convolution in step (1) is: where w k and f i represent the k-th parameter of the convolution kernel within the receptive field and the i-th feature on the corresponding feature map, respectively, denotes the j-th high-order feature output of the corresponding high-order neural network output. The structure of the high-order convolution is as follows: The input features are divided into G groups, the proposed high-order convolution operation is performed on each group respectively, the learned high-order pair-dependent feature maps are connected, and the channel number of the feature map is changed by 1×1×1 convolution operation; The implementation process of step (6) is as follows: First, use 2×2×2 max pooling operation with a step of 2 to downsample the fused multi-order features; the remaining high-order neural network is composed of a high-order feature learning path and a low-order feature learning path, the high-order feature learning path is used to model the pair-dependent relationship between the voxels in the local receptive field, and the low-order feature learning path is used to learn the high-level context features of the image; Each encoding path block in the high-order feature learning path is composed of a high-order convolution block and a traditional convolution layer, wherein the high-order convolution block is used to learn the high-order pair-dependent features between the voxels in the local receptive field, and the high-order and traditional convolution operations both use 3×3×3 convolution kernel; each encoding path block in the low-order feature learning path is composed of two traditional convolution layers, and batch normalization and ReLU nonlinear operation are further performed after each convolution layer; then, a channel attention fusion block is used to fuse the feature maps extracted by the high-order feature learning path and the low-order feature learning path; finally, 2×2×2 max pooling operation with a step of 2 is performed to downsample the fused feature maps; In the decoding path of the high-order feature learning path, each decoding path block is composed of a deconvolution layer, a high-order convolution block and a traditional convolution layer; the deconvolution layer with a 2x2x2 kernel is used to restore the low-resolution feature map to a high-resolution feature map, and meanwhile, the feature map derived from the deconvolution layer is combined with the corresponding high-resolution feature map in the encoding path through the use of a skip connection; then, the output feature map is input to a 3x3x3 high-order convolution layer and a 3x3x3 traditional convolution layer; in the decoding path of the low-order feature learning path, each block is composed of a 2x2x2 deconvolution layer and two 3x3x3 traditional convolution layers; similarly, the feature map derived from the deconvolution layer is combined with the corresponding high-resolution feature map in the encoding path through the use of a skip connection; batch normalization and ReLU are further performed after each convolution layer; in addition, a channel attention module after each decoding path block is used to fuse the up-sampled high-order feature map and the advanced context feature map into the network for brain region of interest segmentation; Finally, a 1x1x1 convolution layer with a softmax nonlinear unit is used to generate a segmentation probability map, and then a maximum a posteriori probability is used to obtain a final segmentation image.
2. The brain MR image segmentation method based on high-order neural network according to claim 1, characterized in that, The traditional convolution in step (2) is: where w k and f i represent the kth parameter of the convolution kernel within the receptive field and the ith feature on the corresponding feature map, respectively, denotes the jth low-order feature output of the corresponding high-order neural network output. 3.The brain MR image segmentation method based on high-order neural network according to claim 1, characterized in that, The implementation process of step (5) is as follows: The high-order neural network has two inputs, i.e. a target brain MR image and multiple label images of an atlas image; for the target image, two 3x3x3 traditional convolution layers are used to extract low-order context information of the target image; meanwhile, a 3x3x3 high-order convolution layer is used to extract high-order pair-dependent features; Then, a 3x3x3 traditional convolution layer is used to learn the context information of the extracted high-order pair-dependent features; A set of atlas label images is used to learn anatomical prior knowledge to guide the segmentation process; the two 3x3x3 traditional convolution layers are used to learn anatomical prior features from the set of atlas label images; meanwhile, a 3x3x3 high-order convolution layer and a 3x3x3 traditional convolution are used to learn high-order anatomical prior features for brain ROI segmentation; Batch normalization and ReLU nonlinear mapping units are used after each convolution layer; finally, a channel attention feature fusion block is introduced to adaptively fuse these features; specifically, first, the learned features are concatenated in the channel dimension, Global pooling is performed, then the dimension is compressed to one quarter through a fully connected operation, then a ReLU nonlinear mapping unit is used, then the dimension is restored to the original dimension through a fully connected operation, and finally a Sigmoid nonlinear mapping unit is used to obtain a weight vector; the weight vector is multiplied with the concatenated original features in the channel to obtain the output features used as the input of the next layer of neural network.