A hybrid 2D and 3D convolutional neural network method for vestibular segmentation in ear CT images

By combining 2D and 3D convolutional neural networks, combined with DenseNet-BC and U-Net architectures, the problems of insufficient boundaries and lack of data in the segmentation of human ear vestibular structures are solved, and higher segmentation accuracy and efficiency are achieved. Suitable for automatic segmentation of complex ear temporal bone structures.

CN113850818BInactive Publication Date: 2024-11-01BEIJING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110992746.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-27
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing deep learning-based image semantic segmentation method is not ideal for segmentation of human ear vestibular structures, mainly due to the small structure, lack of obvious boundaries, lack of data samples, and the inability of existing methods to effectively utilize the context semantic information of three-dimensional images, resulting in segmentation. Low accuracy.

Method used

Using a hybrid 2D and 3D convolutional neural network method, 2D CNN is designed for precise segmentation of the vestibular structure, and then 3D DenseUNet network is trained using the model parameters obtained by 2D CNN training, combining DenseNet-BC and U-Net architectures to integrate low-level space Information and high-level semantic information realize accurate segmentation of the vestibular.

Benefits of technology

It improves the accuracy and efficiency of vestibular segmentation, can better utilize the semantic information of the image, reduce the optimization burden of 3D network, significantly improves segmentation performance, and is suitable for the automatic segmentation of complex ear temporal bone structures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113850818B_ABST
    Figure CN113850818B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for vestibular segmentation of ear CT images by hybrid 2D and 3D convolutional neural networks, including three steps of constructing a data set, designing a 2DCNN segmentation network based on multiple deep feature fusion strategies, and designing a 3DDenseUNet segmentation network. The 2D network uses an encoder-decoder structure as the backbone network to extract the vestibular features of the ear CT image; then the DenseNet-BC and U-Net network architectures are integrated to build a 3DDenseUNet network, which fuses low-level spatial information and high-level semantic information, and finally realizes accurate vestibular segmentation. The segmentation network designed for the vestibular structure can obtain better segmentation performance than the general segmentation method, and improve the work efficiency and quality of medical staff in the radiology department. The present invention can accurately perform automatic segmentation of key ear structures, help doctors complete a large amount of repetitive work, and effectively reduce the burden on doctors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and medical image processing, and specifically relates to computer image processing, deep learning, ear CT image diagnosis, etc. Background Art

[0002] The ear has a unique structure, precise structure, and complex functions, and most of the structure is located in the temporal bone. The temporal bone is divided into three parts: the inner ear, the middle ear, and the outer ear, mainly including the malleus, incus, stapes, the outer wall of the cochlea, the inner cavity of the cochlea, the vestibule, the anterior semicircular canal, the lateral semicircular canal, the posterior semicircular canal, the internal auditory canal, the jugular bulb, and more than 30 other organs.

[0003] The vestibule is one of the important organs of the inner ear, located between the cochlea and the semicircular canals. It appears as an irregular oval cavity on CT images and is the body's sensor for its own motion state and the head's position in space. Vestibular abnormalities are the most common inner ear diseases. Accurate segmentation of the vestibule is the basis for quantitative analysis of the ear's anatomical structure, which can help doctors diagnose abnormalities and plan treatments.

[0004] At present, image semantic segmentation and object detection based on deep learning have made significant progress, but when applied to vestibular segmentation of human ear CT images, they face the following problems:

[0005] (1) The vestibule is located between the cochlea and the semicircular canals. It has a small and precise structure with very few voxels. The temporal bone CT sequence of the ear contains 6 to 9 layers, and a single-layer CT image has only a dozen voxels. The boundaries between the vestibule and the cochlea and semicircular canals are not obvious, and there are problems of multi-scale changes at different levels and drastic changes in spatial position. When the existing deep learning-based image semantic segmentation method is applied to the segmentation task of the vestibular structure of the human ear, the effect is not ideal, and overfitting problems usually occur, resulting in low segmentation accuracy.

[0006] (2) Most of the existing medical image segmentation methods are based on two-dimensional convolutional neural networks (2D CNN) research solutions, but ear CT image data is three-dimensional. The 2D CNN method is to divide the three-dimensional body into multiple two-dimensional slices and then send them into the network for training. This method has low complexity and is easy to train, but it cannot make good use of the contextual semantic information between slices, making it difficult to improve the segmentation performance. Although the 3D CNN method can utilize the correlation between slices, it requires huge computing resources and is difficult to train. Therefore, how to give full play to the advantages of both is a key issue that needs to be addressed.

[0007] (3) Due to objective reasons such as the difficulty in collecting sample data and the high threshold for annotation, the labeled sample data is very scarce in both quantity and quality. Deep neural networks require large-scale data samples for training to achieve excellent performance. The lack of labeled samples and the imbalance of categories have brought great technical challenges to the segmentation of the vestibule.

[0008] The present invention proposes a vestibular segmentation method for ear CT images that mixes 2D and 3D convolutional neural networks based on the specific structural characteristics of the human vestibule. In order to reduce the optimization burden of the 3D network, the present invention first designs a 2D CNN to perform a relatively accurate segmentation of the vestibular structure. Then, the 3D network is trained using the model parameters obtained by 2D CNN training to output the segmentation results. With this network structure, not only can the semantic information within the slice be utilized, but also the contextual semantic information between slices can be well utilized, giving full play to the advantages of the two network structures to obtain a more accurate vestibular segmentation result. Summary of the invention

[0009] The purpose of the present invention is to propose a vestibular segmentation method for ear CT images, which adopts a deep convolutional neural network architecture to learn the semantic information within and between ear CT image slices to achieve accurate segmentation of the ear vestibular structure.

[0010] In order to achieve the above objectives, the present invention proposes the following technical solutions: first, the 2D network adopts the encoder-decoder structure as the backbone network to extract the vestibular features of the ear CT images; then, the DenseNet-BC and U-Net network architectures are integrated to build a 3D DenseUNet network, which integrates low-level spatial information and high-level semantic information, and finally realizes the accurate segmentation of the vestibule.

[0011] The scheme includes three steps: constructing a dataset, designing a 2D CNN segmentation network based on multiple deep feature fusion strategies, and designing a 3D DenseUNet segmentation network. Each step is described in detail below.

[0012] Step 1: Build the dataset

[0013] Since there is no data set for vestibular segmentation at home and abroad, it is necessary to build a vestibular segmentation data set for network training. The present invention builds a data set VestibuleDataSet. The data samples are collected clinically and include 102 cases and a total of 714 normal vestibular CT images.

[0014] Step 2: Design of 2D CNN segmentation network based on multiple deep feature fusion strategies

[0015] In view of the unclear structure of the vestibular boundary, the present invention adopts the basic architecture of encoder-decoder and designs a 2D vestibular segmentation network. The encoder and decoder are connected and cascaded to fuse the features, effectively fusing the shallow rough edge positioning information with the deep detailed pixel segmentation information.

[0016] At the same time, in order to solve the problem of multi-scale changes in different levels of the vestibule, a dilated spatial convolutional pooling pyramid (ASPP) is added at the connection of the codec and the jump connection. By using multiple dilated convolutions with different sampling rates to extract features of different receptive fields, the information of vestibules of different shapes and sizes can be accurately mined to achieve more accurate vestibular segmentation.

[0017] Step 3: 3D DenseUNet segmentation network design

[0018] Since the vestibule has the characteristics of few voxel points and unclear boundaries, the present invention designs a 3D CNN based on the U-Net network architecture, and adopts a DenseNet network structure in the encoder of the 3D CNN. The dense connection method in DenseNet ensures that the input of each layer is the output of all previous layers. This connection method strengthens the transmission of information, can maximize the use of information of different scales, and better extract the 3D features of the vestibule.

[0019] Compared with the prior art, the present invention has important theoretical research significance and application value:

[0020] 1. High segmentation accuracy. Medical images have unique characteristics different from natural images, and the vestibular structure is small and precise with unclear boundaries. Therefore, the segmentation network designed for the vestibular structure can achieve better segmentation performance than the general segmentation method.

[0021] 2. Greatly improve the work efficiency and quality of medical staff in the radiology department. The temporal bone structure of the ear is complex, containing more than 30 organs. It takes more than 20 minutes for doctors to view a set of temporal bone CT images, and they need to watch and confirm repeatedly. The present invention can accurately perform automatic segmentation of key ear structures, help doctors complete a large amount of repetitive work, and effectively reduce the burden on doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 : 2D CNN segmentation network structure diagram.

[0023] Figure 2 : 3D CNN segmentation network structure diagram. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] A hybrid 2D and 3D convolutional neural network method for vestibular segmentation of ear CT images is proposed. The implementation process of the method is as follows:

[0026] Step 1: Build the dataset

[0027] The data set used in the present invention is divided into three parts, including a training set of 82 cases with a total of 570 CT images, a validation set of 10 cases with a total of 72 CT images, and a test set of 10 cases with a total of 72 CT images. The data were annotated by radiologists with rich clinical experience, and all CT images were annotated at the voxel level of the vestibule.

[0028] In the present invention, only the axial image is used. Therefore, the spiral CT scan image of the ear temporal bone is subjected to multi-planar reconstruction and standardization operation, so that the imaging parameters are kept consistent within a certain range. All the collected CT images are obtained by spiral scanning the temporal bone CT cross section using a Phillips 64-row spiral CT scanner.

[0029] Step 2: Design of 2D CNN segmentation network based on multiple deep feature fusion strategies

[0030] The present invention uses the encoder-decoder network as the basic architecture and designs a vestibular segmentation 2D network, whose structure is as follows: Figure 1 shown.

[0031] Specifically:

[0032] (1) In order to better extract the depth features, the present invention uses 5 layers of blocks in the downsampling and upsampling processes respectively;

[0033] (2) Downblock consists of Conv layer, SEblock layer, BatchNorm layer, Relu layer and Pooling layer, and the maximum pooling (Maxpooling) is used in Downblock1, while the average pooling (Averagepooling) is used in Downblock2, Downblock3 and Downblock4. Because the shallow layer of the network contains more useless information, the use of Maxpooling can better retain the boundary feature information and reduce useless information. The deep layer of the network pays more attention to high-level semantic information, and the use of Averagepooling can retain the background information around the target;

[0034] (3) The codec is connected using a cascade operation to fuse the underlying spatial information with the high-level semantic information to improve segmentation accuracy.

[0035] Step 3: 3D DenseUNet segmentation network design

[0036] In view of the specific structural characteristics of the vestibule, the present invention integrates the DenseNet-BC and U-Net network architectures to build a 3D DenseUNet network when designing the 3D CNN. The network structure is as follows: Figure 2 shown.

[0037] Although the output of each layer of Denseblock has a fixed number of channels, its input channel number is large. Therefore, a 1×1 convolution layer is used before the 3×3 convolution layer to reduce the number of parameters, thereby improving computational efficiency. This network structure is called DenseNet-B. In order to reduce the number of feature maps and achieve downsampling, transition layers are added between Denseblocks. The transition layers include Conv layers and average pooling layers. Such a network structure is called DenseNet-C. These two network structures are combined and called DenseNet-BC. Due to the high memory consumption of 3D convolution and the limitation of GPU memory, the present invention reduces the number of convolution layers in each Denseblock to half of the original on the basis of DenseNet-121, and sets the growth rate K to 32.

[0038] Step 3.1: Training of 3D DenseUNet segmentation network

[0039] In order to reduce the optimization burden of 3D CNN, the semantic information generated by 2D CNN is added to guide the training of 3D CNN. First, the 2D CNN part is trained, and the parameters of the entire training process are randomly initialized. Then the output 8×224×224×2 of 2D CNN is converted to a 224×224×8 3D image and fused with the original 224×224×8 3D image through a connection operation. Then the fused image is sent to 3D DenseUNet, and the model parameters obtained by fixing the 2D CNN training are used to train the 3D CNN part. The parameters are also randomly initialized, and the final output is a 224×224×8 segmentation result.

[0040] According to the specific structural characteristics and segmentation requirements of the vestibule, this paper designs a vestibular segmentation method that mixes 2D and 3D convolutional neural networks. Compared with traditional segmentation methods, this method effectively combines the advantages of the two structures and can achieve accurate segmentation of the vestibular structure of the ear.

Claims

1. A method for vestibular segmentation of ear CT images using a hybrid 2D and 3D convolutional neural network, characterized in that: The 2D network uses an encoder-decoder structure as the backbone network to extract vestibular features of ear CT images; Then, we integrate the DenseNet-BC and U-Net network architectures to build a 3D DenseUNet network, which integrates low-level spatial information and high-level semantic information, and finally achieves accurate segmentation of the vestibule. The method includes three steps: constructing a dataset, designing a 2D CNN segmentation network based on multiple deep feature fusion strategies, and designing a 3DDenseUNet segmentation network; Step 1: Build a dataset; Self-built vestibular segmentation data set for network training; the present invention self-built the data set VestibuleDataSet; the data samples therein were collected through clinical practice, including 102 cases and a total of 714 normal vestibular CT images; Step 2: Design of 2D CNN segmentation network based on multiple deep feature fusion strategies; A 2D vestibular segmentation network is designed using the basic architecture of encoder-decoder. The features of encoder and decoder are fused by connection and cascading, fusing the shallow rough edge positioning information with the deep detailed pixel segmentation information. The atrous spatial convolutional pooling pyramid (ASPP) is added to the connection between the codec and the skip connection. By using multiple atrous convolutions with different sampling rates to extract features of different receptive fields, the information of vestibules of different shapes and sizes can be accurately mined to achieve more accurate vestibular segmentation. Step 3: 3D DenseUNet segmentation network design; A 3D CNN is designed based on the U-Net network architecture, and a DenseNet network structure is used in the encoder of the 3D CNN. The dense connection method in DenseNet ensures that the input of each layer is the output of all previous layers. The 3D features of the vestibule are extracted using information of different scales. The implementation process of step 2 is as follows: (1) Five layers of blocks are used in the downsampling and upsampling processes respectively; (2) Downblock consists of Conv layer, SEblock layer, BatchNorm layer, Relu layer and Pooling layer. Maxpooling is used in Downblock1, while Averagepooling is used in Downblock2, Downblock3 and Downblock4. Because the shallow layer of the network contains more useless information, Maxpooling can better retain the boundary feature information and reduce useless information. The deep layer of the network pays more attention to high-level semantic information, and Averagepooling can retain the background information around the target. (3) The codec is connected using a cascade operation to fuse the underlying spatial information with the high-level semantic information to improve segmentation accuracy; The implementation process of step 3 is as follows: In view of the specific structural characteristics of the vestibule, when designing 3DCNN, the 3DDenseUNet network was built by integrating the DenseNet-BC and U-Net network architectures; A 1×1 convolution layer is used before a 3×3 convolution layer to reduce the number of parameters and improve computational efficiency. This network structure is called DenseNet-B. In order to reduce the number of feature maps and achieve downsampling, a transition layer TransitionLayers is added between Denseblocks. The transition layer includes a Conv layer and an average pooling layer Averagepooling. This network structure is called DenseNet-C. These two network structures are combined and called DenseNet-BC. Due to the high memory consumption of 3D convolution and the limitation of GPU memory, based on DenseNet-121, the number of convolution layers in each Denseblock is reduced to half of the original number, and the growth rate K is set to 32. In order to reduce the optimization burden of 3DCNN, the semantic information generated by 2DCNN is added to guide the training of 3DCNN. First, the 2DCNN part is trained, and the parameters of the entire training process are randomly initialized. The output 8×224×224×2 of 2DCNN is then converted into a 3D image of 224×224×8 and fused with the original 3D image of 224×224×8 through a connection operation; the fused image is then sent to 3DDenseUNet, and the model parameters obtained by fixing the 2DCNN training are used to train the 3DCNN part. The parameters are also randomly initialized, and finally a segmentation result of 224×224×8 is output.