An automatic segmentation method for the temporal bone inner ear bone cavity structure based on a coarse-to-fine dense encoder-decoder network

Abstract

A method for automatic segmentation of the temporal bone inner ear bone cavity structure based on a coarse-to-fine dense encoding and decoding network belongs to the field of medical imaging. The present invention adopts a coarse-to-fine framework to first perform coarse segmentation on the anatomical structure to be segmented in the temporal bone region and calculate the coordinates of the center point. Around the center point, the image is expanded to the outside to a region that can completely contain the inner ear bone cavity structure, and a part of the background information can be retained as a sub-region for further precise segmentation. In the fine segmentation stage, dense connection modules are introduced in the encoding process to extract more sufficient features, and hollow convolutions are added to the dense connection modules, so that the segmentation algorithm obtains a larger receptive field for the segmentation target, so as to extract more sufficient surrounding features and spatial information. In the decoding stage, the features extracted in the encoding stage are upsampled by transposed convolution, and after each transposed convolution, a dense connection module is introduced to enhance the reuse of the decoded information. The segmentation of the present invention is more accurate.

Description

Technical Field

[0001] The invention belongs to the field of medical image processing, and in particular relates to a method for automatically segmenting the temporal bone inner ear bone cavity structure using a coarse-to-fine dense encoding and decoding network. Background Art

[0002] Temporal bone CT is an important reference for doctors to examine ear diseases. In the temporal bone area, it is divided into three areas: the outer ear, the middle ear, and the inner ear, which contain more than 30 tiny anatomical structures. Among them, the inner ear area is one of the important areas of the temporal bone, which helps the human body hear sounds and maintain balance. This area mainly includes structures such as the cochlea, vestibule, external semicircular canal, posterior semicircular canal, and anterior semicircular canal. These structures are mainly composed of connected bone cavity structures, and they play different roles in ensuring human hearing and balance. The cochlea works together with the outer ear and the middle ear. External sound waves cause the fluid in the cochlear wall to flow, causing the fine hairs in the cochlea to bend and move, converting the motion signals of the sound waves into electrical signals and sending them to the brain through the auditory nerve; the vestibule and the three semicircular canals play an important role in maintaining human balance. The vestibule is the connecting part of the cochlea and the semicircular canals, which contains more fluid and fine hairs, and senses human movement through the flow of fluid; the three semicircular canals are at right angles to each other. When the human body moves, the fluid in the semicircular canals flows, promoting the movement of the internal fine hairs, and helping people feel the direction of movement. Therefore, the inner ear area is an important reference for symptoms such as hearing loss and dizziness. In recent years, with the development of medical imaging technology, temporal bone CT image data has grown rapidly. However, due to the lack of automatic analysis tools, a large amount of temporal bone CT image data is difficult to be effectively applied to related analysis and research.

[0003] Medical image segmentation is a complex and critical basic step in automatic processing and analysis of medical images, medical research and clinical diagnosis. Its purpose is to segment out parts of medical images that have certain special significance, and provide a reliable reference for clinical diagnosis, surgical planning, clinical teaching and other tasks. Segmentation is an important step in medical image analysis. Accurate segmentation can not only reduce the workload of doctors, but also help doctors further understand the characteristics of anatomical structures and conduct physiological analysis, such as the number of cochlear modiolus, vestibule size, semicircular canal curvature angle and other physical quantities.

[0004] The segmentation of small objects in anatomical structures has always been a challenging task in medical image segmentation. In the anatomical structure of the temporal bone, the anatomical structure that needs to be segmented accounts for less than 1% of the voxels in a complete CT data. In addition, due to the particularity of the anatomical structure of the temporal bone, there is usually no obvious boundary between the anatomical structure and the surrounding anatomical structure, which brings challenges to the segmentation of the anatomical structure of the temporal bone.

[0005] Classic medical image segmentation algorithms include threshold method, region growing and other methods. However, due to the unclear boundaries between the anatomical structures of temporal bone CT, the precise structures and small volumes, the above methods are difficult to obtain accurate segmentation results. Therefore, there is an urgent need for more efficient and accurate segmentation algorithms.

[0006] In recent years, with the development of deep learning technology, many medical image segmentation algorithms based on convolutional neural networks have emerged. In order to fully capture the intra-layer information and inter-layer information, such algorithms usually use a three-dimensional U-shaped neural network with an encoding and decoding structure. Due to the limitation of computing resources, such algorithms usually use a sliding window to sequentially divide the CT blocks in a complete CT data to predict the voxel category. Since the temporal bone to be segmented is small and easily disturbed by the complex background, the segmentation speed and accuracy are poor.

[0007] The present invention designs a temporal bone CT image segmentation algorithm of a densely connected encoding and decoding network from a coarse to fine framework, which is used for the automatic segmentation of the ear bone cavity structure in the temporal bone region. A coarse to fine framework is adopted, and the anatomical structure to be segmented in the temporal bone region is firstly segmented by an efficient and lightweight segmentation algorithm, and the approximate area is obtained. The center point coordinates are calculated according to the foreground point set predicted by the coarse segmentation result. Then, around the center point, the image is expanded to the area that can completely contain the inner ear bone cavity structure and can retain a part of the background information as a sub-area for further precise segmentation. In the fine segmentation stage, dense connection modules are introduced in the encoding process to extract more sufficient features, and hollow convolution is added to the dense connection module, so that the segmentation algorithm obtains a larger receptive field for the segmentation target to extract more sufficient surrounding features and spatial information. In the decoding stage, the features extracted in the encoding stage are upsampled by transposed convolution, and after each transposed convolution, a dense connection module is introduced to strengthen the reuse of the decoded information. For other parts, the 3D deep supervision mechanism and 3D multi-pooling feature fusion strategy adopted in the patent (publication number: 110544264A, publication date: December 6, 2019) will continue to be used in the fine segmentation network to guide the training of the segmentation algorithm. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of existing medical image small target segmentation methods. The segmentation of small anatomical structures has always been a challenging task in medical image segmentation. This problem also exists in the anatomical structure of the temporal bone. In a temporal bone CT sequence, the number of voxels of the target anatomical structure accounts for less than 1% of the number of voxels in the complete data. Therefore, the complex and large background will have a negative impact on the segmentation result of the target to be segmented. At the same time, due to the particularity of temporal bone CT, the boundaries between the anatomical structures are not clear, which brings challenges to the segmentation of anatomical structures. In response to the above problems, a coarse-to-fine automatic segmentation framework for key anatomical structures of the temporal bone is designed. In the fine segmentation algorithm, a new innovation is proposed on the basis of the small target segmentation method of key anatomical structures of the temporal bone based on the 3D deep supervision mechanism proposed in the previously submitted patent (publication number: 110544264A, publication date: December 6, 2019). By introducing densely connected blocks and dilated convolutions, a larger target receptive field and more sufficient features are obtained, and the temporal bone anatomical structure is automatically segmented more accurately.

[0009] The present invention is achieved by adopting the following technical means:

[0010] A method for automatic segmentation of the temporal bone inner ear bone cavity structure from coarse to fine dense network. The method is divided into three stages: coarse positioning of the temporal bone inner ear bone cavity structure based on coarse segmentation, extraction of candidate regions of the temporal bone inner ear bone cavity structure, and fine segmentation of the temporal bone inner ear bone cavity structure. The specific process is as shown in the attached manual. Figure 1 shown.

[0011] The method specifically comprises the following steps:

[0012] 1) Rough positioning of the temporal bone inner ear bone cavity structure based on rough segmentation:

[0013] Step 1: Use a conventional medical image segmentation method with a lightweight network structure and fast segmentation speed to roughly segment the temporal bone inner ear bone cavity structure. In the coarse segmentation model training stage, a 48×48×48 cube (the actual physical size of the CT image in the front-back and left-right directions is the product of the number of pixels and the pixel spacing, and the up-down direction is the product of the slice span and the layer spacing. Therefore, as long as the actual physical distance is greater than 24mm, the cube containing the inner ear bone cavity structure can be completely extracted. In this method, the pixel spacing and layer spacing of the data used are both 0.5mm. Therefore, the voxel block size corresponding to the extraction of a 24mm×24mm×24mm cube is 48×48×48) is truncated if the cube contains the annotation of the target structure. min The value is set to T min , HU value is greater than T max The setting is Tmax , where T min and T max The value range is between the lowest HU value and the highest HU value. Usually, in temporal bone CT, the HU value of air is -1024, and the HU value of bone is above 300. The data can be truncated to the lowest -1024 and the highest 2000 or above to completely retain the temporal bone area (this method uses 2347 as the truncation upper limit) for further temporal bone analysis, so as to reduce the influence of irrelevant background on the segmentation of the target structure, and the truncated HU value is normalized to a data distribution with a mean of 0 and a variance of 1, and sent to the coarse segmentation network for training. Otherwise, the same size cube is re-extracted until the annotation containing the anatomical structure is extracted within this range.

[0014] Step 2: The complete temporal bone CT image data is divided into multiple groups of blocks in order according to the overlapping sliding window method, and then sent to the trained coarse segmentation network for segmentation, and the segmentation results are superimposed and restored to the same size as the complete temporal bone CT in the input order. In the coarse segmentation test, in order to find a balance between coarse positioning time and segmentation accuracy, the overlap rate is selected as 2, and the input complete temporal bone CT image data is divided into multiple groups of overlapping blocks and input into the segmentation network for prediction.

[0015] Step 3: Based on the voxel-level annotations of each structure, the absolute median difference method is used to remove outliers generated in the rough segmentation. Assume that the absolute median difference calculation is as shown in formula (1):

[0016] MAD = median(|Xi-median(X)|) (1) where X is the set of all points predicted as foreground points, X i ∈X is the i-th point in the point set X, median(·) is the median of the point set, and the absolute median difference algorithm process is as follows:

[0017] (1) Calculate the median (X) of all predicted foreground point coordinates;

[0018] (2) Calculate the absolute deviation value abs(X) of each predicted prospect point from the median i -median(X));

[0019] (3) Calculate the median absolute deviation (MAD) in (2);

[0020] (4) Divide the value in (2) by the value in (3) to obtain a set of distances Dis from the center of all predicted foreground points based on MAD. The calculation formula is as follows:

[0021]

[0022] (5) The points whose maximum Dis value in the three dimensions of x, y, and z is greater than the threshold Th are removed as outliers. Th is the screening threshold of the ratio Dis between the absolute deviation value and the median of the absolute deviation value. It can be selected according to the proportional relationship between the true foreground point and the outlier point. When the ratio of the current point to the outlier point is small, Th is selected to be larger, which means that the position difference of the foreground point is more robust and more foreground points will be retained, but the outliers may not be completely removed; when the ratio of the current point to the outlier point is large, Th is selected to be smaller, which means that the foreground point screening conditions are more stringent. While removing outliers and retaining most of the foreground points, the correct foreground point may be deleted. The conventional MAD algorithm threshold value is between 1 and 10. When removing the rough positioning of the five inner ear bone cavity structures, the outliers will interfere with the center coordinates of the target area. When removing the outliers, the number of foreground points is large, so the Th value is selected to be smaller. This method uses 3.5 as the screening threshold to remove outliers far away from the target area.

[0023] 2) Extraction of candidate regions of key anatomical structures of the temporal bone:

[0024] The first step is to count the extraction size of the candidate region of the temporal bone inner ear bone cavity structure. Based on the statistical voxel-level annotation data, the maximum and minimum values ​​of all annotated voxel points of the temporal bone inner ear bone cavity structure in the three dimensions of x, y, and z are counted, and the voxel coordinate span of the anatomical structure is preliminarily calculated by the difference between the maximum value and the minimum value of x, y, and z. Since the pixel spacing and inter-slice spacing parameters between CTs may be inconsistent, extracting voxel regions of the same size cannot guarantee the extraction of temporal bone regions of the same size. Therefore, according to the pixel spacing and inter-slice spacing of each CT, the actual physical bounding box size corresponding to the inner ear bone cavity structure of each CT is calculated, and the maximum actual physical size of each structure is extended outward to 24mm×24mm×24mm. Taking the temporal bone CT with inter-slice spacing and pixel spacing of 0.5mm as an example, a 48×48×48 cube can be taken to ensure that the target segmentation structure is completely surrounded while meeting the input of the segmentation algorithm, as the extraction size of the candidate region.

[0025] In the second step, the region of interest is extracted by combining the roughly located center point of the anatomical structure to be segmented and the size information of the prior bounding box of the anatomical structure. The region of interest is described by the center point of the region and the three-dimensional size. The cube data is extracted outward from the center point of the key anatomical structure of the temporal bone to be segmented predicted in the first stage. The three-dimensional size of the cube is calculated according to the bounding box size corresponding to each anatomical structure of the temporal bone to be segmented in the first step. The extracted sub-region is used as the candidate region for further precise segmentation, and the position of the extracted sub-region is recorded.

[0026] 3) Fine segmentation of the temporal bone inner ear bone cavity structure:

[0027] The fine segmentation stage specifically includes two processes: encoding and decoding. The overall network architecture is shown in the attached manual. Figure 2 shown.

[0028] a) Coding stage

[0029] Step 1: Data truncation and normalization. The 48×48×48 voxel length sub-region of the temporal bone inner ear bone cavity structure candidate region extracted in stage 2) is sent to the precise segmentation algorithm. In order to reduce the influence of complex irrelevant background on the segmentation target, the CT value of the sub-region of the CT image data is truncated according to the HU value distribution range of the temporal bone CT. For the temporal bone CT, the sub-region CT value less than T min The value is truncated to T min , greater than T max The value is truncated to T max , T min With T max The selection of is consistent with the rough positioning of the temporal bone inner ear bone cavity structure based on rough segmentation in the first stage, and is normalized to a data distribution with a mean of 0 and a variance of 1.

[0030] Step 2: Use a densely connected network with dilated convolution to extract features. The 48×48×48 sub-region that completely contains the temporal bone inner ear bone cavity structure to be segmented after truncating and normalizing the data in the first step is sent to the network. In the encoding stage, a total of 3 groups of densely connected modules are designed. This module can enhance the transmission of information in the network and reduce the gradient disappearance or gradient explosion. At the same time, the extracted features can be reused to obtain rich semantic information. This module consists of batch normalization-rectified linear unit-convolution layer, splicing and bottleneck layer, as shown in the attached instructions. Figure 3 As shown in the figure, all convolutions in the dense connection module have a size of 3×3×3. The batch normalization-rectified linear unit-convolution layer in the dense connection block is composed of three operations: batch normalization, rectified linear unit, and convolution layer; concatenation is to cascade feature maps at the channel level; the bottleneck layer is used to reduce the number of feature maps output by the dense connection block. In the third group of dense connection modules, a dilated convolution module is used. The schematic diagram of this module is shown in the attached description. Figure 4 , the convolution output can increase the receptive field, that is, contain a larger range of spatial information around the anatomical structure. This helps to extract the tiny temporal bone key anatomical structure and its surrounding spatial information in three-dimensional data.

[0031] Step 3: Multi-pooling feature fusion. The multi-pooling feature fusion strategy of the patent (publication number: 110544264A, publication date: December 6, 2019) is adopted. Batch normalization-corrected activation unit-convolution layer is used after the output of each level of dense connection module. In order to prevent overfitting, a Dropout layer is usually used thereafter, and the Dropout rate is set to 0.5. After that, 3D maximum pooling and 3D average pooling are used at the same time, and the results after pooling are spliced. 3D maximum pooling can retain the edge features of volumetric data, and 3D average pooling can retain the background information of volumetric data. The splicing of the two can provide rich feature information for subsequent segmentation.

[0032] b) Decoding stage

[0033] Step 1: Use transposed convolution and dense connection modules to upsample features. During the decoding process, transposed convolution with a dense connection module is used to decode semantic information. For the tensor feature data with a size of 12×12×12 in the last layer of the encoding stage, two transposed convolutions are used for upsampling to restore the tensor feature data to the original input size of 48×48×48. Different from the conventional method of using ordinary convolution to extract features after transposed convolution, in the decoding stage of the present invention, a dense connection block is used instead of an ordinary convolution layer after the two transposed convolutions, so that the features after the transposed convolution can be used more efficiently, thereby better predicting the voxel category.

[0034] Step 2: 3D deep supervision mechanism. The 3D deep supervision mechanism of the patent (publication number: 110544264A, publication date: December 6, 2019) is adopted. In the encoding stage, the features output by the first densely connected network block are extracted using 64 convolution kernels, and then pass through a 1×1×1 convolution, followed by a softmax layer to output the auxiliary segmentation result. The second layer of the decoding stage performs a convolution operation on the spliced ​​features to further extract features. The obtained features are first subjected to a transposed convolution to improve the resolution, and then passed through a softmax layer with a 1×1×1 convolution kernel to obtain the second auxiliary segmentation result. The last layer of the decoding stage performs a convolution operation on the spliced ​​features using different convolution kernels, and then outputs the prediction result of the backbone network. The prediction result of the backbone network and the prediction result of the branch network jointly guide the training of the network. During the network training process, the loss function of the backbone network and the loss function of the branch network together constitute a joint objective function, which includes Coefficient (DSC) loss function and cross entropy loss function. The DSC loss function is defined as shown in formula (3):

[0035]

[0036] Where X and Y represent the predicted voxel and the real target voxel respectively, n represents the number of categories of the target to be segmented (including the background), x i With y i Respectively represent the number of target labeled voxels contained in the predicted voxel data and the true target voxel data. The weight introduced into the cross entropy loss function is recorded as W, as shown in formula (4):

[0037]

[0038] Where N k Represents the number of target voxel labels in the voxel data to be segmented, N c Represents the number of all voxels in the voxel data to be segmented. The cross entropy loss function is shown in formula (5):

[0039]

[0040] Based on the loss function defined above, the joint objective function is constructed as shown in formula (6):

[0041]

[0042] Where m is the number of supervised hidden layers, λ 1k and λ 2k is the hyperparameter of the kth supervised hidden layer loss function, and its value range is an open interval of 0 to 1. This is because the joint loss function should be based on the loss function of the network backbone and supplemented by the loss function of the hidden layer. In the present invention, m is 2, and the hyperparameter λ 1k and λ 2k The values ​​are 0.6 and 0.3 respectively, L k and H k They are the kth supervised hidden layer DSC loss function and cross entropy loss function respectively. The target loss function is constructed based on the loss functions of the backbone network and the branch network to jointly guide the network training, reduce the gradient disappearance, and speed up the convergence of the network.

[0043] Step 3: Use the precise segmentation algorithm to predict the segmentation result with a resolution of 48×48×48. According to the candidate region extraction position recorded in the second stage, restore it to the corresponding position in the complete CT as the final segmentation result.

[0044] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0045] The present invention proposes an automatic segmentation algorithm for the bone cavity structure of the temporal bone inner ear with a dense encoding and decoding network from coarse to fine. In view of the small size of the anatomical structure of the temporal bone, a lightweight and efficient general medical image anatomical segmentation algorithm is first used to roughly segment the anatomical structure to be segmented, obtain the spatial position distribution range of the anatomical structure, and extract small sub-regions from the center of the distribution range to narrow the candidate area for precise segmentation of the structure. On the basis of narrowing the area to be segmented, an algorithm for precise segmentation with dense connections and dilated convolution is used to perform more accurate automatic segmentation of the bone cavity structure of the temporal bone. The method adopts a coarse-to-fine segmentation framework, by first roughly segmenting the anatomical structure to be segmented to delineate the candidate area, and then accurately segmenting within the candidate area, further solving the problem that the anatomical structure target in the temporal bone region is small and is greatly affected by the complex background, and improving the segmentation accuracy and segmentation speed of the bone cavity structure of the temporal bone inner ear. The invention can replace manual delineation to achieve computer automatic segmentation of the bone cavity structure of the temporal bone inner ear.

[0046] Features of the present invention:

[0047] 1. Aiming at the complex and fine structure of the inner ear bone cavity in temporal bone CT, the algorithm proposes a coarse-to-fine segmentation framework, which uses a lightweight coarse segmentation method to determine the candidate area for accurate segmentation of the anatomical structure, and uses a more complex fine segmentation algorithm in the area for further accurate segmentation;

[0048] 2. In the encoding stage of the fine segmentation algorithm, the algorithm introduces a densely connected module with dilated convolution to capture the complex spatial information around the anatomical structure to be segmented; in the decoding stage, transposed convolution is combined with densely connected blocks to enhance the reuse of decoded information. The segmentation performance of the tiny anatomical structure of the temporal bone is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 , the overall framework diagram of the temporal bone inner ear bone cavity structure segmentation algorithm from coarse to fine;

[0050] Figure 2 , the overall network architecture diagram of the fine segmentation algorithm of the temporal bone inner ear bone cavity structure;

[0051] Figure 3 , schematic diagram of quasi-dense connection module and dense connection module with dilated convolution;

[0052] Figure 4 , Schematic diagram of standard convolution and dilated convolution;

[0053] Figure 5 , the segmentation result of the temporal bone inner ear bone cavity structure. DETAILED DESCRIPTION

[0054] The following is a description of the implementation examples of the present invention in conjunction with the accompanying drawings:

[0055] With the approval of the Ethics Committee of Beijing Friendship Hospital Affiliated to Capital Medical University, we collected 64 manually standardized temporal bone CT data. According to the requirements of the hospital, the patient information in all data was desensitized. Among the 64 normal human temporal bone CT data, 33 were male and 31 were female, with an average age of 44 years. Experienced radiologists from Friendship Hospital were invited to perform voxel-level annotation of the 5 inner ear bone cavity structures in the temporal bone CT. In the experiment, 56 data were used as the training set and 8 data were used as the training set.

[0056] The data preprocessing adopted by the present invention includes resampling of CT images and annotation data.

[0057] In order to avoid inconsistent CT data distribution caused by different pixel spacing and layer spacing of CT image data due to different brands and parameters of CT acquisition equipment, the CT image data was resampled to a unified pixel spacing and layer spacing of 0.5 mm using the B-spline interpolation algorithm; the annotation data was also resampled to a unified pixel spacing and layer spacing of 0.5 mm using the nearest neighbor algorithm.

[0058] 1) Rough positioning of key anatomical structures of the temporal bone:

[0059] Step 1: Use the lightweight 3D Unet algorithm, which is widely used in the field of medical image segmentation, to roughly segment the 5 inner ear bone cavities. Use 56 training sets and 8 test sets for segmentation testing. During the training phase, randomly select one of the 56 complete temporal bone CT image data and the corresponding inner ear bone cavity structure annotation data, and randomly extract a cube area with an actual physical size of 24mm at the same position in the CT image data and the annotation. Since all CTs have a uniform pixel spacing and layer spacing of 0.5mm after resampling, the corresponding voxel area of ​​this area is unified into a 48×48×48 cube. If the annotated cube contains the annotated foreground of the target structure, that is, the extracted annotated cube contains a label value of 1, then the HU value of the extracted temporal bone CT image cube is truncated, and the T min Set to -1024, T maxIt is set to 2347 to reduce the impact of irrelevant background on the target structure segmentation, and the truncated HU value is normalized to a data distribution with a mean of 0 and a variance of 1, and finally sent to the coarse segmentation network for training. Otherwise, cubes of the same size are re-extracted until they are labeled within this range. The batch size of the network training is set to 2, the initial learning rate is 0.0001, and the momentum coefficient is set to 0.5. The DSC loss function and the weighted cross entropy loss function are used. After every 20 batches of training are completed, a data set is randomly selected from the validation set for validation, and the DSC coefficient is used as the evaluation result. The formula is shown in formula (7). The model is saved every 2000 times to guide the loss convergence of the precise segmentation model, and the DSC index is verified to be close to saturation. The model with the highest verification result index is selected as the optimal model for precise segmentation network training.

[0060]

[0061] Step 2: In the coarse segmentation algorithm testing phase, the complete 8 test data are fed into the 3D Unet algorithm for segmentation testing in the form of overlapping sliding windows with an overlap rate of 2 and a cube size of 48×48×48 for each sliding window.

[0062] Step 3: For the rough segmentation results, the absolute median difference algorithm is used with a threshold of 2.5 to remove abnormal outliers in the 3D Unet segmentation;

[0063] Step 3: Use the center point of the voxel-level segmentation result of the inner ear bone cavity structure as the center point for further candidate region extraction.

[0064] 2) Extraction of candidate regions of key anatomical structures of the temporal bone:

[0065] Step 1: Count the size of the key anatomical candidate regions of the temporal bone CT. Based on the voxel-level annotation data of the inner ear bone cavity structure, in the 56 training data sets, the maximum and minimum values ​​of all annotated voxel points of the inner ear bone cavity structure in the three dimensions of x, y, and z are counted, and the corresponding actual physical distance is calculated by the difference between the maximum value of x, y, and z and the minimum value of x, y, and z, and extended outward to meet the three-time downsampling requirements of the 3D Unet segmentation algorithm. Finally, the actual physical distance of the candidate bounding box area of ​​the inner ear bone cavity structure is determined to be 24×24×24, and the corresponding voxel size is 48×48×48.

[0066] Step 2: Based on the center point of the inner ear bone cavity structure in the first stage, expand 12mm in the positive and negative directions of x, y, and z, that is, 24 voxel distances, as the candidate area for further precise segmentation.

[0067] 3) Accurate segmentation of key anatomical structures of the temporal bone:

[0068] The precise segmentation stage of the key anatomical structures of the temporal bone is mainly divided into the encoding stage and the decoding stage.

[0069] a) Coding stage

[0070] Step 1: The 48×48×48 sub-region of the inner ear bone cavity extracted in stage 2) is fed into the precise segmentation algorithm. The sub-region CT values ​​of the CT image data with CT values ​​less than -1024 are truncated to -1024, and those greater than 2347 are truncated to 2347, and normalized to a data distribution with a mean of 0 and a variance of 1.

[0071] Step 2: Densely connected modules combined with dilated convolutions for feature extraction. The inner ear bone cavity structure is small in size. In order to more fully extract the structure to be segmented, densely connected blocks are used for feature extraction. Each layer of densely connected modules consists of a set of batch normalization-rectified linear units-convolutional layers, splicing and bottleneck layers. In the encoding stage, we used three 3-layer densely connected modules. In order to make the convolution output have a larger receptive field and extract a larger range of spatial information, the convolution layer of the third densely connected module is replaced with a dilated convolution with a dilation rate of 2.

[0072] Step 3: Multi-pooling feature fusion. The multi-pooling feature fusion strategy is adopted. Batch normalization-corrected activation unit-convolution layer is used after the output of each level of dense connection module. In order to prevent overfitting during training, a dropout layer with a dropout rate of 0.5 is usually used. After that, 3D maximum pooling and 3D average pooling are used at the same time to splice the pooled results. 3D maximum pooling can retain the edge features of volumetric data, and 3D average pooling can retain the background information of volumetric data. The splicing of the two can provide rich feature information for subsequent segmentation.

[0073] b) Decoding stage

[0074] i. Use transposed convolution and densely connected modules to upsample features.

[0075] Step 1: The features output by the first, second and third densely connected network blocks in the encoding stage are F1, F2 and F3, with resolutions of 48×48×48, 24×24×24 and 12×12×12 respectively. A transposed convolution operation is applied to F3, with a convolution step of 2 and border padding of 0. After the first set of transposed convolutions, the output feature T2 has a size of 24×24×24;

[0076] Step 2: Concatenate the output F2 of the second densely connected block in the encoding stage with T2 to form a new feature group C2. Extract the features of feature group C2 through densely connected block 4 in the decoding stage to obtain feature D2.

[0077] Step 3: The output F1 of the first densely connected block in the encoding stage passes through a 3D convolution layer to obtain 64 features M1. The transposed convolution operation is used to upsample D2 to 48×48×48. The output feature is recorded as T1. The features F1, M1, and T1 are concatenated to obtain the feature group C1, where the concatenation of M1 and F1 is concatenated in the form of short connections and long connections respectively to reduce the semantic gap between low-level spatial features and high-level semantic features. The feature group C1 is input into the densely connected block 5 in the decoding stage to increase the reuse of features and obtain feature D1. Finally, D1 is channel-filtered through a 1×1×1 bottleneck convolution as the final output.

[0078] ii.3D deep supervision mechanism.

[0079] Step 1: In the decoding stage, feature D2 is transposed and convolved to upsample the feature data resolution to 48×48×48, and then convolved with a 1×1×1 bottleneck to output two feature cubes. The probability of each voxel being the target anatomical structure is calculated based on softmax, which is recorded as auxiliary prediction aux_pred1.

[0080] Step 2: Use 1×1×1 convolution on the feature group M1 in the encoding stage, and calculate the probability of each voxel being the target anatomical structure according to softmax, which is recorded as auxiliary prediction aux_pred2;

[0081] Step 3: For feature group D1, input 1×1×1 bottleneck convolution to perform channel screening, and calculate the classification probability of each voxel according to softmax, which is recorded as the main prediction main_pred;

[0082] Step 4: Joint loss guides network training. The network prediction results aux_pred1, aux_pred2 and main_pred are compared with the manually annotated gold standard GT, and the cross entropy loss and DSC loss are calculated. The loss obtained from the auxiliary prediction results and the loss obtained from the main prediction results together constitute the joint loss to guide the network training.

[0083] The following is the training and testing process of the precise segmentation network:

[0084] a) Model training

[0085] In the model training phase, the 48×48×48 subvolume extracted in the first phase is input into the segmentation algorithm for training. During the training process, the batch size is set to 2, the initial learning rate is set to 0.0001, and the momentum coefficient is set to 0.5. After each 20 batches of training are completed, a data set is randomly selected from the validation set for verification. The DSC coefficient is used as the evaluation result. The model is saved every 2000 times until the loss of the accurate segmentation model converges and the verification DSC index is close to saturation. The model with the highest verification result index is selected as the optimal model for accurate segmentation network training.

[0086] b) Model testing

[0087] In the model testing phase, the 48×48×48 sub-region extracted in the second stage was accurately segmented and predicted. After the prediction was completed, the accurate segmentation result was restored to the corresponding position according to the position recorded in the sub-region extracted in the first stage as the final segmentation result of a temporal bone CT image.

[0088] The subjective diagram of the algorithm's segmentation results for the inner ear bone cavity structure is shown in the attached description. Figure 5 .

Claims

1. A method for automatic segmentation of the temporal bone inner ear bone cavity structure from a coarse-to-fine dense network, characterized in that: The whole process is divided into three stages: coarse positioning of the temporal bone inner ear bone cavity structure based on coarse segmentation, extraction of candidate regions of the temporal bone inner ear bone cavity structure, and fine segmentation of the temporal bone inner ear bone cavity structure. The specific steps include: 1) Rough positioning of the temporal bone inner ear bone cavity structure based on rough segmentation: Step 1: Roughly segment the temporal bone inner ear bone cavity structure; in the rough segmentation model training stage, randomly extract a 48×48×48 cube at the same position in a complete temporal bone CT image data and annotation file. If the cube contains the annotation of the target structure, the HU value of the extracted temporal bone CT image cube is truncated, and the HU value is less than T min The value is set to T min , HU value is greater than T max The setting is T max , where T min and T max The value range is between the lowest HU value and the highest HU value. In temporal bone CT, the HU value of air is -1024, and the HU value of bone is above 300. The data can be truncated to the lowest -1024 and the highest 2000 or above to completely retain the temporal bone area for further temporal bone analysis. The truncated HU values ​​are normalized to a data distribution with a mean of 0 and a variance of 1, and sent to the coarse segmentation network for training. Otherwise, cubes of the same size are re-extracted until annotations containing anatomical structures are extracted within this range. Step 2: The complete temporal bone CT image data is divided into multiple groups of blocks in sequence according to the overlapping sliding window method, and the blocks are sent to the trained rough segmentation network for segmentation in sequence, and the segmentation results are superimposed and restored to the same size as the complete temporal bone CT in the input order; the overlap rate is selected as 2, and the input complete temporal bone CT image data is divided into multiple groups of overlapping blocks and input into the segmentation network for prediction; Step 3: Based on the voxel-level annotations of each structure, the absolute median difference method is used to remove outliers generated in the rough segmentation; the absolute median difference calculation is assumed to be as shown in formula (1): MAD=median(|Xi-median(X)|) (1) Among them, X is the set of all points predicted as foreground points, X i ∈X is the i-th point in the point set X, median(·) is the median of the point set, and the absolute median difference algorithm process is as follows: (1) Calculate the median (X) of all predicted foreground point coordinates; (2) Calculate the absolute deviation value abs(X) of each predicted prospect point from the median i -median(X)); (3) Calculate the median absolute deviation (MAD) of the values ​​in (2); (4) Divide the value in (2) by the value in (3) to obtain a set of distances Dis from the center of all predicted foreground points based on MAD; the calculation formula is as follows: (5) The points whose maximum Dis value in the three dimensions of x, y, and z is greater than the threshold Th are removed as outliers; 3.5 is used as the screening threshold to remove outliers far away from the target area; 2) Extraction of candidate regions of key anatomical structures of the temporal bone: The first step is to count the extraction size of the candidate region of the temporal bone inner ear bone cavity structure; based on the statistical voxel-level annotation data, the maximum and minimum values ​​of all annotated voxel points of the temporal bone inner ear bone cavity structure in the three dimensions of x, y, and z are counted, and the voxel coordinate span of the anatomical structure is preliminarily calculated by the difference between the maximum value of x, y, and z and the minimum value of x, y, and z. According to the pixel spacing and inter-slice spacing of each CT, the actual physical bounding box size corresponding to the inner ear bone cavity structure of each CT is calculated, and the maximum value of the actual physical size of each structure is extended outward to 24mm×24mm×24mm. A 48×48×48 cube is taken from the temporal bone CT with a inter-slice spacing and pixel spacing of 0.5mm, which can ensure the complete encirclement of the target segmentation structure while meeting the input of the segmentation algorithm, as the extraction size of the candidate region; In the second step, the region of interest is extracted by combining the roughly located center point of the anatomical structure to be segmented and the size information of the prior bounding box of the anatomical structure; the region of interest is described by the center point and three-dimensional size of the region, wherein the center point of the key anatomical structure to be segmented of the temporal bone predicted in the first stage is taken as the center, and the cube data is extracted by extending outward, and the three-dimensional size of the cube is based on the bounding box size corresponding to each anatomical structure to be segmented of the temporal bone calculated by the statistics in the first step, and the extracted sub-region is used as the candidate region for further precise segmentation, and the position of the extracted sub-region is recorded; 3) Fine segmentation of the temporal bone inner ear bone cavity structure: The fine segmentation stage specifically includes two processes: encoding and decoding; a) Coding stage Step 1: Data truncation and normalization; the 48×48×48 voxel length sub-region of the candidate region of the temporal inner ear bone cavity structure extracted in stage 2) is sent to the precise segmentation algorithm; the sub-region CT value of the CT image data is truncated according to the HU value distribution range of the temporal bone CT. For the temporal bone CT, the sub-region CT value less than T min The value is truncated to T min , greater than T max The value is truncated to T max , T min With T max The selection of is consistent with the rough positioning of the temporal bone inner ear bone cavity structure based on rough segmentation in the first stage, and is normalized to a data distribution with a mean of 0 and a variance of 1; Step 2: Use a densely connected network with dilated convolution to extract features; send the 48×48×48 sub-region that completely contains the temporal bone inner ear bone cavity structure to be segmented after truncating and normalizing the data in the first step into the network; 3 groups of densely connected modules are designed in the encoding stage; the module consists of batch normalization-rectified linear unit-convolution layer, splicing and bottleneck layer, among which all convolution sizes of the densely connected module are 3×3×3; the batch normalization-rectified linear unit-convolution layer in the densely connected block is composed of three operations: batch normalization, rectified linear unit and convolution layer; splicing is to cascade the feature map at the channel level; the role of the bottleneck layer is to reduce the number of feature maps output by the densely connected block; in the third group of densely connected modules, a dilated convolution module is used; Step 3: Multi-pooling feature fusion; batch normalization-corrected activation unit-convolution layer is used after the output of each level of dense connection module. In order to prevent overfitting, a Dropout layer is usually used afterwards, and the Dropout rate is set to 0.

5. After that, 3D maximum pooling and 3D average pooling are used at the same time, and the results after pooling are spliced; b) Decoding stage Step 1: Use transposed convolution and dense connection modules to upsample features; in the decoding process, use transposed convolution with dense connection modules to decode semantic information; use two transposed convolutions to upsample the tensor feature data of the last layer with a size of 12×12×12 in the encoding stage, and restore the tensor feature data to the original input size of 48×48×48; after the two transposed convolutions in the decoding stage, replace the ordinary convolution layer with a dense connection block; Step 2: 3D deep supervision mechanism; in the encoding stage, the features output by the first densely connected network block are extracted using 64 convolution kernels, and then pass through a 1×1×1 convolution, followed by a softmax layer to output the auxiliary segmentation result; the second layer of the decoding stage performs a convolution operation on the spliced ​​features to further extract features, and the obtained features are first subjected to a transposed convolution to improve the resolution, and then pass through a softmax layer with a 1×1×1 convolution kernel to obtain the second auxiliary segmentation result; the last layer of the decoding stage performs a convolution operation on the spliced ​​features containing different convolution kernels and then outputs the prediction result of the backbone network; during the network training process, the loss function of the backbone network and the loss function of the branch network together constitute a joint objective function, including Coefficient is the DSC loss function and the cross entropy loss function; the DSC loss function is defined as shown in formula (3): Where X and Y represent the predicted voxel and the real target voxel respectively, n represents the number of categories of the target to be segmented, and x i With y i They represent the number of target labeled voxels contained in the predicted voxel data and the true target voxel data respectively; the weight introduced into the cross entropy loss function is recorded as W, as shown in formula (4): Where N k Represents the number of target voxel labels in the voxel data to be segmented, N c Represents the number of all voxels in the voxel data to be segmented; the cross entropy loss function is shown in formula (5): Based on the loss function defined above, the joint objective function is constructed as shown in formula (6): Where m is the number of supervised hidden layers, λ 1k and λ 2k is the hyperparameter of the kth supervised hidden layer loss function, m is 2, and the hyperparameter λ 1k and λ 2k The values ​​are 0.6 and 0.3 respectively, L k and H k They are the DSC loss function and cross entropy loss function of the kth supervised hidden layer respectively; the target loss function is constructed based on the loss functions of the backbone network and the branch network to jointly guide the network training, reduce the gradient disappearance, and speed up the convergence of the network; Step 3: Use the precise segmentation algorithm to predict the segmentation result with a resolution of 48×48×48. According to the candidate region extraction position recorded in the second stage, restore it to the corresponding position in the complete CT as the final segmentation result.

Images