Airway stenosis automatic segmentation method, system and device based on CT image

Abstract

The invention relates to a computer vision technology, in particular to a medical image processing technology based on deep learning and a medical diagnosis auxiliary method. The invention provides an airway stenosis automatic segmentation method, system and device based on a CT image, and the method specifically comprises the following technical means: 1) building an airway stenosis automatic segmentation model based on a marked training set and a verification set, so as to achieve the automatic recognition, positioning and segmentation of a narrow airway; (2) reasoning by using the established airway stenosis automatic segmentation model, and automatically identifying, positioning and segmenting a stenosis airway from an input CT (Computed Tomography) image; and 3) performing quantitative parameter measurement based on the narrow airway obtained by automatic segmentation. According to the automatic segmentation system and device for the airway stenosis, whether the airway stenosis exists in the CT image or not can be rapidly and accurately recognized, if yes, automatic positioning and segmentation of the stenosis can be achieved, and the system and device assist imaging departments or clinicians in rapidly and accurately diagnosing the narrow airway; meanwhile, based on the airway stenosis obtained through automatic segmentation, the stenosis range and the maximum, minimum and average lumen diameters of the stenosis part can be rapidly and accurately calculated, on one hand, the method can be used for evaluating the severity of the stenosis, and on the other hand, the method can be used for assisting clinical treatment.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a medical image processing technology based on deep learning and a medical diagnosis auxiliary method Background Art

[0002] The application of computer vision technology in the field of medical image recognition has made significant progress and is continuously promoting the development of medical image analysis. Deep learning technology, especially convolutional neural network (CNN), has performed well in the automatic segmentation of medical images. For example, the 3D-Net architecture has been widely used in various medical image segmentation tasks, including the recognition and segmentation of cells, organs, and lesions. Deep learning algorithms can analyze medical image data and assist doctors in the diagnosis of diseases, such as tumor detection and cardiovascular disease diagnosis. These algorithms can recognize specific lesion patterns by learning from large amounts of medical image data.

[0003] Airway stenosis is one of the diseases related to the respiratory system. Its respiratory interventional treatment methods vary according to factors such as the cause, airway involvement, and degree of lesions. At present, respiratory interventional doctors often need to measure the type, diameter, range, severity, and other factors of airway stenosis under CT to pre-plan interventional treatment methods. However, at present, such assessments need to be manually measured, and the results vary depending on the implementer. Insufficient training for grassroots respiratory interventional doctors may lead to inaccurate results. Therefore, there is an urgent need to provide a technology that can accurately and quickly identify, locate, and segment airway stenosis. Summary of the invention

[0004] To solve the above problems, the present invention provides a method, system and device for automatic segmentation of airway stenosis based on CT images, which specifically include the following technical means:

[0005] A method for automatic segmentation of airway stenosis based on CT images comprises the following steps:

[0006] 1) Establish an automatic airway stenosis segmentation model based on the labeled training set and validation set to achieve automatic identification, location and segmentation of stenotic airways;

[0007] 2) Use the established automatic airway stenosis segmentation model for inference to automatically identify, locate, and segment stenotic airways from input CT images;

[0008] 3) Quantitative parameter measurement based on the narrow airway obtained by automatic segmentation.

[0009] Preferably, the segmentation network in step 1 adopts a 3D-Unet network based on a convolutional neural network.

[0010] Preferably, the quantitative parameters in step 3 include the airway length of the narrowed portion, the airway lumen diameter of the narrowed portion (maximum value, minimum value, average value), etc.

[0011] Wherein, step 1 specifically includes the following steps:

[0012] S1.1 Input training set and validation set;

[0013] S1.2 perform data verification on the training set and the validation set to obtain a verified training set and a validation set;

[0014] S1.3 performing data preprocessing on the verified training set and validation set;

[0015] S1.4 uses the preprocessed training set to train the model and obtain an automatic airway stenosis segmentation model.

[0016] Furthermore, each of the semantic segmentation models includes 9 convolution modules, wherein the Nth convolution module includes N output channels, where N is a natural number, N∈[1,2,...,9]. Convolution modules 1 to 4 are downsampling modules, each of which includes two convolution layers (each convolution layer is followed by a batch normalization layer BN and an activation layer ReLU) and a downsampling layer (Max Pool). After each downsampling module, the feature map size becomes smaller and the number of output channels increases. Convolution modules 5 to 8 are upsampling modules, each of which includes two convolution layers and an upsampling layer (Upsampling). After each upsampling module, the feature map size becomes larger and the number of output channels decreases. The output connection of convolution module 4 and convolution module 5 is used as the input of convolution module 6, the output connection of convolution module 3 and convolution module 6 is used as the input of convolution module 7, the output connection of convolution module 2 and convolution module 7 is used as the input of convolution module 8, and the output connection of convolution module 1 and convolution module 8 is used as the input of convolution module 9. The input of the model is a single-channel CT image and its corresponding annotations, and the output of the model is a two-channel segmentation result, representing the background and foreground respectively. The output of the model has the same spatial resolution scale as the input.

[0017] Furthermore, in step 2, the trained airway stenosis automatic segmentation model is used to realize automatic segmentation of the stenotic airway. The process includes the following steps:

[0018] S2.1 Input the test set, which includes CT plain scan images that may contain airway stenosis, and perform data verification and data preprocessing on the test set.

[0019] S2.2 obtains the airway stenosis pre-segmentation model by loading the model parameters and network structure of the trained airway stenosis automatic segmentation model.

[0020] S2.3 performs predictive analysis on the test set through the pre-segmentation model to obtain the airway stenosis segmentation result.

[0021] In order to further improve the degree of automation, the present invention also provides a quantitative parameter measurement method based on the above airway stenosis segmentation result, comprising the following steps:

[0022] S3.1 Smooths the segmentation results to eliminate the jagged shape of the airway boundary and false positive points

[0023] S3.2 Extract the centerline or skeleton line of the narrow airway through skeletonization or thinning algorithm

[0024] S3.3 Drawing the smoothed narrow airway surface using the isosurface extraction algorithm

[0025] S3.4 Generate a cross section perpendicular to the center line

[0026] S3.5 Quantitative parameter measurement: calculate the extent or length of the stenotic airway based on the skeleton centerline of the stenotic airway, or calculate the minimum diameter, maximum diameter, and average diameter of the stenotic airway based on the cross-section generated perpendicular to the centerline

[0027] Preferably, the smoothing process comprises dilation and erosion operations.

[0028] Preferably, the smoothing process includes the following two steps: ① open operation: first perform a scale ([1,1,1], i.e., one pixel in each of the three directions of the spatial dimension) erosion operation on the input annotation result, and then perform a scale dilation operation; ② close operation: first perform a scale dilation operation on the input processing result of step ①, and then perform a scale erosion operation.

[0029] The automatic segmentation system and device for airway stenosis proposed in the present invention can quickly and accurately identify whether airway stenosis exists in CT images. If so, the automatic positioning and segmentation of the stenosis can be achieved, assisting the imaging department or clinicians in making a quick and accurate diagnosis of the airway stenosis. At the same time, based on the airway stenosis obtained by automatic segmentation, the stenosis range, the maximum, minimum and average lumen diameter of the stenosis part can be quickly and accurately calculated, which can be used to evaluate the severity of the stenosis on the one hand, and assist clinical treatment on the other hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall flow chart of the airway stenosis identification segmentation and quantitative parameter determination scheme proposed by the present invention

[0031] Figure 2 The training process of the airway stenosis automatic segmentation model proposed in this invention

[0032] Figure 3 The network structure of the semantic segmentation model proposed in this invention

[0033] Figure 4 The automatic segmentation process of airway stenosis proposed by the present invention DETAILED DESCRIPTION

[0034] Based on the big image data of the National Respiratory Center, the present invention automatically annotates CT images of airway stenosis and uses the method of deep learning of artificial intelligence for image segmentation, allowing the machine to automatically identify and segment the narrow airway, and automatically calculate indicators such as the diameter, length, and severity of the stenosis, thereby assisting respiratory interventional treatment.

[0035] 1) Training set and validation set description

[0036] A total of 499 patients with airway stenosis who were diagnosed with airway stenosis by bronchoscopy and CT between March 2012 and December 2023 were included.

[0037] The inclusion criteria were as follows: (1) patients older than 14 years old, (2) patients diagnosed with airway stenosis based on bronchoscopy and CT findings, (3) patients with airway stenosis pathologically diagnosed as tumor and inflammation, and (4) patients undergoing 1-2 mm CT examination.

[0038] The marking rules for CT were as follows: (1) 1-2 mm CT was used; (2) a special window width and window level (HU 500, HU-100) was used and CT was marked at this window width and window level to match the real airway data; (3) the narrow airways in CT were marked by an independent professional respiratory interventionist, and any disagreement between the first two researchers was resolved by a third researcher.

[0039] 2) Task 1: Establish an automatic airway stenosis segmentation model. The overall process is as follows: Figure 2 shown.

[0040] The input of the training process is the training set and the validation set. The format of medical images is usually dicom sequence, and the format of the corresponding manually annotated files is usually nifti file (extension .nii.gz). The manually annotated files are usually called "gold standard". During the training process, the output results of the model will be compared with the "gold standard", and the model parameters will be optimized based on the comparison. The training process mainly includes three stages: data verification, data preprocessing and segmentation model training.

[0041] (1) Data verification

[0042] ① Availability check: If the number of DICOM sequence layers of a case in the training set and validation set is less than a certain threshold (such as 60), or the slice thickness of a case is greater than a certain threshold (such as 2 mm), such cases will be excluded.

[0043] ②Integrity check: It mainly checks whether there are missing layers in the DICOM sequence. The method is to first sort the DICOM sequence based on the unique sequence code (Instance Number) of each slice, and then check the layer spacing (Spacing Between Slices) between adjacent slices based on the relative coordinate position (Slice Location) of each slice. If the layer spacing between two adjacent slices is twice or more than the layer spacing of other adjacent slices, it means that there is a missing slice between the two slices, and such cases are directly discarded.

[0044] ③ Repeatability check: Check whether there are duplicate slices, that is, check whether there are duplicate slice sequence codes. If there are duplicate slices, delete the redundant slices.

[0045] ④Consistency check: mainly to check whether the spatial size of the annotation file and the corresponding CT image file is the same, and whether the difference in voxel spacing is less than a certain threshold. If not, the case will be discarded.

[0046] (2) Data preprocessing

[0047] ① Establish a data dictionary: For each case in the training set and validation set, establish a mapping relationship between its CT image and the corresponding annotation file, and encapsulate it as a dictionary list in Python language dictionary format.

[0048] ② Data loading: Load CT images and their corresponding annotation files, and add channel dimensions before their spatial dimensions.

[0049] ③ Labeling file processing: Check and unify the category values ​​of the narrow airway labels, because multiple people may use different labeling tags when labeling, and these tags have different category values. Another situation may be that different labels are used for narrowing in different locations, which also needs to be checked and unified.

[0050] ④ Data normalization: Standardize the image based on the CT value range (for example, the CT value range of the mediastinal window is -350 to 150), and standardize the CT value (i.e. HU value) from -350 to 150 to [0,1]. The values ​​below -350 are standardized to 0, and the values ​​above 150 are standardized to 1. The normalization method is as follows:

[0051]

[0052] The denominator is the maximum HU value (150) minus the minimum HU value (-350), and the numerator is the HU value of the pixel minus the minimum HU value, and the value of the pixel after annotation is calculated.

[0053] ⑤ Foreground cropping: The foreground is the manually annotated lesion mask, and the part outside the lesion is the background. The effective part of the image is cropped based on the annotation to improve the efficiency and effect of training and evaluation.

[0054] ⑥ Resampling: Resample the CT images and annotations to unify the anatomical coordinates and voxel spacing so that the model can learn consistent target features of interest.

[0055] ⑦ Data scaling: Scale the spatial dimension of the processed images and annotation data to a fixed-size input block (crops) in order to perform batch training on the input samples.

[0056] ⑧Data augmentation: Perform random affine transformation (rotation, scaling in different directions, etc.) on the above input blocks, on the one hand to weaken the sensitivity of the convolutional neural network to narrow locations, on the other hand to improve the generalization ability of the segmentation model.

[0057] (3) Airway stenosis semantic segmentation model training

[0058] The semantic segmentation model uses 3D Unet as the basic backbone network. The network structure is as follows Figure 3 As shown. The network structure contains 9 convolution modules, and the number above each module is the number of output channels. Convolution modules 1 to 4 are downsampling modules, each of which contains two convolution layers (each convolution layer is followed by a batch normalization layer BN and an activation layer ReLU) and a downsampling layer (Max Pool). After each downsampling module, the feature map size becomes smaller and the number of output channels increases. Convolution modules 5 to 8 are upsampling modules, each of which contains two convolution layers and an upsampling layer (Upsampling). After each upsampling module, the feature map size becomes larger and the number of output channels decreases. The output connection of module 4 and module 5 is used as the input of module 6, the output connection of module 3 and module 6 is used as the input of module 7, the output connection of module 2 and module 7 is used as the input of module 8, and the output connection of module 1 and module 8 is used as the input of module 9. The input of the model is a single-channel CT image and its corresponding annotation, and the output of the model is a two-channel segmentation result, representing the background and foreground respectively. The output of the model has the same spatial resolution scale as the input.

[0059] 3) Task 2: Automatic segmentation of narrow airways (if present)

[0060] The overall process of Task 2 is as follows Figure 4 shown.

[0061] The input test set is a CT plain scan image that may contain airway stenosis. Since the test set does not have an annotation file, there is no consistency check in the data verification part, and the rest of the verification process is the same as Task 1. The data preprocessing part also does not process the annotation file, and the rest of the processing is the same as Task 1. By loading the model parameters and network structure maintained in Task 1, the airway stenosis pre-segmentation model is obtained. Then the test set is predicted and analyzed by the pre-segmentation model to obtain the airway stenosis segmentation result.

[0062] 4) Task 3: Quantitative parameter measurement, see the processing flow Figure 1 The following is a detailed description of each step.

[0063] (1) Smoothing

[0064] Since the training data is manually annotated layer by layer and pixel by pixel, it is difficult to accurately identify and annotate the airway boundaries for narrow airways across layers. When observing the annotation results from the coronal or sagittal plane, it can be seen that the airway boundaries have some jagged shapes, which is caused by more or less annotations on the boundaries of some layers. This situation will be transmitted to the automatic segmentation results of the segmentation model, that is, the airway boundaries obtained by automatic segmentation may also appear jagged. In addition, the pre-segmentation model will inevitably identify some false positive pixels. For this reason, it is necessary to eliminate the jagged shapes of the airway boundaries and these false positive points through smoothing, otherwise it will affect the subsequent airway centerline extraction results or the final quantitative parameter measurement results.

[0065] Smoothing can be done through dilation and erosion operations. Usually it includes two steps: ①Open operation: first perform an erosion operation of one scale ([1,1,1], i.e. one pixel in each of the three directions of the spatial dimension) on the input annotation result, and then perform an expansion operation of one scale; ②Close operation: first perform an expansion operation of one scale, and then perform an erosion operation of one scale on the input result of step ①.

[0066] (2) Extracting the centerline of the narrow airway

[0067] The center line or skeleton line is extracted from the smoothed narrow airway through a skeletonization or thinning algorithm.

[0068] (3) Reconstruction of airway surface

[0069] The smoothed narrow airway surface is drawn using the Marching Cubes (also known as isosurface extraction) algorithm.

[0070] (4) Generate a cross section perpendicular to the center line

[0071] Based on the skeleton centerline of the stenotic airway and the surface of the stenotic airway, cross sections perpendicular to the centerline are generated at certain intervals along the centerline.

[0072] (5) Quantitative parameter measurement

[0073] The extent or length of the stenotic airway can be calculated based on the skeleton centerline of the stenotic airway. The minimum, maximum and average diameters of the stenotic airway can be calculated based on the generated cross section perpendicular to the centerline.

[0074] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.

Claims

1. A method for automatic segmentation of airway stenosis, characterized in that: The following steps are involved: S1. Establish an automatic airway stenosis segmentation model based on the labeled training set and validation set to achieve automatic identification, location and segmentation of stenotic airways; S2. Use the established automatic airway stenosis segmentation model for inference to automatically identify, locate and segment the stenotic airway from the input CT images; S3. Quantitative parameter measurement based on the stenotic airways obtained by automatic segmentation; The segmentation model in step S1 adopts a 3D-Unet network based on a convolutional neural network; The quantitative parameters in step 3 include the airway length of the narrowed portion, and the maximum value, minimum value and average value of the airway lumen diameter of the narrowed portion.

2. The method for automatic segmentation of airway stenosis according to claim 1, characterized in that: Each of the segmentation models comprises 9 convolution modules, wherein the Nth convolution module comprises N output channels, wherein N is a natural number, N∈[1,2,...,9]; wherein the 1st to 4th convolution modules are downsampling modules, each of which comprises two convolution layers and one downsampling layer; the 5th convolution module and the 8th convolution module are upsampling modules, each of which comprises two convolution layers and one upsampling layer; the output of the 4th convolution module is connected to the 5th convolution module as the input of the 6th convolution module, the output of the 3rd convolution module is connected to the 6th convolution module as the input of the 7th convolution module, the output of the 2nd convolution module is connected to the 7th convolution module as the input of the 8th convolution module, and the output of the 1st convolution module 1 is connected to the 8th convolution module as the input of the 9th convolution module.

3. The method for automatic segmentation of airway stenosis according to claim 1, characterized in that: Step 1 specifically includes the following steps: S1.1 Input training set and validation set; S1.2 perform data verification on the training set and the validation set to obtain a verified training set and a validation set; S1.3 performing data preprocessing on the verified training set and validation set; S1.4 uses the preprocessed training set to train the model and obtain an automatic airway stenosis segmentation model; The data verification method in step S1.2 includes availability verification, integrity verification, repeatability verification, and consistency verification; The data preprocessing in step S1.2 includes establishing a data dictionary, loading data, processing annotation files, normalizing data, foreground cropping, resampling, scaling data, and augmenting data.

4. The method for automatic segmentation of airway stenosis according to claim 3, characterized in that: The data normalization refers to standardizing the image based on the CT value range, standardizing the CT values ​​of -350 to 150 to [0, 1], those below -350 to 0, and those above 150 to 1.

5. The method for automatic segmentation of airway stenosis according to claim 1, characterized in that: Step 2 specifically includes the following steps: S2.1 Input the test set, which includes CT plain scan images that may contain airway stenosis, and perform data verification and data preprocessing on the test set. S2.2 obtains the airway stenosis pre-segmentation model by loading the model parameters and network structure of the trained airway stenosis automatic segmentation model. S2.3 performs predictive analysis on the test set through the pre-segmentation model to obtain the airway stenosis segmentation result.

6. The method for automatic segmentation of airway stenosis according to claim 1, characterized in that: The quantitative parameter measurement in step 3 includes the following steps: S3.1 Smooths the segmentation results to eliminate the jagged shape of the airway boundary and false positive points S3.2 Extract the centerline or skeleton line of the narrow airway through skeletonization or thinning algorithm S3.3 Drawing the smoothed narrow airway surface using the isosurface extraction algorithm S3.4 Generate a cross section perpendicular to the center line S3.5 Calculate the extent or length of the stenotic airway based on the skeleton centerline of the stenotic airway, or calculate the minimum diameter, maximum diameter, and average diameter of the stenotic airway based on a cross section generated perpendicular to the centerline.

7. An automatic airway stenosis segmentation system, characterized in that: include: A model building unit, used to build an automatic airway stenosis segmentation model based on the labeled training set and validation set, so as to realize automatic identification, positioning and segmentation of the stenotic airway; An inference unit, used for performing inference using the established automatic airway stenosis segmentation model to automatically identify, locate and segment the stenotic airway from the input CT image; The quantitative parameter measurement unit is used to measure the quantitative parameters of the narrow airway based on the automatic segmentation. The steps applied to the automatic airway stenosis segmentation system are as follows: ① Availability check: If the number of DICOM sequence layers of a case in the training set and validation set is less than the first threshold, or the slice thickness of a case is greater than the second threshold, the data will be excluded. ②Integrity check: Check whether there are missing layers in the DICOM sequence. First, sort the DICOM sequence based on the unique sequence code of each slice, and then check the inter-layer spacing between adjacent slices based on the relative coordinate position of each slice. If the inter-layer spacing between two adjacent slices is twice or more than the inter-layer spacing of other adjacent slices, it means that there is a missing slice between the two slices, and such data is directly discarded. ③ Repeatability check: Check whether there are duplicate slices, that is, check whether there are duplicate slice sequence codes. If there are duplicate slices, delete the redundant slices. ④Consistency check: mainly to check whether the annotation file and the corresponding CT image file are the same in spatial size, and whether the difference in voxel interval is less than a certain threshold. If not, the data will be discarded. The first threshold is 60, and the second threshold is 2 mm.

8. A data preprocessing method, applied to the airway stenosis automatic segmentation system according to claim 10, characterized in that: The following steps are involved: ① Establish a data dictionary: For each case in the training set and validation set, establish a mapping relationship between its CT image and the corresponding annotation file, and encapsulate it as a dictionary list in Python language dictionary format; ② Data loading: Load CT images and their corresponding annotation files, and add channel dimensions before their spatial dimensions; ③ Annotation file processing: check and unify the categories of narrow airway annotations; ④Data normalization: standardize the image based on the CT value range; ⑤Foreground cropping: The foreground is the manually annotated lesion mask, and the part outside the lesion is the background. The effective part of the image is cropped based on the annotation. ⑥ Resampling: Resample the CT images and annotations to unify the anatomical coordinates and voxel spacing so that the model can learn consistent target features of interest. ⑦ Data scaling: Scale the spatial dimension of the processed image and annotation data to a fixed-size input block; ⑧Data augmentation: Perform random affine transformation on the above input blocks. The normalization method is as follows: Where HU is the CT value of the current pixel, HU max is the maximum value of HU, HU min is the minimum HU value; The random affine transformation includes rotation and scaling along different directions.

9. A quantitative parameter measurement method, applied to the airway stenosis automatic segmentation system according to claim 10, characterized in that: The following steps are involved: S1 smoothes the segmentation results to eliminate the jagged shape of the airway boundary and false positive points S2 extracts the center line or skeleton line of the narrow airway through a skeletonization or thinning algorithm. S3 draws the smoothed narrow airway surface through an isosurface extraction algorithm. S4 generates a cross section perpendicular to the center line S5 quantitative parameter measurement, calculates the range or length of the stenotic airway based on the skeleton centerline of the stenotic airway, or calculates the minimum diameter, maximum diameter and average diameter of the stenotic airway based on the generated cross section perpendicular to the centerline.

10. The quantitative parameter measurement method according to claim 9, characterized in that: The smoothing process includes dilation and erosion operations; The smoothing process includes the following two steps: ① open operation: first perform a scale erosion operation on the input annotation result, and then perform a scale dilation operation; ② close operation: first perform a scale dilation operation on the input processing result of step ①, and then perform a scale erosion operation.

Images