Airway classification method and device, and computer readable storage medium
Through the deep learning network, the airway image data is segmented and feature extracted, and whether the airway is a difficult airway is solved, which solves the problem of low airway classification accuracy in the prior art and achieves higher classification accuracy and reliability.
Patent Information
- Application Number
- CN202110852858.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-07-27
AI Technical Summary
In the prior art, the airway classification accuracy is low, making it difficult to effectively predict difficult airways, resulting in serious consequences such as asphyxiation and cardiac arrest in patients.
Using a deep learning network, by acquiring airway image data for segmentation and feature extraction, two-channel data is constructed to identify whether the airway is a difficult airway, and a pre-trained deep learning feature extraction network model and classification network model are used for identification.
It improves the accuracy of airway classification, is simple in methods and reliable in results, and can effectively identify whether the airway is a difficult airway, reducing individual differences and misjudgment rates.
Smart Images

Figure CN113657458B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an airway classification method and device thereof, and a computer-readable storage medium. Background Art
[0002] Difficult airway situation refers to the situation in which artificial ventilation is required due to illness or surgery during general anesthesia, emergency treatment, and diagnosis and treatment of critically ill patients, but difficulties are encountered in establishing an artificial ventilation tube.
[0003] Clinical difficult airway situations include difficulty in mask ventilation, difficulty in exposing the glottis with a laryngoscope, difficulty in intubation, intubation failure, and difficulty in supraglottic airway ventilation. Regardless of whether it is emergency intubation for critically ill patients or endotracheal intubation after induction of general anesthesia, the patient has lost his normal respiratory function. Once the patient cannot be successfully intubated in time and cannot be effectively ventilated, the patient will suffer from cardiac arrest, brain damage, or even death due to suffocation. Therefore, predicting difficult airway is particularly important.
[0004] The current preoperative difficult airway assessment methods mainly include understanding medical history, physical examination and auxiliary examination. Among them, understanding medical history includes age, BMI (Body Mass Index), dental abnormalities, sleep apnea syndrome and snoring history; physical examination includes the length of the upper incisors, the relationship between the upper and lower incisors when the mouth is closed, the development and protrusion ability of the mandible, the degree of mouth opening, and the grading of pharyngeal structure; auxiliary examinations include the inspection and evaluation of tools such as video laryngoscopes or video intubation soft mirrors under conscious sedation and surface anesthesia to clarify the grading of laryngoscope exposure. The above methods require a large amount of information and some information is not easy to obtain. In addition, these methods have certain specificity and sensitivity in predicting difficult airways, with large individual differences and mostly relying on experience. Summary of the invention
[0005] The main technical problem solved by the present invention is to provide an airway classification method and device thereof, and a computer-readable storage medium to solve the problem of low airway classification accuracy in the prior art.
[0006] To solve the above technical problems, the first technical solution adopted by the present invention is: to provide an airway classification method, the method comprising: acquiring airway image data; segmenting the airway image data to obtain airway area image data; connecting the airway image data and the airway area image data to obtain two-channel data; inputting the two-channel data into a pre-trained deep learning network to confirm whether the airway in the airway image is a difficult airway.
[0007] Among them, the deep learning network includes a deep learning feature extraction network model and a deep learning classification network model. The step of inputting the two-channel data into the pre-trained deep learning network to confirm whether the airway in the airway image is a difficult airway specifically includes: inputting the two-channel data into the pre-trained deep learning feature extraction network model to extract features to obtain an airway feature map; inputting the airway feature map into the pre-trained deep learning classification network model to identify and determine whether the airway is a difficult airway.
[0008] Among them, the steps of segmenting the airway image data to obtain the airway region image data specifically include: setting a threshold, using threshold segmentation to binarize the airway image data to extract the initial airway region image; preprocessing the initial airway region image, the preprocessing includes hole filling and small area removal; constructing an airway template and determining the seed point in the initial airway region image through the airway template; determining whether the connected domain of the seed point is the airway region image according to the proportion of voxel points in the connected domain of the seed point in the voxel points of the airway image data.
[0009] Among them, the steps of constructing an airway template and determining seed points in the initial airway area image through the airway template specifically include: constructing the airway template; matching the airway template with the connected domain of each voxel point in the initial airway area image; and determining the voxel point with the highest overlap between the connected domain and the airway template as the seed point.
[0010] Among them, the step of determining whether the connected domain of the seed point is an airway region image according to the proportion of voxel points in the connected domain of the seed point in the voxel points of the airway image data specifically includes: judging whether the proportion of voxel points in the connected domain of the seed point in the voxel points of the airway image data is within the range of 0.2% to 0.8%; if the proportion is within the range, determining that the connected domain of the seed point is an airway region image.
[0011] Among them, before the step of obtaining airway image data, it also includes: training a deep learning network, wherein the deep learning feature extraction network model and the deep learning classification network model have a three-dimensional convolution kernel.
[0012] Among them, the steps of training the deep learning network specifically include: obtaining a first training sample set, the first training sample set includes multiple first image samples in a public data set of medical CT images, the first image samples include image data and a real mask area image corresponding to the image data; wherein the same image data is stacked into two-channel data; the two-channel data is input into an initial feature extraction network model for feature extraction to obtain a feature map; the feature map is input into a decoder for segmentation to obtain a predicted mask area image of the image data; a first loss function is constructed by using the predicted mask area image and the real mask area image; and the initial feature extraction network model is iteratively pre-trained using the first loss function to obtain a feature extraction network model.
[0013] Among them, the initial feature extraction network model at least includes an input layer, a first feature extraction layer, a pooling layer, a second feature extraction layer and a third feature extraction layer; the step of inputting the two-channel data into the initial feature extraction network model for feature extraction to obtain a feature map specifically includes: inputting the two-channel data into the input layer, and preprocessing the two-channel data; the first feature extraction layer extracts features from the preprocessed two-channel data to obtain a first feature map; the pooling layer performs a pooling operation on the first feature map; the second feature extraction layer extracts features from the first feature map after the pooling operation to obtain a second feature map; the third feature extraction layer extracts features from the second feature map to obtain a third feature map; the third feature map is connected with the second feature map after upsampling to obtain a feature map.
[0014] Among them, the number of input channels of the first feature extraction layer is 2.
[0015] Among them, the step size of the second feature extraction layer and the third feature extraction layer is s=(1, 2, 2).
[0016] Among them, the step of training the deep learning network specifically also includes: obtaining a second training sample set, the second training sample set includes multiple second image samples, the second image samples include airway data, airway area image data corresponding to the airway data, and the real category of the airway in the airway data; wherein the airway data is connected with the airway area image data; inputting the second image sample into the trained feature extraction network model for feature extraction to obtain an airway feature map; inputting the airway feature map into the initial classification network model for classification to obtain the predicted category of the airway; constructing a second loss function through the real category of the airway and the predicted category of the airway; using the second loss function to iteratively train the feature extraction network model and the initial classification network model to obtain a deep learning feature extraction network model and a deep learning classification network model.
[0017] Wherein, a second training sample set is obtained, the second training sample set includes multiple second image samples, the second image samples include airway data, airway area image data corresponding to the airway data, and the real category of the airway in the airway data; wherein, the step of connecting the airway data with the airway area image data also includes: screening valid data points in the airway image data; calculating the mean and variance of the valid data points and normalizing them; connecting the normalized airway image data with the airway area image data to obtain two-channel data, and performing data enhancement processing on the two-channel data.
[0018] The data enhancement processing includes at least one of data translation processing, data rotation processing and data scaling processing.
[0019] Among them, the initial classification network model includes a fourth feature extraction layer, a filling layer and a linear layer. Before the step of inputting the second image sample into the deep learning feature extraction network model for feature extraction to obtain an airway feature map, it also includes: determining the maximum number of slices of the airway image data in the second training sample set; determining the maximum number of slices as the number of airway feature maps input to the linear layer; the step of inputting the airway feature map into the initial classification network model for classification to obtain the predicted category of the airway specifically includes: the fourth feature extraction layer extracts features from the airway feature map and performs size conversion; the filling layer supplements the number of converted airway feature maps so that the number of airway feature maps input to the linear layer is the same as the maximum number of slices; the linear layer identifies the input airway feature map to obtain the predicted category of the airway.
[0020] In order to solve the above technical problems, the second technical solution adopted by the present invention is: to provide an airway classification device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to implement the steps in the above airway classification method.
[0021] In order to solve the above technical problems, the third technical solution adopted by the present invention is: to provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above airway classification method are implemented.
[0022] The beneficial effects of the present invention are as follows: Different from the prior art, an airway classification method and device thereof, and a computer-readable storage medium are provided. The airway classification method in the present application obtains airway image data, and obtains airway region image data based on the airway image data, generates two-channel data by connecting the airway image data with the airway region image data, and inputs the two-channel data into a deep learning network model to identify and determine whether the airway image is a difficult airway. The present application inputs the two-channel data into a deep learning network, so that the deep learning network can effectively use the airway information in the airway region image data to identify whether the airway is a difficult airway, and can also retain the deep learning network's ability to learn other parts other than the airway information in the airway image data, thereby improving the accuracy of airway classification. The method is simple and the classification results are reliable. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a flow chart of an embodiment of an airway classification method provided by the present invention;
[0024] Figure 2 yes Figure 1 A flowchart of a specific embodiment of step S1 in the provided airway classification method;
[0025] Figure 3 It is a structural diagram of a specific embodiment of a deep learning network provided by the present invention;
[0026] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step S11 in the airway classification method provided;
[0027] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step S12 in the airway classification method provided;
[0028] Figure 6 It is a flowchart of a specific embodiment of step S121 in the airway classification method provided by the present invention;
[0029] Figure 7 yes Figure 1 A flowchart of a specific embodiment of step S3 in the airway classification method provided;
[0030] Figure 8 yes Figure 1 A flowchart of a specific embodiment of step S5 in the airway classification method provided;
[0031] Fig. 9 is a schematic block diagram of an embodiment of an airway classification device provided by the present invention;
[0032] Fig.10 It is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION
[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0035] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0036] In order to enable those skilled in the art to better understand the technical solution of the present invention, a target tracking method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0037] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an airway classification method provided by the present invention. In this embodiment, an airway classification method is provided, and the airway classification method includes the following steps.
[0038] S1: Train the deep learning network.
[0039] See also Figure 2 , Figure 2 yes Figure 1 A flow chart of a specific embodiment of step S1 in the provided airway classification method. Specifically, the deep learning network includes a deep learning feature extraction network model and a deep learning classification network model, and both the deep learning feature extraction network model and the deep learning classification network model have a three-dimensional convolution kernel. Specifically, the feature extraction network model is first obtained by pre-training with a public medical CT image data set, and then the deep learning feature extraction network model and the deep learning classification network model are obtained by training with an airway image data set, so that the deep learning feature extraction network model can extract features more accurately, thereby completing the training of the deep learning network. Among them, the airway image data set is an airway scanning image data set.
[0040] S11: Pre-train the deep learning feature extraction network model.
[0041] See also Figure 3 , Figure 3It is a structural diagram of a specific embodiment of the deep learning network provided by the present invention. Specifically, a segmentation network is constructed, and the segmentation network includes an initial feature extraction network model and a decoder. Among them, the initial feature extraction network model includes an input layer, a first feature extraction layer, a pooling layer, a second feature extraction layer and a third feature extraction layer. In another optional embodiment, the initial feature extraction network model also includes a fifth feature extraction layer and a sixth feature extraction layer, and the fifth feature extraction layer and the sixth feature extraction layer are sequentially arranged after the third feature extraction layer. In a specific embodiment, the initial feature extraction network model is modified based on the ResNet-50 network. Among them, the number of input channels of the first feature extraction layer is 2, the number of output channels is 64, and the step size is s=(2, 2, 2); the number of input channels of the pooling layer is 64, the number of output channels is 64, and the step size is s=(2, 2, 2); the number of input channels of the second feature extraction layer is 64, the number of output channels is 256, and the step size is s=(1, 1, 1); the number of input channels of the third feature extraction layer is 128, the number of output channels is 512, and the step size is s=(1, 2, 2); the number of input channels of the fifth feature extraction layer is 256, the number of output channels is 1024, and the step size is s=(1, 2, 2); the number of input channels of the sixth feature extraction layer is 512, the number of output channels is 2048, and the step size is s=(1, 2, 2).
[0042] The decoder includes a transposed convolution layer, a seventh feature extraction layer, an eighth feature extraction layer, a ninth feature extraction layer and a first output layer, which are arranged in sequence. Among them, the number of input channels of the transposed convolution layer is 2048, the number of output channels is 256, the step size s=(2,8,8), and the convolution kernel is 3×3×3; the number of input channels of the seventh feature extraction layer is 512, the number of output channels is 32, and the step size s=(1,1,1); the number of input channels of the eighth feature extraction layer is 32, the number of output channels is 32, the step size s=(1,1,1), and the convolution kernel is 3×3×3; the number of input channels of the ninth feature extraction layer is 32, the number of output channels is 2, the step size s=(1,1,1), and the convolution kernel is 1×1×1. Among them, the initial feature extraction network model is used to extract features, and the decoder is used to realize the category labeling of each pixel in the feature map.
[0043] See also Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step S11 in the airway classification method is provided. The training steps of the deep learning feature extraction network model include the following steps.
[0044] S111: Acquire a first training sample set, where the first training sample set includes a plurality of first image samples in a public medical CT image dataset.
[0045] Specifically, the medical CT image public dataset may include the KiTS19 dataset, the LiTS dataset, and the MSD dataset as public datasets. The first image sample includes scanned image data and a real mask area image corresponding to the scanned image data; wherein the same scanned image data is stacked into two-channel data. In a specific embodiment, the scanned image data may be three-dimensional scanned image data.
[0046] S112: Input the two-channel data into the initial feature extraction network model to extract features and obtain a feature map.
[0047] Specifically, the two-channel data is input into the input layer, the input layer inputs the two-channel data into the first feature extraction layer, the first feature extraction layer extracts features from the two-channel data, and the extracted first feature map is input into the pooling layer, the pooling layer performs a pooling operation on the first feature extraction layer to achieve size reduction of the first feature map. The pooled first feature map is input into the second feature extraction layer for feature extraction to obtain the second feature map, the second feature map is input into the third feature extraction layer for feature extraction to obtain the third feature map, the third feature map is input into the fifth feature extraction layer for feature extraction to obtain the fifth feature map, the fifth feature map is input into the sixth feature extraction layer for feature extraction to obtain the sixth feature map, and the sixth feature map is the feature map output by the initial feature extraction network model. By performing multiple feature extractions on the second feature map, the accuracy of the feature map extracted by the initial feature extraction network model is higher, and the learning ability of the initial feature extraction network model is improved.
[0048] S113: Input the feature map into the decoder for segmentation to obtain a predicted mask area image of the scanned image data.
[0049] Specifically, the initial feature extraction network model inputs the sixth feature map into the transposed convolution layer for upsampling to improve the resolution of the sixth feature map, and connects the processed sixth feature map with the second feature map in the initial feature extraction network model and inputs it into the seventh feature extraction layer for feature extraction to obtain the seventh feature map. In a specific embodiment, the second feature map and the sixth feature map after resolution enhancement processing are connected to obtain data to be input into the seventh feature extraction layer. The eighth feature extraction layer extracts features from the seventh feature map to obtain the eighth feature map, and the ninth feature extraction layer extracts features from the ninth feature map to obtain the ninth feature map, thereby realizing the category labeling of each pixel in the feature map, that is, segmentation, and the first output layer outputs the ninth feature map to obtain a predicted mask area image.
[0050] S114: constructing a first loss function by predicting the mask area image and the true mask area image.
[0051] Specifically, a cross-entropy loss function is used to calculate the error value between the predicted mask region image and the true mask region image. In a specific embodiment, the first loss function is a cross-entropy loss.
[0052] S115: Using the first loss function to iteratively train the initial feature extraction network model to obtain a feature extraction network model.
[0053] Specifically, the initial feature extraction network model is iteratively trained by predicting the error value between the mask area image and the true mask area image to obtain the feature extraction network model.
[0054] In an optional embodiment, the result of the initial feature extraction network model is back-propagated, and the weight of the initial feature extraction network model is corrected according to the loss value fed back by the first loss function. In an optional embodiment, the parameters in the initial feature extraction network model can also be corrected to achieve the training of the initial feature extraction network model.
[0055] The scanned image data is input into the initial feature extraction network model, and the initial feature extraction network model predicts the mask area image in the scanned image data. When the error value between the predicted mask area image and the true mask area image is less than a preset threshold value, which can be set by itself, such as 1%, 5%, etc., the training of the initial feature extraction network model is stopped and the feature extraction network model is obtained.
[0056] By training the segmentation network to obtain a feature extraction network model, the generalization of the network can be increased, the feature extraction network model can be promoted to learn robust high-level feature extraction, and the accuracy of feature extraction can be improved.
[0057] S12: Train the feature extraction network model and the deep learning classification network model.
[0058] See also Figure 5 , Figure 5 yes Figure 2A flowchart of a specific embodiment of step S12 in the airway classification method provided. Specifically, an initial classification network model is constructed. The initial classification network model includes a fourth feature extraction layer, a filling layer, a reshaping layer, a linear layer, and a second output layer. The number of input channels of the fourth feature extraction layer is 2048, the number of output channels is 32, and the step length s=(1, 1, 1). The fourth feature extraction layer extracts features from the feature map and performs size conversion; the number of converted feature maps is supplemented by the filling layer so that the number of airway feature maps input to the linear layer is the same as the maximum number of slices; the linear layer identifies the input feature map to obtain the predicted category of the airway. Based on the feature extraction network model obtained by the above training, the airway scanning image data is feature extracted and the initial classification network model is pre-trained. The airway scanning image data is input into the feature extraction network model, and while the initial classification network model is pre-trained, the parameters in the feature extraction network model are also trained so that the obtained deep learning feature extraction network model can extract features more accurately. Specifically, the training of the feature extraction network model and the deep learning classification network model specifically includes the following steps.
[0059] S121: Acquire a second training sample set, where the second training sample set includes a plurality of second image samples.
[0060] Specifically, the second image sample includes airway scanning data, airway region image data corresponding to the airway scanning data, and the true category of the airway in the airway scanning data. The airway scanning image data is connected to the airway region image data. In a specific embodiment, the airway scanning image data is connected to the airway region image to obtain two-channel data. In a specific embodiment, the airway scanning image data is a three-dimensional CT airway scanning image. The three-dimensional CT airway scanning image includes multiple CT airway slice images. In a specific embodiment, please refer to Figure 6 , Figure 6 It is a flowchart of a specific embodiment of step S121 in the airway classification method provided by the present invention. Before the airway scanning image data is connected with the airway region image data, the airway scanning image data needs to be preprocessed, and the preprocessing includes the following steps.
[0061] S1211: Screening valid data points in the airway scanning image data.
[0062] Specifically, invalid values in the airway scanning image data can be removed to reduce the situation where some pixels in the airway scanning image have abnormal values caused by factors such as noise and metal. In a specific embodiment, all pixel data in the CT airway slice image are sorted, and pixel points with pixel values in the range of [0.005, 0.995] are taken as valid data points. The value range of the valid data point can be set as needed.
[0063] S1212: Calculate the mean and variance of the valid data points and perform normalization processing.
[0064] Specifically, the mean and variance of the valid data points obtained in the above steps are calculated, and then the calculated mean and variance are normalized. In another optional embodiment, in order to ensure the integrity of the airway scanning image data, the invalid data points in the CT airway slice image are replaced with standard normal distribution random numbers, thereby completing the data points and maintaining the data integrity of the CT airway slice image.
[0065] S1213: Connect the normalized airway scanning image data with the airway region image data to obtain two-channel data.
[0066] Specifically, the normalized airway scanning image data obtained in step S1212 is connected with the airway region image corresponding to the airway scanning image data to obtain two-channel data.
[0067] S1214: Perform data enhancement processing on the two-channel data.
[0068] Specifically, the data enhancement processing includes at least one of data translation processing, data rotation processing and data scaling processing. In a specific embodiment, the two-channel data can be translated within 10% of the original size to increase the image samples, the two-channel data can be rotated within the range of [-5°, 5°] to enhance the image samples, and the two-channel data can be scaled within the range of [0.8, 1.2] times the original size to increase the image samples.
[0069] The size of common head and neck three-dimensional CT airway scanning images is generally Z×H×W, where H and W represent the length and width of the image, and Z represents the number of slices. Data of size Z×H×W is input into the feature extraction network model, and the feature map size output after the feature extraction network model is Z'×H'×W', where Z'=Z / 4, H'=H / 32, and W'=W / 32. For three-dimensional CT airway scanning images of different patients, H and W are the same, while the number of slices Z will vary with the size of the patient and the position of the patient in the scanner. Therefore, the feature maps obtained after different three-dimensional CT airway scanning images pass through the feature extraction network model are different. In an optional embodiment, the maximum number of slices of the airway scanning image data in the second training sample set is determined. In other words. Determine the maximum number Z of CT airway slice images corresponding to all three-dimensional CT airway scanning images in the second training sample set max , Z maxAs the input size of the linear layer, the input size of the linear layer must be a fixed value. The three-dimensional CT airway scanning image includes a CT airway slice image.
[0070] S122: Input the second image sample into the feature extraction network model to extract features and obtain an airway feature map.
[0071] Specifically, the two-channel data obtained by connecting the airway scanning image data in the second image sample with the airway region image is input into the feature extraction network model, and the feature extraction network model extracts features from the two-channel data to obtain an airway feature map. Specifically, the feature extraction network model is pre-trained in step S11, and the step of the feature extraction network model extracting features from the two-channel data in the second image sample is the same as the step of step S112, which will not be repeated here.
[0072] S123: Input the airway feature map into the initial classification network model for classification to obtain the predicted category of the airway.
[0073] Specifically, the airway feature map is input into the fourth feature extraction layer for feature extraction to obtain the fourth feature map, and the filling layer fills the size of the fourth feature map to the size set by the linear layer to avoid increasing the computational complexity and the loss of important spatial information. In a specific embodiment, the airway feature map is converted into a fixed size, that is, the airway feature map is adaptively filled to a predefined size Z' max , where Z' max =Z max / 4,Z max is the maximum number of CT airway slice images corresponding to the three-dimensional CT airway scan image in the second training data set, the fourth feature map supplemented by the reshaping layer is tiled into a one-dimensional tensor, the linear layer performs linear combination on the tiled one-dimensional tensor and then performs recognition, and the second output layer outputs the predicted category of the identified airway. In a specific embodiment, the predicted category of the airway includes two types: difficult airway and non-difficult airway.
[0074] S124: Construct a second loss function by using the true category of the airway and the predicted category of the airway.
[0075] Specifically, a cross-entropy loss function is used to calculate the error value between the true category of the airway and the predicted category of the airway. In a specific embodiment, the second loss function is a cross-entropy loss.
[0076] S125: Using the second loss function, iteratively train the feature extraction network model and the initial classification network model to obtain a deep learning feature extraction network model and a deep learning classification network model.
[0077] Specifically, the feature extraction network model and the initial classification network model are iteratively trained through the error value between the true category of the airway and the predicted category of the airway to obtain the deep learning feature extraction network model and the deep learning classification network model.
[0078] In an optional embodiment, the result of the initial classification network model is back-propagated, and the weights of the feature extraction network model and the initial classification network model are modified according to the loss value fed back by the second loss function. In an optional embodiment, the parameters in the feature extraction network model and the initial classification network model can also be modified to achieve the training of the feature extraction network model and the initial classification network model.
[0079] The second image sample is input into the feature extraction network model for feature extraction, and the airway category in the airway scan image data is predicted by the initial classification network model. When the error value between the true category of the airway and the predicted category of the airway is less than a preset threshold value, which can be set by itself, such as 1%, 5%, etc., the training of the parameters in the feature extraction network model and the initial classification network model is stopped, and a deep learning feature extraction network model and a deep learning classification network model are obtained.
[0080] Through the above steps, the initial feature extraction network model and the initial classification network model are trained in sequence to obtain a deep learning feature extraction network model and a deep learning classification network model.
[0081] S2: Acquire airway image data.
[0082] Specifically, the airway scanning image data to be detected is obtained, wherein the airway scanning image data is a CT airway slice image.
[0083] S3: Segment the airway image data to obtain airway region image data.
[0084] See also Figure 7 , Figure 7 yes Figure 1 A flow chart of a specific embodiment of step S3 in the airway classification method is provided. The airway region image data can be obtained according to the airway scanning image data, which specifically includes the following steps.
[0085] S31: Setting a threshold, and using threshold segmentation to perform binarization processing on the airway image data to extract an initial airway region image.
[0086] Specifically, the CT value in the CT airway slice image is determined based on the linear absorption coefficient of various tissues to X-rays. There is a large amount of air in the airway, and the CT value of the air is about -1000HU. In a specific embodiment, the threshold is set to -200HU, and the area with a CT value less than or equal to -200HU is set to 1, and the area with a CT value greater than -200HU is set to 0. The CT airway slice image is then segmented using the threshold of -200HU to obtain an initial airway region image. At this point, the initial airway region image is a black and white image, with white being the airway region and the airway being the non-airway region.
[0087] S32: Preprocessing the initial airway region image, where the preprocessing includes hole filling and small region removal.
[0088] Specifically, due to the influence of noise, there are a large number of black holes and small white areas in the initial airway region image obtained after threshold segmentation. In a specific embodiment, the initial airway region image obtained after threshold segmentation is processed by an opening operation to make the contour of the airway in the initial airway region image smooth, disconnect the narrow neck and eliminate thin protrusions; the initial airway region image after threshold segmentation is processed by a closing operation to bridge the narrow discontinuity and the elongated gully, eliminate small holes, and fill the cracks in the contour line.
[0089] S33: construct an airway template and determine a seed point in the initial airway region image using the airway template.
[0090] Specifically, an airway template is constructed; the airway template is matched with a connected domain of each voxel point in the initial airway region image; and a voxel point with the highest overlap between the connected domain and the airway template is determined as a seed point.
[0091] The shape of the airway varies from patient to patient and from different positions of the same patient, but the overall feature of the airway is a slender cylindrical structure. In a specific embodiment, the airway template is constructed by simulating a section of a cylinder with a diameter of 32 and a height of 16 in a cube with a length, width, and height of 64, 64, and 16, respectively, as the airway template. The unit of the airway template is voxel, and the cross section of the specific airway template is circular.
[0092] In a specific embodiment, the three-dimensional CT airway scan image is generally from the nostrils to the upper part of the lungs. Since the sinuses and other parts also contain a lot of air, the slices containing the nostrils have different shapes after threshold segmentation, while the result of the threshold segmentation of the slices above the lungs is closer to a tubular shape, and the cross section is also circular. There are multiple voxel points in the initial airway area image. The connected domains with a diameter of 32 and a height of 16, centered on each voxel point, are matched with the airway template respectively, and the similarity between each connected domain and the airway template is determined. The voxel point corresponding to the connected domain with the highest similarity to the airway template is determined as the seed point.
[0093] S34: Determine whether the connected domain of the seed point is an airway region image according to the proportion of the voxel points in the connected domain of the seed point in the voxel points of the airway image data.
[0094] Specifically, the voxels in the connected domain where the seed point is located are counted, and the proportion of the voxels in the connected domain where the seed point is located in the voxel points of the three-dimensional CT airway scanning image is calculated. Among them, the proportion of the airway in the three-dimensional CT airway scanning image is generally in the range of 0.2%-0.8%. In a specific embodiment, it is determined whether the proportion of the voxel points in the connected domain of the seed point in the voxel points of the three-dimensional CT airway scanning image is in the range of 0.2% to 0.8%. If the proportion is in the range of 0.2%-0.8%, it is determined that the connected domain of the seed point is an airway region image; if the proportion is not in the range of 0.2%-0.8%, it is determined that the connected domain of the seed point is not an airway region image.
[0095] In a specific embodiment, if it is determined that the connected domain of the seed point is not an airway region image, the diameter of the airway template needs to be adjusted, and a new airway template is regenerated after adding 2 to the diameter of the original airway template, and then steps S33 and S34 are performed again. Until the connected domain where the airway is located is found, and the connected domain is displayed, the airway region image data is obtained. This method can accurately segment the airway from the three-dimensional CT airway scan image.
[0096] S4: Connect the airway image data and the airway region image data to obtain two-channel data.
[0097] Specifically, in order to enable the deep learning network to focus on the airway area in the airway scan image while retaining the learning ability of other parts in the airway scan image, thereby improving the accuracy of extracting the feature map of the airway, the CT airway slice image data and the airway area image data are connected to obtain two-channel data. In this step, it is not necessary to perform enhancement processing on the two-channel data.
[0098] S5: Input the two-channel data into a pre-trained deep learning network to confirm whether the airway in the airway image is a difficult airway.
[0099] See also Figure 8 , Figure 8 yes Figure 1 A flow chart of a specific embodiment of step S5 in the airway classification method is provided. Specifically, the two-channel data obtained in step S4 is input into the deep learning feature extraction network model to obtain an airway feature map, and the airway feature map is identified by the deep learning classification network model to determine whether the airway in the airway scanning image is a difficult airway. Among them, determining whether the airway in the airway scanning image is a difficult airway through the two-channel data specifically includes the following steps.
[0100] S51: Input the two-channel data into a pre-trained deep learning feature extraction network model to extract features and obtain an airway feature map.
[0101] Specifically, the two-channel data is input into the deep learning feature extraction network model for feature extraction. In a specific embodiment, the step of the deep learning feature extraction network model for feature extraction of the two-channel data is the same as the specific operation of the above-mentioned step S112, which is not repeated here. Among them, the parameters of the first feature extraction layer, the pooling layer, the second feature extraction layer, the third feature extraction layer, the fifth feature extraction layer and the sixth feature extraction layer in the deep learning feature extraction network model are different from the parameters of the first feature extraction layer, the pooling layer, the second feature extraction layer, the third feature extraction layer, the fifth feature extraction layer and the sixth feature extraction layer in the above-mentioned step S112.
[0102] S52: Input the airway feature map into a pre-trained deep learning classification network model to identify and determine whether the airway is a difficult airway.
[0103] Specifically, the airway feature map obtained in the above step S51 is input into the deep learning classification network model to identify the airway feature map, and then the airways in the airway feature map are classified. In a specific embodiment, the step of the deep learning classification network model to identify and classify the airway feature map is the same as the specific operation in the above step S123, which will not be repeated here. Among them, the parameters of the fourth feature extraction layer, filling layer, reshaping layer and linear layer in the deep learning classification network model are different from the parameters of the fourth feature extraction layer, filling layer, reshaping layer and linear layer in the initial classification network model. Among them, the input size of the linear layer in this embodiment is determined according to the maximum number of CT airway slice images in the three-dimensional CT airway scanning image, so that the deep learning network can focus on the airway area in the CT airway scanning image, and is suitable for three-dimensional CT airway scanning images with different numbers of slices, thereby improving the classification accuracy.
[0104] The airway classification method provided in this embodiment obtains airway scanning image data, obtains airway region image data based on the airway scanning image data, generates two-channel data by connecting the airway scanning image data with the airway region image data, and inputs the two-channel data into the deep learning network model to determine whether the airway in the airway scanning image is a difficult airway. This application inputs the two-channel data into the deep learning network, so that the deep learning network can effectively use the airway information in the airway region image data to identify whether the airway is a difficult airway. It can also retain the deep learning network's learning ability for other parts besides the airway information in the airway scanning image data, thereby improving the accuracy of airway classification. The method is simple and the classification results are reliable.
[0105] See also Fig. 9 , Fig. 9 FIG. 1 is a schematic block diagram of an embodiment of an airway classification device provided by the present invention. Fig. 9 As shown, the airway classification device 80 in this embodiment includes: a processor 81, a memory 82, and a computer program stored in the memory 82 and executable on the processor 81. When the computer program is executed by the processor 81, the steps in the above-mentioned airway classification method are implemented. To avoid repetition, they are not described one by one here.
[0106] See also Fig.10 , Fig.10 It is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention.
[0107] A computer-readable storage medium 90 is also provided in an embodiment of the present application. The computer-readable storage medium 90 stores a computer program 901. The computer program 901 includes program instructions. The processor executes the program instructions to implement the steps of any airway classification method provided in the embodiment of the present application.
[0108] The computer-readable storage medium 90 may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium 90 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.
[0109] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. An airway classification method, characterized in that: The method comprises: Acquiring airway image data; Segmenting the airway image data to obtain airway region image data; Connecting the airway image data and the airway region image data to obtain two-channel data; Inputting the two-channel data into a pre-trained deep learning network to confirm whether the airway in the airway image is a difficult airway; wherein the deep learning network includes a deep learning feature extraction network model and a deep learning classification network model, and the deep learning classification network model includes a fourth feature extraction layer, a filling layer, and a linear layer; The step of inputting the two-channel data into a pre-trained deep learning network to confirm whether the airway in the airway image is a difficult airway specifically includes: Inputting the two-channel data into the pre-trained deep learning feature extraction network model to extract features and obtain an airway feature map; Determine the maximum number of slices of the airway characteristic map of the two-channel data; Determine the maximum number of slices as the number of the airway feature maps input into the linear layer; The fourth feature extraction layer extracts features from the airway feature map and performs size conversion; The number of the converted airway feature maps is supplemented by the filling layer so that the number of the airway feature maps input into the linear layer is the same as the maximum number of slices; The linear layer identifies the input airway feature map to obtain a classification result of the airway.
2. The airway classification method according to claim 1, characterized in that: The step of segmenting the airway image data to obtain airway region image data specifically includes: Setting a threshold, and performing binarization processing on the airway image data using threshold segmentation to extract an initial airway region image; Preprocessing the initial airway region image, wherein the preprocessing includes hole filling and small region removal; Constructing an airway template and determining a seed point in the initial airway region image by using the airway template; Whether the connected domain of the seed point is the airway region image is determined according to the proportion of the voxel points in the connected domain of the seed point in the voxel points of the airway image data.
3. The airway classification method according to claim 2, characterized in that: The step of constructing an airway template and determining a seed point in the initial airway region image by using the airway template specifically includes: Construct an airway template; Matching the airway template with the connected domain of each voxel point in the initial airway region image; The voxel point with the highest overlap between the connected domain and the airway template is determined as the seed point.
4. The airway classification method according to claim 2, characterized in that: The step of determining whether the connected domain of the seed point is the airway region image according to the proportion of the voxel points in the connected domain of the seed point in the voxel points of the airway image data specifically includes: Determine whether the proportion of voxel points in the connected domain of the seed point in the voxel points of the airway image data is within a range of 0.2% to 0.8%; If the ratio is within the range, the connected domain of the seed point is determined to be the airway region image.
5. The airway classification method according to claim 1, characterized in that: The step of acquiring airway image data also includes: The deep learning network is trained, wherein the deep learning feature extraction network model and the deep learning classification network model have three-dimensional convolution kernels.
6. The airway classification method according to claim 5, characterized in that: The step of training the deep learning network specifically includes: Acquire a first training sample set, the first training sample set comprising a plurality of first image samples in a public medical CT (Computed Tomography) image data set, the first image samples comprising image data and a real mask region image corresponding to the scanned image data; wherein the same image data is stacked into two-channel data; Inputting the two-channel data into an initial feature extraction network model to extract features and obtain a feature map; Inputting the feature map into a decoder for segmentation to obtain a predicted mask area image of the image data; Constructing a first loss function through the predicted mask area image and the real mask area image; The initial feature extraction network model is iteratively pre-trained using the first loss function to obtain a feature extraction network model.
7. The airway classification method according to claim 6, characterized in that: The initial feature extraction network model at least includes an input layer, a first feature extraction layer, a pooling layer, a second feature extraction layer and a third feature extraction layer; The step of inputting the two-channel data into the initial feature extraction network model to extract features and obtain a feature map specifically includes: Inputting the two-channel data into the input layer, and preprocessing the two-channel data; The first feature extraction layer performs feature extraction on the preprocessed two-channel data to obtain a first feature map; Performing a pooling operation on the first feature map using the pooling layer; The second feature extraction layer performs feature extraction on the first feature map of the pooling operation to obtain a second feature map; The third feature extraction layer extracts features from the second feature map to obtain a third feature map; The third feature map is connected with the second feature map after upsampling to obtain the feature map.
8. The airway classification method according to claim 7, characterized in that: The number of input channels of the first feature extraction layer is 2.
9. The airway classification method according to claim 7, characterized in that: The step size of the second feature extraction layer and the third feature extraction layer is s=(1, 2, 2).
10. The airway classification method according to claim 6, characterized in that: The step of training the deep learning network specifically includes: Acquire a second training sample set, wherein the second training sample set includes a plurality of second image samples, wherein the second image samples include airway data, airway region image data corresponding to the airway data, and a true category of the airway in the airway data; wherein the airway data is connected to the airway region image data; Inputting the second image sample into the trained feature extraction network model to extract features and obtain the airway feature map; Inputting the airway characteristic map into an initial classification network model for classification to obtain a predicted category of the airway; Constructing a second loss function by using the true category of the airway and the predicted category of the airway; The feature extraction network model and the initial classification network model are iteratively trained using the second loss function to obtain the deep learning feature extraction network model and the deep learning classification network model.
11. The airway classification method according to claim 10, characterized in that: The step of obtaining a second training sample set, wherein the second training sample set includes a plurality of second image samples, and the second image samples include airway data, airway region image data corresponding to the airway data, and a true category of the airway in the airway data; wherein the step of connecting the airway data with the airway region image data further includes: Screening valid data points in the airway image data; Calculating the mean and variance of the valid data points and performing normalization processing; The normalized airway image data is connected with the airway region image data to obtain the two-channel data, Perform data enhancement processing on the two-channel data.
12. The airway classification method according to claim 11, characterized in that: The data enhancement processing includes at least one of data translation processing, data rotation processing and data scaling processing.
13. The airway classification method according to claim 10, characterized in that: The initial classification network model includes a fourth feature extraction layer, a filling layer and a linear layer, Before the step of inputting the second image sample into the deep learning feature extraction network model to extract features to obtain the airway feature map, the step further includes: Determining a maximum number of slices of the airway image data in the second training sample set; Determine the maximum number of slices as the number of the airway feature maps input into the linear layer; The step of inputting the airway characteristic map into the initial classification network model for classification to obtain the predicted category of the airway specifically includes: The fourth feature extraction layer extracts features from the airway feature map and performs size conversion; The number of the converted airway feature maps is supplemented by the filling layer so that the number of the airway feature maps input into the linear layer is the same as the maximum number of slices; The linear layer identifies the input airway feature map to obtain the predicted category of the airway.
14. An airway classification device, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to implement the steps in the airway classification method as described in any one of claims 1 to 13.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the airway classification method according to any one of claims 1 to 13 are implemented.
Citation Information
Patent Citations
Pulmonary nodule detection device and method based on shape template matching and combining classifier
CN104751178A
Blood vessel extracting method
CN106296664A
Pulmonary airway image segmentation method, terminal and storage medium
CN108510478A
Image classification method and device, computer readable storage medium and computer equipment
CN110321920A
Image classification method based on CNN fragmentation multi-scale feature fusion
CN111079795A
