A method for automatic segmentation of pulmonary artery and vein based on neural network

Abstract

The application discloses a lung artery and vein automatic segmentation method based on a neural network, which comprises the following steps: (S100) enhancing a bronchus part in CT through an image enhancement algorithm; (S200) constructing a neural network, combining the enhanced CT data and original CT images as input data of the neural network, and performing neural network training to complete artery and vein segmentation or classification training; (S300) performing image enhancement on a lung bronchus for a CT scanning data to be analyzed, combining the enhanced CT data and original CT images as input data of the neural network for prediction, outputting a probability maximum value of each voxel point as a prediction target, i.e. background, artery or vein, generating a corresponding organ mask, and performing three-dimensional reconstruction. The application effectively improves the accuracy of lung artery and vein segmentation and classification by introducing the anatomical information of the lung bronchus in the neural network training process.

Description

TECHNICAL FIELD

[0001] The present application relates to a lung artery and vein automatic segmentation method, in particular to a lung artery and vein automatic segmentation method based on a neural network. BACKGROUND

[0002] With the development of medical technology towards digitization and precision, three-dimensional reconstruction models of human organs are increasingly applied. Medical image information collected by CT (computed tomography), MRI (magnetic resonance imaging), color ultrasound and other devices is processed by computers to reconstruct three-dimensional models of organs. Through three-dimensional models, doctors can accurately and intuitively understand the spatial relationship between lesions and surrounding tissues, plan preoperative procedures and guide surgery.

[0003] In thoracic surgery, doctors need to determine surgical cutting plans based on the spatial positional relationship between tumors and arteries, veins and bronchi, and position tumors during surgery. Lung artery and vein segmentation and classification based on preoperative thin-layer CT scanning is crucial for accurate preoperative planning and intraoperative guidance. Existing lung artery and vein segmentation and classification tools require a lot of manual post-processing and manual adjustment, consuming a lot of manpower.

[0004] Lung arteries and veins grow in a complex tree-like structure, and have the same gray value in CT. Due to limited CT spatial resolution and local volume effects, lung artery and vein pixels appear adjacent to each other in CT. When an artery and vein segmentation and classification algorithm attempts to distinguish the artery and vein system through connected component analysis, errors often occur, with arteries being labeled as veins or vice versa. Due to the complex anatomical structure of the artery and vein system, manual post-processing is very difficult, time-consuming and labor-intensive, and is prone to omissions, affecting surgical planning. An artery and vein segmentation and classification system based on deep neural networks can achieve higher blood vessel recognition accuracy than traditional algorithms based on local second-order differential analysis, and can learn the connectivity of arteries and veins from the mediastinal artery and vein to the peripheral artery and vein through multi-scale context information analysis, achieving automatic classification of arteries and veins. However, current deep learning algorithms still have many errors in artery and vein classification. SUMMARY

[0005] The present application provides a lung artery and vein automatic segmentation method based on a neural network, which solves the problem of errors in artery and vein classification in existing methods and improves the accuracy of artery and vein and bronchial segmentation.

[0006] To achieve the above-mentioned purpose, the present application provides a lung artery and vein automatic segmentation method based on a neural network, which comprises:

[0007] (S100) Enhance the bronchial part in CT through an image enhancement algorithm;

[0008] (S200) Constructing a neural network, merging the enhanced CT data as an independent channel with the original CT image to form double-channel data, taking the double-channel data as input data of the neural network, and performing neural network training to complete the training of artery and vein segmentation or classification;

[0009] (S300) For the CT scan data to be analyzed, using the image enhancement algorithm of step (S100) to enhance the lung bronchus, taking the enhanced CT data as an independent channel and merging it with the original CT image to form double-channel data, inputting the input data into the neural network for prediction, outputting the maximum probability of each voxel point as the prediction target, i.e. background, artery or vein, generating a corresponding organ mask, and performing three-dimensional reconstruction.

[0010] Preferably, in step (S100), the size of each chest lung CT image is (w, h), and there are a total of d images, so the three-dimensional size is (d, w, h). The number of pixel points in the x-axis direction in the Hessian matrix is w, the number of pixel points in the y-axis direction is h, and the number of pixel points in the z-axis direction is d. The Hessian matrix of each voxel of the CT scan data is calculated as follows:

[0011]

[0012] In formula (1), v is the value of each voxel of the CT scan; v" x,x represents the second-order partial derivative of the voxel point in the x-axis direction; v" x,y represents the second-order partial derivative of the voxel point in the x-axis and y-axis directions; v" x,z represents the second-order partial derivative of the voxel point in the x-axis and z-axis directions; v" y,x represents the second-order partial derivative of the voxel point in the y-axis and x-axis directions; v" y,y represents the second-order partial derivative of the voxel point in the y-axis direction; v" y,z represents the second-order partial derivative of the voxel point in the y-axis and z-axis directions; v" z,x represents the second-order partial derivative of the voxel point in the z-axis and x-axis directions; v" z,y represents the second-order partial derivative of the voxel point in the z-axis and y-axis directions; v" z,z represents the second-order partial derivative of the voxel point in the z-axis direction.

[0013] The three eigenvalues λ1, λ2 and λ3 of the Hessian matrix are calculated, and their absolute values are sorted. Assuming |λ1|<|λ2|<|λ3|, if |λ1|~0, |λ2|, |λ3|>>0, it is considered that the local around this voxel is a low gray value pipe shape, and it is considered to be a bronchus part. The corresponding voxel is enhanced, the voxel value of the bronchus covering position is increased, and the value of the voxel outside the bronchus is inhibited.

[0014] Preferably, in step (S200), assuming that the neural network is trained patch by patch with the size of input patch as (patch_d, patch_w, patch_h), the size of the merged input data is (channel=2, patch_d, patch_w, patch_h), and the size of the input data is used as the input of the trained neural network.

[0015] Preferably, in step (S200), the neural network is trained by the mini-batch method, and each time a training sample with a mini-batch size is input, the size of the overall input sample tensor is (batch_size, channel, patch_d, patch_w, patch_h), and the training label is the manually labeled pulmonary artery and vein label.

[0016] Preferably, in step (S300), the enhanced CT data is combined with the original CT data to form dual-channel data, and the batch dimension is expanded through dimension expansion, and the batch_size size is fixed as 1, and the final input data size is (1, 2, patch_d, patch_w, patch_h).

[0017] The neural network-based automatic pulmonary artery and vein segmentation method of the present application solves the problem of errors in the classification of arteries and veins in existing methods, and has the following advantages:

[0018] (1) The present application effectively improves the accuracy of pulmonary artery and vein segmentation and classification by introducing the anatomical information of the pulmonary bronchus in the neural network training process.

[0019] (2) The present application effectively improves the accuracy of automatic pulmonary artery and vein segmentation and classification by optimizing the neural network training method.

[0020] (3) The present application improves the contrast of bronchus position pixels through an image enhancement algorithm, so that the bronchus information is maintained in the pooling process during the neural network downsampling process. The neural network training automatically learns the accompanying features of the bronchus and the artery through the artery and vein labels and the bronchus information, and automatically introduces the anatomical prior information in the artery and vein classification.

[0021] (4) The present application enhances the bronchus through Hessian matrix analysis, automatically introduces the pulmonary bronchus information through neural network training, and does not need to additionally increase the neural network labeling function. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1The lung bronchus enhancement data of the application is merged with the original CT data into a dual-channel data graph.

[0023] Figure 2 The lung pulmonary vein labeling data graph of the application.

[0024] Figure 3 The neural network architecture diagram of the training patch of the application is [128, 112, 160].

[0025] Figure 4 The lung pulmonary vein reconstruction graph of the application.

[0026] Figure 5 The different lung pulmonary vein segmentation method graph of the application.

[0027] Note: Figure 5 In the figure, blue represents the lung bronchus; red represents the pulmonary artery; green represents the pulmonary vein; the yellow arrow in (b), (e) and (h) points to the enlarged area in (a), (d) and (g) respectively; (c), (f) and (i) are enlarged views of the positions of the yellow arrows in (b), (e) and (h) respectively; the yellow enlarged area in (a) is a cross section of the 5th right pulmonary artery in the label data; the yellow enlarged area in (d) is a non-artery misclassified as a pulmonary vein; the yellow enlarged area in (g) is a correctly segmented pulmonary artery. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the application will be described below clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0029] Embodiment 1

[0030] A lung pulmonary vein automatic segmentation method based on a neural network, the method comprising:

[0031] (S100) bronchus information enhancement based on second-order differentiation

[0032] According to the characteristic that the pulmonary artery and the bronchus grow together, the bronchus part in the CT is enhanced through an image enhancement algorithm. Assuming that each chest lung CT image has a size of (w, h), and there are d images, the three-dimensional size is (d, w, h), and the three-dimensional size is the number of pixel points in the x, y and z axis directions, that is, the number of pixel points w in the x axis direction, the number of pixel points h in the y axis direction and the number of pixel points d in the z axis direction.

[0033] The Hessian matrix of each voxel of the CT scan data is calculated, which is:

[0034]

[0035] In formula (1), v is a CT scanning value of each voxel; v' x ' ,x represents a second-order partial derivative of the voxel point in the x-axis direction; v' x ' ,y represents a second-order partial derivative of the voxel point in the x-axis and y-axis directions; v' x ' ,z represents a second-order partial derivative of the voxel point in the x-axis and z-axis directions; v' y ' ,x represents a second-order partial derivative of the voxel point in the y-axis and x-axis directions; v' y ' ,y represents a second-order partial derivative of the voxel point in the y-axis direction; v' y ' ,z represents a second-order partial derivative of the voxel point in the y-axis and z-axis directions; v' z ' ,x represents a second-order partial derivative of the voxel point in the z-axis and x-axis directions; v' z ' ,y represents a second-order partial derivative of the voxel point in the z-axis and y-axis directions; v' z ' ,z represents a second-order partial derivative of the voxel point in the z-axis direction.

[0036] Three eigenvalues λ1, λ2 and λ3 of the Hessian matrix are calculated (the three eigenvalues represent the change size of the three largest directions of the local gray value of the voxel), and the absolute values are sorted. Assuming λ1<λ2<λ3, if λ1~0 and λ2, λ3>>0 are simultaneously satisfied, it is considered that the local around the voxel is a low gray value pipe shape, and it is considered to be a bronchial part. The corresponding voxel is enhanced, the voxel value of the bronchial covering position is increased, and the value of the voxel outside the bronchial is inhibited. If any one of λ1<λ2<λ3, λ1~0 and λ2, λ3>>0 is not satisfied, the voxel point does not belong to the point of the target pipe tissue, and a lower value is given to it. Finally, through threshold judgment, it is excluded from the segmentation area of the target pipe tissue.

[0037] (S200) Constructing a neural network and performing neural network training

[0038] According to the traditional 3DUnet architecture, a neural network is constructed, the enhanced CT data is merged as an independent channel with the original CT image, the input size of the neural network is the merging result of the original input size and the enhanced CT data, and Figure 1 is used as 2-channel data for arteriovenous segmentation / classification training.

[0039] Assuming that the neural network is trained patch by patch with the size of the input patch being (patch_d, patch_w, patch_h), the size of the combined input data is (channel=2, patch_d, patch_w, patch_h), and the input data size is used as the input of the trained neural network.

[0040] The neural network is trained by the mini-batch method, and each time the training sample of the mini-batch size is input, the overall input sample tensor size is (batch_size, channel, patch_d, patch_w, patch_h), and the training label is the manually labeled lung artery and vein label.

[0041] (S300) Neural network prediction

[0042] New CT scan data is obtained, the lung bronchus is enhanced in step (S100), the enhanced CT data is combined with the original CT data to form double-channel data, and the batch dimension is expanded through dimension expansion, and the batch_size size is fixed as 1. The final input data size is (1, 2, patch_d, patch_w, patch_h). The input data is input into the neural network for prediction, the probability of each voxel point belonging to different categories in the output result is compared, the target with the maximum probability is taken as the prediction target, i.e. background, artery or vein, the corresponding organ mask is generated, and three-dimensional reconstruction is performed, see Figure 4 .

[0043] As shown in Figure 3 , it is a neural network architecture diagram of the training patch of the application, and the neural network architecture and corresponding parameters used by the application can be seen.

[0044] As shown in Figure 5 , it is a different lung artery and vein segmentation method of the application, and the improved neural network bronchus segmentation accuracy Dice value parameter reaches 91%, which is effectively improved compared with before the improvement.

[0045] Although the content of the application has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as a limitation of the application. After reading the above content, various modifications and alternatives of the application will be obvious to those skilled in the art. Therefore, the protection scope of the application should be defined by the appended claims.

Claims

1. A method for automatic segmentation of pulmonary artery and vein based on neural network, characterized in that, The method comprises: (S100) enhancing the bronchial part in the CT by an image enhancement algorithm; (S200) constructing a neural network, merging the enhanced CT data as an independent channel and the original CT image to form double-channel data, taking the double-channel data as input data of the neural network, and performing neural network training to complete arteriovenous segmentation or classification training; (S300) for the CT scan data to be analyzed, using the image enhancement algorithm of step (S100) to enhance the pulmonary bronchus, merging the enhanced CT data as an independent channel and the original CT image to form double-channel data, inputting the input data into the neural network for prediction, outputting the maximum probability of each voxel point as the prediction target, i.e. background, artery or vein, generating a corresponding organ mask, and performing three-dimensional reconstruction; In step (S100), assuming that the size of each chest lung CT image is (w, h), and there are d images, the three-dimensional size is (d, w, h), and the Hessian matrix in the Hessian matrix is x The number of pixel points in the axis direction is w, the number of pixel points in the y-axis direction is h, and the number of pixel points in the z-axis direction is d. The Hessian matrix of each voxel of the CT scan data is calculated as follows: ； In equation (1), v For each voxel value in a CT scan; Indicates voxel points at x Second-order partial derivative in the axial direction; Indicates voxel points at x shaft and y Second-order partial derivative in the axial direction; Indicates voxel points at x shaft and z Second-order partial derivative in the axial direction; Indicates voxel points at y shaft and x Second-order partial derivative in the axial direction; Indicates the voxel point at y Second-order partial derivative in the axial direction; Indicates the voxel point at y shaft and z Second-order partial derivative in the axial direction; Indicates the voxel point at z shaft and x Second-order partial derivative in the axial direction; Indicates the voxel point at z shaft and y Second-order partial derivative in the axial direction; Indicates the voxel point at z Second-order partial derivative in the axial direction; Three eigenvalues of the Hessian matrix are calculated , and are sorted by their absolute values, and it is assumed that if , , , then the local area around this voxel is considered to be of a low- intensity tubular shape, and it is considered to be a bronchial portion. The corresponding voxel is enhanced, and the voxel value at the bronchial covering position is increased, and the value of the voxel outside the bronchus is suppressed.

2. The method of claim 1, wherein, In step (S200), assuming that the neural network is trained patch by patch according to the size of the input patch (patch_d, patch_w, patch_h), the size of the merged input data is (channel=2, patch_d, patch_w, patch_h), and the input data size is taken as the input of the trained neural network.

3. The method of claim 1, wherein, In step (S200), the neural network is trained by the mini-batch method, and each time a training sample of a mini-batch size is input, and the total input sample tensor size is (batch_size, channel, patch_d, patch_w, patch_h), and the training label is the manually labeled pulmonary arteriovenous label.

4. The method of claim 1, wherein, In step (S300), the enhanced CT data and the original CT data are combined to form double-channel data, and the dimension is expanded, the batch dimension is expanded, and the batch_size size is fixed as 1. The final input data size is (1, 2, patch_d, patch_w, patch_h).

Images