An automatic segmentation method and system for blood vessels in organs in enhanced CT images

Abstract

The present invention provides a method and system for automatically segmenting blood vessels within organs in enhanced CT images. The method includes performing organ segmentation processing on the enhanced CT images to extract the organ regions, and obtaining three-dimensional regions of interest in the organs from the organ regions; within the three-dimensional regions of interest in the organs, performing global blood vessel segmentation within the organs and local blood vessel segmentation within the organs respectively to generate a global automatic blood vessel segmentation result within the organs and a local automatic blood vessel segmentation result within the organs; and fusing the global automatic blood vessel segmentation result within the organs and the local automatic blood vessel segmentation result within the organs to obtain the final automatic blood vessel segmentation result within the organs. The present invention proposes a fully automatic progressive deep learning solution for segmenting blood vessels within organs, aiming to maintain the integrity of blood vessels in terms of their shape details and continuity. Experimental results show that this solution has high accuracy, efficiency, and robustness.

Description

Technical Field

[0001] The present invention relates to the field of image machine learning processing, and particularly to an automatic segmentation method and system for blood vessels in organs in enhanced CT images. Background Art

[0002] Cardiovascular and cerebrovascular diseases seriously endanger human health. In recent years, the incidence and mortality of cardiovascular and cerebrovascular diseases have generally shown an upward trend, and the cardiovascular and cerebrovascular mortality rate continues to show a rapid upward trend. Therefore, making an early diagnosis of cardiovascular and cerebrovascular patients and taking scientific and effective treatment measures have very important practical significance. At present, the gold standard for diagnosing cardiovascular and cerebrovascular diseases is angiography within organs, but this method is an invasive examination, with a relatively high price, and there will be problems such as complications, which is not suitable for routine physical examinations and is relatively difficult to promote in grass-roots hospitals. In contrast, the diagnostic method of enhanced CT imaging is safe, reliable and non-invasive, and has been widely used in clinics.

[0003] Through enhanced CT images, the organs and the blood vessel structures within the organs of patients can be reconstructed and evaluated. For a series of tasks of analyzing enhanced CT images, such as stenosis calculation, centerline extraction and plaque analysis, automatic segmentation of blood vessels within organs is a key step. Since the blood vessels within organs may have rich diameter variations and complex trajectories, manual segmentation of blood vessels within organs is very laborious and requires high technical skills, which has led to an increasing demand for automatic segmentation of blood vessels within organs.

[0004] Existing methods for segmenting blood vessels within organs have utilized classical machine learning and modern deep learning methods to segment blood vessels in enhanced CT images. The former methods include region-based, edge-based, tracking-based, graph cut-based and level set-based methods, but all these methods require some kind of manual assistance in their pre-processing or post-processing steps. The latter methods are mainly based on FCN and UNET, and they analyze two-dimensional or three-dimensional patches for automatic segmentation of blood vessels within organs. These two existing types of methods both face two common challenges, namely: 1) relatively small volumes of blood vessels within organs are scattered in the surrounding tissues with a larger volume; 2) there is a high perceptual similarity and close spatial adjacency between the blood vessels within organs and their adjacent vascular structures (such as pulmonary blood vessels). Summary of the Invention

[0005] In order to perform accurate and robust automatic segmentation of blood vessels within organs efficiently, the present invention proposes an automatic segmentation method and system for blood vessels within organs in enhanced CT images, which can preserve the integrity of blood vessels in terms of shape details and continuity.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An automatic segmentation method for blood vessels in organs in enhanced CT images, comprising the following steps:

[0008] Perform organ segmentation processing on the enhanced CT image to extract the organ region, and obtain a three-dimensional region of interest in the organ from the organ region;

[0009] In the three-dimensional region of interest in the organ, perform global blood vessel segmentation in the organ and local blood vessel segmentation in the organ respectively to generate an automatic global blood vessel segmentation result in the organ and an automatic local blood vessel segmentation result in the organ;

[0010] Fuse the automatic global blood vessel segmentation result in the organ and the automatic local blood vessel segmentation result in the organ to obtain the final automatic blood vessel segmentation result in the organ.

[0011] As a further improvement of the present invention, the specific steps of using the organ segmentation module to extract the organ region and the three-dimensional region of interest in the organ from the original CT scan image include:

[0012] After preprocessing the original enhanced CT image by gray mapping, input it into the HVessel-Net deep learning model, and train it through supervised learning and the first loss function to generate an organ segmentation mask;

[0013] Filter the original enhanced CT image through the organ segmentation mask to obtain the organ region, and obtain the corresponding bounding box, which has the same length in the coronal and sagittal directions; extract the three-dimensional region of interest in the organ from the detected organ region through three-dimensional cropping operation.

[0014] As a further improvement of the present invention, the formula of the first loss function is as follows:

[0015]

[0016] y true and y pred respectively represent the ground truth mask and the mask predicted by organ segmentation.

[0017] As a further improvement of the present invention, the global blood vessel segmentation in the organ specifically includes:

[0018] Adjust the input three-dimensional region of interest in the organ to a three-dimensional matrix, input the three-dimensional matrix into the HVessel-Net deep learning model, and train it through supervised learning and the second loss function to generate a three-dimensional mask of the same size as the three-dimensional matrix, corresponding to the blood vessels in the organ detected in the three-dimensional region of interest in the organ; adjust the size of the three-dimensional mask to be consistent with the size of the original enhanced CT image through the input three-dimensional region of interest in the organ to obtain a three-dimensional global automatic blood vessel segmentation mask for the whole case.

[0019] As a further improvement of the present invention, the formula of the second loss function is as follows:

[0020]

[0021] Among them, V t and V p respectively represent the true value mask and the mask generated by automatic segmentation of blood vessels in the global organ, and their skeletons are respectively represented as S t and S p , T prec (S p , V t ) and T sens (S t , V p ) represent the proportion of S p in V t and the proportion of S t in V p , which are also respectively called the accuracy and sensitivity indexes of the topological structure.

[0022] As a further improvement of the present invention, the automatic segmentation of blood vessels in the local organ specifically includes:

[0023] Apply a three-dimensional sliding window to the input three-dimensional organ region of interest, with the same step size in all three dimensions, to generate a series of three-dimensional patches, each patch having a size of 128×128×128; each patch is separately input into the HVessel-Net deep learning model and trained through supervised learning and the third loss function to generate a three-dimensional mask for the blood vessels in the organ within the three-dimensional space represented by the patch; all the generated three-dimensional masks are combined together to form a full-size three-dimensional mask, whose size is the same as that of the input three-dimensional organ region of interest; the size of the three-dimensional mask is adjusted to be consistent with the size of the original enhanced CT image through the input three-dimensional organ region of interest, and a three-dimensional automatic segmentation mask of blood vessels in the local organ of the whole case is obtained.

[0024] As a further improvement of the present invention, the formula of the third loss function is as follows:

[0025]

[0026] Suos_Dice loss = 1 - (α×Suos+(1 - α)×Dice) (6)

[0027] Among them, y true and y pred respectively represent the true value mask and the mask predicted by the segmentation of blood vessels in the organ, and the parameter α is empirically adjusted to 0.3 to maintain the balance between suppressing over-segmentation and improving the accuracy.

[0028] As a further improvement of the present invention, the HVessel-Net deep learning model is constructed by incorporating dilated convolutions into an encoder-decoder convolutional network structure; the specific structure is as follows:

[0029] In the encoding stage, a convolutional layer with a 3×3×3 voxel receptive field reduces the resolution of the feature map, followed by a dilated convolutional layer that preserves the resolution of the input feature map; another dilated convolutional layer further processes the resulting feature map and preserves the resolution of the map.

[0030] The final embedding vector output by the encoding part is transformed by a ResConv block; the transformed embedding vector is fed into the decoding part of the network, which has the same structure as the encoding part of the network but in reverse order.

[0031] A fully convolutional layer is applied, using sigmoid as its activation function, to obtain the final network output.

[0032] As a further improvement of the present invention, the fusion of the global intra-organ vascular automatic segmentation result and the local intra-organ vascular automatic segmentation result specifically refers to a method of using the local segmentation result as the main part and the global segmentation result as a supplement for fusion; specifically including:

[0033] The global intra-organ vascular segmentation result is skeletonized, and the skeletonized result is superimposed on the local intra-organ vascular segmentation result. The maximum connected component of the superimposed result is retained, and the parts lacking in the local intra-organ vascular segmentation result are obtained to remove false positives such as impurities; the global intra-organ vascular segmentation regions corresponding to all the lacking parts are superimposed on the local intra-organ vascular segmentation result, which is used as the final intra-organ vascular segmentation result.

[0034] An automatic intra-organ vascular segmentation system for enhanced CT images, comprising:

[0035] An organ segmentation module for performing organ segmentation processing on the enhanced CT image to extract the organ region and obtaining a three-dimensional organ region of interest from the organ region;

[0036] An intra-organ vascular segmentation module for respectively performing global intra-organ vascular segmentation and local intra-organ vascular segmentation within the three-dimensional organ region of interest to generate a global intra-organ vascular automatic segmentation result and a local intra-organ vascular automatic segmentation result;

[0037] A result fusion module for fusing the global intra-organ vascular automatic segmentation result and the local intra-organ vascular automatic segmentation result to obtain the final intra-organ vascular automatic segmentation result.

[0038] The beneficial effects of the present invention are reflected in:

[0039] Automatic Segmentation Method of Vessels in Organs in Enhanced CT Images of the Present Invention. This study proposes a fully automatic and progressive deep learning solution for segmenting vessels in organs. Based on the region of interest of the organ obtained by the organ segmentation module, the vessels in the organ are segmented, which can eliminate the interference of ribs and pulmonary vessels and make the segmentation more accurate. In the stage of segmenting vessels in the organ, the global vessel segmentation in the organ benefits from focusing on the overall profile features of the vessels in the organ, which can maintain the continuity of the whole vessels, while the local vessel segmentation in the organ focuses on the detailed features of the vessels in the organ, thus ensuring the integrity of the vessel shape. Therefore, a method of fusing global and local segmentation is adopted, aiming to maintain the integrity of the vessels in terms of both their shape details and continuity. Through experimental verification, the proposed solution reaches 91.24% and 1.16 mm respectively in the evaluation indicators Dice (Dice Similarity Coefficient) and ASSD (Average Symmetric Surface Distance), and the average time consumption for each image is only 0.124 seconds, which is higher than the performance of the comparative method. At the same time, for the involved enhanced CT images, even in the case of poor contrast, motion artifacts and plaques in the vessels in the organ, the solution can still robustly produce satisfactory results. Therefore, this solution has high accuracy, efficiency and robustness. Description of the Drawings

[0040] Figure 1 It is the network structure diagram of the HVessel-Net. In the network, Conv represents the convolutional layer, which may also be the dilated convolutional layer; ReLu represents the rectified linear unit layer; BN represents the batch normalization layer; ConvTrans represents the transposed convolutional layer, which may also be the transposed dilated convolutional layer.

[0041] Figure 2 It is the key processing workflow of the progressive solution for segmenting vessels in organs based on deep learning

[0042] Figure 3 It is the schematic diagram of the fusion scheme;

[0043] Figure 4 It shows the organ segmentation results randomly selected from 9 cases out of 20 cases.

[0044] Figure 5 It shows the vessel segmentation results in organs of 9 challenging cases selected from 30 test cases.

[0045] Figure 6 It is the schematic diagram of the process of the automatic segmentation method of vessels in organs in the preferred embodiment of the present invention for enhanced CT images;

[0046] Figure 7Schematic structural diagram of the automatic segmentation system of blood vessels in organs in the enhanced CT image of the preferred embodiment of the present invention;

[0047] Figure 8 Schematic structural diagram of the electronic device of the preferred embodiment of the present invention. Detailed implementation manners

[0048] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0049] The following detailed descriptions are all exemplary descriptions, aiming to provide further detailed descriptions of the present invention. Unless otherwise specified, all technical terms adopted by the present invention have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs. The terms used in the present invention are only for describing specific implementation manners, and are not intended to limit the exemplary embodiments according to the present invention.

[0050] As Figure 6 shown, an automatic segmentation method for blood vessels in organs in the enhanced CT image of the present invention includes the following steps:

[0051] Perform organ segmentation processing on the enhanced CT image to extract the organ region, and obtain a three-dimensional organ region of interest from the organ region;

[0052] In the three-dimensional organ region of interest, perform global blood vessel segmentation in the organ and local blood vessel segmentation in the organ respectively to generate a global automatic blood vessel segmentation result in the organ and a local automatic blood vessel segmentation result in the organ;

[0053] Fuse the global automatic blood vessel segmentation result in the organ and the local automatic blood vessel segmentation result in the organ to obtain the final automatic blood vessel segmentation result in the organ.

[0054] For the automatic segmentation method of blood vessels in organs in the enhanced CT image of the present invention, this study proposes a fully automatic progressive deep learning solution for blood vessel segmentation in organs, aiming to maintain the integrity of blood vessels in terms of their shape details and continuity. Experimental results show that this solution has high accuracy, efficiency and robustness. The specific steps of using the organ segmentation module to extract the organ region and the three-dimensional organ region of interest from the original CT scan image include:

[0055] As a preferred embodiment, the original enhanced CT image is input into the HVessel-Net deep learning model on the basis of gray mapping preprocessing, and is trained through supervised learning and the first loss function to generate an organ segmentation mask;

[0056] Filter the original enhanced CT image through the organ segmentation mask to obtain the organ region and obtain the corresponding bounding box, which has the same length in its coronal and sagittal directions; extract the three-dimensional organ region of interest from the detected organ region through a three-dimensional cropping operation.

[0057] As a preferred embodiment, the global intra-organ vascular segmentation specifically includes:

[0058] Adjust the input three-dimensional organ region of interest to a three-dimensional matrix, input the three-dimensional matrix into the HVessel-Net deep learning model, train through supervised learning and the second loss function to generate a three-dimensional mask of the same size as the three-dimensional matrix, corresponding to all intra-organ vessels detected within the three-dimensional organ region of interest; adjust the size of the three-dimensional mask to be consistent with the size of the original enhanced CT image through the input three-dimensional organ region of interest to obtain the three-dimensional global intra-organ vascular automatic segmentation mask for the entire case.

[0059] As a preferred embodiment, the local intra-organ vascular automatic segmentation specifically includes:

[0060] Apply a three-dimensional sliding window to the input three-dimensional organ region of interest, with the same step size in all three dimensions, to generate a series of three-dimensional patches, each patch having a size of 128×128×128 voxels; each patch is individually input into the HVessel-Net deep learning model, trained through supervised learning and the third loss function to generate a three-dimensional mask for the intra-organ vessels within the three-dimensional space represented by the patch; all the generated three-dimensional masks are combined together to form a full-size three-dimensional mask, whose size is the same as the input three-dimensional organ region of interest; adjust the size of the three-dimensional mask to be consistent with the size of the original enhanced CT image through the input three-dimensional organ region of interest to obtain the three-dimensional local intra-organ vascular automatic segmentation mask for the entire case.

[0061] Specifically, the HVessel-Net deep learning model adopted in the present invention is constructed by integrating dilated convolutions into the backbone of the Vnet network; the specific structure is:

[0062] In the encoding stage, a convolutional layer with a 3×3×3 voxel receptive field reduces the resolution of the feature map, followed by a dilated convolutional layer that retains the resolution of the input feature map; another extended convolutional layer further processes the obtained feature map and retains the resolution of the map.

[0063] The final embedding vector output by the encoding part is transformed by the ResConv block; the transformed embedding vector is fed into the decoding part of the network, whose structure is the same as that of the encoding part of the network but in reverse order.

[0064] Apply a fully convolutional layer, using sigmoid as its activation function, to obtain the final network output.

[0065] Finally, the fusion of the global and local automatic vascular segmentation results within the organ specifically refers to a method of fusion that mainly uses the local segmentation results and supplements them with the global segmentation results; specifically, it includes:

[0066] Skeletonize the global vascular segmentation results within the organ, overlay the skeletonized results on the local vascular segmentation results within the organ, retain the largest connected component of the overlaid results, and obtain the parts lacking in the local vascular segmentation results to remove impurities and other false positives; overlay the global vascular segmentation regions corresponding to all the lacking parts on the local vascular segmentation results within the organ, and use it as the final vascular segmentation result within the organ.

[0067] The method of the present invention will be described in detail below.

[0068] I. Steps of data preprocessing algorithm design and feasibility analysis:

[0069] Data preprocessing refers to the processing algorithm performed on the data before it is put into the deep learning network. The data preprocessing algorithm of the present invention is mainly the gray mapping algorithm.

[0070] The CT value (Hounsfield Unit) range of enhanced CT images is very large, while the data value range put into the deep learning network is usually 0 to 255. If the original enhanced CT images are directly put into the deep learning network, many vascular features will be lost. Therefore, it is necessary to select a Hounsfield Unit window to highlight the vascular features. By statistically analyzing the enhanced CT images, the optimal Hounsfield Unit window for highlighting vascular features is determined to be [-250, 450]. On this basis, a linear gray mapping of the CT values of the original DICOM data from 0 to 255 is performed, as shown in Formula 1.

[0071]

[0072] Where x and y represent the CT values before and after mapping respectively, and min and max are set to -250 HU and 450 HU respectively.

[0073] II. The deep learning model HVessel-Net is specifically described as follows:

[0074] There are already many excellent network architectures for image segmentation, such as the Deeplab series, the unet series, and RefineNet, etc. However, most of these network architectures are for 2D images. In clinical practice, many medical images are 3D volume data (such as MRI, CT). The 3D Vnet network proposed by Fausto Milletari et al. performs well in 3D medical image segmentation. This is mainly because Vnet directly uses 3D convolution for image segmentation and learns the residual function during the convolution stage. However, most networks for image segmentation need to perform convolution and then pooling on the image during the learning process, reducing the image size while increasing the receptive field to obtain more information. But since the image segmentation prediction is pixel-wise output, the smaller image size after pooling needs to be upsampled to the original image size for prediction. During the process of first reducing and then increasing the size, some information is definitely lost. The dilated convolution algorithm can enable the deep learning network to have a larger receptive field and obtain more information without pooling.

[0075] Based on this, the present invention proposes a new model, named the HVessel-Net neural network model, for automatic organ segmentation and automatic blood vessel segmentation within the organ.

[0076] HVessel-Net is constructed by incorporating dilated convolution into an encoder-decoder convolutional network architecture. Figure 1 Shows the network architecture of HVessel-Net. In its encoding stage, a convolutional layer with a 3×3×3 voxel receptive field reduces the resolution of the feature map by half (see ① and ②), followed by a dilated convolutional layer that retains the resolution of its input feature map but achieves a 7×7×7 voxel receptive field with only 3×3×3 parameters (see ③). Another extended convolutional layer further processes the resulting feature map, obtaining a 15×15×15 voxel receptive field using 3×3×3 parameters while also retaining the resolution of these maps (see ④). The final embedded vector output by the encoding part of the network (see ⑤) is transformed by the ResConv block. The transformed embedded vector is fed into the decoding part of the network, which has the same structure as the encoding part of the network but in reverse order (see Figure 1 for details). Finally, a fully convolutional layer is applied, which uses sigmoid as its activation function to obtain the final network output.

[0077] A key advantage of the proposed HVessel-Net model is its ability to directly process three-dimensional data, thus effectively utilizing spatial information. At the same time, with various dilated convolutional layers built into HVessel-Net, a relatively small set of network parameters can be used to obtain a receptive field of sufficient size without affecting the resolution of the feature map, which is contrary to the traditional encoder-decoder convolutional network structure design that uses traditional convolutional layers and thus requires more network parameters. Through the experimental results of the present invention, the performance advantage of HVessel-Net over the encoder-decoder convolutional network structure has been comprehensively improved.

[0078] III. The progressive deep learning solution for automatic segmentation of blood vessels in organs is as follows:

[0079] Based on the newly introduced HVessel-Net model, the present invention further proposes a new progressive deep learning solution for automatic segmentation of blood vessels in organs, and its processing workflow is as Figure 2 shown, and its key operations are listed in Table 1.

[0080] Specifically, it includes:

[0081] First, use its organ segmentation module to extract the organ region and the three-dimensional organ region of interest (3D Cardiac ROI) from the original CT scan image of a case on the basis of gray-scale mapping preprocessing. According to the three-dimensional organ region of interest, the global and local features are independently analyzed by the global automatic segmentation of blood vessels in organs and the local automatic segmentation of blood vessels in organs modules respectively, generating two sets of automatic segmentation results of blood vessels in organs, and finally fused together to synthesize the final automatic segmentation result of blood vessels in organs.

[0082] Table 1: Lists Figure 2 the key operations in it and the dimensions of their respective inputs and outputs.

[0083]

[0084] Specifically, the organ segmentation specifically includes the following contents:

[0085] A capable organ segmentation model can provide two important clinical benefits. First, high-quality organ segmentation results contribute to organ reconstruction, and the results enable doctors to diagnose coronary heart disease more intuitively and accurately; second, the three-dimensional organ region of interest generated by organ segmentation can effectively eliminate ribs and pulmonary blood vessels, thereby obtaining a more accurate automatic segmentation result of blood vessels in organs.

[0086] In the proposed solution, the organ segmentation module is implemented using HVessel-Net, trained through supervised learning, plus the Dice_loss function. The original enhanced CT images are first preprocessed with a gray-scale mapping algorithm before entering the network, and then the data size is adjusted from 512×512×N to 128×128×128.

[0087] The loss function is defined according to the Dice similarity coefficient (Dice) in Equation (2).

[0088]

[0089] where y true and y pred represent the ground truth mask and the mask of the organ segmentation prediction, respectively.

[0090] Subsequently, the three-dimensional organ region of interest detection procedure is performed as follows. First, the size of the organ segmentation mask generated in the first stage is adjusted to 512×512×N, and the original enhanced CT images are filtered to obtain the organ region and derive the corresponding bounding box, which has the same length in its coronal and sagittal directions. The three-dimensional cropping operation further extracts the three-dimensional organ region of interest from the detected organ region.

[0091] IV. The segmentation of blood vessels within the organ includes the following:

[0092] Blood vessels within the organ often exhibit rich diameter variations and complex trajectories, making the automatic segmentation of blood vessels within the organ a highly laborious and technically demanding process. Therefore, the present invention introduces an automatic blood vessel segmentation program within the organ. Given a three-dimensional organ region of interest, it is independently analyzed by the global blood vessel segmentation module and the local blood vessel segmentation module within the organ to generate two sets of automatic blood vessel segmentation results within the organ. These two results are fused together to synthesize the final automatic blood vessel segmentation result within the organ. Benefiting from the global and local automatic blood vessel segmentation within the organ independently completed by the above two modules, the proposed solution can retain the integrity of the blood vessels in terms of shape details and continuity in its final output of automatic blood vessel segmentation within the organ.

[0093] (a) Global blood vessel segmentation within the organ:

[0094] The global intra-organ vessel segmentation module first adjusts the input three-dimensional organ region of interest to a 256×256×128 matrix. Then, this matrix is input into HVessel-Net and trained through supervised learning and the clDice_loss function to generate a three-dimensional mask with a size of 256×256×128, corresponding to all intra-organ vessels detected within the three-dimensional organ region of interest. Then, this mask is adjusted to the size of the input three-dimensional organ region of interest, and finally, through its result, it is made consistent with the size of the original enhanced CT image, generating a three-dimensional global intra-organ vessel automatic segmentation mask for the entire case.

[0095] The clDice_loss defined in Equation (3) aims to maintain the continuity of the global intra-organ vessel automatic segmentation result, where V t and V p represent the ground truth mask and the mask generated by the global intra-organ vessel automatic segmentation respectively, and their skeletons are represented as S t and S p , T prec (S p , V t ) and T sens (S t , V p ) represent the proportion of S p in V t and the proportion of S t in V p , which are also called the accuracy and sensitivity indicators of the topological structure respectively.

[0096]

[0097] (b) Local intra-organ vessel automatic segmentation:

[0098] The local intra-organ vessel segmentation module first applies a three-dimensional sliding window (with a step size of 64 pixels in all three dimensions) to the input three-dimensional organ region of interest to generate a series of three-dimensional patches, each with a size of 128×128×128 voxels. Each patch is individually fed into HVessel-Net and trained through supervised learning and the designed new loss function Suos_Dice_loss to generate a three-dimensional mask for the intra-organ vessels within the three-dimensional space represented by the patch. Then, all the generated three-dimensional masks are combined together to form a full-size three-dimensional mask with the same size as the input three-dimensional organ region of interest. Finally, this full-size three-dimensional mask is adjusted to the size of the original enhanced CT image, generating a three-dimensional local intra-organ vessel automatic segmentation mask for the entire case.

[0099] The Suos_Dice loss, defined in Equation (6), aims to suppress over-segmentation while ensuring the smoothness of the vascular segmentation results within local organs.

[0100]

[0101] Suos_Dice loss = 1 - (α × Suos + (1 - α) × Dice) (6)

[0102] Where y true and y pred represent the ground truth mask and the mask of the predicted blood vessels within the organ, respectively. In all experiments of the present invention, the parameter α is empirically adjusted to 0.3.

[0103] V. The specific fusion method is as follows:

[0104] The global blood vessel segmentation within the organ focuses on the overall characteristics of the blood vessels within the organ, thereby ensuring the overall continuity. However, since both the input and output of the network are sampled, the smoothness of the outer contour of the blood vessels may not be very good. The local blood vessel segmentation within the organ focuses on the detailed characteristics of the blood vessels within the organ, thereby ensuring the morphological integrity. However, since the data input into the network is local small patches, this may lead to situations such as poor continuity and breaks in some blood vessels.

[0105] Combining the advantages and disadvantages of the above two schemes, fusing the two results can obtain better blood vessel segmentation results within the organ. During fusion, the two results are not directly superimposed, but a method mainly based on the local segmentation result and supplemented by the global segmentation result is used for fusion, as shown in Figure 3 . Specifically, first, the global blood vessel segmentation result within the organ is skeletonized; then, the skeletonized result is superimposed on the local blood vessel segmentation result within the organ. To remove impurities and other false positives, the maximum connected component is retained for the superimposed result and the part lacking in the local blood vessel segmentation result (the part with only the skeleton, such as Figure 3 the red part of -(A)) is obtained; finally, the global blood vessel segmentation regions corresponding to all the lacking parts are superimposed on the local blood vessel segmentation result within the organ, and it is used as the final blood vessel segmentation result within the organ.

[0106] VI. The training strategy and model evaluation are specifically as follows:

[0107] Training Strategy: The proposed solution is implemented in Python using the deep learning library Keras, which runs on an Ubuntu 16.04.4 LTS system equipped with an NVIDIA GeForce RTX 2080Ti GPU. The key parameters of the solution are empirically optimized as follows: batch_size = 2, epochs = 90, and initial learning rate = 0.001, monitor = val_loss, patience = 10, factor = 0.1, min_lr = 1e-8. In the training phase, to improve the robustness of the deep learning network, data augmentation methods are needed to increase the diversity of data. In the present invention, online data augmentation is performed in the training phase, using random translation transformation, flipping transformation, and Gaussian blur, etc. The translation transformation performs random translation transformation in three directions: axial, coronal, and sagittal; the flipping transformation only performs random flipping transformation in two directions: coronal and sagittal; the introduction of Gaussian blur is to improve the robustness to data with different image qualities. Gaussian blur processes the data by randomly selecting a blur factor, where the blur factor takes values in [0.1, 0.3, 0.5, 0.7, 1, 1.2].

[0108] Evaluation Metrics: In the present invention, Dice (Dice Similarity Coefficient) and ASSD (Average Symmetric Surface Distance) are used to evaluate the quality of organ segmentation and intra-organ vessel segmentation results. Dice has been widely used to evaluate the quality of image segmentation, which calculates the overlap degree between the segmentation result and its ground truth. The larger the Dice, the better the segmentation quality. Given the extreme importance of the vessel boundary in the automatic intra-organ vessel segmentation result, the average surface distance metric is also used as its measurement standard in the quantitative evaluation process to evaluate the quality of the automatic intra-organ vessel segmentation in terms of the vessel segmentation boundary. The smaller the ASSD, the better the quality of the automatic intra-organ vessel segmentation result.

[0109] Statistical analysis was performed using Python 3.7 on a Linux computer. The diagnostic performance of the deep learning algorithm was evaluated using Dice and ASSD by comparing with the ground truth, and the results were expressed as mean ± standard deviation. Subgroup analysis and comparison of gender, age, and number of slices for Dice and ASSD were performed by Wilcoxon's test. In the present invention, the statistical significance level was defined as P < 0.05.

[0110] For the quantitative results, Table 2 shows the performance of the organ segmentation and the automatic segmentation of blood vessels within the organ produced by the proposed solution in terms of Dice and ASSD. For easy comparison, the table also shows the corresponding performance of the automatic segmentation of blood vessels within the organ of the peer methods. Note that the peer methods cannot perform organ segmentation.

[0111] Twenty cases were randomly selected from the enhanced CT dataset in this invention, and each case had its true organ mask label. The performance of the proposed solution in the first-stage organ segmentation task was tested. The solution achieved (94.08 ± 1.85)% and (4.53 ± 0.63) mm in Dice and ASSD respectively. The performance of the second-stage task of automatic segmentation of blood vessels within the organ was tested on 30 cases randomly selected from the enhanced CT dataset, and each case had its true mask label of blood vessels within the organ. The Dice and ASSD of this solution reached (91.24 ± 1.29)% and (1.16 ± 0.19) mm respectively. A Wilcoxon test was conducted on the results of the automatic segmentation of blood vessels within the organ of the proposed solution and the results of the peer methods, and the P-value was obtained as 9.871e-7. The Dice and ASSD scores and the P-value consistently indicate that the proposed solution is superior to the peer methods with a statistically significant advantage. In terms of its efficiency, when performing the automatic segmentation of blood vessels within the organ, the proposed solution only consumes 0.112 seconds per image and an average of 30 seconds per case; in contrast, manual segmentation of blood vessels within the organ by an experienced doctor requires at least 10 minutes per case.

[0112] Table 2 Performance of the proposed solution in organ segmentation and the segmentation of blood vessels within the organ, and performance comparison of the blood vessels within the organ with the peer methods

[0113]

[0114] For the qualitative results, the effect of the segmentation of blood vessels within the organ was analyzed through visualization. Figure 4 The organ segmentation results of 9 cases randomly selected from 20 cases were used for 3D reconstruction using the RadiAnt software package. Figure 4 The results of the 3D reconstruction are shown, which indicates that the organ segmentation results produced by the proposed solution can satisfactorily ensure the following two points: 1) The blood vessels within the organ to be segmented are indeed located within its organ region, and 2) The ribs and pulmonary blood vessels are successfully eliminated from the region occupied by the segmented blood vessels within the organ. These two characteristics achieved by the organ segmentation results provide a good basis for the subsequent task of blood vessels within the organ. Figure 5The results of intra-organ vessel segmentation for 9 challenging cases selected from 30 test cases are shown. Visualization was performed using the 3D reconstruction software package RadiAnt. The results indicate that the proposed model can indeed perform intra-organ vessel segmentation and can even handle complex situations such as images with poor contrast, plaques, and left or right intra-organ vessel dominance types.

[0115] In summary, the present invention introduces an end-to-end automatic intra-organ vessel segmentation solution based on deep learning. Enhanced CT plays an important role in the diagnosis of cardiovascular diseases, and intra-organ vessel automatic segmentation is one of the most challenging tasks. To computationally assist this task, the present invention proposes a new deep learning solution. The HVessel-Net network was designed, and a fully automatic progressive deep learning solution was proposed. This solution aims to preserve the integrity of blood vessels in terms of shape details and continuity. This solution was developed using 360 enhanced CT cases, among which 150 cases have pre-labeled true organ masks and 210 cases have pre-labeled true intra-organ vessel masks. Dice and ASSD scores were used to measure the accuracy of this solution in intra-organ vessel automatic segmentation. The proposed solution achieved 91.24% in Dice, 1.16 mm in ASSD, consumed an average of 0.124 seconds per image, and an average of 30 seconds per case. The novel deep learning solution (based on the HVessel-Net deep learning model) adopted by the present invention can automatically learn to perform intra-organ vessel automatic segmentation in an end-to-end manner, achieving high accuracy, efficiency, and robustness, even for images with poor contrast, motion artifacts, and vessel distortions caused by severe disease progression.

[0116] The present invention proposes a fully automatic progressive deep learning solution for intra-organ vessel segmentation, aiming to maintain the integrity of blood vessels in terms of their shape details and continuity. The experimental results show that this solution has high accuracy, efficiency, and robustness.

[0117] As Figure 7 shown, another object of the present invention is to propose an automatic intra-organ vessel segmentation system for enhanced CT images, including:

[0118] An organ segmentation module for performing organ segmentation processing on enhanced CT images to extract an organ region and obtaining a three-dimensional organ region of interest from the organ region;

[0119] An intra-organ vessel segmentation module for performing global intra-organ vessel segmentation and local intra-organ vessel segmentation on the three-dimensional organ region of interest respectively to generate a global intra-organ vessel automatic segmentation result and a local intra-organ vessel automatic segmentation result;

[0120] A result fusion module is used to fuse the automatic segmentation results of blood vessels in the global organ and the automatic segmentation results of blood vessels in the local organ to obtain the final automatic segmentation result of blood vessels in the organ.

[0121] As Figure 8 shown, the third object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for automatically segmenting blood vessels in an organ of an enhanced CT image are implemented.

[0122] The method for automatically segmenting blood vessels in an organ of an enhanced CT image includes the following steps:

[0123] Perform organ segmentation processing on the enhanced CT image to extract the organ region, and obtain a three-dimensional organ region of interest from the organ region;

[0124] Perform global blood vessel segmentation in the three-dimensional organ region of interest and local blood vessel segmentation in the three-dimensional organ region of interest respectively to generate an automatic segmentation result of global blood vessels in the organ and an automatic segmentation result of local blood vessels in the organ;

[0125] Fuse the automatic segmentation result of global blood vessels in the organ and the automatic segmentation result of local blood vessels in the organ to obtain the final automatic segmentation result of blood vessels in the organ.

[0126] The fourth object of the present invention is to provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method for automatically segmenting blood vessels in an organ of an enhanced CT image are implemented.

[0127] The method for automatically segmenting blood vessels in an organ of an enhanced CT image includes the following steps:

[0128] Perform organ segmentation processing on the enhanced CT image to extract the organ region, and obtain a three-dimensional organ region of interest from the organ region;

[0129] Perform global blood vessel segmentation in the three-dimensional organ region of interest and local blood vessel segmentation in the three-dimensional organ region of interest respectively to generate an automatic segmentation result of global blood vessels in the organ and an automatic segmentation result of local blood vessels in the organ;

[0130] Fuse the automatic segmentation result of global blood vessels in the organ and the automatic segmentation result of local blood vessels in the organ to obtain the final automatic segmentation result of blood vessels in the organ.

[0131] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. An automatic segmentation method for blood vessels in organs of enhanced CT images, characterized in that, it includes the following steps: Perform organ segmentation processing on the original enhanced CT image to extract the organ region, and obtain a three-dimensional organ region of interest from the organ region; In the three-dimensional organ region of interest, perform global blood vessel segmentation in the organ and local blood vessel segmentation in the organ respectively to generate a global automatic blood vessel segmentation result in the organ and a local automatic blood vessel segmentation result in the organ; Fuse the global automatic blood vessel segmentation result in the organ and the local automatic blood vessel segmentation result in the organ to obtain the final automatic blood vessel segmentation result in the organ; The performing organ segmentation processing on the original enhanced CT image to extract the organ region and obtaining a three-dimensional organ region of interest from the organ region specifically includes: After the original enhanced CT image is preprocessed by gray mapping, input it into the HVessel-Net deep learning model, and train it through supervised learning and the first loss function to generate an organ segmentation mask; Filter the original enhanced CT image through the organ segmentation mask to obtain the organ region, and obtain the corresponding bounding box, which has the same length in the coronal and sagittal directions; Extract the three-dimensional organ region of interest from the detected organ region through three-dimensional cropping operation; The HVessel-Net deep learning model is constructed by integrating dilated convolutions into the encoder-decoder convolutional network structure; The specific structure is: In the encoding stage, a convolutional layer with a 3×3×3 voxel receptive field reduces the resolution of the feature map, and then is a dilated convolutional layer, which retains the resolution of the input feature map; Another extended convolutional layer further processes the obtained feature map and retains the resolution of the map; The final embedding vector output by the encoding part is transformed by the ResConv block; The transformed embedding vector is sent to the decoding part of the network, whose structure is the same as that of the encoding part of the network, but in the reverse order; Apply a fully convolutional layer, using sigmoid as its activation function, to obtain the final network output; Adopt the HVessel-Net deep learning model to be able to automatically learn to perform automatic blood vessel segmentation in the organ in an end-to-end manner; The global blood vessel segmentation in the organ specifically includes: Adjust the input three-dimensional organ region of interest to a three-dimensional matrix, input the three-dimensional matrix into the HVessel-Net deep learning model, and train it through supervised learning and the second loss function to generate a three-dimensional mask of the same size as the three-dimensional matrix, corresponding to the blood vessels in the organ detected from the three-dimensional organ region of interest; Adjust the size of the three-dimensional mask to be consistent with the size of the original enhanced CT image through the input three-dimensional organ region of interest to obtain a three-dimensional global automatic blood vessel segmentation mask for the whole case; The local automatic blood vessel segmentation in the organ specifically includes: Apply a three-dimensional sliding window within the input three-dimensional organ region of interest with the same step size in all three dimensions to generate a series of three-dimensional patches, each with a size of 128×128×128; each patch is independently input into the HVessel-Net deep learning model and trained through supervised learning and the third loss function to generate a three-dimensional mask for the blood vessels within the organ in the three-dimensional space represented by the patch; all the generated three-dimensional masks are combined together to form a full-size three-dimensional mask with the same size as the input three-dimensional organ region of interest; adjust the size of the three-dimensional mask to be consistent with the size of the original enhanced CT image through the input three-dimensional organ region of interest to obtain the automatic segmentation mask of the three-dimensional local organ blood vessels for the entire case; Fusing the automatic segmentation results of global organ blood vessels and local organ blood vessels specifically refers to using a method that mainly takes the local segmentation results and supplements with the global segmentation results; specifically including: Skeletonize the global organ blood vessel segmentation results, overlay the skeletonized results on the local organ blood vessel segmentation results, retain the largest connected component of the overlaid results and obtain the parts lacking in the local organ blood vessel segmentation results; overlay the global organ blood vessel segmentation regions corresponding to all the lacking parts on the local organ blood vessel segmentation results, and use it as the final organ blood vessel segmentation result.

2. The method according to claim 1, wherein, the formula of the first loss function is as follows: y true and y pred represent the true value mask and the mask for organ segmentation prediction, respectively.

3. The method according to claim 1, wherein, the formula of the second loss function is as follows: Among them, and represent the true value mask and the mask generated by automatic segmentation of blood vessels in the global organ respectively, and their skeletons are represented as and , and represent in the proportion of and in the proportion of, which are also respectively called the accuracy and sensitivity indexes of the topological structure.

4. The method according to claim 1, wherein, the formula of the third loss function is as follows: = （4） (5) (6) Among them, and represent the true value mask and the mask for predicting the segmentation of blood vessels in the organ respectively. The parameter α is empirically adjusted to 0.3 to maintain the balance between suppressing over-segmentation and improving the accuracy.

5. An automatic segmentation system for organ blood vessels in enhanced CT images, based on the method according to any one of claims 1 to 4, wherein, comprising: An organ segmentation module for performing organ segmentation processing on the enhanced CT image to extract the organ region and obtaining a three-dimensional organ region of interest from the organ region; An organ blood vessel segmentation module for performing global organ blood vessel segmentation and local organ blood vessel segmentation respectively within the three-dimensional organ region of interest to generate an automatic global organ blood vessel segmentation result and an automatic local organ blood vessel segmentation result; A result fusion module for fusing the automatic global organ blood vessel segmentation result and the automatic local organ blood vessel segmentation result to obtain the final automatic organ blood vessel segmentation result.

Images