Lesion image segmentation method guided by health image

Abstract

The invention discloses a multi-temporal lesion image segmentation method guided by a healthy multi-temporal image, and the method mainly comprises the steps: training a deep learning network through employing a healthy multi-temporal contrast enhanced CT, freezing the parameters of the deep learning network, and then the lesion CT image is sent to the trained healthy image network and another lesion image network to be trained, wherein the network structure of the lesion image network is consistent with that of the lesion image network to be trained. The health image network serves as a normal tissue feature extractor and assists in lesion image network training; wherein the loss function of the health image network adopts a combination of a Dice loss function and a cross entropy loss function, and the loss function of the lesion image network also comprises a mean square error loss function of the health image network and the lesion image network for feature maps extracted from the lesion image besides the two loss functions. According to the scheme, the lesion image network is guided through the health image to distinguish the healthy tissue and the lesion tissue, and the segmentation precision of the lesion tissue is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medicine, and more specifically to a method for automatic segmentation of pancreas and pancreatic tumors by multi-phase contrast-enhanced CT. Background Art

[0002] The pancreas is the second largest gland in the human body. It directly participates in human metabolic activities through the secretion of pancreatic juice by its exocrine glandular cells and insulin by its endocrine cells. However, due to the irregular and unhealthy eating habits of modern humans, one of the most deadly diseases has occurred: pancreatic cancer. Pancreatic cancer is a malignant tumor with a poor prognosis. Since patients do not show specific symptoms in the early stages, most patients are already in the advanced stage when they are diagnosed, which makes the 5-year survival rate of pancreatic cancer patients remain at around 10% for a long time.

[0003] To this end, the academic community has been conducting research on this, either by using the vascular structure around the pancreas to select areas in the image that are highly similar to it to achieve pancreatic segmentation, or by using the different imaging characteristics of tumors in multi-phase contrast-enhanced CT to improve the segmentation accuracy of pancreatic tumors.

[0004] Existing deep learning methods focus on the design of network structures or post-processing strategies when improving the performance of pancreatic tumor segmentation, while ignoring the rich information contained in healthy images. Healthy medical images not only occupy a large proportion in the data set, but also show a clearer tissue structure, highly consistent morphological features, and are usually accompanied by a lower noise level. These characteristics make healthy images a potential high-quality reference source that can provide support for the accuracy and robustness of the model. In addition, the anatomical symmetry and stability of specific areas of healthy images help to improve the contextual perception of the model, thereby better distinguishing lesions from normal tissues. If these significant advantages in healthy images can be fully utilized, not only can the generalization ability of the model be enhanced, but it may also provide a new technical approach to further improve the segmentation accuracy. The present invention proposes to use healthy pancreatic multi-phase contrast enhanced CT images to improve the segmentation accuracy of the pancreatic tumor segmentation network, provide doctors with a more accurate and reliable reference, improve the efficiency of target area delineation, and improve the success rate of surgery. Summary of the invention

[0005] The healthy image-guided lesion image segmentation method proposed in the present invention comprises the following steps:

[0006] Step 1: Preprocess the healthy images and pathological images and their labels, align the images of different phases, unify the image resolution by resampling, extract the region of interest (ROI) of each image, use grayscale truncation to eliminate the influence of irrelevant areas, and normalize them.

[0007] Step 2: Use the processed healthy images and their labels to train the deep learning network, and send the images of different phases to the two 3D-UNet encoder branches of the network respectively. A feature fusion module is introduced between the two branches to fuse the image features from different phases. The fused features are decoded by the decoder to obtain the predicted segmentation result map, and the Dice loss function and cross entropy loss function between the healthy image labels are calculated. The gradient is backpropagated, the network parameters are updated, and the network is iterated until the network converges.

[0008] Step 3: Freeze the healthy network parameters, build a network with the same structure as the healthy image, and input the preprocessed lesion network into the frozen healthy network and the newly built lesion network to be trained. In addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean square error loss between the features extracted from the lesion image by the healthy network and the lesion network in the healthy tissue part. Iterate until the network converges.

[0009] Step 4: Input the pre-processed multi-phase contrast-enhanced CT images of the lesion patient into the trained lesion network, and the corresponding segmentation map of healthy tissue and lesion tissue can be output.

[0010] The contents of step 1 specifically include:

[0011] (1) Images of the same case at different times are registered, and the target area is aligned using rigid registration and non-rigid registration (B-spline transformation).

[0012] (2) Resample the images and labels and use trilinear interpolation to unify the images and labels to the same resolution (1×1×1mm).

[0013] (3) According to the label size, the image and label are cropped to a size that is 25 pixels larger than the label along the three axes. At the same time, it is confirmed whether the image and label size after expansion is larger than (128×192×192 pixels). If the size after expansion is larger than this value, the cropping result is retained. Otherwise, the image and label are cropped to this size according to the center of the label.

[0014] (4) The grayscale of the image is truncated. The points with grayscale values ​​below -70 and above 230 are set to -70 and 230 respectively. Then the grayscale value of the image is normalized to [0, 1].

[0015] The contents of step 2 specifically include:

[0016] The 3D-UNet encoder is used to encode it, and a feature fusion module is introduced to achieve feature fusion. The network has two encoding paths and one decoding path, and the two encoding paths are connected by a feature fusion module.

[0017] The encoding path consists of five layers, each of which consists of two consecutive convolutional modules, which consist of a 3×3×3 convolution, an instance normalization (IN) and a nonlinear activation function (Leaky ReLU). The layers are connected by 2×2×2 maximum pooling.

[0018] The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branch calculates the difference between two phase features, passes through the above convolution module and sigmoid function to obtain the difference weight matrix, and multiplies it with the original feature to obtain the enhanced difference feature map. The long-distance relationship modeling branch uses the hollow space pyramid convolution to capture the feature's distance relationship. Finally, the feature maps obtained by the four branches are sent to the above convolution module to obtain the fused features as the input of the decoding branch.

[0019] The structure of each layer of the decoder is the same as that of the encoder. The upsampling method of transposed convolution is used to restore the original image size, and finally a 1×1 convolution is used to reduce the channel dimension to the number of categories.

[0020] The contents of step 3 specifically include:

[0021] The Dice loss and cross entropy loss between the predicted segmentation results and the true segmentation labels are calculated as:

[0022]

[0023] Among them, TP is True Positive, that is, the number of voxels predicted as positive samples are actually positive samples; FP is False Positive, that is, the number of voxels predicted as positive samples are actually negative samples; FN is False Negative, that is, the number of voxels predicted as negative samples are actually positive samples, C is the number of categories, y i is the true label of category i, p i is the probability that the model predicts class i.

[0024] Step 4 specifically includes:

[0025] The mean square error loss function between the lesion image features extracted by the healthy network and the lesion network at the healthy tissue location is as follows:

[0026]

[0027] Among them, N, C, D, H, and W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network respectively. i represents the network layer number where the feature map is located. For feature mean square error calculation with i>1, the mask is generated by downsampling the original label.

[0028] The beneficial effects of the present invention are as follows:

[0029] The present invention utilizes healthy pancreatic images and diseased pancreatic images and their labels, based on a deep learning network, and utilizes the rich information in healthy images to improve the segmentation accuracy of the diseased image segmentation network for the pancreas and pancreatic tumors. The improvement in the segmentation accuracy of the two types of targets is helpful for the application of clinical surgery, providing doctors with more accurate and reliable references, improving the efficiency of target area delineation, and increasing the success rate of surgery.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the prior art description. Obviously, the drawings in the following description are Figure 1 This is an implementation mode of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A schematic diagram of a flow chart of a healthy image-guided lesion image segmentation method according to an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the network structure of the present invention;

[0034] Figure 3 It is a structural schematic diagram of the feature fusion module of the present invention;

[0035] Figure 4 Schematic diagram of the training method of the diseased network. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0037] Figure 1 4 is a flow chart of a healthy image-guided lesion image segmentation method according to an embodiment of the present invention. The method includes the following four steps.

[0038] Step 1: Preprocess the healthy images and pathological images and their labels, align the images of different phases, unify the image resolution by resampling, extract the region of interest (ROI) of each image, use grayscale truncation to eliminate the influence of irrelevant areas, and normalize them.

[0039] (1) Images of the same case at different times are registered, and the target area is aligned using rigid registration and non-rigid registration (B-spline transformation).

[0040] (2) Resample the images and labels and use trilinear interpolation to unify the images and labels to the same resolution (1×1×1mm).

[0041] (3) According to the label size, the image and label are cropped to a size that is 25 pixels larger than the label along the three axes. At the same time, it is confirmed whether the image and label size after expansion is larger than (128×192×192 pixels). If the size after expansion is larger than this value, the cropping result is retained. Otherwise, the image and label are cropped to this size according to the center of the label.

[0042] (4) The grayscale of the image is truncated. The points with grayscale values ​​below -70 and above 230 are set to -70 and 230 respectively. Then the grayscale value of the image is normalized to [0, 1].

[0043] Step 2: Use the processed healthy images and their labels to train the deep learning network, and send the images of different phases to the two 3D-UNet encoder branches of the network respectively. A feature fusion module is introduced between the two branches to fuse the image features from different phases. The fused features are decoded by the decoder to obtain the predicted segmentation result map, and the Dice loss function and cross entropy loss function between the healthy image labels are calculated. The gradient is backpropagated, the network parameters are updated, and the network is iterated until the network converges.

[0044] The 3D-UNet encoder is used to encode it, and a feature fusion module is introduced to achieve feature fusion. The network has two encoding paths and one decoding path, and the two encoding paths are connected by a feature fusion module.

[0045] The encoding path consists of five layers, each of which consists of two consecutive convolutional modules, which consist of a 3×3×3 convolution, an instance normalization (IN) and a nonlinear activation function (Leaky ReLU). The layers are connected by 2×2×2 maximum pooling.

[0046] The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branch calculates the difference between two phase features, passes through the above convolution module and sigmoid function to obtain the difference weight matrix, and multiplies it with the original feature to obtain the enhanced difference feature map. The long-distance relationship modeling branch uses the hollow space pyramid convolution to capture the feature's distance relationship. Finally, the feature maps obtained by the four branches are sent to the above convolution module to obtain the fused features as the input of the decoding branch.

[0047] The structure of each layer of the decoder is the same as that of the encoder. The upsampling method of transposed convolution is used to restore the original image size, and finally a 1×1 convolution is used to reduce the channel dimension to the number of categories.

[0048] Step 3: Freeze the healthy network parameters, build a network with the same structure as the healthy image, and input the preprocessed lesion network into the frozen healthy network and the newly built lesion network to be trained. In addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean square error loss between the features extracted from the lesion image by the healthy network and the lesion network in the healthy tissue part. Iterate until the network converges.

[0049] The Dice loss and cross entropy loss between the predicted segmentation results and the true segmentation labels are calculated as:

[0050]

[0051] Among them, TP is True Positive, that is, the number of voxels predicted as positive samples are actually positive samples; FP is False Positive, that is, the number of voxels predicted as positive samples are actually negative samples; FN is False Negative, that is, the number of voxels predicted as negative samples are actually positive samples, C is the number of categories, y i is the true label of category i, p i is the probability that the model predicts class i.

[0052] Step 4: Input the pre-processed multi-phase contrast-enhanced CT images of the lesion patient into the trained lesion network, and the corresponding segmentation map of healthy tissue and lesion tissue can be output.

[0053] The mean square error loss function between the lesion image features extracted by the healthy network and the lesion network at the healthy tissue location is as follows:

[0054]

[0055] Among them, N, C, D, H, and W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network respectively. i represents the network layer number where the feature map is located. For feature mean square error calculation with i>1, the mask is generated by downsampling the original label.

Claims

1. A healthy image-guided lesion image segmentation method, characterized in that The following steps are involved: Step 1: Preprocess the healthy images and pathological images and their labels, align the images of different phases, unify the image resolution by resampling, extract the region of interest (ROI) of each image, use grayscale truncation to eliminate the influence of irrelevant areas, and normalize them. Step 2: Use the processed healthy images and their labels to train the deep learning network, and send the images of different phases to the two 3D-UNet encoder branches of the network respectively. A feature fusion module is introduced between the two branches to fuse the image features from different phases. The fused features are decoded by the decoder to obtain the predicted segmentation result map, and the Dice loss function and cross entropy loss function between the healthy image labels are calculated. The gradient is backpropagated, the network parameters are updated, and the network is iterated until the network converges. Step 3: Freeze the healthy network parameters, build a network with the same structure as the healthy image network, and input the preprocessed lesion network into the frozen healthy network and the newly built lesion network to be trained. In addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean square error loss between the features extracted from the lesion image by the healthy network and the lesion network in the healthy tissue part. Iterate until the network converges. Step 4: Input the pre-processed multi-phase contrast-enhanced CT images of the lesion patient into the trained lesion network, and the corresponding segmentation map of healthy tissue and lesion tissue can be output.

2. The method according to claim 1, characterized in that Preprocessing operations are performed on healthy images, diseased images and their labels, including: The images of the same case at different time phases were registered, and the target areas were aligned using rigid registration and non-rigid registration (B-spline transformation). The images and labels were resampled and trilinear interpolation was used to unify the images and labels to the same resolution (1×1×1mm). According to the label size, the image and label are cropped to a size of 25 pixels along the three axes. At the same time, it is confirmed whether the image and label size after expansion is larger than (128×192×192 pixels). If the size after expansion is larger than this value, the cropping result is retained. Otherwise, the image and label are cropped to this size according to the center of the label. The grayscale of the image is truncated, and the points with grayscale values ​​lower than -70 and higher than 230 are set to -70 and 230 respectively, and then the grayscale value of the image is normalized to [0,1].

3. The method according to claim 1, characterized in that The 3D-UNet encoder is used to encode it, and a feature fusion module is introduced to achieve feature fusion, including: The network has two encoding paths and one decoding path, and the two encoding paths are connected by a feature fusion module. The encoding path includes five layers, each layer consists of two consecutive convolution modules, and the convolution module consists of a 3×3×3 convolution, an instance normalization (IN) and a nonlinear activation function (Leaky ReLU). The layers are connected by a 2×2×2 maximum pooling. The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branch calculates the difference between two phase features, passes through the above convolution module and sigmoid function to obtain the difference weight matrix, and multiplies it with the original feature to obtain the enhanced difference feature map. The long-distance relationship modeling branch uses the hollow space pyramid convolution to capture the feature's distance relationship. Finally, the feature maps obtained by the four branches are sent to the above convolution module to obtain the fused features as the input of the decoding branch. The structure of each layer of the decoder is the same as that of the encoder. The upsampling method of transposed convolution is used to restore the original image size, and finally a 1×1 convolution is used to reduce the channel dimension to the number of categories.

4. The method according to claim 1, characterized in that: The Dice loss and cross entropy loss between the predicted segmentation results and the true segmentation labels are calculated as: Among them, TP is True Positive, that is, the number of voxels predicted as positive samples are actually positive samples; FP is False Positive, that is, the number of voxels predicted as positive samples are actually negative samples; FN is False Negative, that is, the number of voxels predicted as negative samples are actually positive samples, C is the number of categories, y i is the true label of category i, p i is the probability that the model predicts class i.

5. The method according to claim 1, characterized in that The mean square error loss function between the lesion image features extracted by the healthy network and the lesion network at the healthy tissue location is as follows: Among them, N, C, D, H, and W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network respectively. i represents the network layer number where the feature map is located. For feature mean square error calculation with i>1, the mask is generated by downsampling the original label.

Images