A health image guided lesion image segmentation method

Abstract

The application discloses a kind of health multi-phase image guided multi-phase lesion image segmentation methods, this method mainly includes: using healthy multi-phase contrast-enhanced CT training deep learning network, freeze its parameters, then send lesion CT image into trained healthy image network and another network structure consistent with the lesion image network to be trained.Healthy image network acts as normal tissue feature extractor, assist lesion image network training;Wherein, the loss function of healthy image network adopts the combination of Dice loss function and cross-entropy loss function, the loss function of lesion image network includes the mean square error loss function of feature map extracted by healthy image network and lesion image network to lesion image in addition to above-mentioned two loss functions.The application scheme distinguishes healthy tissue and lesion tissue by guiding lesion image network with healthy image, and the segmentation precision of lesion tissue is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medicine, and more specifically to an automatic segmentation method for pancreas and pancreatic tumors using contrast-enhanced CT. Background Technology

[0002] The pancreas, the second largest gland in the human body, directly participates in metabolic activities through the secretion of pancreatic juice by its exocrine glandular cells and insulin by its endocrine gland cells, making it one of the most important organs in the body. However, due to the irregular and unhealthy eating habits of modern people, it has led to one of the most deadly diseases: pancreatic cancer. Pancreatic cancer is a malignant tumor with a poor prognosis, and because patients do not show specific symptoms in the early stages, most patients are diagnosed at an advanced stage; resulting in a 5-year survival rate of around 10% for pancreatic cancer patients for a long time.

[0003] Therefore, the academic community has been studying it, either by using the vascular structures around the pancreas to select highly similar regions in the image to achieve pancreatic segmentation, or by using the different imaging characteristics of tumors in multi-time-contrast enhanced CT to improve the segmentation accuracy of pancreatic tumors.

[0004] Existing deep learning methods, when improving pancreatic tumor segmentation performance, focus on network architecture design or post-processing strategies, neglecting the rich information contained in healthy images. Healthy medical images not only constitute a large proportion of datasets, but also exhibit clearer tissue structures, highly consistent morphological features, and typically lower noise levels. These characteristics make healthy images a potential high-quality reference source, supporting the model's accuracy and robustness. Furthermore, the anatomical symmetry and regional stability of healthy images help improve the model's contextual awareness, thus better distinguishing lesions from normal tissue. Fully utilizing these significant advantages of healthy images can not only enhance the model's generalization ability but also potentially provide new technical avenues for further improving segmentation accuracy. This invention proposes using multi-time-contrast enhanced CT images of healthy pancreas to improve the segmentation accuracy of pancreatic tumor segmentation networks, providing doctors with more accurate and reliable references, improving target delineation efficiency, and increasing surgical success rates. Summary of the Invention

[0005] The present invention proposes a method for segmenting lesion images guided by healthy images, comprising the following steps:

[0006] Step 1: Preprocess healthy and diseased images and their labels, register images from different time phases, unify image resolution through resampling, extract the region of interest for each image, use grayscale truncation to remove the influence of irrelevant regions, and normalize.

[0007] Step 2: Train a deep learning network using the processed health images and their labels. Images from different time phases are fed into two 3D-UNet encoder branches of the network. A feature fusion module is introduced between the two branches to fuse image features from different time phases. The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference enhancement branches perform difference operations on features from different time phases and generate weights to enhance the original features. The long-distance relationship modeling branches extract multi-scale contextual information of features through dilated spatial pyramid convolution. Finally, the features output from the four branches are fused to obtain fused features. The fused features are decoded by the decoder to obtain the predicted segmentation result map. The Dice loss function and cross-entropy loss function between the health image labels are calculated, gradients are backpropagated, network parameters are updated, and the iteration continues until the network converges.

[0008] Step 3: Freeze the parameters of the healthy network and build a network with the same structure as the healthy image; input the preprocessed lesion image into the frozen healthy network and the newly built lesion network to be trained; in addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean squared error loss between the features extracted from the lesion image by the healthy network and the lesion network in the healthy tissue part, and iterates until the network converges.

[0009] Step 4: Input the pre-processed multi-time contrast-enhanced CT images of the lesion patient into the trained lesion network, which can output the corresponding segmentation maps of healthy tissue and lesion tissue.

[0010] Step 1 specifically includes:

[0011] (1) Register images of different time phases of the same case using rigid registration and non-rigid registration. Non-rigid registration uses B-spline transformation to align the target region.

[0012] (2) Resample the images and labels using trilinear interpolation to unify the images and labels to a resolution of 1×1×1mm.

[0013] (3) Based on the label size, crop the image and label to the size of the label by 25 pixels outward along the three axes. At the same time, confirm whether the size of the expanded image and label is greater than 128×192×192 pixels. If the size after expansion is greater than this value, retain the cropping result; otherwise, crop the image and label to this size according to the center of the label.

[0014] (4) Perform gray-scale truncation on the image, set the values ​​of points with gray values ​​below -70 and above 230 to -70 and 230 respectively, and then normalize the image gray values ​​to [0,1].

[0015] Step 2 specifically includes:

[0016] The network is encoded using a 3D-UNet encoder, and a feature fusion module is introduced to achieve feature fusion. The network has two encoding paths and one decoding path, and the two encoding paths are connected by the feature fusion module.

[0017] The encoding path consists of five layers, each consisting of two consecutive convolutional modules. Each convolutional module consists of a 3×3×3 convolution, an instance normalization, and a Leaky ReLU non-linear activation function. The layers are connected by a 2×2×2 max pooling connection.

[0018] The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branches calculate the difference between two temporal features, pass them through the convolution module and sigmoid function to obtain a difference weight matrix, and multiply it with the original features to obtain an enhanced difference feature map. The long-distance relationship modeling branches use dilated spatial pyramid convolution to capture the long-distance relationships of features. Finally, the feature maps obtained from the four branches are fed into the convolution module to obtain the fused features, which are used as the input to the decoding branch.

[0019] Each layer of the decoder has the same structure as the encoder. It uses transposed convolution upsampling to restore the original image size, and finally uses 1×1 convolution to reduce the channel dimension to the number of categories.

[0020] Step 3 specifically includes:

[0021] Calculate the Dice loss and cross-entropy loss between the predicted segmentation result and the true segmentation label, respectively: (1)

[0022] (2)

[0023] Where 𝑇𝑃 is True Positive, i.e., the number of voxels predicted as positive and actually being positive; 𝐹𝑃 is False Positive, i.e., the number of voxels predicted as positive but actually being negative; 𝐹𝑁 is False Negative, i.e., the number of voxels predicted as negative but actually being positive, C is the number of classes, and y i The true label for category i, p i It is the probability that the model predicts for category i.

[0024] Step 4 specifically includes:

[0025] The mean squared error loss function at the location of healthy tissue is as follows: (This is the difference between the lesion image features extracted by the healthy network and the lesion network.)

[0026] (3)

[0027]

[0028] Where N, C, D, H, W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network, respectively, i represents the network layer number where the feature map is located, and the feature mean square error is calculated for i>1. The mask is generated by downsampling the original label.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention utilizes healthy pancreatic images and diseased pancreatic images and their labels. Based on a deep learning network, it leverages the rich information in healthy images to improve the segmentation accuracy of the diseased image segmentation network for pancreatic and pancreatic tumors. The improved segmentation accuracy of these two types of targets is beneficial for clinical surgical applications, providing doctors with more accurate and reliable references, improving target delineation efficiency, and increasing surgical success rates.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, Figure 1 in the following description is one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 is a flowchart illustrating a method for segmenting lesion images guided by health imaging according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of the network structure of the present invention;

[0035] Figure 3 is a schematic diagram of the feature fusion module structure in this invention;

[0036] Figure 4 is a schematic diagram of the lesion network training method in this invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are some embodiments of the present invention, but not all embodiments.

[0038] Figure 1 is a flowchart of a health image-guided lesion image segmentation method according to an embodiment of the present invention, which includes the following four steps.

[0039] Step 1: Preprocess healthy and diseased images and their labels, register images from different time phases, unify image resolution through resampling, extract the region of interest for each image, use grayscale truncation to remove the influence of irrelevant regions, and normalize.

[0040] (1) Register images of different time phases of the same case using rigid registration and non-rigid registration. Non-rigid registration uses B-spline transformation to align the target region.

[0041] (2) Resample the images and labels using trilinear interpolation to unify the images and labels to a resolution of 1×1×1mm.

[0042] (3) Based on the label size, crop the image and label to the size of the label by 25 pixels outward along the three axes. At the same time, confirm whether the size of the expanded image and label is greater than 128×192×192 pixels. If the size after expansion is greater than this value, retain the cropping result; otherwise, crop the image and label to this size according to the center of the label.

[0043] (4) Perform gray-scale truncation on the image, set the values ​​of points with gray values ​​below -70 and above 230 to -70 and 230 respectively, and then normalize the image gray values ​​to [0,1].

[0044] Step 2: Train a deep learning network using the processed health images and their labels. Images from different time phases are fed into two 3D-UNet encoder branches of the network. A feature fusion module is introduced between the two branches to fuse image features from different time phases. The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference enhancement branches perform difference operations on features from different time phases and generate weights to enhance the original features. The long-distance relationship modeling branches extract multi-scale contextual information of features through dilated spatial pyramid convolution. Finally, the features output from the four branches are fused to obtain fused features. The fused features are decoded by the decoder to obtain the predicted segmentation result map. The Dice loss function and cross-entropy loss function between the health image labels are calculated, gradients are backpropagated, network parameters are updated, and the iteration continues until the network converges.

[0045] The network is encoded using a 3D-UNet encoder, and a feature fusion module is introduced to achieve feature fusion. The network has two encoding paths and one decoding path, and the two encoding paths are connected by the feature fusion module.

[0046] The encoding path consists of five layers, each consisting of two consecutive convolutional modules. Each convolutional module consists of a 3×3×3 convolution, an instance normalization, and a Leaky ReLU non-linear activation function. The layers are connected by a 2×2×2 max pooling connection.

[0047] The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branches calculate the difference between two temporal features, pass them through the convolution module and sigmoid function to obtain a difference weight matrix, and multiply it with the original features to obtain an enhanced difference feature map. The long-distance relationship modeling branches use dilated spatial pyramid convolution to capture the long-distance relationships of features. Finally, the feature maps obtained from the four branches are fed into the convolution module to obtain the fused features, which are used as the input to the decoding branch.

[0048] Each layer of the decoder has the same structure as the encoder. It uses transposed convolution upsampling to restore the original image size, and finally uses 1×1 convolution to reduce the channel dimension to the number of categories.

[0049] Step 3: Freeze the parameters of the healthy network, build a network with the same structure as the healthy image, and input the preprocessed lesion network into the frozen healthy network and the newly built lesion network to be trained; in addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean squared error loss between the features extracted from the lesion image by the healthy network and the lesion network in the healthy tissue part, and iterates until the network converges.

[0050] The Dice loss and cross-entropy loss between the predicted segmentation result and the true segmentation label are calculated as follows: (1)

[0051] (2)

[0052] Where 𝑇𝑃 is True Positive, i.e., the number of voxels predicted as positive and actually being positive; 𝐹𝑃 is False Positive, i.e., the number of voxels predicted as positive but actually being negative; 𝐹𝑁 is False Negative, i.e., the number of voxels predicted as negative but actually being positive, C is the number of classes, and y i The true label for category i, p i It is the probability that the model predicts for category i.

[0053] Step 4: Input the pre-processed multi-time contrast-enhanced CT images of the lesion patient into the trained lesion network, which can output the corresponding segmentation maps of healthy tissue and lesion tissue.

[0054] The mean squared error loss function at the location of healthy tissue is as follows: (This is the difference between the lesion image features extracted by the healthy network and the lesion network.)

[0055] (3)

[0056]

[0057] Where N, C, D, H, W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network, respectively, i represents the network layer number where the feature map is located, and the feature mean square error is calculated for i>1. The mask is generated by downsampling the original label.

Claims

1. A method for segmenting lesion images guided by healthy images, characterized in that... Includes the following steps: Step 1: Preprocess healthy images and disease images and their labels, register images from different time phases, unify image resolution through resampling, extract the region of interest for each image, use grayscale truncation to remove the influence of irrelevant regions, and normalize. Step 2: Train a deep learning network using the processed health images and their labels. Feed the images from different time phases into the two 3D-UNet encoder branches of the network. Introduce a feature fusion module between the two branches to fuse the image features from different time phases. The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference enhancement branches perform difference operations on features from different time phases and generate weights to enhance the original features. The long-distance relationship modeling branches extract multi-scale contextual information of features through dilated spatial pyramid convolution. Finally, the features output by the four branches are fused to obtain fused features. The fused features are decoded by the decoder to obtain the predicted segmentation result map. The Dice loss function and cross-entropy loss function between the health image labels are calculated, gradient backpropagation is performed, network parameters are updated, and the iteration continues until the network converges. Step 3: Freeze the parameters of the healthy network, build a network with the same structure as the healthy image network, and input the preprocessed lesion image into the frozen healthy network and the newly built lesion network to be trained; in addition to calculating the same loss function as the healthy network, the training of the lesion network also adds the mean square error loss between the features extracted by the healthy network and the lesion network from the lesion image in the healthy tissue part, and iterates until the network converges. Step 4: Input the pre-processed multi-time contrast-enhanced CT images of the lesion patient into the trained lesion network, which can output the corresponding segmentation maps of healthy tissue and lesion tissue.

2. The method according to claim 1, characterized in that, Preprocessing operations are performed on healthy and diseased images and their labels, including: For images of the same case at different time points, registration was performed using rigid registration and non-rigid registration. Non-rigid registration used B-spline transformation to align the target regions. The images and labels were resampled using trilinear interpolation to unify them to a resolution of 1×1×1mm. Based on the label size, crop the image and label to the size of the label by expanding outward by 25 pixels along the three axes. At the same time, confirm whether the expanded image and label size is greater than 128×192×192 pixels. If the expanded size is greater than this value, retain the cropping result; otherwise, crop the image and label to this size according to the center of the label. The image is truncated to grayscale. Points with grayscale values ​​below -70 and above 230 are set to -70 and 230 respectively. Then the image grayscale values ​​are normalized to [0,1].

3. The method according to claim 1, characterized in that, Encode it using a 3D-UNet encoder, and introduce a feature fusion module to achieve feature fusion, including: The network has two encoding paths and one decoding path. The two encoding paths are connected by a feature fusion module. The encoding path consists of five layers, each consisting of two consecutive convolutional modules. Each convolutional module consists of a 3×3×3 convolution, an instance normalization, and a Leaky ReLU non-linear activation function. The layers are connected by a 2×2×2 max pooling. The feature fusion module consists of two difference enhancement branches and two long-distance relationship modeling branches. The difference branches calculate the difference between two temporal features, pass them through the convolution module and sigmoid function to obtain a difference weight matrix, and multiply it with the original features to obtain an enhanced difference feature map. The long-distance relationship modeling branches use dilated spatial pyramid convolution to capture the long-distance relationships of features. Finally, the feature maps obtained from the four branches are fed into the convolution module to obtain the fused features, which are used as the input to the decoding branch. Each layer of the decoder has the same structure as the encoder. It uses transposed convolution upsampling to restore the original image size, and finally uses 1×1 convolution to reduce the channel dimension to the number of categories.

4. The method according to claim 1, characterized in that, The Dice loss and cross-entropy loss between the predicted segmentation result and the true segmentation label are calculated as follows: （1） （2） Where 𝑇𝑃 is True Positive, i.e., the number of voxels predicted as positive and actually being positive; 𝐹𝑃 is False Positive, i.e., the number of voxels predicted as positive but actually being negative; 𝐹𝑁 is False Negative, i.e., the number of voxels predicted as negative but actually being positive, C is the number of classes, and y i The true label for category i, p i It is the probability that the model predicts for category i.

5. The method according to claim 1, characterized in that, The mean squared error loss function at the location of healthy tissue is as follows: (This is the difference between the lesion image features extracted by the healthy network and the lesion network.) (3) Where N, C, D, H, W are the image batch size, number of feature channels, feature map depth, feature map height, and feature map width of the input network, respectively, i represents the network layer number where the feature map is located, and the feature mean square error is calculated for i>1. The mask is generated by downsampling the original label.

Images