Image segmentation method, system and device using feature similarity for segmentation guidance
By introducing a feature similarity-guided segmentation method into the UNet neural network model, the problem of insufficient information utilization in medical image segmentation is solved, achieving higher accuracy and better generalization performance in medical image segmentation.
Patent Information
- Application Number
- CN202211574610.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-08
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-12-08
AI Technical Summary
Existing medical image segmentation methods fail to utilize sufficient information, resulting in inaccurate segmentation results and a lack of generalization ability. In particular, they neglect shape information, leading to significant errors in the segmentation results.
A UNet-based neural network model is adopted, which combines an image downsampling encoder, a feature upsampling decoder, and a same-layer fusion module. By guiding the fusion of label features and segmentation result features through feature similarity, the segmentation accuracy is improved.
Without increasing testing time, it significantly improves the segmentation accuracy and generalization performance of medical images, and improves the accuracy of segmentation results through similarity-guided information fusion.
Smart Images
Figure CN115880308B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and relates to an image segmentation method, system and device, and particularly to a medical image segmentation method, system and device that uses feature similarity to guide segmentation. Background Technology
[0002] In medical diagnosis, many internal medicine diseases, such as those affecting the heart, liver, and kidneys, are typically diagnosed using medical imaging techniques such as ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI). These technologies help doctors understand the specific condition of a patient's internal organs without invasive procedures, and are therefore widely used.
[0003] Over the past few decades, the development of deep learning, especially deep convolutional neural networks (CNNs), has led to significant progress in medical image segmentation, making it possible to use deep learning to assist doctors in diagnosis. Currently, the most widely used network architecture in medical image segmentation is UNet. UNet proposes using skip connections to fuse features from shallow and deep layers, achieving good segmentation results in most cases. Although many other segmentation methods based on UNet have emerged since, UNet remains the most frequently used medical image segmentation network. However, most current medical image segmentation methods treat the task as classifying individual pixels, causing them to ignore information such as shape that could improve the segmentation results, resulting in segmentation outcomes with significant shape errors. Summary of the Invention
[0004] To address the problem that existing medical image segmentation methods suffer from inaccurate segmentation results and lack of generalization ability due to insufficient utilization of various information, this invention provides a medical image segmentation method, system, and device that utilizes feature similarity to guide segmentation.
[0005] The technical solution adopted by the method of the present invention is: an image segmentation method guided by feature similarity, comprising the following steps:
[0006] Step 1: Acquire medical images;
[0007] Step 2: Obtain the segmentation result image using a UNet-based neural network model;
[0008] The UNet-based neural network model includes an image downsampling encoder, a same-layer fusion module, a feature upsampling decoder for predicting segmentation results, and a feature upsampling decoder for predicting similarity.
[0009] The image downsampling encoder includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence.
[0010] The feature upsampling encoder for predicting segmentation results includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0011] The feature upsampling decoder for predicting similarity includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0012] The same-layer fusion module is used to concatenate the output features F1, F2, F3, and F4 of the first four stages of the image downsampling encoder with the output features G1, G2, G3, and G4 of each stage of the feature upsampling encoder used to predict the segmentation result in the order G4, G3, G2, and G1.
[0013] The technical solution adopted by the system of the present invention is: an image segmentation system that utilizes feature similarity to guide segmentation, comprising the following modules:
[0014] Module 1 is used to acquire medical images;
[0015] Module 2 is used to obtain the segmentation result image using a UNet-based neural network model;
[0016] The UNet-based neural network model includes an image downsampling encoder, a same-layer fusion module, a feature upsampling decoder for predicting segmentation results, and a feature upsampling decoder for predicting similarity.
[0017] The image downsampling encoder includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence.
[0018] The feature upsampling encoder for predicting segmentation results includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0019] The feature upsampling decoder for predicting similarity includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0020] The same-layer fusion module is used to concatenate the output features F1, F2, F3, and F4 of the first four stages of the image downsampling encoder with the output features G1, G2, G3, and G4 of each stage of the feature upsampling encoder used to predict the segmentation result in the order G4, G3, G2, and G1.
[0021] The technical solution adopted by the device of the present invention is: an image segmentation device that utilizes feature similarity to guide segmentation, comprising:
[0022] One or more processors;
[0023] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image segmentation method guided by feature similarity.
[0024] This invention employs a UNet-based neural network model that utilizes similarity-guided segmentation to segment medical images. By leveraging the similarity between the label features and the segmentation result features, the neural network can obtain the shape features in the label. Simultaneously, by predicting similarity, the neural network can obtain the confidence level of its own segmentation results. Through the fusion of these two types of similarity-guided information, the segmentation accuracy of medical images is improved without increasing testing time. Attached Figure Description
[0025] Figure 1 The diagram shows the structure of a UNet-based neural network model provided in this embodiment of the invention.
[0026] Figure 2 This is a flowchart illustrating the training process of a UNet-based neural network model, as provided in an embodiment of this application.
[0027] Figure 3 The diagram shows the structure of the reconstructed network model provided in this embodiment. Detailed Implementation
[0028] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0029] This invention provides an image segmentation method guided by feature similarity, comprising the following steps:
[0030] Step 1: Acquire medical images;
[0031] Step 2: Obtain the segmentation result image using a UNet-based neural network model;
[0032] Please see Figure 1 The UNet-based neural network model in this embodiment includes an image downsampling encoder, a feature upsampling decoder for predicting segmentation results, and a feature upsampling decoder for predicting similarity.
[0033] The image downsampling encoder of this embodiment includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence.
[0034] This embodiment of the feature upsampling encoder for predicting segmentation results includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0035] This embodiment of the feature upsampling decoder for predicting similarity includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0036] The same-layer fusion module in this embodiment is used to concatenate the output features F1, F2, F3, and F4 of the first four stages of the image downsampling encoder with the output features G1, G2, G3, and G4 of each stage of the feature upsampling encoder used to predict the segmentation result in the order G4, G3, G2, and G1.
[0037] Please see Figure 2 This embodiment uses a UNet-based neural network model, which is a pre-trained UNet-based neural network model; its training process includes the following steps:
[0038] Step 2.1: Obtain training label images. First, adjust the training label images to a uniform size of 256*256, and then obtain the features of the labels by reconstructing the network model.
[0039] To acquire training medical images, the training medical images are first uniformly adjusted to a size of 256*256. Then, a UNet-based neural network model is used to obtain segmentation results and predict similarity. Finally, a reconstruction network model is used to obtain the features of the segmentation results.
[0040] Please see Figure 3The reconstruction network model in this embodiment includes a downsampling encoder and an upsampling encoder;
[0041] The downsampling encoder in this embodiment includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence.
[0042] The upsampling encoder in this embodiment includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
[0043] The reconstruction network model in this embodiment is a pre-trained reconstruction network model. During training, labels are used as input, and the output is required to be the same as the input. The loss function used is the cross-entropy loss function. The training ends after reaching a preset number of iterations (200 in this embodiment). At the end of each training iteration, the network performance is tested using a validation set, and the parameters of the reconstruction network model with the best performance are saved as the final reconstruction network model parameters.
[0044] Step 2.2: Calculate the cosine similarity between the features of the segmentation result and the features of the label;
[0045] In this embodiment, the similarity between the features of the segmentation result and the features of the label is calculated by using the cosine similarity between the features of the segmentation result and the features of the label as the similarity.
[0046] When obtaining label features and segmentation results, only the output at the end of the first stage of the reconstruction network downsampling encoder is used as the features.
[0047] Step 2.3: Train the UNet-based neural network model based on the training labeled images, labels, and cosine similarity; the training ends when the preset number of times is reached (200 in this embodiment). At the end of each training session, the network performance is tested using a validation set, and the parameters of the network model with the best performance are saved as the final network model parameters.
[0048] In this embodiment, the parameters of the label reconstruction network are fixed during the process of building a UNet-based neural network model. The loss function is composed of the cross-entropy loss CELoss between the prediction result and the label, the mean squared error loss between similarity and 1, and the mean squared error loss between predicted similarity and similarity as the final loss function used to train the UNet-based neural network model.
[0049] In this embodiment, the network segmentation loss function is the cross-entropy loss function:
[0050]
[0051] Where N represents the number of pixels, i represents the current i-th pixel, M represents the number of categories, c represents the current category, and y ic Let c and p represent the labels of the current i-th point. ic This represents the probability that the current i-th point belongs to category c;
[0052] In this embodiment, the similarity-guided loss function is the mean squared error loss function, which is:
[0053]
[0054] Where n represents the number of pixels, and i represents the current i-th pixel. Let represent the predicted value of the current i-th point, and yi represent the label value of the current i-th point;
[0055] The overall loss function in this embodiment is expressed as follows:
[0056] Loss=CE(pred, gt)+MSE(cos_sim, cos_pred)+MSE(1, cos_sim);
[0057] Wherein, pred represents the predicted segmentation result, gt represents the segmentation label, cos_sim represents the similarity calculated between the features of the segmentation result and the features of the label, and cos_pred represents the predicted similarity.
[0058] Through experiments, using the commonly used medical evaluation metrics Dicescore (Dice) and Hausdorff Distance (HD) as evaluation standards, this application was validated on three commonly used medical image segmentation datasets. The experimental results show that the segmentation method proposed in this application can significantly improve the segmentation effect of neural networks on medical images and has good generalization performance.
[0059] The datasets used were the ACDC cardiac MRI dataset, the Synapse abdominal multi-organ CT dataset, and the colorectal polyp endoscopic dataset. The colorectal polyp endoscopic dataset included five datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, ETIS, and Endoscene. Experimental results are as follows:
[0060]
[0061] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.
Claims
1. An image segmentation method using feature similarity-guided segmentation, characterized in that, Includes the following steps: Step 1: Acquire medical images; Step 2: Obtain the segmentation result image using a UNet-based neural network model; The UNet-based neural network model includes an image downsampling encoder, a same-layer fusion module, a feature upsampling decoder for predicting segmentation results, and a feature upsampling decoder for predicting similarity. The image downsampling encoder includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence. The feature upsampling encoder for predicting segmentation results includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1. The feature upsampling decoder for predicting similarity includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1. The same-layer fusion module is used to concatenate the output features F1, F2, F3, and F4 of the first four stages of the image downsampling encoder with the output features G1, G2, G3, and G4 of each stage of the feature upsampling encoder used to predict the segmentation result in the order G4, G3, G2, and G1.
2. The image segmentation method using feature similarity-guided segmentation according to claim 1, characterized in that: The UNet-based neural network model is a pre-trained UNet-based neural network model. Its training process includes the following steps: Step 2.1: Obtain the training label images and reconstruct the network model to obtain the features of the labels; Acquire training medical images, use a UNet-based neural network model to obtain segmentation results and predict similarity, and use the reconstruction network model to obtain features of the segmentation results; Step 2.2: Calculate the similarity between the features of the segmentation result and the features of the label; Step 2.3: Train the UNet-based neural network model based on the training labeled images, labels, and similarities. The training ends after a preset number of iterations. At the end of each training iteration, the network performance is tested using a validation set, and the parameters of the network model with the best performance are saved as the final network model parameters.
3. The image segmentation method using feature similarity-guided segmentation according to claim 2, characterized in that: The reconstructed network model described in step 2.1 includes a downsampling encoder and an upsampling encoder; The downsampling encoder includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence. The upsampling encoder includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The structure of the upsampling module is as follows: a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence. The convolutional layer for result output has a kernel size of 1 and a stride of 1.
4. The image segmentation method using feature similarity-guided segmentation according to claim 2, characterized in that: The reconstructed network model is a pre-trained reconstructed network model. During training, the labels are used as input, and the output is required to be the same as the input. The loss function used is the cross-entropy loss function. The training ends after a preset number of iterations. At the end of each training iteration, the network performance is tested using a validation set, and the parameters of the best-performing reconstructed network model are saved as the final reconstructed network model parameters.
5. The image segmentation method using feature similarity-guided segmentation according to claim 2, characterized in that: The calculation of the similarity between the features of the segmentation result and the features of the label in step 2.2 is to calculate the cosine similarity between the features of the segmentation result and the features of the label as the similarity.
6. The image segmentation method using feature similarity-guided segmentation according to any one of claims 2-5, characterized in that: In step 2.3, the loss functions used for training include the network segmentation loss function and the similarity-guided loss function; The network segmentation loss function is the cross-entropy loss function: ； Where N represents the number of pixels. i Indicates the current number i There are 100 pixels, M represents the number of categories, and c represents the current category. y ic Let c represent the label of the current i-th point. p ic This represents the probability that the current i-th point belongs to category c; The similarity-guided loss function is the mean squared error loss function: ； Where n represents the number of pixels, and i represents the current i-th pixel. This represents the predicted value of the current i-th point. yi This represents the label value of the current i-th point; The overall loss function is expressed as: Loss=CE(pred,gt)+ )+ ; Wherein, pred represents the predicted segmentation result, gt represents the segmentation label, cos_sim represents the similarity calculated between the features of the segmentation result and the features of the label, and cos_pred represents the predicted similarity.
7. An image segmentation system guided by feature similarity, characterized in that, Includes the following modules: Module 1 is used to acquire medical images; Module 2 is used to obtain the segmentation result image using a UNet-based neural network model; The UNet-based neural network model includes an image downsampling encoder, a same-layer fusion module, a feature upsampling decoder for predicting segmentation results, and a feature upsampling decoder for predicting similarity. The image downsampling encoder includes five stages of convolutional modules arranged in series. Each stage of the convolutional module includes two convolutional layers arranged in series. The kernel size of the two convolutional layers is 3, and the stride is 1. After each convolutional layer, a normalization layer and an activation layer are added in sequence. The feature upsampling encoder for predicting segmentation results includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1. The feature upsampling decoder for predicting similarity includes four convolutional modules arranged in series and a convolutional layer for result output. Each convolutional module includes an upsampling module and a convolutional module arranged in series. The upsampling module structure consists of a bilinear interpolation upsampling layer, a convolutional layer, a normalization layer, and an activation layer. The convolutional layer has a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional module includes two convolutional layers arranged in series. Both convolutional layers have a kernel size of 3 and a stride of 1. After each convolutional layer, a normalization layer and an activation layer are added sequentially. The convolutional layer for result output has a kernel size of 1 and a stride of 1. The same-layer fusion module is used to concatenate the output features F1, F2, F3, and F4 of the first four stages of the image downsampling encoder with the output features G1, G2, G3, and G4 of each stage of the feature upsampling encoder used to predict the segmentation result in the order G4, G3, G2, and G1.
8. An image segmentation device that utilizes feature similarity-guided segmentation, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image segmentation method using feature similarity-guided segmentation as described in any one of claims 1 to 6.