Image segmentation method and system based on semi-supervised learning and detail feature enhancement

Abstract

The invention provides an image segmentation method and system based on semi-supervised learning and detail enhancement. The method comprises the following steps: collecting a medical image and labeling part of data; constructing a medical image segmentation model; inputting the marked image into a medical image segmentation model to calculate the cross entropy loss of the marked image; weak disturbance and strong disturbance are applied to the unlabeled image; inputting the weak disturbance image and the strong disturbance image into a medical image segmentation model, outputting the weak disturbance image through an encoder, adding a feature disturbance strategy to the weak disturbance image, and calculating consistency loss by using an output result obtained by unlabeled data; the total loss of the set model comprises cross entropy loss and consistency loss; performing optimization and reverse gradient propagation on the total loss, and updating model network parameters; and inputting the target image into the medical image segmentation model, wherein the medical image segmentation model outputs the category to which each pixel point of the target image belongs. According to the invention, the efficiency of medical image segmentation can be improved, and higher precision and robustness can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image segmentation, and in particular to an image segmentation method and system based on semi-supervised learning and detail feature enhancement. Background Art

[0002] Medical image segmentation is a technology that divides each pixel in a medical image into different categories based on the characteristics of different tissues. It can help doctors accurately locate the lesion area.

[0003] With the continuous development of deep learning technology, many researchers are committed to applying deep learning technology to the field of medical image segmentation and have achieved remarkable results. However, the success of deep learning often depends on large-scale annotated data. The acquisition of medical images requires manual annotation by clinicians, which requires a lot of labor costs and it is difficult to obtain sufficient annotated data. To solve this problem, semi-supervised learning provides an effective way to train the model by making full use of labeled data and a large amount of unlabeled data, so as to achieve better performance on a relatively small labeled data set and have stronger generalization ability. The use of semi-supervised learning methods in medical image segmentation can effectively reduce the cost of medical image data annotation, and use a small amount of annotated data to achieve the same or even better segmentation effect as full supervised learning. Most traditional semi-supervised learning methods are based on generating pseudo-labels to supervise unlabeled data. The quality of generated pseudo-labels has a huge impact on model training, and semi-supervised learning based on strong and weak consistency regularization can well avoid this problem. Therefore, semi-supervised learning technology based on strong and weak consistency has important research significance.

[0004] In addition, the current mainstream medical image segmentation model mainly adopts the Transformer+unet combined architecture. Although this architectural feature can better retain the global feature information and local feature information of medical images during the semantic segmentation process, the Transformer architecture has the problem of high computing resource requirements, and the semantic information of different channels and positions in medical images is unevenly distributed. The traditional medical image segmentation model has limited interaction between the semantic information obtained from different channels, so the semantic information learned by the model is biased.

[0005] Therefore, it is also of great research significance to study a method that can solve the uneven distribution of image features and a method that can maximize the retention of image features using less computing resources. Summary of the invention

[0006] The purpose of the present invention is to propose an image segmentation method and system based on semi-supervised learning and detail feature enhancement, which can significantly improve the efficiency of image segmentation and achieve higher accuracy with lower parameter quantity and computational complexity.

[0007] To achieve the above object, in a first aspect, the present invention provides an image segmentation method based on semi-supervised learning and detail feature enhancement, comprising the following steps: Step S1: collect original medical images, annotate some of the data, and obtain a partially annotated training data set; Step S2: constructing a medical image segmentation model, wherein the medical image segmentation model includes three parts: an encoder, a feature transformation, and a decoder; Step S3: Annotated images x Input the medical image segmentation model and calculate its cross entropy loss as the supervised loss; Step S4: Unlabeled images x u Apply weak perturbation strategy and strong perturbation strategy to obtain weak perturbation image x w and strongly perturbed images x s1 、x s2 ; Step S5: The weakly perturbed image x w and strongly perturbed images x s1 、x s2 Input medical image segmentation model; the weakly perturbed image x w After the encoder output, add a feature perturbation strategy to obtain a feature-enhanced image x d and x vat ; Weakly perturbed image x w , image after feature enhancement x d and x vat , strongly disturbed image x s1 and x s2 Pass through the decoder respectively and finally output the corresponding results p w , p d , p vat , p s1 ,p s2 ; Step S6: Calculate the consistency loss using the output result obtained from the unlabeled data as the unsupervised loss; Step S7: setting the total model loss to include the supervised loss obtained in step S3 and the unsupervised loss obtained in step 6; Step S8: optimizing the total loss and performing reverse gradient propagation, and updating the model network parameters to achieve semi-supervised learning of the medical image segmentation model; Step S9: Input the target image into a medical image segmentation model, which outputs the category to which each pixel of the target image belongs, and selects the category with the highest probability as the pixel category.

[0008] Furthermore, the weak perturbation strategy includes image flipping and image cropping strategies; the strong perturbation strategy includes cutmix and st++ data perturbation methods, wherein: x s1 Obtained through the cutmix strategy, x s2 It is obtained through the st++ strategy; the feature perturbation strategy includes a dropout strategy and a virtual adversarial training strategy.

[0009] Furthermore, the encoder, decoder, and feature transformation parts of the medical image segmentation model all use adaptive dynamic large kernel convolution to quickly extract image features and model the long- and short-distance dependencies of the image, thereby improving the adaptability of the medical image segmentation model to objects of different scales; a pixel feature fusion attention mechanism is used in the connection part and feature transformation part of the encoder and decoder to fuse the position information and channel information of different channels, thereby helping the model learn the semantic information of different channels.

[0010] Furthermore, the adaptive dynamic large kernel convolution is composed of a prior knowledge fusion branch and an adaptive large kernel convolution branch. The prior knowledge fusion branch includes horizontal differential convolution and vertical differential convolution. By selecting pixel pairs of the image, the differential strategy is used to explicitly encode the prior information into the convolutional neural network; the adaptive large kernel convolution branch includes a convolution layer, a GeLU activation function layer, two adaptive convolution layers, and a convolution layer in sequence.

[0011] Furthermore, the pixel feature fusion attention mechanism includes a position information layer, a channel information layer, a channel fusion layer, a channel cleaning layer, a convolution layer and an activation function layer. The position information layer and the channel information layer respectively obtain the position information of the image. w s and channel information w cThe channel cleaning layer cleans the semantic information of each channel, and after passing through the convolution layer and the activation function layer, the important feature information of the image is obtained. w fin .

[0012] Furthermore, the data tensor shape of the medical image segmentation model input is [B, C, H, W], where B represents the number of images input for training once, C is channel, representing the number of channels of each image, H represents the height of each image, and W represents the width of each image; the specific process of the medical image segmentation model performing the segmentation task includes the following steps: Step a: Input the data into the encoder part of the model, which is divided into 3 layers, each layer including adaptive dynamic large kernel convolution and downsampling module; Step b: The feature transformation part of the model of the medical image input after three downsamplings, the feature transformation part sequentially includes a common convolution with a convolution kernel size of 3x3, an adaptive dynamic large kernel convolution and a pixel feature fusion attention mechanism; the image obtained after the medical image after three downsamplings is subjected to a common convolution with a convolution kernel size of 3x3 and an adaptive dynamic large kernel convolution, and the image obtained after the medical image after the last downsampling is subjected to feature fusion with the medical image output by the last downsampling layer through the pixel feature fusion attention mechanism; Step c: the image after feature fusion of the feature transformation part is input into the decoder part of the model, wherein the decoder part is divided into 3 layers corresponding to the encoder, and each layer includes an adaptive dynamic large kernel convolution and an upsampling module; wherein the feature images of the first two layers in the upsampling are fused with the feature images of the downsampling layer corresponding to the encoder part through a pixel fusion attention mechanism; Step d: The model outputs the result, that is, the category to which each pixel of the target image belongs.

[0013] Furthermore, in step S3, the supervised loss L s The specific expressions are as follows: Where, for each pixel, x i represents the true category of each pixel, y i Represents the model's predicted category for each pixel.

[0014] Furthermore, in step S6, the unsupervised loss L u The specific expressions are as follows: in, B urepresents the batch size of unlabeled data, τ is a predefined confidence threshold for filtering noisy labels; H represents minimizing the entropy between two probability distributions, λ 1 , λ 2 and μ Both represent loss weights, p w represents the segmentation result of the weakly perturbed image, p s1 and p s2 represents the segmentation result of the strongly perturbed image, p vat Used to represent the segmentation result after applying VAT perturbation to the image. p d Represents the result after applying dropout perturbation.

[0015] Furthermore, the total loss in S7 L The specific representation is as follows: in, α A hyperparameter ranging from 0 to 1 that adjusts the importance of unlabeled data in the model.

[0016] In a second aspect, the present invention provides an image segmentation system based on semi-supervised learning and detail feature enhancement, comprising a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes any one of the above-mentioned image segmentation methods based on semi-supervised learning and detail feature enhancement.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention avoids the shortcomings of the semi-supervised learning method based on pseudo-labels by using a semi-supervised learning method based on strong and weak consistency regularization, and has higher generalization.

[0018] (2) The present invention can better model the long- and short-distance dependencies in medical images through the adaptive large-kernel convolution module, and can improve the adaptability of the model to objects of different scales without consuming a large amount of computing resources.

[0019] (3) The present invention fuses the position information and channel information of different channels through the pixel fusion attention module, thereby improving the interaction ability between the semantic information of different channels, thereby helping the model to better learn the semantic information of different channels and better solving the problem of uneven distribution of feature information in traditional medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the structure of the medical image segmentation model of the present invention. Figure 2 It is a schematic diagram of the structure of the pixel feature fusion attention mechanism of the present invention; FIG3 (a) is a schematic diagram of the adaptive dynamic large kernel convolution of the present invention; FIG3 (b) is a diagram of the branch difference strategy of prior knowledge fusion of the present invention; FIG3 (c) is a schematic diagram of an adaptive convolution module in an adaptive large kernel convolution branch of the present invention; Figure 4 Schematic diagram of the semi-supervised learning method of the present invention; Figure 5 It is a flowchart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0022] The present invention proposes an image segmentation method based on semi-supervised learning and detail feature enhancement. Figure 5 , including the following steps: Step S1: collect clinical medical image data sets, annotate a part of the data sets, and obtain a partially annotated training data set.

[0023] In step S1, the data set is preprocessed: the original data set and labels are cropped into a size of 512×512, and then data enhancement such as random flipping and random scaling is performed.

[0024] Step S2: for the unlabeled data set, a part of the data set is subjected to weak perturbation, and the other part is subjected to strong perturbation strategy; In step S2, the weak perturbation strategies used for the unlabeled images are random cropping and random flipping strategies; the strong perturbation strategies used are cutmix and st++ algorithm strategies.

[0025] Step S3: construct a medical image semantic segmentation model through pixel feature fusion attention mechanism and adaptive dynamic large kernel convolution, and set a batch input of 8 images.

[0026] In step S3, Figure 1As shown in the figure, the medical image semantic segmentation model is mainly divided into three parts: encoder, decoder, and feature transformation. All three parts use adaptive dynamic large kernel convolution to quickly extract image features and model the long- and short-distance dependencies of images and improve the adaptability of the medical image segmentation model to targets of different scales. The pixel feature fusion attention mechanism is used in the connection part between the encoder and decoder and the feature transformation part to fuse the position information and channel information of different channels, which helps the model learn the semantic information of different channels.

[0027] Step S4: input the labeled data in a batch into the medical image semantic segmentation model, calculate the function loss of the model output, and use the calculated cross entropy loss as the supervised loss.

[0028] In step S4, the supervised loss function used is specifically expressed as follows: Where, for each pixel, x i represents the true category of each pixel, y i Represents the model's predicted category for each pixel.

[0029] Step S5: Reference Figure 4 , input the weakly perturbed data in a batch into the medical image semantic segmentation model, add the feature perturbation strategy to it after outputting the model encoder part, and then output the segmentation result of the weakly perturbed data through the model decoder part. The feature perturbation strategy includes the dropout strategy and the virtual adversarial training strategy.

[0030] Step S6: input the strongly perturbed data in a batch into the medical image semantic segmentation model, and output the segmentation result of the strongly perturbed data.

[0031] Step S7, calculate the consistency loss of the model output results of steps S5 and S6, and use the calculated result as the unsupervised loss of the model.

[0032] In step S7, the unsupervised loss function used is specifically expressed as follows: in, B u represents the batch size of unlabeled data, τ is a predefined confidence threshold for filtering noisy labels. H It means minimizing the entropy between two probability distributions. λ 1 , λ 2 and μ Both represent loss weights.p w represents the segmentation result of the weakly perturbed image, and p s1 and p s2 Represents the segmentation result of the strongly perturbed image. p vat Used to represent the segmentation result after applying VAT perturbation to the image. p d Represents the result after applying dropout perturbation.

[0033] Step S8: setting the total model loss to the sum of the cross entropy loss obtained in step S4 and the consistency loss of the strong and weak perturbation data outputs obtained in step S7 multiplied by the weight coefficients respectively.

[0034] The total loss of S8 L The specific representation is as follows: in, α A hyperparameter ranging from 0 to 1 that adjusts the importance of unlabeled data in the model.

[0035] Step S9, optimize the overall loss and perform reverse gradient propagation to update network parameters; Step S10: The model outputs the category to which each pixel of the target image belongs, and selects the category with the highest probability as the pixel category.

[0036] In this embodiment, the pixel feature fusion attention mechanism is as follows: Figure 2 As shown in the figure, the feature map first passes through the position information layer and the channel information layer to obtain the position information and channel information, then fuses the two types of information and performs channel cleaning operations, and finally passes through a convolution layer for feature transformation and uses a sigmoid activation function to regularize it.

[0037] In this embodiment, the adaptive dynamic large kernel convolution module is shown in Figure 3 (a), and consists of two parts: a prior knowledge fusion branch and an adaptive large kernel convolution branch. The prior knowledge fusion branch is composed of horizontal differential convolution and vertical differential convolution. By selecting pixel pairs of the image, the prior information is explicitly encoded into the convolutional neural network using a differential strategy. The differential strategy is shown in Figure 3 (b). Taking the horizontal differential convolution as an example, the horizontal gradient is first calculated by calculating the difference between the selected pixel pairs. After the training is completed, we rearrange the learned kernel weights equivalently and directly convolve the unchanged input features. The number of network layers of the adaptive large kernel convolution branch is 5 layers, including a convolution layer, a GeLU activation function layer, two adaptive convolution layers, and a convolution layer in sequence. Among them, the adaptive convolution used in this branch is shown in Figure 3 (c). The image information is first compressed by global average pooling, and then two fully connected layers, a ReLU activation function, and a softmax activation function are used to generate normalized attention weights for n convolution kernels. By using this special large-kernel convolution, the convolution kernel size can be flexibly changed according to pictures of different sizes, thereby increasing the expressive power of the model without increasing the depth or width of the network.

[0038] The present invention also proposes an image segmentation system based on semi-supervised learning and detail feature enhancement, comprising a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically executes any one of the above-mentioned image segmentation methods based on semi-supervised learning and detail feature enhancement.

[0039] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. Image segmentation method based on semi-supervised learning and detail feature enhancement, characterized in that: The following steps are included: Step S1: collect original medical images, annotate some of the data, and obtain a partially annotated training data set; Step S2: constructing a medical image segmentation model, wherein the medical image segmentation model includes three parts: an encoder, a feature transformation, and a decoder; Step S3: Annotated images x Input the medical image segmentation model and calculate its cross entropy loss as the supervised loss; Step S4: Unlabeled images x u Apply weak perturbation strategy and strong perturbation strategy to obtain weak perturbation image x w and strongly perturbed images x s1 、x s2 ; Step S5: The weakly perturbed image x w and strongly perturbed images x s1 、x s2 Input medical image segmentation model; the weakly perturbed image x w After the encoder output, add a feature perturbation strategy to obtain a feature-enhanced image x d and x vat ; Weakly perturbed image x w , image after feature enhancement x d and x vat , strongly disturbed image x s1 and x s2 Pass through the decoder respectively and finally output the corresponding results p w , p d , p vat , p s1 , p s2 ; Step S6: Calculate the consistency loss using the output result obtained from the unlabeled data as the unsupervised loss; Step S7: setting the total model loss to include the supervised loss obtained in step S3 and the unsupervised loss obtained in step 6; Step S8: optimizing the total loss and performing reverse gradient propagation, and updating the model network parameters to achieve semi-supervised learning of the medical image segmentation model; Step S9: Input the target image into a medical image segmentation model, which outputs the category to which each pixel of the target image belongs, and selects the category with the highest probability as the pixel category.

2. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: The weak perturbation strategy includes image flipping and image cropping strategies; the strong perturbation strategy includes cutmix and st++ data perturbation methods, wherein: x s1 Obtained through the cutmix strategy, x s2 It is obtained through the st++ strategy; the feature perturbation strategy includes a dropout strategy and a virtual adversarial training strategy.

3. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: The encoder, decoder and feature transformation of the medical image segmentation model all use adaptive dynamic large kernel convolution to quickly extract image features and model the long- and short-distance dependencies of the image, thereby improving the adaptability of the medical image segmentation model to objects of different scales. The pixel feature fusion attention mechanism is used in the connection part and feature transformation part between the encoder and decoder to fuse the position information and channel information of different channels to help the model learn the semantic information of different channels.

4. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 3, characterized in that: The adaptive dynamic large kernel convolution is composed of a prior knowledge fusion branch and an adaptive large kernel convolution branch. The prior knowledge fusion branch includes horizontal differential convolution and vertical differential convolution. By selecting pixel pairs of the image, the differential strategy is used to explicitly encode the prior information into the convolutional neural network; the adaptive large kernel convolution branch includes a convolution layer, a GeLU activation function layer, two adaptive convolution layers, and a convolution layer in sequence.

5. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 3, characterized in that: The pixel feature fusion attention mechanism includes a position information layer, a channel information layer, a channel fusion layer, a channel cleaning layer, a convolution layer and an activation function layer. The position information layer and the channel information layer respectively obtain the position information of the image. w s Channel Information w c The channel cleaning layer cleans the semantic information of each channel, and after passing through the convolution layer and the activation function layer, the important feature information of the image is obtained. w fin .

6. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: The data tensor shape of the medical image segmentation model input is [B, C, H, W], where B represents the number of images input for training once, C is channel, representing the number of channels of each image, H represents the height of each image, and W represents the width of each image; the specific process of the medical image segmentation model performing the segmentation task includes the following steps: Step a: Input the data into the encoder part of the model, which is divided into 3 layers, each layer including adaptive dynamic large kernel convolution and downsampling module; Step b: The feature transformation part of the model of the medical image input after three downsamplings, the feature transformation part sequentially includes a common convolution with a convolution kernel size of 3x3, an adaptive dynamic large kernel convolution and a pixel feature fusion attention mechanism; the image obtained after the medical image after three downsamplings is subjected to a common convolution with a convolution kernel size of 3x3 and an adaptive dynamic large kernel convolution, and the image obtained after the medical image after the last downsampling is subjected to feature fusion with the medical image output by the last downsampling layer through the pixel feature fusion attention mechanism; Step c: the image after feature fusion of the feature transformation part is input into the decoder part of the model, wherein the decoder part is divided into 3 layers corresponding to the encoder, and each layer includes an adaptive dynamic large kernel convolution and an upsampling module; wherein the feature images of the first two layers in the upsampling are fused with the feature images of the downsampling layer corresponding to the encoder part through a pixel fusion attention mechanism; Step d: The model outputs the result, that is, the category to which each pixel of the target image belongs.

7. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: In step S3, the supervised loss L s The specific expressions are as follows: Where, for each pixel, x i represents the true category of each pixel, y i Represents the model's predicted category for each pixel.

8. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: In step S6, the unsupervised loss L u The specific expressions are as follows: in, B u represents the batch size of unlabeled data, τ is a predefined confidence threshold for filtering noisy labels; H represents minimizing the entropy between two probability distributions, λ 1 , λ 2 and μ Both represent loss weights, p w represents the segmentation result of the weakly perturbed image, p s1 and p s2 represents the segmentation result of the strongly perturbed image, p vat Used to represent the segmentation result after applying VAT perturbation to the image. p d Represents the result after applying dropout perturbation.

9. The image segmentation method based on semi-supervised learning and detail feature enhancement according to claim 1, characterized in that: The total loss in S7 L The specific representation is as follows: in, α A hyperparameter ranging from 0 to 1 that adjusts the importance of unlabeled data in the model.

10. Image segmentation system based on semi-supervised learning and detail feature enhancement, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, the method specifically performs the image segmentation method based on semi-supervised learning and detail feature enhancement as described in any one of claims 1 to 9.

Images