A polyp segmentation method applicable to different data distributions

Abstract

The present invention discloses a polyp segmentation method that can be used under different data distributions, comprising the following steps: S1, obtaining a data set of polyps and preprocessing the data set; S2, constructing a polyp segmentation model and a discriminator for adversarial training; S3, pre-training the segmentation model and the discriminator respectively using adversarial learning; S4, formally training the pre-trained segmentation model; S5, using the formally trained segmentation model to perform polyp segmentation; the present invention dynamically combines adversarial training and pseudo-labeling methods, can solve cross-domain problems when data distributions are different, and has good generalization ability on test sets with different data distributions.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a polyp segmentation method that can be used under different data distributions. Background Art

[0002] Colon cancer is a common malignancy worldwide, causing nearly one million deaths in 2020. Polyps are considered a precursor to colon cancer, making their effective detection and removal crucial. Clinically, polyp diagnosis is performed by experienced endoscopists, a process that carries the disadvantages of high labor costs and a high rate of missed diagnoses.

[0003] With the development of artificial intelligence, numerous deep learning-based polyp segmentation methods have been proposed. Initially, convolutional neural networks (CNNs), UNets, and ResUnets were applied to polyp segmentation. Later, with the development of transformers, methods based on parallel anti-attention networks, such as ParNet and SANet, were proposed and achieved impressive results. These models all rely on the premise that the test and training sets share the same data distribution (same image color, lighting, texture, and other styles), and their excellent performance relies on extensive image annotation. However, in real-world scenarios, due to the wide variation in medical equipment configurations across hospitals and individual patient data, the distribution of acquired polyp image data varies significantly. This is reflected in differences in color, texture, lighting, and texture between the target domain polyp images to be detected and the pre-prepared source domain polyp training set. Furthermore, labels for these target domain polyp images are difficult to obtain in a timely manner, making methods that rely solely on source domain datasets suboptimal in the target domain. In these more practical scenarios, these models suffer from poor generalization and significantly reduced performance, making them difficult to meet user needs. Summary of the Invention

[0004] The purpose of the present invention is to provide a polyp segmentation method that can be used under different data distributions. The polyp segmentation method that can be used under different data distributions dynamically combines adversarial training and pseudo-labeling methods, can solve cross-domain problems when data distributions are different, and has good generalization ability on test sets with different data distributions.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A polyp segmentation method applicable to different data distributions includes the following steps:

[0007] S1. Obtain a polyp dataset and preprocess the dataset;

[0008] S2. Build a polyp segmentation model and a discriminator for adversarial training;

[0009] S3. Use adversarial learning to pre-train the segmentation model and discriminator respectively;

[0010] S4. Formal training of the pre-trained segmentation model;

[0011] S5. Use the formally trained segmentation model to perform polyp segmentation.

[0012] Preferably, the specific process of step S1 is:

[0013] S11. Obtain four public datasets of intestinal polyps: Kvasir, ClinicDB, ColonDB, and ETIS;

[0014] S12. Use the images and labels of one of the datasets as the source domain, and the images of the other three datasets as the target domain;

[0015] S13. The colorectal polyp segmentation image is cropped to a size of 352*352 using a bilinear interpolation algorithm.

[0016] Preferably, the segmentation model in step S2 adopts the deeplabv2 model in the domain adaptation scenario; the discriminator consists of 5 convolutional layers and uses leakyReLU as the activation function.

[0017] Preferably, the specific process of step S3 is:

[0018] S31, the three-channel source domain polyp image with width and height W*H is represented as X S ∈S W*H*3 , whose label is y s ; The three-channel target domain polyp image with width and height of W*H is represented as X t ∈T W*H*3 ; Segmentation model Among them, f represents the feature extractor and g represents the final classifier;

[0019] S32, the source domain polyp image X S and the target domain polyp image X t Input into the segmentation model h, the corresponding source domain polyp image and target domain polyp image prediction output are P S (h,w) and P t (h,w) , and then use the entropy graph formula I x (h,w) =-P x (h ,w) logP x (h,w) , obtain the source domain entropy graph I s(h,w) and target domain entropy graph I t (h,w) ;

[0020] S33, training discriminator D: When training the discriminator, the source domain is represented as 1 and the target domain is represented as 0; the target domain entropy map I t (h,w) and source domain entropy graph I s (h,w) As the input of the discriminator, the domain classification loss L of the cross entropy of the discriminator is calculated D , and use gradient descent to train a discriminator consisting of convolutional layers, the formula is as follows:

[0021]

[0022] in, Represents the cross entropy domain classification loss, 1 represents the source domain, 0 represents the target domain, represents the source domain dataset, represents the target domain dataset, represents the entropy map of the source domain image output, An entropy map representing the output of the target domain image;

[0023] S34. Training segmentation model h: The training of the segmentation model consists of two parts;

[0024] Calculate cross entropy loss: The formula for calculating cross entropy loss is as follows:

[0025]

[0026] in, represents the pseudo label of the target domain image after denoising, represents the probability prediction for the target domain image, (h, w)∈K represents the pixels with higher confidence in the pseudo label, and c represents the number of categories;

[0027] Calculate the source domain cross entropy loss: The formula for calculating the cross entropy loss is as follows:

[0028]

[0029] in, represents the true label of the source domain image, Represents the probability prediction for the source domain image;

[0030] Calculate the adversarial loss: the target domain entropy map I obtained by the segmentation model h t (h,w) It can successfully deceive the discriminator, which will think it is an entropy map from the source domain. The specific loss formula is as follows:

[0031]

[0032] in, represents the target domain dataset, θ F represents the parameters of the segmentation model that needs to be learned, Indicates that the target domain entropy map needs to be The prediction is the cross entropy domain classification loss from source domain 1;

[0033] The final objective loss function is:

[0034]

[0035] Among them, λ acv Is the balance L seg and L adv Hyperparameters of weights;

[0036] After training for 150 epochs, we obtain the pre-trained segmentation model h1 and the pre-trained discriminator D1.

[0037] Preferably, the specific process of step S4 is:

[0038] S41. Input the target domain polyp image into the pre-trained segmentation model h1 to obtain the target domain entropy map I t (h,w) ;

[0039] S42, target domain entropy graph I t (h,w) Continue to use it as the input of the pre-trained discriminator D1, using Decide on the next branch of training;

[0040] S43, if Then the source domain polyp images and corresponding labels as well as the target domain polyp images are used for adversarial training; conversely, if Pseudo-label training is performed using source domain polyp images and corresponding labels as well as target domain polyp images and pseudo-labels.

[0041] Preferably, the specific process of generating the pseudo-label in step S43 is:

[0042] S431: Input all images of the source domain and their corresponding labels into the pre-trained segmentation model h1 to obtain the prototype value. The calculation formula is as follows:

[0043]

[0044] Among them, x t refers to the target domain image, f(x t ) i Point to x t The features on the i-th index extracted from Refers to the pre-trained segmentation model h1 predicting x t The i-th pixel is the k-th class;

[0045] S432: The target domain image, the target domain image reduced to 0.7 times the original size, and the target domain image enlarged to 1.5 times the original size are used as inputs of the pre-trained segmentation model h1, and three corresponding predictions are obtained. The average prediction value of the pixels corresponding to the three predictions is the final output P. t ;

[0046] S433, P t Perform noise reduction and generate corresponding pseudo labels: f(x t ) i represents the feature of the i-th pixel of the target domain image x extracted by the encoder f, if f(x t ) i Away from prototype η (k) , then the probability of it belonging to class k is weakened, and vice versa, it is enhanced, which is achieved through the following formula:

[0047]

[0048]

[0049] in, refers to the momentum encoder of the feature extractor f, T is set to 1, η (k) Refers to the feature center of category k; the weight on pixel i is calculated and multiplied by the prediction, and the soft prediction is finally Convert to hard label one-hot encoding Set the highest predicted label of a class to 1, and the rest to 0;

[0050] S434, using the image and label of the source domain, the image and pseudo label of the target domain, through the cross entropy loss L seg Continue to train the pre-trained segmentation model h1. During the training process, the pseudo labels will be dynamically adjusted as the prototype and weight ω change.

[0051] Preferably, the specific process of step S5 is: taking the target domain image, the target domain image reduced in size to 0.7 times the original size, and the target domain image enlarged in size to 1.5 times the original size as the input of the formally trained segmentation model, obtaining three corresponding predictions, and the predicted average value of the pixel points corresponding to the three predictions is obtained as the final output.

[0052] After adopting the above technical solution, the present invention has the following beneficial effects: by analyzing the different applicable scenarios of pseudo-label training and adversarial training in domain adaptation scenarios, the present invention uses a discriminator to dynamically combine adversarial training and pseudo-label methods, thereby better avoiding the adverse effects of the two methods, being able to solve cross-domain problems when data distribution is different, having good generalization ability on test sets with different data distributions, and realizing the polyp segmentation task in domain adaptation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the present invention;

[0054] Figure 2 are polyp images of four datasets of the present invention;

[0055] Figure 3 The image of the source domain and its label and the image of the target domain of the present invention;

[0056] Figure 4 Schematic diagram of the structure of the discriminator of the present invention;

[0057] Figure 5 This is a diagram of the pre-training structure of adversarial training of the present invention;

[0058] Figure 6 This is a structural diagram of the adversarial training and pseudo-label training of the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of the present invention using a segmentation model to generate target domain polyp image prediction. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] Example

[0062] like Figures 1 to 7 As shown, a polyp segmentation method that can be used under different data distributions includes the following steps:

[0063] S1. Obtain a data set of polyps and preprocess the data set;

[0064] The specific process of step S1 is:

[0065] S11. Obtain four public datasets of intestinal polyps: Kvasir, ClinicDB, ColonDB, and ETIS;

[0066] S12. Use the images and labels of one of the datasets as the source domain, and the images of the other three datasets as the target domain;

[0067] S13, cropping the colorectal polyp segmentation image to a size of 352*352 using a bilinear interpolation algorithm;

[0068] S2. Build a polyp segmentation model and a discriminator for adversarial training;

[0069] The segmentation model in step S2 adopts the deeplabv2 model in the domain adaptation scenario; the discriminator consists of 5 convolutional layers and uses leakyReLU as the activation function;

[0070] S3. Use adversarial learning to pre-train the segmentation model and discriminator respectively;

[0071] The specific process of step S3 is:

[0072] S31, the three-channel source domain polyp image with width and height W*H is represented as X S ∈S W*H*3 , whose label is y s ; The three-channel target domain polyp image with width and height of W*H is represented as X t ∈T W*H*3 ; Segmentation model Among them, f represents the feature extractor and g represents the final classifier;

[0073] S32, the source domain polyp image X S and the target domain polyp image X t Input into the segmentation model h, the corresponding source domain polyp image and target domain polyp image prediction output are P S (h,w) and P t (h,w) , and then use the entropy graph formula I x (h,w) =-P x (h ,w) logP x (h,w) , obtain the source domain entropy graph I s (h,w) and target domain entropy graph I t (h,w) ;

[0074] S33, training discriminator D: When training the discriminator, the source domain is represented as 1 and the target domain is represented as 0; the target domain entropy map I t (h,w) and source domain entropy graph I s (h,w)As the input of the discriminator, the domain classification loss L of the cross entropy of the discriminator is calculated D , and use gradient descent to train a discriminator consisting of convolutional layers, the formula is as follows:

[0075]

[0076] in, Represents the cross entropy domain classification loss, 1 represents the source domain, 0 represents the target domain, represents the source domain dataset, represents the target domain dataset, represents the entropy map of the source domain image output, An entropy map representing the output of the target domain image;

[0077] S34. Training segmentation model h: The training of the segmentation model consists of two parts;

[0078] Calculate cross entropy loss: The formula for calculating cross entropy loss is as follows:

[0079]

[0080] in, represents the pseudo label of the target domain image after denoising, Represents the probability prediction for the target domain image, (h, w)∈K represents the pixels with higher confidence in the pseudo-label, and c represents the number of categories. In this paper, the number of categories is 2, which is divided into polyp category and normal tissue category;

[0081] Calculate the source domain cross entropy loss: The formula for calculating the cross entropy loss is as follows:

[0082]

[0083] in, represents the true label of the source domain image, Represents the probability prediction for the source domain image;

[0084] Calculate the adversarial loss: the target domain entropy map I obtained by the segmentation model h t (h,w) It can successfully deceive the discriminator, which will think it is an entropy map from the source domain. The specific loss formula is as follows:

[0085]

[0086] in, represents the target domain dataset, θ F represents the parameters of the segmentation model that needs to be learned, Indicates that the target domain entropy graph needs to be The prediction is the cross entropy domain classification loss from source domain 1;

[0087] The final objective loss function is:

[0088]

[0089] Among them, λ acv Is the balance L seg and L adv Hyperparameters of weights;

[0090] After training for 150 epochs, we obtain the pre-trained segmentation model h1 and the pre-trained discriminator D1;

[0091] S4. Formal training of the pre-trained segmentation model;

[0092] The specific process of step S4 is:

[0093] S41. Input the target domain polyp image into the pre-trained segmentation model h1 to obtain the target domain entropy map I t (h,w) ;

[0094] S42, target domain entropy graph I t (h,w) Continue to use it as the input of the pre-trained discriminator D1, using Decide on the next branch of training;

[0095] S43, if Then the source domain polyp images and corresponding labels as well as the target domain polyp images are used for adversarial training; conversely, if Then, pseudo-label training is performed using source domain polyp images and corresponding labels as well as target domain polyp images and pseudo-labels;

[0096] The specific process of generating the pseudo-label in step S43 is as follows:

[0097] S431: Input all images of the source domain and their corresponding labels into the pre-trained segmentation model h1 to obtain the prototype value. The calculation formula is as follows:

[0098]

[0099] Among them, x t refers to the target domain image, f(x t ) i Point to x t The features on the i-th index extracted from Refers to the pre-trained segmentation model h1 predicting x t The i-th pixel is the k-th class;

[0100] S432: The target domain image, the target domain image reduced to 0.7 times the original size, and the target domain image enlarged to 1.5 times the original size are used as inputs of the pre-trained segmentation model h1, and three corresponding predictions are obtained. The average prediction value of the pixels corresponding to the three predictions is the final output P. t ;

[0101] S433, P t Perform noise reduction and generate corresponding pseudo labels: f(x t ) i represents the feature of the i-th pixel of the target domain image x extracted by the encoder f, if f(x t ) i Away from prototype η (k) , then the probability of it belonging to class k is weakened, and vice versa, it is enhanced, which is achieved through the following formula:

[0102]

[0103]

[0104] in, refers to the momentum encoder of the feature extractor f, T is set to 1, η (k) Refers to the feature center of category k; the weight on pixel i is calculated and multiplied by the prediction, and the soft prediction is finally Convert to hard label one-hot encoding Set the highest predicted label of a class to 1, and the rest to 0;

[0105] S434, using the image and label of the source domain, the image and pseudo label of the target domain, through the cross entropy loss L seg Continue to train the pre-trained segmentation model h1. During the training process, the pseudo labels will be dynamically adjusted as the prototype and weight ω change;

[0106] S5. Use the formally trained segmentation model to perform polyp segmentation;

[0107] The specific process of step S5 is as follows: the target domain image, the target domain image reduced to 0.7 times the original size, and the target domain image enlarged to 1.5 times the original size are used as the input of the formally trained segmentation model to obtain three corresponding predictions, and the predicted average value of the pixel points corresponding to the three predictions is obtained as the final output.

[0108] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A polyp segmentation method that can be used under different data distributions, characterized in that: The following steps are involved: S1. Obtain a data set of polyps and preprocess the data set; S2. Build a polyp segmentation model and a discriminator for adversarial training; S3. Use adversarial learning to pre-train the segmentation model and discriminator respectively; S4. Formal training of the pre-trained segmentation model; The specific process of step S4 is: S41. Input the target domain polyp image into the pre-trained segmentation model h1 to obtain the target domain entropy map I t (h,w) ; S42, target domain entropy graph I t (h,w) Continue to use it as the input of the pre-trained discriminator D1, using Decide on the next branch of training; S43, if Then the source domain polyp images and corresponding labels as well as the target domain polyp images are used for adversarial training; conversely, if Then, pseudo-label training is performed using source domain polyp images and corresponding labels as well as target domain polyp images and pseudo-labels; The specific process of generating the pseudo-label in step S43 is as follows: S431: Input all images of the source domain and their corresponding labels into the pre-trained segmentation model h1 to obtain the prototype value. The calculation formula is as follows: Among them, x t refers to the target domain image, f(x t ) (i) Point to x t The features on the i-th index extracted from Refers to the pre-trained segmentation model h1 predicting x t The i-th pixel is the k-th class; S432: The target domain image, the target domain image reduced to 0.7 times the original size, and the target domain image enlarged to 1.5 times the original size are used as inputs of the pre-trained segmentation model h1, and three corresponding predictions are obtained. The average prediction value of the pixels corresponding to the three predictions is the final output P. t ; S433, P t Perform noise reduction and generate corresponding pseudo labels: f(x t ) (i) represents the feature of the i-th pixel of the target domain image x extracted by the encoder f, if f(x t ) (i) Away from prototype η (k) , then the probability of it belonging to class k is weakened, and vice versa, it is enhanced, which is achieved through the following formula: in, refers to the momentum encoder of the feature extractor f, T is set to 1, η (k) Refers to the feature center of category k; the weight on pixel i is calculated and multiplied by the prediction, and the soft prediction is finally Convert to hard label one-hot encoding Set the highest predicted label of a class to 1, and the rest to 0; S434, using the image and label of the source domain, the image and pseudo label of the target domain, through the cross entropy loss L seg Continue to train the pre-trained segmentation model h1. During the training process, the pseudo labels will be dynamically adjusted as the prototype and weight ω change; S5. Use the formally trained segmentation model to perform polyp segmentation.

2. The polyp segmentation method applicable to different data distributions according to claim 1, characterized in that: The specific process of step S1 is: S11. Obtain four public datasets of intestinal polyps: Kvasir, ClinicDB, ColonDB, and ETIS; S12. Use the images and labels of one of the datasets as the source domain, and the images of the other three datasets as the target domain; S13. The colorectal polyp segmentation image is cropped to a size of 352*352 using a bilinear interpolation algorithm.

3. The polyp segmentation method applicable to different data distributions according to claim 2, characterized in that: The segmentation model in step S2 adopts the deeplabv2 model in the domain adaptation scenario; the discriminator consists of 5 convolutional layers and uses leakyReLU as the activation function.

4. The polyp segmentation method applicable to different data distributions according to claim 3, characterized in that: The specific process of step S3 is: S31, the three-channel source domain polyp image with width and height W*H is represented as X S ∈S W*H*3 , whose label is y s ; The three-channel target domain polyp image with width and height of W*H is represented as X t ∈T W*H*3 ; Segmentation model Among them, f represents the feature extractor and g represents the final classifier; S32, the source domain polyp image X S and the target domain polyp image X t Input into the segmentation model h, the corresponding source domain polyp image and target domain polyp image prediction output are P S (h,w) and P t (h,w) , and then use the entropy graph formula I x (h,w) =-P x (h,w) logP x (h,w) , obtain the source domain entropy graph I s (h,w) and target domain entropy graph I t (h,w) ; S33, training discriminator D: When training the discriminator, the source domain is represented as 1 and the target domain is represented as 0; the target domain entropy map I t (h,w) and source domain entropy graph I s (h,w) As the input of the discriminator, the domain classification loss L of the cross entropy of the discriminator is calculated D , and use gradient descent to train a discriminator consisting of convolutional layers, the formula is as follows: in, Represents the cross entropy domain classification loss, 1 represents the source domain, 0 represents the target domain, represents the source domain dataset, represents the target domain dataset, represents the entropy map of the source domain image output, An entropy map representing the output of the target domain image; S34. Training segmentation model h: The training of the segmentation model consists of two parts; Calculate cross entropy loss: The formula for calculating cross entropy loss is as follows: in, represents the pseudo label of the target domain image after denoising, represents the probability prediction for the target domain image, (h, w)∈K represents the pixels with higher confidence in the pseudo label, and c represents the number of categories; Calculate the source domain cross entropy loss: The formula for calculating the cross entropy loss is as follows: in, represents the true label of the source domain image, Represents the probability prediction for the source domain image; Calculate the adversarial loss: the target domain entropy map I obtained by the segmentation model h t (h,w) It can successfully deceive the discriminator, which will think it is an entropy map from the source domain. The specific loss formula is as follows: in, represents the target domain dataset, θ F represents the parameters of the segmentation model that needs to be learned, Indicates that the target domain entropy graph needs to be The prediction is the cross entropy domain classification loss from source domain 1; The final objective loss function is: Among them, λ acv Is the balance L seg and L adv Hyperparameters of weights; After training for 150 epochs, we obtain the pre-trained segmentation model h1 and the pre-trained discriminator D1.

5. The polyp segmentation method applicable to different data distributions according to claim 4, characterized in that: The specific process of step S5 is as follows: the target domain image, the target domain image reduced to 0.7 times the original size, and the target domain image enlarged to 1.5 times the original size are used as the input of the formally trained segmentation model to obtain three corresponding predictions, and the predicted average value of the pixel points corresponding to the three predictions is obtained as the final output.

Images