Unsupervised domain adaptation method for retinal vessel segmentation of eye fundus image

Through the unsupervised domain adaptation method, combined with student network and teacher network, the full resolution refinement model and EMA algorithm are used to solve the domain offset problem of the retinal vascular segmentation model under different data distributions, and achieve high levels in different target domains Performance segmentation, save manpower and time, and improve diagnostic efficiency.

CN119091135APending Publication Date: 2024-12-06HUNAN FIRST NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019820.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Prior art When applying retinal vascular segmentation models trained on one dataset to another dataset, segmentation accuracy drops sharply, resulting in a domain offset problem, and a method is needed to solve this problem to be distributed across different data. Maintain high-performance segmentation in the target domain.

Method used

Unsupervised domain adaptation method is adopted to calculate the loss of the source domain and the target domain through the combination of student network and teacher network using a full resolution refining model, and update the teacher network weights through the EMA algorithm, combining the losses of multiple semantic levels and resolutions. Training segmentation model to achieve domain adaptation.

Benefits of technology

Maintaining high-performance retinal vascular segmentation in target domains with different data distribution saves manpower and time, improves diagnostic efficiency, and does not require professionals to manually label large amounts of images to train models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091135A_ABST
    Figure CN119091135A_ABST
Patent Text Reader

Abstract

The invention discloses an eye fundus image retinal vessel segmentation unsupervised domain adaptation method, which comprises the following four steps of: firstly, inputting source domain data into a student network, obtaining a prediction probability by using a full-resolution refining model, and performing down-sampling on a label of an image to calculate source domain loss; inputting the target domain image into a student and teacher network, generating a pseudo label and a prediction probability for the target domain image, and calculating domain adaptation loss; the source domain loss and the domain adaptation loss are combined into general loss to supervise training of the student network; and finally, iteratively updating the weight of the teacher network according to an EMA algorithm for segmenting a target domain image. According to the method, the retinal blood vessel in the fundus image can be automatically segmented by using a computer, a professional does not need to manually mark a large number of images to train a retinal blood vessel segmentation model, high-performance segmentation can be kept in target domains with different data distribution, and the diagnosis efficiency is improved while a large amount of manpower and time are saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of fundus image segmentation, and in particular to an unsupervised domain adaptation method for retinal blood vessel segmentation of fundus images. Background Art

[0002] Fundus images are produced when ophthalmologists use fundus cameras to examine patients. Retinal blood vessels in fundus images are the only blood vessels in the human body that can be observed non-invasively. Changes in vascular morphology are closely related to systemic diseases such as hypertension, diabetes, cardiovascular disease, and stroke, and can be used as long-term prognostic indicators for hypertension and diabetes. Automatically segmenting retinal blood vessels in fundus images using computers can save doctors' time and improve diagnostic efficiency. However, this method requires professionals to manually annotate a large number of images to train the retinal blood vessel segmentation model, which requires a lot of manpower and time. When we train a high-precision segmentation model on one dataset (source domain) and apply it to another dataset with different data distribution (target domain), the segmentation accuracy will drop sharply. We usually call this phenomenon the domain shift problem, and a new method is needed to solve the above problem. Summary of the invention

[0003] To solve the above problems, the present invention proposes an unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images, which is characterized by comprising the following steps:

[0004] Step S1: The source domain image set X s and the source domain label Y s Input to the student network, and refine the model at full resolution to get the predicted probability With image tags pass and Calculate the source domain loss L SD ;

[0005] Step S2: Set the target domain image set X t Input to the student network, and refine the model at full resolution to get the predicted probability pass Generate pseudo labels for target domain images

[0006] Step S3: Set the target domain image set X t Input to the teacher network, and refine the model at full resolution to get the predicted probability use and Compute domain adaptation loss L DA ;

[0007] Step S4: L SD and LDA Merged into a general loss to supervise the training of the student network;

[0008] Step S5: According to the EMA algorithm, the weight of the teacher network is updated by the weight of the student network, and it is iterated for segmentation of the target domain;

[0009] Among them, the student network and the teacher network have the same encoder and decoder structure, both of which contain aggregation modules, residual modules, upsampling and downsampling.

[0010] Furthermore, in step S1, the full-resolution refined model is obtained by s With Y s Downsample to obtain images of different resolutions and corresponding labels Its definition is:

[0011] G 1 =Y;

[0012] G l =f(G l-1 );

[0013]

[0014] Among them, the label of the first resolution is the given label Y, G l Represents the label of the lth layer. Each subsequent downsampled label comes from the previous resolution. x is the threshold manually set by the user, which defaults to 0.5.

[0015] Furthermore, in step S1 to step S3, the full-resolution refinement model combines the prediction probability maps of multiple semantic layers into a prediction result for each resolution, which is defined as:

[0016]

[0017] Among them, P l represents the combined prediction probability of the lth resolution, K l represents the total number of predicted probability maps in the lth resolution, represents the weight assigned to the k-th prediction probability map at the l-th resolution, represents the k-th predicted probability map in the l-th resolution.

[0018] Furthermore, in step S1, the formula for calculating the source domain loss of the full-resolution refinement model is:

[0019]

[0020]

[0021] Where L is the number of different resolutions and N is the number of samples per iteration.

[0022] Furthermore, in step S3, the formula for calculating the domain adaptation loss of the full-resolution refinement model is:

[0023]

[0024]

[0025] Where L is the number of different resolutions and N is the number of samples per iteration.

[0026] Furthermore, in step S2, a threshold of 0.5 is used to Convert to pseudo labels for the target domain

[0027] Furthermore, in step S5, the EMA algorithm is defined as:

[0028]

[0029] in, and denote the weights of the teacher network in the i-th iteration and the (i-1)-th iteration, respectively. represents the weight of the corresponding student network in the i-th iteration.

[0030] Furthermore, the aggregation module uses cascaded and parallel multiple convolutions to integrate the upsampling and downsampling results of adjacent stages; the residual module reduces overfitting by adding a Dropout layer after each BatchNorm layer.

[0031] Furthermore, the general loss calculation formula is: L = L SD +L DA .

[0032] The beneficial effects of the present invention are as follows: The present invention reconstructs the encoder and decoder of the network into a refined segmentation model with full resolution by calculating losses at multiple semantic levels and multiple resolutions, and uses a teacher-student network as a framework to train the segmentation model in combination with source domain loss and domain adaptation loss, so that the segmentation model performs well in target domains with different data distributions. The method uses a computer to automatically segment retinal vessels in fundus images, without the need for professionals to manually annotate a large number of images to train the retinal vessel segmentation model. It can maintain high-performance segmentation in target domains with different data distributions, saving a lot of manpower and time while improving diagnostic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the algorithm framework based on the teacher-student network in the present invention;

[0034] Figure 2 Flow chart of the full-resolution refinement model in the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings:

[0036] like Figure 1 and Figure 2 As shown in the figure, an unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images includes a teacher-student based algorithm framework, which consists of a student network and a teacher network. They have the same encoder and decoder structure, and both contain aggregation modules, residual modules, upsampling and downsampling. Among them, the encoder learns the feature map of the blood vessels; the decoder restores the size of the image and predicts whether a certain pixel is a blood vessel pixel or a background pixel; the aggregation module uses cascaded and parallel multiple convolutions to integrate the upsampling and downsampling results of adjacent stages; the residual module reduces overfitting by adding a Dropout layer after each BatchNorm layer. The specific steps are as follows:

[0037] Step S1: The source domain image set X s and the source domain label Y s Input to the student network, and refine the model at full resolution to get the predicted probability With image tags pass and Calculate the source domain loss L SD ;

[0038] Step S2: Set the target domain image set X t Input to the student network, and refine the model at full resolution to get the predicted probability pass Generate pseudo labels for target domain images

[0039] Step S3: Set the target domain image set X t Input to the teacher network, and refine the model at full resolution to get the predicted probability use and Compute domain adaptation loss L DA ;

[0040] Step S4: L SD and L DA Merged into a general loss to supervise the training of the student network;

[0041] Step S5: According to the exponential moving average (EMA) algorithm, the weight of the teacher network is updated by the weight of the student network, and it is iterated for the segmentation of the target domain. The definition of the EMA algorithm is:

[0042]

[0043] in, and denote the weights of the teacher network in the i-th iteration and the (i-1)-th iteration, respectively. represents the weight of the corresponding student network in the i-th iteration. Figure 1 The main process of the framework is shown, and the English annotations in the figure are translated as follows: Source Data: source domain data; Target Data: target domain data; Student Net: student network; TeacherNet: teacher network; Encoder: encoder; Decoder: decoder; EMA Update: EMA algorithm update; SourceDomain Loss: source domain loss; Domain Adaption Loss: domain adaptation loss; Stop Gradient: do not use gradient descent to update the network.

[0044] The full-resolution refinement model used in the framework is used to extract the features of fundus images and predict retinal vascular pixels. The specific processing steps of the model are as follows:

[0045] First, introduce the general definition:

[0046]

[0047]

[0048]

[0049] The present invention uses a threshold of 0.5 to Convert to pseudo labels for the target domain

[0050] Source domain loss

[0051] When calculating the source domain loss, X s With Y s Input into the student network, downsample the image and label to obtain images of different resolutions and corresponding labels The label of the first resolution is the given label, and each subsequent downsampled label comes from the previous resolution, defined as follows:

[0052] G 1 =YG l =f(Gl-1 );

[0053] Where Y is the given label, G l represents the label of the lth layer, and the definition of f is:

[0054]

[0055] Where x represents the threshold manually set by the user, usually defaulting to 0.5. For each resolution, the probability maps of multiple semantic layers are combined into a prediction result, which is defined as:

[0056]

[0057] Among them, P l represents the combined prediction probability of the lth resolution, K l Represents the total number of probability maps in the lth resolution, which may be different at different resolutions. represents the weight assigned to the k-th probability map at the l-th resolution, and represents the kth probability map in the lth resolution.

[0058] The source domain loss formula is defined as follows:

[0059]

[0060]

[0061] Where L is the number of different resolutions and N is the size of the number of samples per iteration.

[0062] Domain Adaptation Loss

[0063] When computing the domain adaptation loss, X t The images are input into the student network and the teacher network at the same time, and the images are downsampled to obtain images of different resolutions. For each resolution, the probability maps of multiple semantic layers are combined into the prediction results, which are defined as follows:

[0064]

[0065] Among them, P l represents the combined prediction probability of the lth resolution, K l Represents the total number of probability maps in the lth resolution, which may be different at different resolutions. represents the weight assigned to the k-th probability map at the l-th resolution, and represents the kth probability map in the lth resolution.

[0066] The prediction output of the student network is binarized into pseudo labels using a threshold of 0.5 The domain adaptation loss formula is defined as follows:

[0067]

[0068]

[0069] Where L is the number of different resolutions and N is the number of samples per iteration.

[0070] General losses

[0071] The source domain loss and domain adaptation loss are combined into a general loss according to the following formula:

[0072] L=L SD +L DA .

[0073] Figure 2 The main process of the model is shown, and the English annotations in the figure are translated as follows: Student Net: student network; Teacher Net: teacher network; Up-sampling: upsampling; Modified Residual Block: improved residual module; Labels and Down-sampling Labels: labels and down-sampling labels; EMA Update: EMA algorithm update; Down-sampling: downsampling; Feature Aggregation Module: feature aggregation module; Prediction of Target Domain on TN: the prediction result of the target domain on the teacher network; Label Down Sample: label down sampling; 1×1 conv: 1×1 convolution; Target Domain Input Image: target domain input image; Source Domain Input Image: source domain input image; Source Domain (SD) Loss: source domain loss; Concatenate: concatenation; Prediction of Source Domain on SN: the prediction result of the source domain on the student network; Prediction: prediction output; Domain Adaption (DA) Loss: domain adaptation loss; Combine Multiple Prediction: combine multiple prediction results; Prediction of Target Domain on SN: Prediction results of the target domain on the student network.

[0074] Comparative experiment

[0075] The method in this application is verified by two public fundus image datasets DRIVE and STARE, and the method is compared with the most advanced methods used by scholars in this field. The results are shown in Tables 1 and 2, which include four evaluation indicators: Acc represents the overall segmentation accuracy, Sn represents the segmentation accuracy of blood vessel pixels, Sp represents the segmentation accuracy of background pixels, and AUC represents the area under the ROC curve.

[0076] Table 1 Segmentation results on the DRIVE dataset

[0077]

[0078] Table 2 Segmentation results on the STARE dataset

[0079]

[0080] It can be seen that the method of the present invention is at an advanced level in all indicators. This method reconstructs the encoder and decoder of the network into a refined segmentation model with full resolution by calculating losses at multiple semantic levels and multiple resolutions, and uses the teacher-student network as a framework, combining source domain loss and domain adaptation loss to train the segmentation model, so that the segmentation model performs well in target domains with different data distributions. This method uses a computer to automatically segment retinal vessels in fundus images, without the need for professionals to manually annotate a large number of images to train the retinal vessel segmentation model. It can maintain high-performance segmentation in target domains with different data distributions, saving a lot of manpower and time while improving diagnostic efficiency.

Claims

1. An unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images, characterized in that: The steps include: Step S1: The source domain image set X s and the source domain label Y s Input into the student network, and refine the model at full resolution to obtain prediction probabilities at different resolutions With image tags pass and Calculate the source domain loss L SD ; Step S2: Set the target domain image set X t Input to the student network, and refine the model at full resolution to get the predicted probability pass Generate pseudo labels for target domain images Step S3: Set the target domain image set X t Input to the teacher network, and refine the model at full resolution to get the predicted probability use and Compute domain adaptation loss L DA ; Step S4: L SD and L DA Merged into a general loss to supervise the training of the student network; Step S5: According to the EMA algorithm, the weight of the teacher network is updated by the weight of the student network, and it is iterated for segmentation of the target domain; Among them, the student network and the teacher network have the same encoder and decoder structure, both of which contain aggregation modules, residual modules, upsampling and downsampling.

2. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S1, the full-resolution refined model is obtained by s With Y s Downsample to obtain images of different resolutions and corresponding labels Its definition is: G 1 =Y; G l =f(G l-1 ); Among them, the label of the first resolution is the given label Y, G l Represents the label of the lth layer. Each subsequent downsampled label comes from the previous resolution. x is the threshold manually set by the user, which defaults to 0.

5.

3. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S1 to step S3, the full-resolution refinement model combines the prediction probability maps of multiple semantic layers into a prediction result for each resolution, which is defined as: Among them, P l represents the combined prediction probability of the lth resolution, K l represents the total number of predicted probability maps in the lth resolution, represents the weight assigned to the k-th prediction probability map at the l-th resolution, represents the k-th predicted probability map in the l-th resolution.

4. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S1, the formula for calculating the source domain loss by the full-resolution refinement model is: Where L is the number of different resolutions and N is the number of samples per iteration.

5. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S3, the formula for calculating the domain adaptation loss of the full-resolution refinement model is: Where L is the number of different resolutions and N is the number of samples per iteration.

6. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S2, a threshold of 0.5 is used to Convert to pseudo labels for the target domain 7. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: In step S5, the EMA algorithm is defined as: in, and denote the weights of the teacher network in the i-th iteration and the (i-1)-th iteration, respectively. represents the weight of the corresponding student network in the i-th iteration.

8. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: The aggregation module uses cascaded and parallel multiple convolutions to integrate the upsampling and downsampling results of adjacent stages; the residual module reduces overfitting by adding a Dropout layer after each BatchNorm layer.

9. The unsupervised domain adaptation method for retinal blood vessel segmentation in fundus images according to claim 1, characterized in that: The general loss calculation formula is: L = L SD +L DA .