Meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance

Abstract

The present invention provides a method for segmenting meibomian gland images based on pseudo-label and deep feature cross-guidance, comprising the following steps: step S1: data set processing; that is, dividing and preprocessing the meibomian gland image data set, the data set including labeled data and unlabeled data; step S2: pseudo-label cross-guidance; step S3: deep feature cross-guidance; an auxiliary branch module is set in the deep layer of the encoder of the segmentation network to reduce the dimension of the output of the deep layer of the model encoder; the shallow convolution of the encoder is supervised and constrained so that the encoder parts of the two models are cross-guided by deep features; the application of this technical solution can train the network to pay attention to the edge information and semantic information of the image, obtain a more comprehensive feature description, reduce the model's dependence on the amount of labeled data, and ultimately obtain better segmentation results.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a meibomian gland image segmentation method based on pseudo-label and depth feature cross-guidance. Background Art

[0002] The meibomian gland image segmentation method refers to automatically segmenting the data according to the given infrared eyelid image to obtain the meibomian glands, and displaying the condition before and after treatment in the form of data. Different from ordinary image segmentation tasks, the difference between the glands and the background in the meibomian gland image is small, the diseased glands show irregular changes, and the glands are densely arranged in one image. Therefore, the existing segmentation method has weak ability to distinguish between the glands and the background area, the gland edges are not clearly segmented, and the segmentation results are not fine enough. More importantly, the labels of the meibomian gland images require a lot of time to be annotated, and it is impossible to obtain a large amount of annotated data, resulting in poor segmentation network effects.

[0003] In order to improve the accuracy of meibomian gland image segmentation, it is necessary to break away from the idea of ​​using only conventional encoders and decoders, and extract appropriate features to guide the model based on the characteristics of the meibomian gland images themselves, so that the obtained features can take into account both the edge information and semantic information of the image. In addition, considering the high labeling cost of meibomian gland images, it is necessary to use semi-supervised learning strategies from a practical application perspective to improve the generalization ability of the model and reduce the model's dependence on labeled data. Therefore, how to extract appropriate features to guide the model, improve the model's generalization ability, and minimize the model's dependence on labeled data has become the key to improving the effect of meibomian gland image segmentation. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a meibomian gland image segmentation method based on cross-guidance of pseudo-labels and deep features, which can train the network to focus on the edge information and semantic information of the image, obtain a more comprehensive feature description, reduce the model's dependence on the amount of labeled data, and ultimately obtain better segmentation results.

[0005] To achieve the above object, the present invention adopts the following technical solution: a meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance, comprising the following steps:

[0006] Step S1: data set processing; that is, dividing and preprocessing the meibomian gland image data set, which includes labeled data and unlabeled data;

[0007] Step S2: Pseudo-label cross-guidance; a segmentation network based on a U-shaped network with an encoding and decoding structure; first initialize two segmentation networks F1 and F2 with exactly the same structure but different parameters; send image data to F1 and F2 at the same time to obtain corresponding segmentation prediction images Y1 and Y2 respectively; for a small amount of labeled data, use the real labels to supervise the two models; for a large amount of unlabeled data, use the segmentation prediction images Y1 and Y2 to obtain the corresponding prediction labels, namely pseudo labels Y1` and Y2` for cross-learning; that is, use Y1` to supervise the segmentation network F2, and use Y2` to supervise the segmentation network F1; the cross-application of real information and prediction information reduces the model's dependence on the amount of labeled sample data;

[0008] Step S3: Deep feature cross-guidance; an auxiliary branch module is set in the deep layer of the encoder of the segmentation network F1 and F2 to reduce the dimension of the output of the deep layer of the model encoder to obtain Y aux1 , Y aux2 ; Combined use of Y aux1 , Y aux2 Perform deep supervised learning with segmentation prediction images Y1, Y2 or pseudo labels Y1`, Y2`; Y aux1 , Y aux2 Get pseudo labels Supervise the shallow convolution of the F2 encoder. The shallow convolution of the F1 encoder is supervised and constrained so that the encoder parts of the two models can cross-guide the deep features.

[0009] In a preferred embodiment, the step S1 of data set processing specifically includes the following steps:

[0010] Step S11: The images are divided according to the ratio of training set: test set: validation set = 7:2:1, where the ratio of labeled data in the training set: unlabeled data = 1:1;

[0011] Step S12: All data are preprocessed using a contrast-limited adaptive histogram equalization image enhancement method.

[0012] In a preferred embodiment, the segmentation network described in the pseudo-label cross-guidance in step S2 is composed of a 5-layer deep U-Net network; Y1` is used to perform supervised learning on the segmentation network F2, and Y2` is used to perform supervised learning on the segmentation network F1.

[0013] In a preferred embodiment, the auxiliary branch module described in step S3 deep feature cross-guidance is composed of 3 layers of convolution, and the number of channels in the module increases from 512 to 256 to 128 to 2; the output of the auxiliary branch module is supervised and constrained by using labels or pseudo-labels to realize cross-guidance of the model by deep features.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) The method proposed in the present invention can train the network to focus on the edge information and global semantic information of the image, obtain a more comprehensive feature description, and ultimately obtain better segmentation results.

[0016] (2) The present invention can guide the network to pay more attention to the learning of deep features of the model and use the deep features to perturb the model, thereby increasing the robustness of the model.

[0017] (3) Compared with existing methods, the present invention can effectively reduce the model's dependence on labeled data, improve segmentation accuracy, and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of a preferred embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of the principle of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0022] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0023] like Figures 1 to 2 As shown, this embodiment provides a few-sample meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance, comprising the following steps:

[0024] Step S1: Dataset processing: dividing and preprocessing the meibomian gland image dataset, which includes labeled data and unlabeled data.

[0025] Step S2 pseudo-label cross-guidance: The present invention uses a U-type network with a codec structure as the basis of a segmentation network. First, initialize two segmentation networks F1 and F2 with exactly the same structure but different parameters. Send the image data to F1 and F2 at the same time to obtain corresponding segmentation prediction images Y1 and Y2, respectively. Among them, for a small amount of labeled data, the real labels are used to perform supervised learning on the two models; for a large amount of unlabeled data, the segmentation prediction images Y1 and Y2 are used to obtain the corresponding predicted labels (hereinafter referred to as pseudo labels) Y1` and Y2` for cross-learning. That is, use Y1` to supervise the learning of the segmentation network F2, and use Y2` to supervise the learning of the segmentation network F1. The cross-application of real information and predicted information can reduce the model's dependence on the amount of labeled sample data.

[0026] Step S3: Deep feature cross-guidance: An auxiliary branch module is set in the deep layer of the encoder of the segmentation network F1 and F2 to reduce the dimension of the output of the deep layer of the model encoder to obtain Y aux1 , Y aux2 . Combined use of Y aux1 , Y aux2 Perform deep supervised learning with segmentation prediction images Y1, Y2 or pseudo labels Y1`, Y2`. aux1 , Y aux2 Get pseudo labels Supervise the shallow convolution of the F2 encoder. The shallow convolution of the F1 encoder is supervised and constrained so that the encoder parts of the two models can cross-guide the deep features, thereby enhancing the segmentation accuracy and ability of the two models to judge data semantics.

[0027] In this embodiment, the use of Y1' to supervise the learning of the segmentation network F2 and the use of Y2' to supervise the learning of the segmentation network F1 in step S2 pseudo-label cross-guidance are for the purpose of training the segmentation consistency of the unlabeled data. By giving pseudo-labels to the unlabeled data and cross-learning the model, the generalization ability of the model is improved and the dependence of the model on the labeled data is reduced.

[0028] In this embodiment, the auxiliary branch module described in step S3 deep feature cross-guidance is composed of 3 layers of convolution, and the number of channels in the module is from 512 to 256 to 128 to 2. Labels or pseudo-labels supervise and constrain the output of the auxiliary branch module so that the deep features of the model guide the model. Through the deep feature cross-guidance method, the network is promoted to pay attention to the edge information and semantic information in the image, and the robustness of the model in meibomian gland image segmentation is improved.

[0029] Preferably, the present embodiment includes an auxiliary branch module, and an auxiliary branch module is set in the fourth layer of the encoder of the segmentation network, so that the number of channels output by the fourth layer of the model encoder is reduced from 512 to 2, and the output of the auxiliary branch module is supervised and constrained with labels or pseudo-labels, so that the network pays attention to the edge information and global semantic information of the image, thereby improving the segmentation accuracy.

[0030] A cross-pseudo-label training strategy, which is divided into three steps. The first step is to declare two segmentation models with the same structure but different initializations; the second step is to send labeled data to the two models to obtain predicted values, and use labels to supervise and constrain the two models; the third step is to send unlabeled data to the two models to obtain predicted values, and obtain corresponding pseudo-labels based on the predicted values. The pseudo-labels predicted by the first model supervise the predicted values ​​of the second model, and the pseudo-labels predicted by the second model supervise the predicted values ​​of the first model.

[0031] Preferably, in this embodiment, a few-sample meibomian gland image segmentation method based on cross-guidance of pseudo-labels and deep features is adopted to improve the accuracy of meibomian gland image segmentation.

[0032] The specific steps include:

[0033] Step S1: Dataset processing: dividing and preprocessing the meibomian gland image dataset, which includes labeled data and unlabeled data.

[0034] Step S2 pseudo-label cross-guidance: The present invention uses a U-type network with a codec structure as the basis of a segmentation network. First, initialize two segmentation networks F1 and F2 with exactly the same structure but different parameters. Send the image data to F1 and F2 at the same time to obtain corresponding segmentation prediction images Y1 and Y2, respectively. Among them, for a small amount of labeled data, the real labels are used to perform supervised learning on the two models; for a large amount of unlabeled data, the segmentation prediction images Y1 and Y2 are used to obtain the corresponding predicted labels (hereinafter referred to as pseudo labels) Y1` and Y2` for cross-learning. That is, use Y1` to supervise the learning of the segmentation network F2, and use Y2` to supervise the learning of the segmentation network F1. The cross-application of real information and predicted information can reduce the model's dependence on the amount of labeled sample data.

[0035] Step S3: Deep feature cross-guidance: An auxiliary branch module is set in the deep layer of the encoder of the segmentation network F1 and F2 to reduce the dimension of the output of the deep layer of the model encoder to obtain Y aux1 , Y aux2 . Combined use of Y aux1 , Y aux2 Perform deep supervised learning with segmentation prediction images Y1, Y2 or pseudo labels Y1`, Y2`. aux1 , Y aux2 Get pseudo labels Supervise the shallow convolution of the F2 encoder. The shallow convolution of the F1 encoder is supervised and constrained so that the encoder parts of the two models can cross-guide the deep features, thereby enhancing the segmentation accuracy and ability of the two models to judge data semantics.

[0036] Figure 2 In step 1, convolution operation.

[0037] Step 2: Move the feature map and splice it with other feature maps.

[0038] step3: upsampling.

[0039] Step 4: Use the auxiliary branch module to reduce the dimension of the fourth layer of the encoder.

[0040] step5: Use labels or pseudo-labels for supervised learning.

[0041] The present invention has the following beneficial effects: the method proposed in the present invention can train the network to focus on the local information and global semantic information of the image, reduce the model's dependence on labeled data, improve the model's generalization ability, and ultimately obtain better segmentation results.

[0042] In summary, in view of the problem that the features directly obtained by using the existing encoder-decoder model structure in the segmentation of meibomian gland images cannot effectively express the edge information and semantic information of the meibomian gland images, resulting in low segmentation accuracy, weak generalization ability, and the need to rely on a large amount of labeled data, the few-sample meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance proposed in this embodiment is used in meibomian gland image segmentation. First, declare two segmentation models with the same structure but different initializations; then send the labeled data into the two models to obtain the predicted value, use the label to supervise the two models, send the unlabeled data into the two models to obtain the predicted value, and obtain the corresponding pseudo-label according to the predicted value. The pseudo-label predicted by the first model supervises the predicted value of the second model, and the pseudo-label predicted by the second model supervises the predicted value of the first model; then set an auxiliary branch module in the fourth layer of the encoder of the two segmentation networks, so that the number of channels output by the fourth layer of the model encoder is reduced from 512 to 2 and is subjected to deep supervised learning with the corresponding label or, finally, obtain the auxiliary branch pseudo-labels according to the outputs of the auxiliary branch modules of the two models, and cross-supervise the loss of the output of the auxiliary branch. The method of this embodiment can guide the network to pay more attention to the learning of model edge information and semantic information, increase the distinguishability of the target, and compared with existing methods, this embodiment can effectively improve the segmentation accuracy, reduce the model's dependence on labeled data, and improve the generalization ability of the model.

[0043] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance, characterized in that: The following steps are involved: Step S1: data set processing; that is, dividing and preprocessing the meibomian gland image data set, which includes labeled data and unlabeled data; Step S2: Pseudo-label cross-guidance; a segmentation network based on a U-shaped network with an encoding and decoding structure; first initialize two segmentation networks F1 and F2 with exactly the same structure but different parameters; send image data to F1 and F2 at the same time to obtain corresponding segmentation prediction images Y1 and Y2 respectively; for a small amount of labeled data, use the real labels to supervise the two models; for a large amount of unlabeled data, use the segmentation prediction images Y1 and Y2 to obtain the corresponding prediction labels, namely pseudo labels Y1` and Y2` for cross-learning; that is, use Y1` to supervise the segmentation network F2, and use Y2` to supervise the segmentation network F1; the cross-application of real information and prediction information reduces the model's dependence on the amount of labeled sample data; Step S3: Deep feature cross-guidance; an auxiliary branch module is set in the deep layer of the encoder of the segmentation network F1 and F2 to reduce the dimension of the output of the deep layer of the model encoder to obtain Y aux1 , Y aux2 ; Combined use of Y aux1 , Y aux2 Perform deep supervised learning with segmentation prediction images Y1, Y2 or pseudo labels Y1`, Y2`; Y aux1 , Y aux2 Get pseudo labels Supervise the shallow convolution of the F2 encoder. The shallow convolution of the F1 encoder is supervised and constrained so that the encoder parts of the two models can cross-guide the deep features.

2. The method for segmenting meibomian gland images based on pseudo-label and deep feature cross-guidance according to claim 1, characterized in that: The step S1 of data set processing specifically includes the following steps: Step S11: The images are divided according to the ratio of training set: test set: validation set = 7:2:1, where the ratio of labeled data in the training set: unlabeled data = 1:1; Step S12: All data are preprocessed using a contrast-limited adaptive histogram equalization image enhancement method.

3. The meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance according to claim 1, characterized in that: The segmentation network described in step S2 pseudo-label cross-guidance is composed of a 5-layer deep U-Net network; Y1` is used to perform supervised learning on the segmentation network F2, and Y2` is used to perform supervised learning on the segmentation network F1.

4. The meibomian gland image segmentation method based on pseudo-label and deep feature cross-guidance according to claim 1, characterized in that: The auxiliary branch module described in step S3 deep feature cross-guidance is composed of 3 layers of convolution, and the number of channels in the module increases from 512 to 256 to 128 to 2. The output of the auxiliary branch module is supervised and constrained by using labels or pseudo-labels to realize the cross-guidance of the model by deep features.

Images