A semi-supervised human eye multi-element segmentation method based on iterative learning

Abstract

The present invention relates to a semi-supervised human eye multi-element segmentation method based on iterative learning, comprising the following steps: dividing an eye dataset under visible light into a labeled eye dataset and an unlabeled eye dataset; training the labeled eye dataset using a supervised deep learning network to obtain a pre-trained model; obtaining pseudo-labels for the unlabeled eye dataset through forward reasoning based on the pre-trained model; filtering the unlabeled eye dataset and the pseudo-labels based on the labeled eye dataset to obtain a trust dataset; inputting the labeled eye dataset and the trust dataset into the supervised deep learning network for retraining; if the training completion requirement is not met, returning to the pseudo-label acquisition step described above; otherwise, terminating the training to obtain a human eye segmentation model; and performing multi-element segmentation of the human eye using the human eye segmentation model. The present invention can improve the accuracy and robustness of the model while using a small amount of labeled data.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a semi-supervised human eye multi-element segmentation method based on iterative learning. Background Art

[0002] Eye feature segmentation techniques primarily focus on extracting specific parts of the eye. Under varying lighting conditions, the distinct characteristics of various eye features are leveraged to segment and extract the target region. Early approaches to eye feature segmentation primarily relied on methods such as the iterative integrand-differential operator or circular Hough transform to determine the target boundary, and on methods using underlying information such as grayscale to detect the target edge. These methods rely heavily on various image features, place high demands on image quality, and are susceptible to noise. Furthermore, while infrared illumination can yield high-quality eye images, significantly aiding in the segmentation of eye features, the image acquisition environment is demanding, making it less straightforward than under visible light. Furthermore, due to significant differences from the visible light image domain, scalability to meet the needs of eye feature segmentation under natural light is difficult. With the development of deep learning, methods for eye feature segmentation based on visible light images have made significant progress. However, due to the difficulty of ground-truth labeling of visible light eye data, existing methods have mostly focused on single eye features, with limited research on methods for segmenting multiple features.

[0003] Existing patent document CN113343943A discloses a multi-element segmentation method for the human eye based on prior information. This method falls within the scope of supervised multi-element segmentation of the human eye based on infrared images and aims to improve the speed and interpretability of obtaining prior information about the sclera, enhance the speed and accuracy of eye image segmentation, and improve segmentation efficiency. The main steps of this scheme are: extracting high-dimensional features of the sclera region through a residual network; using these high-dimensional features to perform attention adjustment on the high-dimensional features of the original eye image; encoding the adjusted high-dimensional features of the original eye image to obtain encoded semantic features; improving the encoded semantic features through cross-connection excitation and inputting them into a decoder for decoding to obtain decoded semantic features; performing channel adjustment on the decoded semantic features and outputting a preliminary segmentation result; calculating the total loss between the preliminary segmentation result and the segmentation label, and by comparing it with a set threshold, determining whether all filters, encoders, and decoders need to be optimized, and outputting the final segmentation results for the pupil, iris, and sclera. Although this solution can segment multiple elements of the human eye, it requires a large amount of labeled eye data. In addition, it is based on the OpenEDS invention research of a large dataset collected under infrared illumination with complete pixel-level annotations (sclera, iris, pupil). It is difficult to apply to visible light human eye element segmentation data, which is difficult to obtain due to the difficulty of labeling. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a semi-supervised human eye multi-element segmentation method based on iterative learning, which can improve the accuracy and robustness of the model while using a small amount of labeled data.

[0005] The technical solution adopted by the present invention to solve the technical problem is to provide a semi-supervised human eye multi-element segmentation method based on iterative learning, comprising the following steps:

[0006] (1) Divide the visible light eye dataset into labeled eye dataset and unlabeled eye dataset;

[0007] (2) training the labeled eye dataset using a supervised deep learning network to obtain a pre-trained model;

[0008] (3) obtaining pseudo labels of the unlabeled eye dataset through forward reasoning based on the pre-trained model;

[0009] (4) screening the unlabeled eye dataset and the pseudo-labels based on the labeled eye dataset to obtain a trust dataset;

[0010] (5) inputting the labeled eye dataset and the trust dataset into the supervised deep learning network for retraining. If the training completion requirement is not met, returning to step (3); otherwise, terminating the training to obtain the human eye segmentation model;

[0011] (6) The human eye segmentation model is used to perform multi-element segmentation on the human eye.

[0012] The step (4) is specifically as follows:

[0013] (41) obtaining standard parameters based on the labels in the labeled eye dataset;

[0014] (42) obtaining a pseudo label parameter for each image data in the unlabeled eye dataset based on the pseudo label;

[0015] (43) Comparing the standard parameters with the pseudo-label parameters, if the total error rate of the pseudo-label parameters is less than a threshold, adding the image data and pseudo-label corresponding to the pseudo-label parameters to the trust data set.

[0016] The step (41) is specifically as follows: extracting sclera labels, iris labels and pupil labels according to the labels, and synthesizing the whole eye labels from the sclera labels, iris labels and pupil labels; calculating the convexity and solidity of the whole eye labels, iris labels and pupil labels respectively; calculating the average convexity and average solidity of the whole eye labels, iris labels and pupil labels respectively, and using the average convexity and average solidity as standard parameters.

[0017] The step (42) is specifically as follows: extracting a pseudo sclera label, a pseudo iris label and a pseudo pupil label according to the pseudo label, and synthesizing a pseudo whole eye label from the pseudo sclera label, the pseudo iris label and the pseudo pupil label; and respectively calculating the convexity and solidity of the pseudo whole eye label, the pseudo iris label and the pseudo pupil label corresponding to each image data.

[0018] The step (43) is specifically as follows: respectively calculating the difference between the convexity and solidity of the pseudo whole eye label, pseudo iris label and pseudo pupil label corresponding to each image data and the average convexity and average solidity; calculating the total error rate by weighted summation based on the difference; if the total error rate of the pseudo label parameters is less than a threshold, adding the image data and pseudo labels corresponding to the pseudo label parameters to the trust data set.

[0019] The step (41) is specifically as follows: extracting sclera labels, iris labels and pupil labels according to the labels; calculating the Hu moments of the sclera labels, iris labels and pupil labels respectively; and calculating the Hu moment statistics of the sclera labels, iris labels and pupil labels as standard parameters.

[0020] The step (42) is specifically as follows: extracting a pseudo sclera label, a pseudo iris label and a pseudo pupil label according to the pseudo label; and respectively calculating the Hu moment of the pseudo sclera label, the pseudo iris label and the pseudo pupil label corresponding to each image data.

[0021] The step (43) is specifically as follows: respectively calculating the difference between the Hu moment of the pseudo sclera label, pseudo iris label and pseudo pupil label corresponding to each image data and the Hu moment statistics of the sclera label, iris label and pupil label; calculating the total error rate by weighted summation based on the difference; if the total error rate of the pseudo label parameter is less than a threshold, adding the image data and pseudo label corresponding to the pseudo label parameter to the trust data set.

[0022] Beneficial effects

[0023] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared to existing technologies: By rationally utilizing a small amount of annotated visible light ocular data, the present invention gradually improves the labels of unlabeled data, allowing the training data to be continuously expanded during the iterative process, thereby improving the algorithm accuracy and increasing the robustness of the model. The pseudo-label screening strategy in the present invention fully counts the shape characteristics of the visible light ocular data labels, identifies the common characteristics of the eye data in this shape, and uses this commonality of eye shape to effectively evaluate and screen pseudo-labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is an overall flow chart of an embodiment of the present invention;

[0025] Figure 2Schematic diagram of pseudo-label screening strategy in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of a filter parameter formulation scheme in an embodiment of the present invention;

[0027] Figure 4 Schematic diagram of another filter parameter formulation scheme in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0029] The embodiment of the present invention relates to a semi-supervised human eye multi-element segmentation method based on iterative learning, such as Figure 1 As shown, the following steps are included: dividing the eye segmentation dataset under visible light into a labeled eye dataset and an unlabeled eye dataset; training the labeled eye dataset using a supervised deep learning network to obtain a pre-trained model; obtaining pseudo-labels of the unlabeled eye dataset through forward reasoning based on the pre-trained model; filtering the unlabeled eye dataset and pseudo-labels based on the labeled eye dataset to obtain a trust dataset; inputting the labeled eye dataset and the trust dataset into the supervised deep learning network for retraining. If the training completion requirements are not met, returning to the above-mentioned pseudo-label acquisition step, otherwise terminating the training to obtain the human eye segmentation model. Specifically:

[0030] Step 1: Process the visible light eye dataset and divide it into a labeled eye dataset (data D1, label L1) and an unlabeled eye dataset D2;

[0031] Step 2: Convert the label L1 in the labeled eye dataset into npy format, obtain the label L1 and data D1, and mark the type of label L1, including background, sclera, iris, and pupil. Some data label types may be missing.

[0032] Step 3: Based on the pre-processed labeled data, a supervised deep learning network is used for training to obtain a pre-trained model; at the same time, a trust data set is initialized, which is used to place the pseudo-label L3 and the corresponding data D3 that meet the conditions.

[0033] In step 4, based on the pre-trained model, the pseudo-label L2 of the unlabeled eye data D2 is obtained through forward reasoning, and the data D1 and label L1 are sent to the pseudo-label screening module together to select the label trustworthy data set (recorded as data D3, pseudo-label L3).

[0034] In step 5, the mixed labeled dataset (data D1, label L1) and the trust dataset (data D3, pseudo label L3) are input into the supervised deep learning network for further training. If the set number of epochs is reached or the loss function Loss value is lower than the threshold, the training ends. Otherwise, the training proceeds to step 4 for iterative learning. The loss function in this embodiment can use a combination of CrossEntropy Loss and Generalized Dice Loss. After the training is completed, the human eye segmentation model can be obtained, and the human eye sclera, iris and pupil can be segmented using the human eye segmentation model.

[0035] Step 4 specifically includes: obtaining standard parameters based on the labels in the labeled eye dataset; obtaining pseudo-label parameters for each image in the unlabeled eye dataset based on the pseudo-labels; comparing the standard parameters with the pseudo-label parameters, and if the total error rate of the pseudo-label parameters is less than a threshold, adding the image data and pseudo-labels corresponding to the pseudo-label parameters to the trust dataset. Figure 2 As shown, in step 4 of this embodiment, when filtering pseudo labels, a filter parameter formulation module is required. There are two schemes for the filter parameter formulation module in this embodiment.

[0036] The filter parameter setting module of the first solution (see Figure 3 ) aims to obtain standard parameters for label L1 and pseudo-label L2, respectively, to effectively evaluate and screen pseudo-labels. Two approaches are possible: one is to develop parameters based on the integration of shape feature parameters. Using label L1 as an example, the specific technical solution is shown below. The pseudo-label L2 approach is similar.

[0037] Extract the sclera label L from the label L1 sclera , Iris Tag L iris , pupil label L pupil ;

[0038] Synthetic Scleral Label L sclera , Iris Tag L iris , pupil label L pupil For full eye label L eye (If some labels are missing, skip this step), the formula is: L eye =L sclera ∪L iris ∪L pupil .

[0039] Calculate the iris label L separately iris , pupil label L pupil , full eye label L eye The shape feature of the eye is selected based on the convexity (C) and solidity (S). The formula is:

[0040]

[0041] The average convexity and average solidity of the whole eye, iris, and pupil are solved and output as standard parameters. The formula is:

[0042]

[0043]

[0044] Among them, C class represents the average convexity, S class represents the average firmness, C(l class ), class∈{eye,iris,pupil} represents the iris label L iris , pupil label L pupil , full eye label L eye Convexity, S(l class ), class∈{eye,iris,pupil} represents the iris label L iris , pupil label L pupil , full eye label L eye The robustness, N class Indicates the total number of data D1.

[0045] Another approach is based on the calculation of Hu moment statistics (see Figure 4 ), also taking label L1 as an example, the specific technical solution is shown below, and the pseudo label L2 is similar.

[0046] Extract the sclera label L from the label L1 sclera , Iris Tag L iris , pupil label L pupil ;

[0047] Calculate the sclera label L separately sclera , Iris Tag L iris , pupil label L pupil Hu moment;

[0048] Calculate the Hu moment statistics of the sclera, iris, and pupil. In this implementation, the median Hu moment is selected as the standard parameter output. The formula is: Hu class =median(Hu(L class)), class∈{sclera,iris,pupil}.

[0049] The pseudo-label screening module proposed in this embodiment has different screening schemes according to different ways of formulating the filter parameters.

[0050] Corresponding to the first type, the specific technical solutions are as follows:

[0051] For the label L1 and pseudo label L2, the standard parameters and pseudo label parameters, namely shape feature quantity, selection convexity and solidity, are obtained through the filter parameter formulation module respectively;

[0052] For the comparison module between standard parameters and pseudo-label parameters, the error rates of the whole eye, iris, and pupil are calculated as comparison criteria. The formula is:

[0053] e class =|C class -C(p class )|,class∈{eye,iris,pupil}

[0054] Among them, C(p class ), class∈{eye,iris,pupil} represents the convexity and robustness of the pseudo-full-eye label, pseudo-iris label and pseudo-pupil label corresponding to a certain image data in data D2

[0055] Calculate the total error rate e = λ1e eye +λ2e iris +λ3e pupil , set the selection threshold. When e is less than the threshold, extract the corresponding image data D2 and pseudo label L2 and add them to the trust dataset to complete the pseudo label screening.

[0056] Corresponding to the second type, the specific technical solutions are as follows:

[0057] For the label L1 and pseudo label L2, the standard parameters and pseudo label parameters, i.e. the corresponding Hu moments, are obtained through the filter parameter formulation module respectively;

[0058] For the comparison module between standard parameters and pseudo-label parameters, the Hu moment error rates of sclera, iris, and pupil are calculated as comparison criteria. The formula is:

[0059] e class =|Hu class -Hu(p class )|,class∈{sclera,iris,pupil}

[0060] Among them, Hu(p class), class∈{sclera,iris,pupil} represents the Hu moment of the pseudo sclera label, pseudo iris label and pseudo pupil label corresponding to a certain image data in data D2

[0061] Calculate the total error rate e = λ1e sclera +λ2e iris +λ3e pupil , set the selection threshold. When e is less than the threshold, the corresponding image data D2 and pseudo label L2 are extracted and added to the trust dataset to complete the pseudo label screening.

[0062] The present invention is further illustrated below using the visible light human eye datasets SBVPI and ESPNet as examples.

[0063] Step 1: Divide the visible light human eye element data into a labeled dataset (data D1, label L1) and an unlabeled dataset (data D2), and prepare a deep learning network (such as ESPNet)

[0064] 1.1) Download the SBVPI dataset from the internet. The dataset contains 1856 eye images, of which 122 have pupil labels, 129 have iris labels, and 1840 have sclera labels.

[0065] 1.2) For experimental purposes, we will focus on 129 images with iris labels. We will place these images and all their corresponding labels into the labeled dataset, while the remaining data will be placed into the unlabeled dataset. The corresponding sclera labels can be used as one of the criteria for pseudo-label screening. We will set a trust dataset and leave the file empty.

[0066] 1.3) Download the ESPNet network model.

[0067] Step 2: Input the labeled data set into the network model for training to obtain a pre-trained model, and use the pre-trained model to obtain the pseudo label L2 of the unlabeled data D2.

[0068] Step 3: Input the labeled dataset data (data D1, label L1) and the pseudo-label data (pseudo-label L2 for data D2) into the filter parameter formulation module proposed in this embodiment to obtain standard parameters and pseudo-label parameters. Taking standard parameter acquisition as an example, you can choose either Option 1 or Option 2. Option 1 includes 2.1)-2.4), and Option 2 includes 2.1), 2.5), and 2.6). The same applies to pseudo-label parameter acquisition.

[0069] 2.1) Extract the sclera label L from the label L1 sclera , Iris Tag L iris , pupil label L pupil ;

[0070] 2.2) By sclera label Lsclera , Iris Tag L iris , pupil label L pupil Synthetic full eye tag L eye ;

[0071] 2.3) Calculate the full eye label L separately eye , Iris Tag L iris , pupil label L pupil The convexity C(l eye )、C(l iris )、C(l pupil ) and robustness S(l eye )、S(l iris )、S(l pupil );

[0072] 2.4) Calculate the full eye label L eye , Iris Tag L iris , pupil label L pupil The average convexity C eye 、C iris 、C pupil and average firmness S eye 、S iris 、S pupil ;

[0073] 2.5) Calculate the sclera label L separately sclera , Iris Tag L iris , pupil label L pupil Hu moment Hu(L sclera )、Hu(L iris )、Hu(L pupil );

[0074] 2.6) Calculate the Hu moment statistics of sclera, iris and pupil, Hu sclera 、Hu iris 、Hu pupil .

[0075] Step 4: Based on the obtained standard parameters and pseudo-label parameters, the pseudo-label screening module proposed in this embodiment is used to screen the unlabeled data D2 and the corresponding pseudo-label L2 to obtain the trust data set (data D3, pseudo-label L3). Specifically, you can select 4.1), 4.2)a), 4.3), 4.4), or 4.1), 4.2)b), 4.3), 4.4):

[0076] 4.1) In the process of obtaining pseudo-label parameters, the sclera pseudo-label P is extracted from the pseudo-label L2 sclera , iris pseudo label P iris , pupil pseudo label P pupil , and by the scleral pseudo-label P sclera, iris pseudo label P iris , pupil pseudo label P pupil Synthesize the whole eye pseudo label P eye ;

[0077] 4.2) a) (Scheme 1) Calculate the error rate e of the whole eye pseudo label, iris pseudo label, and pupil pseudo label corresponding to a single image eye 、e iris 、e pupil As a standard of comparison;

[0078] 4.2) b) (Scheme 2) Calculate the error rate e of the sclera pseudo-label, iris pseudo-label, and pupil pseudo-label corresponding to a single image sclera 、e iris 、e pupil As a standard of comparison;

[0079] 4.3) Based on the error rate comparison criterion, a threshold discrimination strategy is designed and selected: the total error rate e is calculated. When e is less than the threshold, the original image data and the corresponding pseudo-label are moved from the unlabeled dataset (pseudo-label L2 for data D2) to the trusted dataset (pseudo-label L3 for data D3);

[0080] 4.4) Repeat 4.2 to 4.4) until all operations on the unlabeled data D2 are completed.

[0081] Step 5: Combine the trust label dataset (data D3, pseudo label L3) and the labeled dataset (data D1, label L1) and input them into the pre-trained model to continue training. If the loss function value or the cycle number threshold set in advance is reached, the human eye segmentation model is obtained, otherwise jump to step 2.

[0082] Step 6: Use the eye segmentation model to perform multi-element segmentation on the sclera, iris, and pupil of the human eye in the image.

[0083] It's easy to see that by rationally utilizing a small amount of annotation for visible light ocular data and gradually improving the labels for unlabeled data, the present invention continuously expands the training data during the iterative process, improving both algorithm accuracy and model robustness. The pseudo-label screening strategy employed in this invention utilizes a comprehensive statistical analysis of the shape characteristics of visible light ocular data labels to identify commonalities in these shape characteristics, effectively evaluating and screening pseudo-labels based on these commonalities.

Claims

1. A semi-supervised human eye multi-element segmentation method based on iterative learning, characterized in that: The following steps are involved: (1) Divide the visible light eye dataset into labeled eye dataset and unlabeled eye dataset; (2) training the labeled eye dataset using a supervised deep learning network to obtain a pre-trained model; (3) obtaining pseudo labels of the unlabeled eye dataset through forward reasoning based on the pre-trained model; (4) Filtering the unlabeled eye dataset and pseudo-labels based on the labeled eye dataset to obtain a trust dataset, specifically: (41) obtaining standard parameters based on the labels in the labeled eye dataset, specifically: extracting sclera labels, iris labels, and pupil labels according to the labels, and synthesizing a full-eye label from the sclera labels, iris labels, and pupil labels; Calculating the convexity and solidity of the whole eye label, the iris label, and the pupil label respectively; respectively calculating the average convexity and average solidity of the whole eye label, the iris label, and the pupil label, and using the average convexity and average solidity as standard parameters; (42) obtaining a pseudo label parameter for each image data in the unlabeled eye dataset based on the pseudo label; (43) comparing the standard parameters with the pseudo-label parameters, and if the total error rate of the pseudo-label parameters is less than a threshold, adding the image data and pseudo-label corresponding to the pseudo-label parameters to the trust data set; (5) inputting the labeled eye dataset and the trust dataset into the supervised deep learning network for retraining. If the training completion requirement is not met, returning to step (3); otherwise, terminating the training to obtain the human eye segmentation model; (6) The human eye segmentation model is used to perform multi-element segmentation on the human eye.

2. The semi-supervised human eye multi-element segmentation method based on iterative learning according to claim 1, characterized in that: The step (42) specifically comprises: extracting a pseudo sclera label, a pseudo iris label and a pseudo pupil label according to the pseudo label, and synthesizing a pseudo whole eye label from the pseudo sclera label, the pseudo iris label and the pseudo pupil label; The convexity and robustness of the pseudo full eye label, pseudo iris label, and pseudo pupil label corresponding to each image data are calculated respectively.

3. The semi-supervised human eye multi-element segmentation method based on iterative learning according to claim 2, characterized in that: The step (43) specifically comprises: calculating the difference between the convexity and the solidity of the pseudo full eye label, the pseudo iris label and the pseudo pupil label corresponding to each image data and the average convexity and the average solidity; The total error rate is calculated by weighted summation according to the difference; if the total error rate of the pseudo label parameter is less than a threshold, the image data and pseudo label corresponding to the pseudo label parameter are added to the trust data set.

4. A semi-supervised human eye multi-element segmentation method based on iterative learning, characterized in that: The following steps are involved: (1) Divide the visible light eye dataset into labeled eye dataset and unlabeled eye dataset; (2) training the labeled eye dataset using a supervised deep learning network to obtain a pre-trained model; (3) obtaining pseudo labels of the unlabeled eye dataset through forward reasoning based on the pre-trained model; (4) Filtering the unlabeled eye dataset and pseudo-labels based on the labeled eye dataset to obtain a trust dataset, specifically: (41) obtaining standard parameters based on the labels in the labeled eye dataset, specifically: extracting sclera labels, iris labels, and pupil labels according to the labels; and calculating the Hu moments of the sclera labels, iris labels, and pupil labels respectively; Calculating the Hu moment statistics of the sclera label, iris label and pupil label as standard parameters; (42) obtaining a pseudo label parameter for each image data in the unlabeled eye dataset based on the pseudo label; (43) comparing the standard parameters with the pseudo-label parameters, and if the total error rate of the pseudo-label parameters is less than a threshold, adding the image data and pseudo-label corresponding to the pseudo-label parameters to the trust data set; (5) inputting the labeled eye dataset and the trust dataset into the supervised deep learning network for retraining. If the training completion requirement is not met, returning to step (3); otherwise, terminating the training to obtain the human eye segmentation model; (6) The human eye segmentation model is used to perform multi-element segmentation on the human eye.

5. The semi-supervised human eye multi-element segmentation method based on iterative learning according to claim 4, characterized in that: The step (42) specifically comprises: extracting a pseudo sclera label, a pseudo iris label and a pseudo pupil label according to the pseudo label; Calculate the Hu moment of the pseudo sclera label, pseudo iris label and pseudo pupil label corresponding to each image data respectively.

6. The semi-supervised human eye multi-element segmentation method based on iterative learning according to claim 5, characterized in that: The step (43) is specifically as follows: respectively calculating the difference between the Hu moment of the pseudo sclera label, the pseudo iris label and the pseudo pupil label corresponding to each image data and the Hu moment statistics of the sclera label, the iris label and the pupil label; The total error rate is calculated by weighted summation according to the difference; if the total error rate of the pseudo label parameter is less than a threshold, the image data and pseudo label corresponding to the pseudo label parameter are added to the trust data set.

Images