Human eye multi-class segmentation method based on topological intersection and constraint loss

Abstract

The invention discloses a human eye multi-class segmentation method based on topological intersection and constraint loss, belongs to the field of computer vision, and solves the problem of topological structure representation among different labels. On the basis of the segmentation model, priori knowledge of a human eye structure is introduced, and segmented images of multiple types of labels are guided in a topological intersection and constraint loss function form. A maximum pooling layer is connected to the exterior of the segmentation model, topological constraints are introduced into the maximum pooling layer, and the maximum pooling difference and topological region constraints form topological intersection and constrain loss. The loss is divided into two modules, the first module is a maximum pooling difference which comprises normalized input and target tensor, maximum pooling is carried out on a prediction image and an annotation image by using different kernels and strides, and the difference value between pooling prediction and each channel label is calculated; the second module is a topological region constraint, an additional topological condition is introduced to ensure that the sclera comprises the iris, the iris comprises the pupil, and the sclera and the pupil are mutually exclusive. And combining the pooling difference and the topological constraint into a final topological intersection and constraining the loss.

Description

Technical Field

[0001] The invention proposes a segmentation method based on topological intersection constraint loss for multi-class segmentation of human eye images, and belongs to the field of computer vision. Background Art

[0002] This method segments the eye area objects in the face image, and performs regional segmentation on multiple parts of the eye image based on medical prior knowledge, which helps to improve the model segmentation performance. Accurate eye image segmentation is conducive to improving the accuracy and reliability of iris recognition technology, and can also assist in the medical detection of eye diseases such as ocular myasthenia gravis and cataracts. In actual scenarios, the eyes have biological structural characteristics. This structural relationship can help us grasp the topological relationship in eye image segmentation, effectively constrain the multi-class labels in the image, and optimize the multi-scale segmentation of eye images.

[0003] Existing studies on eye image segmentation mainly focus on the segmentation method of the iris boundary. In this regard, Zhaofeng He et al. [1] A fast and accurate eye iris segmentation method is proposed to address noise sensitivity. First, the rough position of the iris center is extracted, the edge points of the iris boundary are detected, and an elastic model is established (under this model, the center and radius of the circular iris boundary are iteratively refined in a manner driven by the restoring force of Hooke's law). Then, the non-circular iris boundary is processed by an edge fitting method based on smooth splines, and the eyelid is located by edge detection and curve fitting. Yung-Hui Li et al. [2] An eye iris boundary segmentation algorithm is proposed, which divides eye iris segmentation into two steps: eye location and iris region segmentation. The segmentation algorithm detects and locates the eyes through the Faster R-CNN model, then fits the pupil area using the Gaussian mixture model (GMM), and finally estimates the iris boundary using the edge boundary location algorithm. Mousumi Sardar et al. [3] A variant of the interactive U-Net network was introduced to segment the iris of the eye. The interactive U-Net can help generate ground truth for datasets that do not have enough labeled samples. They reduce training time by squeezing and expanding modules, and improve storage efficiency by reducing the number of designed parameters. However, these methods can obtain more accurate iris regions, but cannot eliminate the influence of other labels and cannot fully complete the task of multi-class segmentation. In the current research on multi-class segmentation of eye images, Aayush et al. [4] The RITnet model is proposed to accurately segment the eye. The model combines the deep neural networks of U-Net and DenseNet, and uses the boundary-aware loss function and loss scheduling strategy to train the segmentation model. Hui Guo et al.[5] The consistency of pupil shape is used as the basis for distinguishing fake faces from real faces. The eye location segmentation is performed on the face image, and then the pupil boundary curve after segmentation is analyzed to determine whether the pupil shape is regular.

[0004] All of the above methods can segment the various regions of the eye image to a certain extent, but they have the following limitations: (1) Manual labeling is time-consuming and labor-intensive, so there are fewer eye labeled images, making it difficult for the segmentation model to achieve the expected effect; (2) The relationship between the multiple eye labels after segmentation is confusing, but the eye region has its own characteristics. Using this prior knowledge, the model can achieve the expected effect even with a small amount of data.

[0005] To overcome these challenges, this patent proposes a segmentation method with topological intersection-union loss, named Topological Intersection-Union Loss (TIU-Loss for short). This loss function is designed based on the multi-region intersection and union constraints between anatomical structures, that is, the topological constraints of the image are added to the maximum pooling layer according to the physiological structure of the eyeball. The internal function performs the calculation of the pooling components of each channel, including normalizing the input and target tensors, and performing maximum pooling using different kernels and strides. This function calculates the difference between the predicted image and the labels of each channel, and introduces additional topological conditions to constrain the pixels. The topological relationship of the eye area is as follows: Figure 1 The method of this patent can make full use of the structural relationship between the various regions of the eye, and determine whether the pixel points of the label meet the topological relationship through the constraints between multiple categories of labels to decide whether to retain or discard the pixel points, which significantly improves the multi-category segmentation quality of the eye image.

[0006] References:

[0007] [1]He Z,Tan T,Sun Z,et al.Toward accurate and fast iris segmentation for iris biometrics[J].IEEE transactions on pattern analysis and machineintelligence,2008,31(9):1670-1684.

[0008] [2]Li YH, Huang PJ, Juan Y. An efficient and robust iris segmentationalgorithm using deep learning[J]. Mobile Information Systems, 2019, 2019.

[0009] [3] Sardar M, Banerjee S, Mitra S. Iris segmentation using interactive deep learning [J]. IEEE Access, 2020, 8: 219322-219330.

[0010] [4]Chaudhary AK, Kothari R, Acharya M, et al.Ritnet: Real-time semanticsegmentation of the eye for gaze tracking[C] / / 2019IEEE / CVF International Conference on Computer Vision Workshop(ICCVW).IEEE,2019:3698-3702.

[0011] [5]Guo H, Hu S, Wang Summary of the invention

[0012] Existing deep learning segmentation methods automatically learn powerful feature representations on images and optimize pixel accuracy to achieve higher performance. However, this method often ignores biological structural features and fails to display biological features in the results, resulting in a chaotic structure of segmentation results. Although existing work can represent the topological relationship of a single label, it has made slow progress in solving the topological structure representation between different labels.

[0013] The present invention adopts the following technical scheme: on the basis of the basic segmentation model, the prior knowledge of the structure of the human eye is introduced in a targeted manner, and the segmented image of multiple classes of labels is guided in the form of a topological intersection constraint loss function. The present invention connects a maximum pooling layer outside the segmentation model, introduces topological constraints in the maximum pooling layer, and the maximum pooling difference and the topological region constraint constitute the topological intersection constraint loss. This loss is divided into two modules. Module one is the maximum pooling difference. The internal function performs the difference component calculation of each channel, including normalizing the input and target tensors, and using different kernels and strides to perform maximum pooling on the predicted image and the annotated image, and calculates the difference between the pooled prediction and the labels of each channel. Module two is the topological region constraint, which introduces additional topological conditions in the form of an activation function to constrain the pixel points of multiple classes of labels, ensuring that the sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive. Finally, we combine these pooling differences and topological constraints into the final topological intersection constraint loss.

[0014] The specific scheme of the present invention is as shown in the attached Figure 2 shown.

[0015] Step 1: Get the base segmentation image

[0016] By passing the eye image through the basic segmentation model, a basic segmentation image of the image can be obtained, and a basic image feature representation can be obtained.

[0017] Step 2: The maximum pooling difference module obtains the pooling difference loss

[0018] The purpose of this step is to use the maximum pooling layer to obtain the per-channel difference component of multiple class labels. It includes three major stages.

[0019] Step 2.1 Input normalization: Normalize the input and target tensors to the range of [0,1]. The input is the base segmentation image obtained in step 1, and the target tensor is the ground truth image of the eye image. The normalized input is to facilitate subsequent difference calculations.

[0020] Step 2.2 Max Pooling Operation: Apply multi-scale max pooling operations to the normalized input and target tensors based on the multi-class label channels, kernels, and strides.

[0021] Step 2.3 Difference calculation: Calculate the absolute difference of the sum of the pooled values ​​of each channel.

[0022] Step 3: Topological region constraint module obtains topological region loss

[0023] The purpose of the topological region constraint module is to use the prior knowledge of the human eye structure to form topological relationship constraints between multiple categories of labels. The topological relationship has two parts, the inclusion relationship and the mutual exclusion relationship. In the eye structure, there are two pairs of inclusion relationships and one pair of mutual exclusion relationships. The sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive. This step consists of three stages.

[0024] Step 3.1 Inclusion relationship representation: Utilize the inclusion relationship between structures and express it as a function representation between label pixels.

[0025] Step 3.2 Mutually exclusive relationship representation: Utilize the mutually exclusive relationship between structures and express it as a function representation between label pixels.

[0026] Step 3.3 Comprehensive calculation of topological relations: The topological constraints between the three labels are comprehensively calculated based on prior knowledge.

[0027] Step 4: Generate topological intersection constraint loss

[0028] The present invention combines the pooling difference loss generated by the maximum pooling difference module with the topological area loss generated by the topological area constraint module to obtain the final topological intersection constraint loss L TIU .

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention adopts a topological intersection constraint loss function method, which uses the idea of ​​prior knowledge in the maximum pooling layer to express the topological relationship between multiple types of eye regions in the form of a loss function. It can use the characteristics of the maximum pooling layer to conveniently introduce the segmentation model and improve the calculation speed and performance of the model.

[0031] 2. The present invention adopts a topological intersection constraint loss function method, which makes full use of the inclusion and mutual exclusion relationship between multiple categories of eye regions and constrains the topological relationship between labels, so that the final multi-category segmented image of the eye conforms to the medical description of the eye, significantly improving the geometric consistency and semantic accuracy of the multi-category image segmentation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the topological relationship between multiple class labels in eye images.

[0033] Figure 2 This is the overall model diagram of the method proposed in the present invention. DETAILED DESCRIPTION

[0034] The following is a detailed description of the implementation examples of the present invention in conjunction with the accompanying drawings:

[0035] The present invention is a loss function method based on topological intersection constraints, and is a method for optimizing the segmentation of multiple eye images in the form of a loss function using the physiological structural relationship of the eye as prior knowledge. The specific process of multi-class segmentation of eye images is as follows: the eye image is input into a basic segmentation model to obtain a basic segmented image, and a topological intersection constraint loss is introduced on this basis. The topological intersection constraint loss passes through two modules in the external maximum pooling layer. First, through the maximum pooling difference module, the internal function performs the difference component calculation of each channel, including normalizing the input and target tensors, and using different kernels and strides to perform maximum pooling on the predicted image and the labeled image, and calculating the difference between the pooled prediction and the labels of each channel; then through the topological region constraint module, additional topological conditions are introduced in the form of activation functions to constrain the pixel points of multiple class labels, ensuring that the sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive; finally, we combine these pooling differences and topological constraints into the final topological intersection constraint loss.

[0036] Specifically, the method comprises the following steps:

[0037] Step 1: Get the base segmentation image

[0038] By passing the eye image through the basic segmentation model, a basic segmentation image of the image can be obtained, and a basic image feature representation can be obtained as a representation of the input tensor. The basic segmentation model can be a classic model FCN, UNet, or other classic segmentation models.

[0039] Step 2: The maximum pooling difference module obtains the pooling difference loss

[0040] The purpose of this step is to use the maximum pooling layer to obtain the topological components of each channel of multiple class labels. It includes three major stages.

[0041] Step 2.1 Input Normalization: Normalize the input tensor and target tensor to the range [0,1]. The input tensor is the base segmentation image obtained in step 1, and the target tensor is the ground truth image of the eye image. The normalized input is to facilitate subsequent difference calculations. Assume that the dimensions of the input tensor I and the target tensor T are N×C×H×W, where N is the batch size, C is the number of channels, and H and W are the height and width, respectively. and It is the normalized tensor representation.

[0042] Step 2.2 Max Pooling Operation: Apply multi-scale max pooling operations to the normalized input and target tensors based on the multi-class label channels, kernels, and strides.

[0043] For each channel c and each combination K of kernel size k and stride s, compute the sum of the input tensor and the target tensor after pooling:

[0044]

[0045] Among them, H' and W' are the height and width of the normalized tensor. [:, c, i, j] is an indexing method that means selecting all elements in the first dimension (i.e., the entire batch), then selecting the cth channel, and then selecting the elements in the i-th row and j-th column.

[0046] Step 2.3 Difference calculation: Calculate the absolute difference of the sum of the pooled values ​​of each channel. The formula for calculating the absolute difference is shown in (3).

[0047] Diff(c)=|S I (c)-S T (c)| (3)

[0048] Average all channels:

[0049]

[0050] Where channel start =1, the background label channel is 0 and is not included in the pooling difference loss.

[0051] Average all kernel and stride combinations (k, s), where |K| is the number of combinations:

[0052]

[0053] Step 3: Topological region constraint module obtains topological region loss

[0054] The purpose of the topological region constraint module is to use the prior knowledge of the human eye structure to form topological relationship constraints between multiple categories of labels. The topological relationship has two parts, the inclusion relationship and the mutual exclusion relationship. In the eye structure, there are two pairs of inclusion relationships and one pair of mutual exclusion relationships. The sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive. This step consists of three stages.

[0055] Step 3.1 Inclusion relationship representation: Utilize the inclusion relationship between structures and express it as a function representation between label pixels.

[0056] Inclusion x,y =ReLU(I[:,channel x ,...]-I[:,channel y ,...]) (6)

[0057] This formula represents the inclusion relationship between label x and label y. Label x contains label y, where channel x Represents the channel of label x, channel yRepresents the channel of label y. I[:, channel x ,...] indicates that the shape of tensor I is [N, C, H, W], where: represents the operation on each sample in the entire batch N, ... represents the remaining two dimensions H, W.

[0058] Step 3.2 Mutually exclusive relationship representation: Utilize the mutually exclusive relationship between structures and express it as a function representation between label pixels.

[0059]

[0060] This formula expresses the mutually exclusive relationship between label x and label y.

[0061] The activation function ReLU is expressed as follows:

[0062]

[0063] Step 3.3 Comprehensive calculation of topological relations: The topological constraints between the three labels are comprehensively calculated based on prior knowledge.

[0064] We bring the above relationship formula into the topological constraints we set, Inclusion sclera，iris Inclusion means that the sclera contains the iris. iris，pupil Indicates that the iris includes the pupil, Exclusion sclera，pupil Indicates that the iris and pupil are mutually exclusive.

[0065] It can be obtained that the designed topological area loss is expressed as shown in formula (13).

[0066]

[0067] Step 4: Generate topological pooling region loss

[0068] The present invention integrates the pooling difference loss generated by the topological pooling module and the topological regional loss generated by the topological regional constraint module to obtain the final topological pooling regional loss L TIU .

[0069] L TIU =MeanDiff ks +MeanConstraints(10).

Claims

1. A human eye multi-class segmentation method based on topological intersection constraint loss, characterized by: Step 1: Get the base segmentation image By passing the eye image through the basic segmentation model, a basic segmentation image of the image can be obtained, and a basic image feature representation can be obtained; Step 2: The maximum pooling difference module obtains the pooling difference loss The purpose of this step is to use the maximum pooling layer to obtain the difference component of each channel of multiple class labels; it includes three major stages; Step 2.1 Input normalization: Normalize the input and target tensors to the range [0,1]. The input is the base segmentation image obtained in step 1, and the target tensor is the ground truth image of the eye image. Step 2.2 Max Pooling Operation: Apply multi-scale max pooling operations to the normalized input and target tensors based on multi-class label channels, kernels, and strides; Step 2.3 Difference calculation: Calculate the absolute difference of the sum of the pooled values ​​of each channel; Step 3: Topological region constraint module obtains topological region loss The purpose of the topological region constraint module is to use the prior knowledge of the human eye structure to form topological relationship constraints between multiple categories of labels; the topological relationship has two parts, the inclusion relationship and the mutual exclusion relationship. In the eye structure, there are two pairs of inclusion relationships and one pair of mutual exclusion relationships. The sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive. This step consists of three stages; Step 3.1 Inclusion relationship representation: Utilize the inclusion relationship between structures and express it as a function representation between label pixels; Step 3.2 Mutually exclusive relationship representation: Utilize the mutually exclusive relationship between structures and express it as a function representation between label pixels; Step 3.3 Comprehensive calculation of topological relations: The topological constraints between the three labels are comprehensively calculated based on prior knowledge; Step 4: Generate topological intersection constraint loss The pooling difference loss generated by the maximum pooling difference module and the topological region loss generated by the topological region constraint module are combined to obtain the final topological intersection constraint loss L TIU .

2. The method for multi-class segmentation of human eyes based on topological intersection constraint loss according to claim 1, characterized in that: Step 1: Get the base segmentation image The eye image is passed through the basic segmentation model to obtain the basic segmentation image of the image, and the basic image feature representation is obtained as the representation of the input tensor; Step 2: The maximum pooling difference module obtains the pooling difference loss It includes three major stages; Step 2.1 Input Normalization: Normalize the input tensor and target tensor to the range [0,1]. The input tensor is the base segmentation image obtained in step 1, and the target tensor is the ground truth image of the eye image. The dimensions of the input tensor I and the target tensor T are N×C×H×W, where N is the batch size, C is the number of channels, and H and W are the height and width respectively. and is the normalized tensor representation; Step 2.2 Max Pooling Operation: Apply multi-scale max pooling operations to the normalized input and target tensors based on multi-class label channels, kernels, and strides; For each channel c and each combination K of kernel size k and stride s, compute the sum of the input tensor and the target tensor after pooling: Among them, H ′ and W ′ is the height and width of the normalized tensor; [:,c,i,j] is an indexing method, which means selecting all elements in the first dimension, then selecting the cth channel, and then selecting the elements in the i-th row and j-th column; Step 2.3 Difference calculation: Calculate the absolute difference of the sum of the pooled values ​​of each channel; the formula for calculating the absolute difference is shown in (3); Diff(c)=|S I (c)-S T (c)| (3) Average all channels: Where channel start =1, the background label channel is 0; Average all kernel and stride combinations (k,s), where |K| is the number of combinations: Step 3: Topological region constraint module obtains topological region loss The prior knowledge of the human eye structure is used to form topological relationship constraints between multiple types of labels. The topological relationship consists of two parts: inclusion relationship and mutual exclusion relationship. In the eye structure, there are two pairs of inclusion relationships and one pair of mutual exclusion relationships. The sclera contains the iris, the iris contains the pupil, and the sclera and the pupil are mutually exclusive. This step consists of three stages. Step 3.1 Inclusion relationship representation: Utilize the inclusion relationship between structures and express it as a function representation between label pixels; Inclusion x,y =ReLU(I[:,channel x ,…]-I[:,channel y ,…]) (6) This formula represents the inclusion relationship between label x and label y. Label x contains label y, where channel x Represents the channel of label x, channel y Represents the channel of label y; I[:,channel x ,…] indicates that the shape of tensor I is [N,C,H,W], where: indicates that each sample in the entire batch N is operated on, and… indicates the remaining two dimensions H and W; Step 3.2 Mutually exclusive relationship representation: Utilize the mutually exclusive relationship between structures and express it as a function representation between label pixels; This formula represents the mutually exclusive relationship between label x and label y; The activation function ReLU is expressed as follows: Step 3.3 Comprehensive calculation of topological relations: The topological constraints between the three labels are comprehensively calculated based on prior knowledge; Substitute the above relationship formula into the set topological constraints, Inclusion sclera,iris Inclusion means that the sclera contains the iris. iris,pupil Indicates that the iris includes the pupil, Exclusion sclera,pupil It means that the iris and pupil are mutually exclusive; It can be obtained that the design topological area loss is expressed as shown in formula (13); Step 4: Generate topological pooling region loss The final topological pooling area loss L is obtained by fusing the pooling difference loss generated by the topological pooling module and the topological area loss generated by the topological area constraint module. TIU ; L TIU =MeanDiff ks +MeanConstraints (10)。

Images