Pupil positioning method for lightweight U-Net and unidirectional connected domain extraction

Abstract

The invention relates to the field of medical image processing and computer vision, in particular to a pupil positioning method for lightweight U-Net and unidirectional connected domain extraction, which comprises the following steps: S1, preprocessing an input ocular surface image, including Gamma correction and histogram equalization; s2, inputting the preprocessed image into a lightweight U-Net segmentation network, recovering a final feature map to an original size, inputting into a segmentation layer, and outputting four types of segmentation regions including a background, an iris, a sclera and a pupil; s3, Canny edge detection is applied to the pupil segmentation area, and an ocular surface area boundary is extracted; s4, screening pupil boundaries based on a one-way connected domain extraction strategy, traversing the edge point set, and extracting connected domains of which the lengths are greater than a preset threshold value through a one-way closed path search algorithm; and S5, calculating the center-of-mass coordinates of all pixels in the pupil connected domain, and taking the center-of-mass coordinates as a pupil center positioning result. The automation level and the imaging success rate of equipment are enhanced, the training requirements of operators of ophthalmic examination related equipment are reduced, and the equipment cost and the popularization difficulty are reduced.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and computer vision, and in particular to automatic pupil localization of ophthalmic image acquisition equipment, and specifically to a pupil localization method based on a lightweight U-Net and unidirectional connected domain extraction. Background Art

[0002] In optical imaging of the human eye and medical device applications, accurately locating the pupil is a critical step in ensuring image acquisition quality and focusing success rates. Existing ophthalmic equipment primarily uses three methods for pupil alignment: The first involves manual adjustment by the physician visually observing the ocular surface image. Typical devices include traditional desktop fundus cameras like the Canon CR-2 and Topcon TRC-NW8. This method relies on operator experience and suffers from long operation times, low standardization, and poor reproducibility, making it difficult to adapt to large-scale screening or primary care settings. The second method utilizes near-infrared imaging-assisted alignment. This involves capturing ocular surface images in real time with an infrared camera and roughly locating the pupil based on its low reflectivity in the infrared image. Some high-end devices (such as the Optos Daytona and Zeiss Clarus 500) combine this with a mechanical motion platform for semi-automatic or fully automatic focusing. However, this method is sensitive to ambient light variations, and the infrared reflectivity differences are small, resulting in blurred boundaries and susceptibility to interference. The overall system is complex and costly, making it unsuitable for low-cost, portable applications. The third method, which infers pupil center position by analyzing corneal reflection points, is commonly used in gaze tracking devices. However, due to the strict requirements for light source arrangement and head fixation, an additional optical path is required, the inference process is complex, and real-time performance is limited. Therefore, it is rarely used in current fundus screening equipment. Overall, existing pupil localization technologies have more or less suffered from poor environmental adaptability, high equipment costs, reliance on manual operation, and an inability to balance high accuracy and low latency.

[0003] At the image processing level, traditional pupil localization methods (such as Hough circle detection, Canny edge detection, and threshold segmentation) are limited by factors such as image noise, illumination variations, and occlusions, resulting in limited accuracy and robustness. They perform particularly poorly in conditions such as small pupils, insufficient mydriasis, and light reflection interference. With the advancement of deep learning, convolutional neural network (CNN)-based segmentation methods are increasingly being applied to eye image analysis. However, these purely neural network-based methods suffer from the following issues: poor adaptability to images with blurred boundaries and low contrast; susceptibility to mis-segmentation and missed segmentation; and high model inference complexity, making them unsuitable for edge devices. Therefore, improvements are needed.

[0004] Therefore, there is an urgent need for an automatic pupil positioning method that combines deep learning image segmentation with traditional image processing connected domain analysis, which can meet the needs while ensuring high robustness and high accuracy, and has the characteristics of lightweight, strong real-time performance, and suitable for embedded edge computing deployment. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a pupil localization method based on a lightweight U-Net and unidirectional connected domain extraction, comprising the following steps: S1. Preprocess the input ocular surface image, including gamma correction and histogram equalization; S2. The preprocessed image is fed into a lightweight U-Net segmentation network. The final feature map is restored to its original size and fed into the segmentation layer, which outputs four segmented regions: background, iris, sclera, and pupil. S3. Apply Canny edge detection to the pupil segmentation area to extract the boundary of the ocular surface area; S4. Screen the pupil boundary based on the one-way connected domain extraction strategy, traverse the edge point set, and extract the connected domain with a length greater than a preset threshold using the one-way closed path search algorithm; S5. Calculate the centroid coordinates of all pixels in the pupil connected domain as the pupil center positioning result.

[0006] Preferably, the structure of the lightweight U-Net segmentation network includes 4 downsampling layers and 4 upsampling layers. The downsampling layer uses the average pooling method to transfer parameters to the next layer. Each downsampling layer contains 5 convolutional layers. The convolutional layer activation function is LeakyReLU, and the convolutional layers use a Dense connection structure to share all parameters.

[0007] Preferably, the upsampling layer amplifies the feature map by a nearest neighbor algorithm and performs feature fusion with the corresponding downsampling layer before amplification.

[0008] Preferably, during the upsampling process, the feature map fusion adopts a skip connection method to splice the output of the downsampling layer with the input of the upsampling layer.

[0009] Preferably, the training of the lightweight U-Net segmentation network adopts a combined loss function:

[0010] in: is the cross entropy loss, is the boundary-aware loss, is the Dice loss, is the surface loss; the weight coefficients are set to λ1=1, λ2=10, λ3=1-α, λ4=α, and α increases linearly from 0 to the set upper limit with the training rounds.

[0011] Preferably, the surface loss The calculation method is,

[0012] in, is the set of all pixels in the target area. is the distribution map of the labeled distance of the true label, that is, the Euclidean distance from each pixel to the true boundary, is the SDM of the predicted mask. The optimization goal is to minimize the difference between the SDM of the predicted mask and the real mask. The SDM defines the Euclidean distance from each pixel in the image to the nearest target boundary and is expressed as,

[0013] in Pixels To the real boundary The negative sign indicates the pixel is in the target area.

[0014] Preferably, the unidirectional connected domain extraction strategy of step S4 includes: A. Starting from the edge point, the initial search direction is the upper left; B. If there is an edge point in the current direction, mark it as the same connected domain and rotate the search direction 90° counterclockwise to continue searching; C. If there is no edge point in the current direction, rotate 45° clockwise until an edge point is found; D. When the path is closed and its length exceeds the threshold, it is determined to be a valid pupil boundary.

[0015] Preferably, when screening connected domains in step S4, connected domains with path lengths less than 100 pixels are discarded.

[0016] Preferably, the Canny edge detection includes median filtering denoising with a kernel size of 3, calculating the gradient change rate in the horizontal and vertical directions by the Sobel operator to obtain the gradient amplitude and direction, non-maximum suppression and edge refinement, and dividing edge pixels into strong edges, weak edges, and non-edges by setting upper and lower thresholds, retaining strong edges and weak edges adjacent to strong edges within an 8-neighborhood or larger adjacent range.

[0017] Preferably, the preprocessing further includes grayscale processing of the original image.

[0018] Compared with the existing technology, the beneficial effect of the present invention lies in: combining the pupil automatic positioning method of the U-Net segmentation network based on the Dense connection structure and the unidirectional connected domain extraction strategy, training a lightweight pupil rough area extraction model through an improved loss function, and then combining the connected domain screening and centroid positioning technology to finally achieve fast and accurate pupil center extraction, which is suitable for refractive detection, fundus imaging, eye tracking and other systems, and achieves high robustness and high precision pupil center positioning and tracking, enhances the automation level and imaging success rate of the equipment, reduces the training requirements for operators of ophthalmic examination-related equipment, and thus reduces the equipment cost and promotion difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be discussed below. Obviously, the technical solutions described in conjunction with the drawings are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments and their drawings can be obtained based on the embodiments shown in these drawings without paying any creative work.

[0020] Figure 1 It is the overall flow chart of the present invention.

[0021] Figure 2 This is a diagram of the pupil segmentation network structure of the present invention.

[0022] Figure 3 are pupil images before and after the preprocessing of the present invention.

[0023] Figure 4 This is the ocular surface segmentation and ocular surface segmentation image edge extraction image of the present invention.

[0024] Figure 5 This is the pupil edge point set extraction and pupil center positioning result diagram of the present invention. DETAILED DESCRIPTION

[0025] The following will provide a clear and complete description of the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments described in the present invention, all other embodiments derived by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The embodiment of the present invention is a pupil automatic positioning method that combines a U-Net segmentation network based on a dense connection structure with a unidirectional connected domain extraction strategy. Figure 1 As shown, Figure 1This is the overall implementation flow chart of this algorithm. This algorithm uses deep learning methods and an improved loss function to train a lightweight pupil rough area extraction model. It then combines connected domain screening and centroid positioning technology to ultimately achieve fast and accurate pupil center extraction.

[0027] The present invention designs a U-Net segmentation network containing a DENSE connection in a feature extraction module and a loss function design for focusing on the boundary that combines multiple loss functions. A data set of segmented ocular surface areas is used to train an ocular surface area segmentation model. After completing the segmentation task on the original image, the Canny operator is used to extract edges of the image to obtain boundary line images of different ocular surface areas. For the above boundary lines, a method of extracting connected domains is used to set a length threshold, and the pupil boundary is screened. The center of mass of this boundary is then calculated to finally obtain a relatively accurate pupil center point.

[0028] First, the ocular surface area is initially segmented. This paper utilizes the traditional U-Net segmentation architecture. Specifically, the overall network structure employs four downsampling layers and four upsampling layers to downsample and upsample the input, respectively. The bottleneck layer reduces the overall information to a small tensor with a size of 1 / 16 of the input resolution.

[0029] Each downsampling layer contains 5 convolutional layers with LeakyReLU activation function. Each convolutional layer shares all the parameters of the previous convolutional layer in this sampling module, that is, the Dense connection method is adopted to reduce the number of parameters; finally, the average pooling method is used to transfer parameters to the next layer.

[0030] During upsampling, the feature map is primarily amplified using the nearest neighbor method. Its internal structure is similar to that of the downsampling layer, but with the opposite effect. Before amplification, the feature map is fused with the downsampling layer to receive additional information. Finally, the feature map is restored to its original size and input into the segmentation layer, which consists of a convolutional layer and a softmax layer. The output has four information channels, completing the segmentation of the background, iris, sclera, and pupil.

[0031] The main innovation of this invention lies in the training method of this network, which mainly designs an efficient loss function for the pupil segmentation task combination.

[0032] The cross entropy loss function (CEL: CrossEntropyLoss) commonly used in traditional segmentation tasks only distinguishes between foreground and background, and has relatively poor ability to distinguish boundary areas. At the same time, its design principle assumes that samples have a relatively balanced distribution, which makes it difficult to adapt to pupil segmentation tasks such as distinguishing small foregrounds in larger backgrounds. In this task, the background, iris, sclera, and pupil occupy an uneven proportion of the image size. Although a weight can be pre-assigned to each category according to the inverse proportion of its proportion to the total population by adopting the training idea of ​​FocalLoss with weights, the actual distribution of the proportion of each data hole in the image is not a constant, and the FocalLoss method still has the problem of not paying attention to boundary pixels in principle. Therefore, the present invention considers combining multiple loss functions to achieve the best boundary segmentation effect. The loss function still uses the CEL loss function. The CEL loss function can be expressed as follows:

[0033] Where C is the number of categories, is the one-hot encoding of the true label. If the sample Belong to category ,but ,otherwise Is the model for the category The predicted probability of .

[0034] Next, we use a common class imbalance segmentation loss function, the Dice coefficient (DL: DiceLoss), which calculates the overlap coefficient between the real image mask pixels and the predicted values. In this way, the similarity between the predicted values ​​and the real values ​​is calculated. In the case of class imbalance, it achieves better classification results by calculating the squared inverse weighted Dice coefficient of the class frequency. The Dice coefficient loss function can be expressed as: in, is a category The weight is usually taken as the inverse of the category frequency, that is: Another point to consider is the insensitivity of the cross entropy loss function to boundaries. It is necessary to find a loss function calculation method that is sensitive to boundaries to modulate it.

[0035] The embodiment of the present invention combines two loss functions to solve the problem, one of which is the edge-guided loss function (BAL: BoundaryAwareLoss). This method is based on category semantic segmentation and consists of two parts. The global part can directly use the aforementioned CEL and DL loss functions, while the local loss focuses on the boundary. It calculates the distance between the two closest areas of each pixel point, and gives higher weights to points with similar distances. Its calculation steps can be expressed as work performed in the following order: first, for boundary extraction, the boundary area is obtained using the expansion-erosion method, and then the loss function is calculated for the points in the boundary area. The BAL loss function can be expressed as: Where B is the index set of boundary pixels, is the predicted probability value, is the true label.

[0036] Another boundary-sensitive loss function is the surface loss function (SL: SurfaceLoss), which focuses on directly optimizing the surface distance between the segmentation result and the true target boundary, and can better preserve small structures with high semantic value. SL can be expressed as: in is the set of all pixels in the target area. It is the signed distance distribution map (SDM: Signed Distance Map) of the true label, that is, the Euclidean distance from each pixel to the true boundary. is the SDM of the predicted mask. The optimization goal is to minimize the difference between the SDM of the predicted mask and the true mask, so that the predicted result is more consistent with the shape of the true boundary. SDM defines the Euclidean distance from each pixel in the image to the nearest target boundary and can be expressed as:

[0037] in It's a pixel To the real boundary The negative sign indicates that the pixels in the target area have the negative Euclidean distance.

[0038] By combining the above loss functions, we can use a lightweight network to obtain the best segmentation effect. The combined loss function can be expressed as: In terms of training details, the hyperparameters are set as: , , , .in is a parameter related to the number of training rounds. It is set to start from 0 and increase evenly with the number of training rounds until it reaches half of the total number of training rounds. In order to further improve the separability of different areas of the image, Gamma correction and histogram equalization are applied to the input image. The images before and after the transformation are as follows: Figure 3 shown.

[0039] After completing the segmentation of the ocular surface area, the next step is to accurately extract the pupil contour. This requires first extracting the image edge and then filtering the edge point set.

[0040] For edge detection, the standard Canny operator can be used to detect the edge to obtain relatively good results. The calculation steps are as follows: After converting the image into a grayscale image, a median filter with a kernel size of 3 is used to remove noise, and then edge detection is performed. The edge detection method can be simply described as calculating the gradient, non-maximum suppression, double threshold detection, and finally completing the edge connection.

[0041] Specifically, in terms of gradient calculation, the Sobel operator is used to calculate the horizontal direction and vertical direction The gradient change rate on the gradient and obtain the gradient amplitude M and direction Then keep the sharpest gradient pixels and remove the adjacent weaker pixels. Then, check the adjacent pixel values ​​of each pixel in the gradient direction, that is, along the gradient direction. If the gradient magnitude M of the pixel is smaller than any adjacent pixel in the gradient direction, it is suppressed, that is, set to 0, otherwise it is retained.

[0042] After calculating the gradient value, two thresholds are set for the gradient value to classify edge pixels into strong edges, weak edges, and non-edges. Strong edges are regarded as edge points, and for weak edges, if they are connected to a strong edge within an 8-neighborhood or larger adjacency range, they are retained as edges; otherwise, they are regarded as noise and suppressed.

[0043] Through the above calculation steps, the edges of different structures on the eye surface, the eye surface area segmentation image and the corresponding edge extraction image can be obtained. Figure 4 shown.

[0044] To further extract pupil information, the extracted edge point set needs to be screened to obtain the edge point set belonging to the pupil. The present invention uses the method of extracting the minimum effective connected domain to extract the pupil boundary and calculate related parameters.

[0045] Specifically, all pixels are traversed to separate the foreground and background. For each foreground pixel, if it has not been marked as visited, its 8 neighboring points are checked. If it is a foreground pixel and has not been visited, it is marked as belonging to the same connected domain. The present invention introduces the concept of searching for a one-way closed connected domain, that is, within a narrow connected domain, for the pixel at the outermost boundary, there will always be a path greater than a certain size threshold, along the edge point of the connected domain back to the original pixel. For an edge point, there must be the next edge point within the range of 8 pixels around it. Therefore, the following search strategy is set: starting from the first boundary point, the initial search direction is set to the upper left; if the pixel in this direction is a foreground point, it is regarded as an edge point, otherwise the search direction is rotated 45 degrees clockwise and the check continues. This cycle is repeated until the next foreground point is found.

[0046] Next, the foreground point is used as the new boundary point, and the search direction is rotated 90 degrees counterclockwise based on the current direction. The same rule is used to continue searching for the next foreground point. In this way, a one-way closed path can be found until it finally returns to the starting boundary point, obtaining a narrow, closed connected domain.

[0047] In addition, considering that the aforementioned edge extraction may lead to some undisputed erroneous extraction areas, which are characterized by a small area, the erroneous areas can be excluded by setting a certain threshold. The present invention adopts 100 as the path length threshold. The connected domains below this threshold are regarded as erroneous extraction areas and discarded.

[0048] Each time a new connected domain is found through the connected domain extraction method, a unique label is assigned to it, and all pixels belonging to the connected domain are marked until all pixels are traversed, and a stack storing the sizes of all unidirectional closed connected domains is obtained. The connected domain with the shortest path length is the pupil edge. After obtaining the through-hole edge connected domain, the pupil center coordinates are finally obtained by calculating the centroid of all pixels in this connected domain. The overall extraction effect is as follows: Figure 5 shown.

[0049] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims, not the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be encompassed within the present invention. Any reference numerals in the claims should not be construed as limiting the claim to which they relate.

[0050] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A pupil localization method based on lightweight U-Net and unidirectional connected domain extraction, characterized by: The following steps are included: S1. Preprocess the input ocular surface image, including gamma correction and histogram equalization; S2. The preprocessed image is fed into a lightweight U-Net segmentation network. The final feature map is restored to its original size and fed into the segmentation layer, which outputs four segmented regions: background, iris, sclera, and pupil. S3. Apply Canny edge detection to the pupil segmentation area to extract the boundary of the ocular surface area; S4. Screen the pupil boundary based on the one-way connected domain extraction strategy, traverse the edge point set, and extract the connected domain with a length greater than a preset threshold using the one-way closed path search algorithm; S5. Calculate the centroid coordinates of all pixels in the pupil connected domain as the pupil center positioning result.

2. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 1, characterized in that: The structure of the lightweight U-Net segmentation network includes 4 downsampling layers and 4 upsampling layers. The downsampling layer uses the average pooling method to transfer parameters to the next layer. Each downsampling layer contains 5 convolutional layers. The activation function of the convolutional layer is LeakyReLU. The convolutional layers use a dense connection structure to share all parameters.

3. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 2, characterized in that: The upsampling layer amplifies the feature map through the nearest neighbor algorithm and performs feature fusion with the corresponding downsampling layer before amplification.

4. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 3 is characterized by: During the upsampling process, the feature map fusion adopts the skip connection method to splice the output of the downsampling layer with the input of the upsampling layer.

5. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 1, characterized in that: The training of the lightweight U-Net segmentation network uses a combined loss function: ; Where: L CEL L is the cross entropy loss, L BAL is the boundary-aware loss, L DL is the Dice loss, L SL is the surface loss; the weight coefficients are set to λ1=1, λ2=10, λ3=1-α, λ4=α, and α increases linearly from 0 to the set upper limit with the training rounds.

6. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 5, characterized in that: Surface loss L SL The calculation method is, ; in, is the set of all pixels in the target area, S(y) is the label distance distribution map of the true label, that is, the Euclidean distance from each pixel to the true boundary, is the SDM of the predicted mask. The optimization goal is to minimize the difference between the SDM of the predicted mask and the real mask. The SDM defines the Euclidean distance from each pixel in the image to the nearest target boundary and is expressed as, ； in Pixels To the real boundary The negative sign indicates the pixel is in the target area.

7. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 1, characterized in that: The one-way connected domain extraction strategy of step S4 includes: A. Starting from the edge point, the initial search direction is the upper left; B. If there is an edge point in the current direction, mark it as the same connected domain and rotate the search direction 90° counterclockwise to continue searching; C. If there is no edge point in the current direction, rotate 45° clockwise until an edge point is found; D. When the path is closed and its length exceeds the threshold, it is determined to be a valid pupil boundary.

8. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 1, characterized in that: When screening connected domains in step S4, connected domains with path lengths less than 100 pixels are discarded.

9. The pupil localization method using lightweight U-Net and unidirectional connected domain extraction according to claim 1, characterized in that: The Canny edge detection includes median filtering denoising with a kernel size of 3, calculating the gradient change rate in the horizontal and vertical directions by the Sobel operator to obtain the gradient amplitude and direction, non-maximum suppression and edge refinement, and classifying edge pixels into strong edges, weak edges, and non-edges by setting upper and lower thresholds. Strong edges and weak edges adjacent to strong edges within an 8-neighborhood or larger adjacency range are retained.

10. The pupil localization method using a lightweight U-Net and unidirectional connected domain extraction according to any one of claims 1 to 9, characterized in that: The preprocessing also includes grayscale processing of the original image.

Images