A pupil rotation angle and direction recognition method based on multi-feature fusion

Abstract

The present application relates to the technical field of pupil recognition, in particular to a pupil rotation angle and direction recognition method based on multi-feature fusion, comprising: collecting eye images of a user and positioning eye regions of interest, segmenting to obtain iris annular texture features, pupil edge key point features and double corneal reflection point reference features, and registering as multi-modal features; predicting the pupil rotation angle and the pupil direction based on the multi-modal features; and merging the recognition results of multiple detection frames and outputting. In view of the problem of low robustness of single mode detection in the prior art, the iris annular texture features, the pupil edge key point features and the double corneal reflection point reference features are introduced to form multi-modal features to realize better representation of the actual angle of the pupil and then prediction. Considering the problem of prediction accuracy, the collection time is also prolonged, and the historical multi-frame prediction results are corrected, so as to realize accurate recognition of the pupil rotation angle and direction.

Description

Technical Field

[0001] This invention relates to the field of pupil recognition technology, specifically to a method for pupil rotation angle and direction recognition based on multi-feature fusion. Background Technology

[0002] Pupil parameter recognition is a core supporting technology in fields such as medical diagnosis (e.g., vestibular dysfunction, oculomotor palsy, strabismus diagnosis), human-computer interaction (e.g., eye-controlled devices, smart wearable interaction), and visual function assessment (e.g., myopia control, visual fatigue monitoring). Generally, pupil parameters include pupil center coordinates, pupil rotation angle and direction, diameter and area, roundness, and pupillary light reflex. Among these, pupil rotation angle and direction are important indicators in neurology and ophthalmology, used to assess symptoms of neurological dysfunction and vestibular system disorders.

[0003] Existing technologies already include solutions for automatically recognizing pupil rotation angles and directions. For example, patent application CN202411810443.7 discloses a method, system, device, and storage medium for eye positioning during ophthalmic surgery. The method includes: acquiring two eye images at adjacent time points and performing image enhancement to obtain a first eye image and a second eye image; comparing the two images to determine if eye rotation has occurred; if eye rotation has occurred, acquiring the center position of the first pupil corresponding to the first eye image, extracting iris feature pairs from the two images, inputting them into an eye rotation recognition model, outputting the eye rotation angle, and adjusting the center position of the surgical pupil based on the eye rotation angle and the center position of the first pupil; if eye rotation has not occurred, performing grayscale processing on the second eye image and using a sliding window-based grayscale analysis method to obtain the center position of the second pupil, thereby adjusting the center position of the surgical pupil. This method enables eye positioning under conditions of eye rotation and pupil dilation, improving the accuracy and reliability of eye positioning.

[0004] For example, patent application CN202111453683.2 discloses a method for measuring iris rotation, including the following steps: detecting the pupil and obtaining the pupil center coordinates and radius; selecting a segment of the iris for sampling based on the pupil center coordinates and pupil radius, then sampling multiple points along the pupil radius direction, and flattening these points into rectangles; using a histogram equalization method to obtain a more contrasting iris image; applying a template matching algorithm, selecting one frame as the reference texture, and then further comparing it with the iris of each subsequent frame to calculate the angular displacement, and finally calculating the iris torsion angle frame by frame; based on the angular displacement of each frame in the video calculated by template matching, obtaining the trajectory curve of the pupil torsion motion in the entire video, and the difference between two adjacent frames is the angular velocity of the pupil torsion; this method solves the problem of how to measure the pupil torsion angle frame by frame in a video, enabling automated torsion angle calculation without manual intervention, and is more robust.

[0005] However, in actual implementation, the inventors found that existing technologies rely on a single feature of the pupil edge for parameter calculation. When the pupil is partially obscured, the recognition accuracy drops sharply and the robustness is insufficient. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, a method for pupil rotation angle and orientation recognition based on multi-feature fusion is provided.

[0007] The specific technical solution is as follows: A method for pupil rotation angle and direction recognition based on multi-feature fusion, comprising: Step S1: acquiring an eye image of the user and locating the region of interest (ROI) of the eye; the eye image includes the user's pupil, iris, and corneal reflective points; Step S2: segmenting iris ring texture features, pupil edge key point features, and dual corneal reflective point reference features from the ROI of the eye, and registering them as multimodal features; Step S3: predicting the pupil rotation angle and pupil direction based on the multimodal features; Step S4: merging the pupil rotation angle and pupil direction according to the recognition results of multiple detection frames and outputting them.

[0008] On the other hand, step S1 includes: step S11: acquiring the eye image of the user; step S12: preprocessing the eye image to obtain a preprocessed image; step S13: identifying the region of interest of the eye from the preprocessed image.

[0009] On the other hand, the preprocessing steps in step S12 include: Gaussian filtering for noise reduction, grayscale conversion, and illumination equalization.

[0010] On the other hand, in step S2, the first extraction process of the iris ring texture feature includes: step A21: using the Canny algorithm to perform edge detection on the region of interest of the eye to segment the outer boundary and inner boundary of the iris and form the iris region; step A22: converting the iris region to polar coordinates and segmenting it to obtain four ring regions; step A23: extracting 16-dimensional LBP features and 32-dimensional grayscale histogram features for each ring region, and splicing them sequentially to obtain the iris ring texture feature.

[0011] On the other hand, in step S2, the second extraction process of the pupil edge key point features includes: step B21: using a least squares ellipse fitting algorithm to fit the pupil contour to the region of interest of the eye; step B22: applying the Shi-Tomasi corner detection algorithm to the pupil contour to extract pupil edge key points; step B23: calculating the center distance of each pupil edge key point relative to the center of the pupil contour, removing outliers of the center distance, and adding the remaining pupil edge key points to the pupil edge key point features.

[0012] On the other hand, in step S2, the third extraction process of the corneal reflective point reference feature includes: step C21: adaptively segmenting and extracting the bright spot region from the region of interest in the eye; step C22: calculating the average brightness of the bright spot region and extracting the two bright spot regions with the highest brightness as the main reflective points; step C23: calculating the main reflective point spacing and comparing it with the main reflective point spacing threshold. If it exceeds the main reflective point spacing threshold, return to step C21 for re-extraction until it meets the main reflective point spacing threshold, and then output the main reflective point as the corneal reflective point reference feature.

[0013] On the other hand, in step S2, the registration process includes: step D21: establishing a reference coordinate system with the midpoint of the reference feature of the corneal reflection point as the origin; step D22: mapping the iris ring texture feature and the pupil edge key point feature to the reference coordinate system; step D23: converting the points in the reference coordinate system to polar coordinates to form the multimodal feature.

[0014] On the other hand, step S3 includes: step S31: performing rotational deviation quantization based on the multimodal features to construct a deviation feature vector; step S32: inputting the deviation feature vector into a pre-trained random forest regression model to obtain the predicted pupil rotation angle and pupil direction.

[0015] On the other hand, step S4 includes: step S41: filtering the pupil rotation angles of the previous few frames and calculating the historical average, then weighting and fusing the historical average and the current pupil rotation angle to obtain the actual pupil rotation angle output; step S42: performing direction calibration based on the historical average and the actual pupil rotation angle to output the pupil direction.

[0016] The above technical solution has the following advantages or beneficial effects: Addressing the issue of low robustness in existing technologies that rely on a single modality for pupil rotation angle detection, this solution introduces multimodal features, including iris ring texture features, pupil edge key point features, and dual corneal reflective point reference features, to achieve a better representation of the actual pupil angle before prediction. Considering the issue of prediction accuracy, the acquisition time is extended, and corrections are made based on historical multi-frame prediction results, thereby achieving accurate identification of the pupil rotation angle and direction. Attached Figure Description

[0017] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0018] Figure 1 This is an overall schematic diagram of an embodiment of the present invention; Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention; Figure 3 This is a schematic diagram of the first extraction process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the second extraction process in an embodiment of the present invention; Figure 5 This is a schematic diagram of the third extraction process in an embodiment of the present invention; Figure 6 This is a schematic diagram of the registration process in an embodiment of the present invention; Figure 7 This is a schematic diagram of step S3 in an embodiment of the present invention; Figure 8 This is a schematic diagram of step S4 in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0022] This invention includes: a method for pupil rotation angle and orientation recognition based on multi-feature fusion, such as... Figure 1 As shown, the process includes: Step S1: Acquiring an eye image of the user and locating the region of interest (ROI) of the eye; the eye image includes the user's pupil, iris, and corneal reflective points; Step S2: Segmenting the iris ring texture features, pupil edge key point features, and dual corneal reflective point reference features from the ROI of the eye, and registering them as multimodal features; Step S3: Predicting the pupil rotation angle and pupil direction based on the multimodal features; Step S4: Merging the pupil rotation angle and pupil direction based on the recognition results of multiple detection frames and outputting the result.

[0023] Specifically, to address the issue of low robustness in existing technologies that rely on a single modality for pupil rotation angle detection, a multimodal feature model is introduced, comprising iris ring texture features, pupil edge key point features, and dual corneal reflective point reference features. This model achieves a better representation of the actual pupil angle before prediction. Considering the accuracy of prediction, the acquisition time is extended, and corrections are made based on historical multi-frame prediction results, thereby achieving accurate identification of both pupil rotation angle and direction.

[0024] Specifically, the above-mentioned technical solution is mainly configured as a software embodiment in computer devices during implementation, used to identify the user's pupil rotation angle and direction. Such computer devices may be medical workstations, such as those used in neurology to observe ocular-neural responses or in ophthalmology to measure strabismus; they may also be general devices involving eye tracking and physiological feature recognition, such as eye trackers in entertainment devices or auxiliary verification modules in iris verification devices. For ease of description, the following mainly focuses on the field of medical recognition, but it is actually applicable to other fields as well.

[0025] At the start of data acquisition, users need to be guided to align their eyes with the optical window of the acquisition device to capture an image of their eyes. For ease of recognition, bright pupil images captured using infrared illumination and a deep eye mask are typically used as eye images. However, depending on the product being applied, the recognition model may be optimized and retrained to extend to dark pupil images for recognition.

[0026] Taking the Bright Eyes image as an example, a near-infrared camera is used for acquisition, with a frame rate of 30~120Hz and a resolution of no less than 1080P. It is integrated into the position of the eye image in a normal eye mask and is equipped with an adjustable near-infrared fill light (wavelength 850nm, light intensity adjustment range 10~50lux) to eliminate ambient light interference.

[0027] During data collection, the subject wears a standard goggle with an integrated image acquisition module. The goggle is adjusted so that the camera is aimed directly at the eyes, ensuring the pupil is fully within the camera's field of view. The system automatically detects the ambient light intensity and adjusts the brightness of the supplementary light accordingly. For example, the supplementary light intensity is set to 20 lux when the ambient light is <20 lux, and to 30 lux when the ambient light is ≥20 lux.

[0028] The camera then captures raw images of the eyes (1920×1080 resolution) and transmits them to a computer via USB, with a stable frame rate of 60Hz.

[0029] The collected images consist of two sets: a baseline image before the test begins and an image from the actual test. Both sets of images undergo feature processing using the same procedure.

[0030] The original captured images of the eye usually include irrelevant peripheral areas such as eyelashes and eyelids. Irrelevant areas are removed by image recognition and segmentation, leaving only the central region of interest, which includes the complete pupil, iris and cornea.

[0031] Then, parallel feature extraction branches are used to extract iris ring texture features, pupil edge key point features, and corneal reflective point baseline features for regions of interest in the eye.

[0032] The iris ring texture feature is derived by extracting LBP and grayscale histogram features from the corresponding ring region of the iris, and then concatenating them to form a vectorized representation. In practical processing, to control the amount of data, the ring region is typically divided into four equal parts, and the LBP and grayscale histogram features are calculated and concatenated in each partition. This feature can be used to characterize the texture information of the pupil.

[0033] Pupil edge key point features are features formed by extracting multiple key points from the edge of the pupil region and splicing them together based on the coordinates of the key points. These features can be used to characterize relevant information about the pupil edge, such as the pupil center position, ellipticity, and size.

[0034] The dual corneal reflective spot reference feature is a feature composed of two reflective bright spots extracted from the complete corneal region in the image and the coordinates of the center points of the bright spots recorded.

[0035] Using the aforementioned features as multimodal inputs to the model, and performing preliminary labeling of deviations based on the baseline state image, the data is then fed into a pre-trained random forest model to achieve direct prediction of pupil rotation angle and pupil direction.

[0036] After completing the prediction process, considering the low accuracy of the prediction process in the existing technology, we chose to merge the prediction results of multiple historical frames with the current prediction results to output a relatively reliable prediction result and mask the jump results, thereby improving the robustness of the algorithm.

[0037] In one embodiment, such as Figure 2 As shown, step S1 includes: step S11: acquiring an eye image from the user; step S12: preprocessing the eye image to obtain a preprocessed image; step S13: identifying the region of interest of the eye from the preprocessed image.

[0038] The preprocessing steps in step S12 include: Gaussian filtering for noise reduction, grayscale conversion, and illumination equalization.

[0039] Specifically, in order to achieve better extraction of the region of interest in the eye, this embodiment first collects eye images of the user, including having the subject wear a regular eye mask with an integrated image acquisition module, adjusting the position of the eye mask so that the camera is aimed at the eye, ensuring that the pupil is completely in the camera's field of view, and the system automatically detects the ambient light intensity and adjusts the brightness of the supplementary light to capture eye images, etc.

[0040] Subsequently, the eye image is preprocessed to obtain a preprocessed image. Specifically, for the acquired original image, high-speed filtering and noise reduction are first performed, including convolution operation on the original image using a 5×5 Gaussian kernel (standard deviation σ=1.5) to remove salt-and-pepper noise and Gaussian noise.

[0041] After removing Gaussian noise, to reduce computational load, the color image was converted to a grayscale image using the weighted average formula I_gray=0.299R+0.587G+0.114B, while preserving key eye features.

[0042] Considering the uneven illumination caused by the curvature of the eyeball and the complex structure of the eyelid, illumination equalization compensation is also required, including the use of adaptive histogram equalization (CLAHE), setting the block size to 8×8 and limiting the contrast gain to 2.0 to compensate for uneven illumination in the eye area.

[0043] After preprocessing, the preprocessed image is segmented, retaining only the complete pupil, iris, and corneal regions. This step mainly relies on grayscale thresholding, which makes it easy to identify the sclera region in the preprocessed image. Subsequently, the outer boundary of the sclera is extracted as the segmentation criterion to extract the region of interest in the eye.

[0044] In one embodiment, such as Figure 3 As shown, in step S2, the first extraction process of iris ring texture features includes: Step A21: using the Canny algorithm to perform edge detection on the region of interest in the eye to segment the outer boundary and inner boundary of the iris and form the iris region; Step A22: converting the iris region to polar coordinates and segmenting it to obtain four ring regions; Step A23: extracting 16-dimensional LBP features and 32-dimensional grayscale histogram features for each ring region, and then stitching them together to obtain the iris ring texture features.

[0045] Specifically, in order to effectively extract the ring-shaped texture features of the iris, in this embodiment, the iris region is first determined from the region of interest of the eye.

[0046] This process primarily relies on the clear boundary between the iris and the pupil and sclera. It involves using the Canny edge detection algorithm (low threshold 50, high threshold 150) to locate the outer boundary (iris-sclera junction) and the inner boundary (pupil-iris junction). The annular region enclosed by these two boundaries corresponds to the iris region. The pupil region can also be determined based on the inner boundary, and the pupil center can be calculated based on pixel coordinates.

[0047] Subsequently, to facilitate unified calculation, the iris region was converted from rectangular coordinates to polar coordinates (ρ, θ) with the center of the pupil as the origin, and evenly divided into 4 annular regions (with a radius interval of 0.5 mm to accommodate different iris sizes).

[0048] For each annular region, 16-dimensional LBP features (8 pixels in the neighborhood and 1 pixel in the radius) and 32-dimensional grayscale histogram features are extracted and concatenated to obtain a 48-dimensional feature vector for each annular region. The four annular regions together form a 192-dimensional vector as the output of the iris annular texture features.

[0049] In one embodiment, such as Figure 4 As shown, in step S2, the second extraction process of pupil edge key point features includes: Step B21: using the least squares ellipse fitting algorithm to fit the pupil contour to the region of interest of the eye; Step B22: applying the Shi-Tomasi corner detection algorithm to the pupil contour to extract pupil edge key points; Step B23: calculating the center distance of each pupil edge key point relative to the center of the pupil contour, removing outliers in the center distance, and adding the remaining pupil edge key points to the pupil edge key point features.

[0050] Specifically, for key points at the pupil edge, this embodiment first uses a least-squares ellipse fitting algorithm to fit the pupil contour for the region of interest in the eye, obtaining the major axis, minor axis, center coordinates, and rotation angle of the ellipse. The fitting error is <0.3 pixels, thus extracting the pupil contour. This includes first performing binarization processing to label the pupil region, then using the Canny or Sobel operator to extract the edge coordinates of the pupil region to obtain a set of edge coordinate points, and then designing and constructing a matrix for each point in the set of edge coordinate points and solving for the feature values ​​to obtain the ellipse coefficients. These coefficients are used for subsequent least-squares ellipse fitting to obtain the major axis, minor axis, center coordinates, and rotation angle of the ellipse. The fitted contour is used as the actual pupil contour to avoid segmentation errors caused by differences in illumination during segmentation.

[0051] Then, the Shi-Tomasi corner detection algorithm is applied to the pupil contour to extract key points of the pupil edge. The corner response threshold R=200 is set to extract 8 key feature points of the pupil edge, which are located at the top, bottom, left, right, upper left, upper right, lower left, and lower right vertices of the ellipse. This part can select points with more significant features from the pupil contour for subsequent model extraction.

[0052] For the extracted key points, a distance check is also performed, including recording the rectangular coordinates (xi, yi) of each key point and calculating the distance between each point and the pupil center. Then, the sequence is compared according to the standard deviation to determine if there are outliers. Outliers are then removed and reselected to avoid matching errors.

[0053] In one embodiment, such as Figure 5 As shown, in step S2, the third extraction process of the baseline features of the dual corneal reflective points includes: Step C21: Adaptive segmentation and extraction of the region of interest in the eye to obtain the bright spot region; Step C22: Calculation of the average brightness of the bright spot region, and extraction of the two bright spot regions with the highest brightness as the primary reflective point and the secondary reflective point; Step C23: Calculation of the reflective point spacing between the primary reflective point and the secondary reflective point and comparison with the reflective point spacing threshold. If it exceeds the reflective point spacing threshold, return to step C21 for re-extraction until it meets the reflective point spacing threshold and output as the baseline features of the dual corneal reflective points.

[0054] Specifically, to extract the baseline features of the corneal reflective points, in this embodiment, an adaptive threshold for brightness is first updated based on the region of interest (ROI) of the eye. The threshold is the average brightness of the current corneal region plus three times the standard deviation. Based on this adaptive threshold, the ROI is segmented to obtain multiple bright spot regions. Subsequently, the average brightness of each bright spot region is calculated, and the two brightest spots are selected as the primary reflective point P1(x1, y1) and the secondary reflective point P2(x2, y2).

[0055] Subsequently, to avoid misidentifying other high-brightness areas, the distance between the primary and secondary reflective points was verified. If it exceeded the range, it was extracted again to ensure that the reflective points were effective corneal reflections.

[0056] In one embodiment, such as Figure 6 As shown, in step S2, the registration process includes: step D21: establishing a reference coordinate system with the midpoint of the reference feature of the corneal reflection point as the origin; step D22: mapping the iris ring texture feature and the pupil edge key point feature to the reference coordinate system; step D23: converting the points in the reference coordinate system to polar coordinates to form multimodal features.

[0057] Specifically, to facilitate subsequent model capture of the relative positional relationships of each feature, this embodiment also registers the above features to form multimodal features, including: A two-dimensional rectangular coordinate system is established with the midpoint O ((x1+x2) / 2, (y1+y2) / 2) between the main reflective point P1 and the secondary reflective point P2 as the origin, the line connecting P1 and P2 as the positive X-axis, and the positive Y-axis perpendicular to the X-axis upward as the positive Y-axis.

[0058] Subsequently, the iris ring texture features and pupil edge keypoint features were mapped to a reference coordinate system. The iris ring texture features were based on the centroid coordinates of each ring region, while the pupil edge keypoints were directly mapped to the reference coordinate system. The coordinates of the eight effective keypoints at the pupil edge and the centroids of the four ring regions of the iris texture were transformed from image pixel coordinates to the aforementioned reference coordinate system. Furthermore, the keypoint and texture centroid coordinates in the reference coordinate system were converted to polar coordinates (ρ, θ), where ρ is the distance from the point to the origin O, and θ is the angle between the point and the positive X-axis (range -180° to +180°).

[0059] In one embodiment, such as Figure 7 As shown, step S3 includes: step S31: performing rotational deviation quantization based on multimodal features to construct a deviation feature vector; step S32: inputting the deviation feature vector into a pre-trained random forest regression model to obtain the predicted pupil rotation angle and pupil direction.

[0060] Specifically, to facilitate the model's measurement of deviations, in this embodiment, based on the extracted baseline state image, the multimodal features are quantized by rotation deviation in combination with the baseline state image to construct a deviation feature vector.

[0061] Specifically, for key points at the pupil edge, the polar angle θi of each key point in the current frame is compared with the polar angle θi0 of the reference state to obtain the angle deviation Δθi=θi-θi0 of a single key point. A total of 8 Δθi (i=1 to 8) are calculated.

[0062] To address the iris ring texture deviation, the deviation Δφj = φj - φj0 between the polar angle φj of the texture centroid of each iris ring region in the current frame and the reference state polar angle φj0 is calculated, and a total of 4 Δφj (j = 1 to 4) are calculated.

[0063] Then, the two sets of deviations are weighted and fused. Considering that the stability of the key points at the pupil edge is better than that of the iris texture, they are assigned a weight of 0.6, while the iris texture is assigned a weight of 0.4. The total angular deviation Δθ is calculated by weighted averaging, using the following formula: Δθ = 0.6×(Δθ1+Δθ2+...+Δθ8) / 8 + 0.4×(Δφ1+Δφ2+Δφ3+Δφ4) / 4; Δθ1 to Δθ8 correspond to the angular deviations of eight individual key points, and Δφ1 to Δφ4 correspond to the polar angle deviations of the centroids of the textures in the four iris annular regions.

[0064] Subsequently, the eight Δθi and four Δφj are concatenated in sequence to construct a 12-dimensional bias feature vector, which is used as input for subsequent models.

[0065] The prediction model is a pre-trained random forest model that can predict the pupil rotation angle based on the given feature vector.

[0066] The model training process includes: (1) Sample collection: 100 subjects of different ages (5-60 years old) and different iris colors (black, brown, blue) were selected. Under different lighting conditions (10 lux, 20 lux, 30 lux, 40 lux, 50 lux), the pupils were simulated to rotate (angle range -15° to +15°, interval 0.5°) by controlling the precision rotation platform. 40 frames of images were collected under each condition, and a total of 20,000 frames of samples were obtained. (2) Labeling: The rotation angle of each frame sample is labeled using a combination of manual labeling and machine-assisted calibration, with a labeling error of <0.1°; (3) Model Training: A random forest regression model is constructed using the Python Scikit-learn library (version 0.24.2 is recommended, as it is stable, compatible, and suited to the feature dimensions and sample size of this solution). Scikit-learn is a mainstream open-source machine learning library in the Python ecosystem, providing a fully encapsulated random forest regression interface. It supports the entire process of custom core parameters, cross-validation, and model evaluation, enabling rapid model training and optimization without the need to develop an algorithm framework from scratch, significantly improving development efficiency and result reproducibility. This solution relies on the `RandomForestRegressor` class in the core dependency library to construct the basic model. At the same time, the `train_test_split` function is used to split the dataset, and the `cross_val_score` function is used to complete 5-fold cross-validation. The specific optimization logic follows the principle of "accuracy-complexity-real-time balance" and uses the "control variable method + 5-fold cross-validation" to determine the optimal value. Details are as follows: ① Optimization of the number of decision trees: This has a core impact on model fitting ability and inference efficiency. The test range was 50, 100, 150, 200, and 250 trees, with other parameters fixed (maximum depth 10, minimum number of splits 2, minimum number of leaf nodes 1). Five-fold cross-validation revealed that when the number of trees was ≤100, the mean absolute error (MAE) of the test set decreased significantly with increasing tree count (MAE = 0.68° for 50 trees, MAE = 0.42° for 100 trees); with 150 trees, the MAE decreased to 0.32°; with >150 trees, the MAE decrease was <0.03° (MAE = 0.30° for 200 trees), but the model inference latency increased from 8ms to 15ms (exceeding the system's real-time requirement of <10ms). Therefore, the optimal number of decision trees was determined to be 150, balancing accuracy and real-time performance.

[0067] ② Maximum Depth Optimization: Core Suppression of Overfitting. The test range was 5, 8, 10, 12, and 15, with a fixed number of decision trees (150) and other parameters. Results showed that: when depth ≤ 8, the model underfitted (MAE = 0.51° at depth 5); when depth = 10, MAE = 0.32° and the difference between the training and test set MAE was < 0.05° (no significant overfitting); when depth > 10, the test set MAE increased (MAE = 0.38° at depth 15), indicating overfitting. Therefore, the maximum depth was determined to be 10.

[0068] ③ Minimum Split Number / Leaf Node Optimization: This is the core optimization for improving generalization ability. The minimum split number was tested in the range of 2-6, and the minimum leaf node number in the range of 1-4, with the above optimal parameters fixed. Results show that when the minimum split number = 2 and the minimum leaf node number = 1, the model has the smallest generalization error on the test set (MAE = 0.35° in cross-population testing); if the values ​​increase (e.g., split number = 6, leaf node number = 4), the model underfits (MAE = 0.45°). Considering the large sample size of 20,000 frames in this scheme, the optimal values ​​were determined to be 2 and 1 respectively.

[0069] The final optimized model parameters are: 150 decision trees, maximum depth 10, minimum number of sample splits 2, and minimum number of sample leaf nodes 1. The parameters are optimized using 5-fold cross-validation. After training, the mean absolute error on the test set is 0.32°, and the inference latency is <10ms, which meets the requirements of high accuracy and real-time performance. (4) Model update: The system supports online training with new samples. When the number of new samples is ≥1000 frames, the model will be automatically updated to improve the adaptability to special groups (such as those with very light iris pigment) and special scenarios (such as extreme low light).

[0070] After obtaining the pupil rotation angle, the pupil rotation direction is determined as follows: 1. Statistical analysis of the consistency of positive and negative directions of Δθi and Δφj: If Δθ>0, and ≥80% of Δθi and Δφj are both>0, it indicates that the feature as a whole is shifted in the counterclockwise direction, and it is initially determined to be a counterclockwise rotation; 2. If Δθ < 0, and ≥ 80% of Δθi and Δφj are both < 0, it indicates that the feature as a whole is shifted in the clockwise direction, and it is preliminarily determined to be a clockwise rotation; 3. If the orientation consistency is less than 80% (e.g., local occlusion causes abnormal shift of some features), the rotation direction of the previous frame will be used to avoid misjudgment of orientation caused by noise in a single frame.

[0071] In one embodiment, such as Figure 8 As shown, step S4 includes: step S41: filter the pupil rotation angles of the previous few frames and calculate the historical average, then perform weighted fusion of the historical average and the current pupil rotation angle to obtain the actual pupil rotation angle output; step S42: perform direction calibration based on the historical average and the actual pupil rotation angle to output the pupil direction.

[0072] Specifically, considering the issue of single-frame recognition error, this embodiment also filters the pupil rotation angles of the previous few frames and calculates the historical average. Then, it performs a weighted fusion of the historical average and the current pupil rotation angle to obtain the actual pupil rotation angle output. This includes: calling the effective rotation angles (θ1, θ2, θ3) of the previous 3 frames stored in the inter-frame buffer, calculating the average value θ_avg=(θ1+θ2+θ3) / 3; and using the weighted fusion formula to calculate the final angle: θ_final=0.8×θ_current +0.2×θ_avg (α=0.8, balancing the accuracy of the current frame with historical stability).

[0073] θ_final is the actual pupil rotation angle, θ_current is the current pupil rotation angle, and θ_avg is the historical average.

[0074] Then, orientation calibration is performed using the following methods, including: If the initial orientation of the current frame is inconsistent with the orientation of the previous two frames, and |θ_current - θ_avg| < 1°, it is determined to be inter-frame noise, and the orientation of the previous frame is retained; if |θ_current - θ_avg| ≥ 1°, it indicates that there is a true orientation change (such as rapid eye movement), the orientation of the current frame is retained and marked as an "orientation change frame". And the process of removing abnormal frames includes: if the absolute value of the deviation between θ_current and θ_avg is greater than 3°, it is judged as an abnormal frame (such as severe occlusion or device jitter), the initial value of the current frame is discarded, θ_avg is used as the final angle of the current frame, and the abnormal frame information is recorded for subsequent system optimization.

[0075] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for pupil rotation angle and direction recognition based on multi-feature fusion, characterized in that, include: Step S1: Acquire an eye image of the user and locate the region of interest in the eye; the eye image includes the user's pupil, iris, and corneal reflective points; Step S2: Iris ring texture features, pupil edge key point features and dual corneal reflective point reference features are segmented from the region of interest of the eye and registered as multimodal features; Step S3: Predict the pupil rotation angle and pupil direction based on the multimodal features; Step S4: Combine the pupil rotation angle and pupil direction according to the recognition results of multiple detection frames and output them.

2. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, Step S1 includes: Step S11: acquiring the eye image of the user; Step S12: preprocessing the eye image to obtain a preprocessed image; Step S13: identifying the region of interest of the eye from the preprocessed image.

3. The pupil rotation angle and direction recognition method according to claim 2, characterized in that, The preprocessing steps in step S12 include: Gaussian filtering for noise reduction, grayscale conversion, and illumination equalization.

4. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, In step S2, the first extraction process of the iris ring texture feature includes: Step A21: using the Canny algorithm to perform edge detection on the region of interest in the eye to segment the outer boundary and inner boundary of the iris and form the iris region; Step A22: converting the iris region to polar coordinates and segmenting it to obtain four ring regions; Step A23: extracting 16-dimensional LBP features and 32-dimensional grayscale histogram features for each ring region, and then stitching them together to obtain the iris ring texture feature.

5. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, In step S2, the second extraction process of the pupil edge key point features includes: step B21: using the least squares ellipse fitting algorithm to fit the pupil contour to the region of interest of the eye; step B22: applying the Shi-Tomasi corner detection algorithm to the pupil contour to extract the pupil edge key points; step B23: calculating the center distance of each pupil edge key point relative to the center of the pupil contour, removing outliers of the center distance, and adding the remaining pupil edge key points to the pupil edge key point features.

6. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, In step S2, the third extraction process of the corneal reflective point reference feature includes: step C21: adaptively segmenting and extracting the bright spot region from the region of interest in the eye; step C22: calculating the average brightness of the bright spot region and extracting the two bright spot regions with the highest brightness as the primary reflective point and the secondary reflective point; step C23: calculating the reflective point distance between the primary reflective point and the secondary reflective point and comparing it with the reflective point distance threshold. If it exceeds the reflective point distance threshold, return to step C21 for re-extraction until it meets the reflective point distance threshold and then output it as the corneal reflective point reference feature.

7. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, In step S2, the registration process includes: step D21: establishing a reference coordinate system with the midpoint of the reference feature of the corneal reflective point as the origin; step D22: mapping the iris ring texture feature and the pupil edge key point feature to the reference coordinate system; step D23: converting the points in the reference coordinate system to polar coordinates to form the multimodal feature.

8. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, Step S3 includes: Step S31: Perform rotation deviation quantization based on the multimodal features to construct a deviation feature vector; Step S32: Input the deviation feature vector into a pre-trained random forest regression model to obtain the predicted pupil rotation angle and pupil direction.

9. The pupil rotation angle and direction recognition method according to claim 1, characterized in that, Step S4 includes: Step S41: Filter the pupil rotation angles of the previous few frames and calculate the historical average, then perform weighted fusion of the historical average and the current pupil rotation angle to obtain the actual pupil rotation angle output; Step S42: Perform direction calibration based on the historical average and the actual pupil rotation angle to output the pupil direction.

Images