Eyeball pose recognition method based on neural network

Abstract

The invention relates to the field of eyeball pose estimation, in particular to an eyeball pose recognition method based on a neural network, and the method comprises the steps: S1, making an eyeball model; s2, shooting a picture of an eyeball body in the eyeball model to complete eyeball body posture image data acquisition; s3, enhancing and processing the attitude image data; s4, building a neural network model; s5, inputting the processed attitude image data into a neural network model, and training and verifying the neural network model; s6, feature matching of the eyeball model and a real human eye is carried out; and the posture estimation of the real human eyes is realized.

Description

Technical Field

[0001] This invention relates to the field of eye pose estimation, and more specifically to an eye pose recognition method based on neural networks. Background Technology

[0002] In the era of information technology and AI, the construction of smart healthcare is a task and foundation for the high-quality development of medical services. The deep integration of artificial intelligence and massive big data constructs cloud data centers, building panoramic intelligent scenarios. 5G and IoT technologies jointly realize information interaction functions. Simultaneously, various fully functional new medical devices and new technologies such as AI models and algorithms are applied to medical scenarios and hospital management. These three aspects represent the key challenges in the current construction of smart healthcare. The technical solutions proposed in this paper address many problems faced in ocular surgery and treatment in medical scenarios, such as the issue of micro-movements of the eyeball during surgery, how to propose personalized treatment plans for different cases, and the issue of patient eyeball training and testing. Furthermore, highly robust tracking technology can better handle natural blinking, brief visual deviations, or slight head movements; rapid, non-contact measurement can shorten the time of routine ophthalmological examinations (such as strabismus screening and eyeball protrusion measurement), improving outpatient efficiency; and non-invasive methods that do not require special devices (such as certain contact electrodes or coils) improve patient comfort and acceptance. These advantages help to further enhance the patient experience.

[0003] In the field of eye pose estimation, in 2010, Schnieders et al. from the University of Hong Kong reconstructed an eye model using a single image and calculated the intersection of binocular vectors. In 2019, Dierkes, Swirski et al. used images from multiple perspectives to obtain geometric information about the eye, and then estimated the shape and position of the eye through mathematical modeling and optimization algorithms. Professor Wolfgang Fuhl's team at the University of Tübingen in Germany employed orbit estimation and aperture prediction techniques to segment the orbit and iris to predict the gaze area, and annotated TEyeD, the world's largest public dataset of eye images captured using a head-mounted device.

[0004] Acquiring eye pose presents numerous challenges. First, eye tracking involves processing and analyzing eye images, which are often affected by ambient lighting and rapid eye movements, resulting in poor image quality. Second, the complex shape of the eye makes it difficult for traditional image processing methods to accurately extract the eye contour, while existing eye recognition methods often suffer from insufficient accuracy. Furthermore, current eye pose recognition methods require high computational complexity, making it difficult to meet the demands of low latency and high frame rates. Summary of the Invention

[0005] The purpose of this invention is to provide a neural network-based eye pose recognition method to achieve pose estimation of real human eyes.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A neural network-based eye pose recognition method, comprising the following steps:

[0008] Step S1: Create an eyeball model;

[0009] Step S2: Take pictures of the eyeball in the eyeball model to complete the acquisition of eyeball pose image data;

[0010] Step S3: Enhancement and processing of pose image data;

[0011] Step S4: Build the neural network model;

[0012] Step S5: Input the processed pose image data into the neural network model to train and validate the neural network model;

[0013] Step S6: Match the features of the eyeball model with those of a real human eye;

[0014] The eyeball model includes a support, which is divided into an upper support and a lower support. A spherical groove capable of covering the eyeball is provided between the lower support and the upper support. The eyeball is installed in the spherical groove. A gyroscope is fixedly connected to the bottom of the eyeball. An eye surface for simulating the upper and lower eyelids is provided on the upper support.

[0015] The upper and lower support seats are provided with vertically arranged limiting grooves on their sides, and a limiting rod is fixedly connected to the side of the eyeball. The limiting rod is fitted with the limiting groove with a gap, and the limiting rod limits the eyeball, so that the eyeball imitates the human eyeball and has only two degrees of freedom in the direction of rotation.

[0016] In step S2, an image of the eyeball is captured and the current posture angle of the eyeball is obtained. During the capture, the gyroscope is connected to the host computer to read the posture angle of the eyeball in real time, and the camera is turned on to capture an image of the eyeball.

[0017] The enhancement of the posture image data in step S3 is achieved by taking pictures of the eyeball rotating and translating in the x, y, and z axes.

[0018] The process of processing the posture image data in step S3 is as follows: to obtain the posture image data, it is necessary to extract the feature regions of the posture image data, convert the posture image data from the RGB color space to the HSV color space, perform threshold segmentation to extract the feature regions, perform dilation and erosion operations to smooth the edges of the regions, and finally convert them into different grayscale values ​​according to different colors to distinguish different features, normalize them into grayscale images, crop out the feature regions and center them to reduce the amount of data, and obtain a standard image of the posture image data.

[0019] In step S4, the neural network model is constructed as follows from front to back: the original image with 1 channel * length 128 * width 128 is processed by convolutional layers and pooling to a channel count of 64 * length 64 * width 64, two residual layers with a channel count of 64 * length 64 * width 64, convolutional layers with doubled channel count and pooling to a channel count of 128 * length 32 * width 32, two residual layers with a channel count of 128 * length 32 * width 32, finally flattened and processed by two fully connected layers with a channel count of 256 * length 32 * width 32 to a channel count of 512 and then two pose outputs, finally obtaining the nonlinear relationship between the image feature information of the pose image data and the human eye pose;

[0020] The training process of the neural network model in step S5 is as follows: During each training session, the dataset is randomly shuffled and divided into a training set and a validation set in a 4:1 ratio; the model processes 64 data points per training batch, with a maximum of 50 training batches per batch, and the initial learning rate is controlled by start...

[0021] The learning rate is set to 0.001. The cosine annealing algorithm is used to dynamically adjust the learning rate. The training termination condition is that if the training effect of the neural network model is not significantly improved after 8 training batches, the training will be terminated.

[0022] Cosine annealing is a method used in neural network training to change the learning rate. The learning rate affects the magnitude of the model change at each step. The algorithm essentially defines an upper limit learning rate (0.005 in this invention) and a lower limit learning rate (0.001 in this invention), and then performs a cosine function transformation according to a certain period (10 learning cycles in this invention). This ensures that the learning rate of each model update is different, making it less likely to get trapped in the local minimum of the reward function.

[0023] The verification process of the neural network model in step S5 is as follows: the trained neural network model is applied to the real-time eye model, and the real data of the eye model is compared with the data predicted by the neural network model to verify the error range.

[0024] In step S6, the process of matching the eyeball model with the real human eye features is as follows: First, semantic segmentation is performed on the intraoperative human eye image using UNet. Then, an ellipse fitting algorithm is used to fit the boundary of the segmented eyeball region to determine the geometric parameters of the iris. Color thresholding is used to segment the upper and lower eyelids, iris, and pupil regions. The geometric parameters of the iris are used to register with the iris region in the eyeball model to obtain the actual ratio between the real patient's eye and the eyeball model. Finally, by extracting and matching the dense feature points of the feature regions in the eyeball model pose image and the intraoperative image into multiple sets of feature point pairs, the feature regions in the eyeball model pose image are transferred to the real intraoperative human eye image, realizing feature matching with the real human eye image, enabling the neural network model to directly process the real human eye image.

[0025] The ellipse fitting algorithm is as follows: The human eye model image is segmented using a three-channel threshold method to convert regions of different colors into grayscale blocks. This method is used to convert the image into a grayscale image. Then, a Gaussian blur algorithm and a Canny algorithm are used to detect each contour. Contours that meet the size requirements are extracted for subsequent processing. Then, the circular Hough transform principle is used to detect the length of the major and minor axes of the contours. The 10 points on the edge are randomly checked to see if they approximately satisfy the ellipse equation. The ellipse in the image is fitted, which also verifies the position and size of the iris and pupil in the human eye model.

[0026] The beneficial effects of this invention are as follows:

[0027] By training a neural network model using eye model images and pose data, the neural network model can directly obtain the human eye pose from human eye images, achieving high-precision human eye pose detection, which can provide necessary information for future automated ophthalmic surgery.

[0028] Compared to existing methods that require wearable devices for eye tracking, this invention uses data from a fabricated eye model to train a neural network model, enabling the determination of eye posture directly from images of the human eye during surgery using only a camera, thus reducing costs and making the procedure more convenient.

[0029] Compared to methods that require strictly fixing the device position, the versatility of this design can be greatly improved due to data augmentation. It can recognize human eye posture from various angles, making it easier for doctors who are not familiar with the engineering principles to get started.

[0030] Neural network models can be transferred to any device with a camera without complicated preparation, and can be updated and upgraded, giving them greater development potential. Attached Figure Description

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and specific implementation methods.

[0032] Figure 1 This is a schematic diagram of the eyeball model structure of the present invention;

[0033] Figure 2 This is a schematic diagram of the eyeball model structure of the present invention;

[0034] Figure 3 This is the process of capturing eyeball pose data according to the present invention;

[0035] Figure 4 This is the original image of the eyeball of the present invention;

[0036] Figure 5 This is the standard image of the present invention;

[0037] Figure 6 This is a schematic diagram of the enhanced dataset of the present invention;

[0038] Figure 7 This is a schematic diagram of the neural network model of the present invention;

[0039] Figure 8 This is a schematic diagram of the training curve of the neural network model of the present invention;

[0040] Figure 9 This is a schematic diagram of a live demonstration of the eyeball model of the present invention;

[0041] Figure 10 This is a schematic diagram demonstrating the feature matching algorithm between the model and the real human eye of the present invention;

[0042] Figure 11 This is a schematic diagram of the eyeball model of the present invention along the X, Y, and Z axes;

[0043] Figure 12 This is a cross-sectional view of the eyeball model of the present invention.

[0044] In the diagram: eyeball 1-1; eye surface 1-2; limiting rod 1-3; gyroscope 1-4; receiving seat 1-5; screw 1-6. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings.

[0046] like Figures 1 to 12 As shown, in order to achieve the technical effect of "realizing the pose estimation of real human eyes", the steps and functions of an eye pose recognition method based on neural networks are explained in detail below.

[0047] A neural network-based eye pose recognition method, comprising the following steps:

[0048] Step S1: Create an eyeball model;

[0049] Step S2: Take a picture of eyeball body 1-1 in the eyeball model to complete the acquisition of eyeball body 1-1 pose image data;

[0050] Step S3: Enhancement and processing of pose image data;

[0051] Step S4: Build the neural network model;

[0052] Step S5: Input the processed pose image data into the neural network model to train and validate the neural network model;

[0053] Step S6: Match the features of the eyeball model with those of a real human eye;

[0054] like Figure 1 and Figure 2 As shown, the eyeball model includes a support 1-5, which is divided into an upper support and a lower support. A spherical groove that can cover the eyeball 1-1 is provided between the lower support and the upper support. The eyeball 1-1 is installed in the spherical groove. A gyroscope 1-4 is fixedly connected to the bottom of the eyeball 1-1. An eye surface 1-2 for simulating the upper and lower eyelids is provided on the upper support.

[0055] The upper and lower support seats are provided with vertically arranged limiting grooves on their sides. A limiting rod 1-3 is fixedly connected to the side of the eyeball 1-1. The limiting rod 1-3 is fitted with the limiting groove with a clearance. The limiting rod 1-3 limits the eyeball 1-1, so that the eyeball 1-1 imitates the human eyeball and has only two degrees of freedom in the direction of rotation.

[0056] To best recreate the ophthalmic surgical scene and record posture information, a replica was created at a 1:3 scale. Figure 1 and Figure 2 The image shows an eyeball model. The following is an explanation of the eyeball model: To mimic the smooth rotation of the human eyeball, an eyeball body 1-1 was designed based on a double-sphere model, and an eye surface 1-2 and a receiving seat 1-5 with a spherical groove were designed based on the upper and lower eyelids of the human eye. To replicate the human eye's rotational freedom in only two directions, a limiting rod 1-3 was designed to lock one rotational degree of freedom. Finally, a magnetically calibrated three-axis gyroscope 1-4 was installed inside the eyeball body 1-1. The magnetically calibrated three-axis gyroscope 1-4 is connected to a host computer via a serial port to transmit the attitude data of the eyeball body 1-1, including the deflection angle and position of the eyeball body 1-1.

[0057] Color stickers that can be clearly distinguished are affixed to the simulated eyelid part of the eye surface 1-2 and the simulated iris and pupil part of the eyeball 1-1 for subsequent analysis. The eyeball model reproduces the scene of the human eye rotating to various angles in actual surgery and can record the posture data of the eyeball 1-1.

[0058] In step S2, an image of the eyeball 1-1 is captured and the current posture angle of the eyeball 1-1 is obtained. During the capture, the gyroscope 1-4 is connected to the host computer to read the posture angle of the eyeball 1-1 in real time, and the camera is turned on to capture an image of the eyeball 1-1.

[0059] This step requires taking corresponding images of the eye model and its current posture angle. Since the patient's eyeball is usually located directly below the surgeon during surgery, the camera is positioned directly above the eye model during data acquisition. During the shooting process, gyroscopes 1-4 are first connected to the host computer to read the posture angle in real time, while the camera is simultaneously turned on to photograph the eye model. First, rotate eyeball 1-1 to the upper left position and record the image and posture data of the eyeball model in this state. Then, continue rotating eyeball 1-1 slightly by about 2° each time, from left to right and from top to bottom. Specifically, to handle situations where eyeball 1-1 is obstructed by the device, a series of images of eyeball 1-1 when obstructed are also recorded. Repeat the above process, recording a total of 400 sets of eyeball model images and posture data. To enhance the applicability of this human eye posture recognition method, the angle between the camera and the eyeball model needs to be adjusted along the x-axis to -10°, -20°, 0°, 10°, and 20°, and then the angle along the y-axis to -10°, -20°, 0°, 10°, and 20°, and then the images need to be taken again. Finally, 4000 sets of data are recorded as raw data. The shooting process is as follows: Figure 3 As shown, the original image is as follows Figure 4 As shown;

[0060] like Figure 11 As shown, the enhancement of the pose image data in step S3 involves taking images of the eyeball 1-1 by rotation and translation along the x, y, and z axes. During data acquisition, to enhance applicability, rotation of the shooting angle along the x and y axes is performed. Next, rotation along the z axis and translation in all three directions are required. Therefore, a program was written to process 4000 standard images. For scaling factors (z-axis translation), three intervals were selected: 0.5–1.5, 0.7–1.3, and 0.9–1.1. For rotation (z-axis rotation), two intervals were selected: -15°–15° and -30°–30°. For x and y axis translation, two intervals were selected: -300 pixels to 300 pixels and -600 pixels to 600 pixels. Data was randomly selected within these intervals, and image operations were performed. Finally, 12 × 4000 extended data images were obtained, ultimately forming a dataset of 13 × 4000 data points. The enhanced dataset is shown below. Figure 6 As shown;

[0061] The processing procedure for the attitude image data in step S3 is as follows: To obtain the attitude image data, the feature regions of the attitude image data need to be extracted. The attitude image data is converted from the RGB color space to the HSV color space, threshold segmentation is performed to extract the feature regions, and then dilation and erosion operations are performed to smooth the edges of the regions. Finally, different colors are converted to different grayscale values ​​to distinguish different features, and the data is normalized to a grayscale image. The feature regions are then cropped and centered to reduce the amount of data, resulting in a standard image of the attitude image data. Figure 5 As shown;

[0062] In step S4, the neural network model is constructed as follows from front to back: the original image with 1 channel * length 128 * width 128 is processed by convolutional layers and pooling to a channel count of 64 * length 64 * width 64, two residual layers with a channel count of 64 * length 64 * width 64, convolutional layers with doubled channel count and pooling to a channel count of 128 * length 32 * width 32, two residual layers with a channel count of 128 * length 32 * width 32, finally flattened and processed by two fully connected layers with a channel count of 256 * length 32 * width 32 to a channel count of 512 and then two pose outputs, finally obtaining the nonlinear relationship between the image feature information of the pose image data and the human eye pose; Figure 7 The structure of the neural network model is shown, and the specific parameters and detailed descriptions of each layer are shown in Table 1.

[0063] Layer type parameter Output size illustrate Input none [1,128,128] Single-channel image Conv2d+BN+ReLU 64 filters, 3×3 [64,128,128] Convolution processing MaxPool2d kernel_size = 2 [64,64,64] Pooling ResBlock×2 64 channels [64,64,64] Residual connection Conv2d+BN+ReLU 128 filters, 3×3 [128,64,64] Convolution processing MaxPool2d kernel_size = 2 [128,32,32] Pooling ResBlock×2 128 channels [128,32,32] Residual connection Conv2d+BN+ReLU 256 filters, 3×3 [256,32,32] Final feature map FullyConnected 256*32*32*512*2 parameters Two pose parameters Fully connected

[0064] The training process of the neural network model in step S5 is as follows: During each training session, the dataset is randomly shuffled and divided into a training set and a validation set in a 4:1 ratio. The model processes 64 data points per training batch, with a maximum epoch of 50 batches. The initial learning rate is set to 0.001, and cosine annealing is used to dynamically adjust the learning speed. Training is terminated if the neural network model's training performance does not significantly improve after 8 training batches. The algorithm has been validated in an environment with an i7-12700K CPU and a GeForce RTX Nvidia 4060 GPU. After training, the training set loss gradually converged from 175 to approximately 1.2°, reaching the expected target range. The validation set loss converged to 1.09°. The training curve is shown below. Figure 8 As shown;

[0065] The verification process of the neural network model in step S5 is as follows: The trained neural network model is applied to a real-time eye model, and the actual data from the eye model is compared with the data predicted by the neural network model to verify the error range. The trained model is then applied to a real-time human eye model, and the actual data from 5000 frames of attitude sensor data is compared with the model's predicted data. The average two-axis attitude error is 1.57°, achieving a relatively ideal effect. A live demonstration is shown below. Figure 9 As shown;

[0066] In step S6, the process of matching the eyeball model with real human eye features is as follows: First, semantic segmentation is performed on the intraoperative human eye image using UNet. An ellipse fitting algorithm is used to fit the boundaries of the segmented eyeball region to determine the geometric parameters of the iris. Color thresholding is used to segment the upper and lower eyelids, iris, and pupil regions. The geometric parameters of the iris are then used to register with the iris region in the eyeball model, thus obtaining the actual ratio between the real patient's eye and the eyeball model. Finally, dense feature points in the feature regions of the eyeball model's pose image and the intraoperative image are extracted and matched into multiple sets of feature point pairs. The feature regions in the eyeball model's pose image are then transferred to the real intraoperative human eye image, achieving feature matching with the real human eye image. This allows the neural network model to directly process real human eye images. The algorithm is demonstrated as follows: Figure 10 As shown, blue, green, and yellow represent the corresponding matching of eyelids, irises, and pupils, respectively, while red represents the missing areas in the real image after processing, which are approximated by the neural network.

[0067] Current methods for human eye posture detection are mostly used in human-computer interaction in virtual reality and emotion detection in behavioral testing. Essentially, these methods track the gaze, but all require a device to collect information. For example, the integrated gaze tracking technology based on eye positioning and helmet positioning developed by Xi'an North Optoelectronic Technology Defense Co., Ltd., and the eye-tracking technology based on an electrostatic interface developed by the Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, both require subjects to wear glasses-like instruments. These methods are unsuitable for ophthalmic surgery and have limitations in accuracy. In contrast, this design proposes using a human eye model that reflects intraoperative eye characteristics and has precise posture reading capabilities as a medium to combine intraoperative eye posture with intraoperative eye images. This requires no external equipment and utilizes neural network algorithms to complete intraoperative eye posture detection. It is not only low-cost and easy to operate, but also specifically designed for intraoperative situations, providing more comprehensive information and higher accuracy.

[0068] like Figure 8 and Figure 9 As shown below, the actual use process of the present invention will be described in detail;

[0069] A dataset is created by acquiring an image (whose feature regions have been segmented using image processing algorithms) and simultaneously reading the pose sensor data from the human eye model. For example, "21.jpg" in the folder "Original Sample Set" corresponds to the data with ID 21 in "Pose Data.csv". Angle_x and Angle_y represent the angles of rotation about the x and y axes, respectively -15.227° and 4.614°.

[0070] The image set was augmented using random coefficients within a certain range. For example, "21.jpg" was transformed into "36021.jpg" by randomly selecting two numbers, 0.93 and -8.21°, from the range of "scaling factor 0.7 to 1.3 and rotation angle -30° to 30°".

[0071] Two numbers, 0.97 and -3.59°, were randomly selected from the range of "scaling factor 0.9 to 1.1 and rotation angle -15° to 15°" to create "44021.jpg" in "Data Augmentation Dataset 2".

[0072] Since the images were only augmented under the same conditions, the two-axis pose data corresponding to "36021.jpg" and "44021.jpg" are still -15.227° and 4.614°, which are the same as those corresponding to "21.jpg". Similarly, the 4000 images in the entire dataset were processed.

[0073] The 4000*13 (4000 sets of labels repeated 13 times) labels corresponding to 52000 images were fed into the neural network for training, and the training curve was obtained as follows. Figure 8 As shown;

[0074] A single test in a real machine, such as Figure 9 As shown in the image, "Prediction X / Y" represents the two-axis rotation angles predicted by the model for the human eye model; "RealX / Y" represents the two-axis rotation angles transmitted back in real time by the attitude sensor, both in degrees.

Claims

1. A method for eye pose recognition based on neural networks, characterized in that: The method includes the following steps: Step S1: Create an eyeball model; Step S2: Take a picture of the eyeball body (1-1) in the eyeball model to complete the acquisition of the pose image data of the eyeball body (1-1); Step S3: Enhancement and processing of pose image data; Step S4: Build the neural network model; Step S5: Input the processed pose image data into the neural network model to train and validate the neural network model; Step S6: Match the features of the eye model with those of a real human eye.

2. The eye pose recognition method based on a neural network according to claim 1, characterized in that: The eyeball model includes a support (1-5), which is divided into an upper support and a lower support. A spherical groove is provided between the lower support and the upper support to cover the eyeball (1-1). The eyeball (1-1) is installed in the spherical groove. A gyroscope (1-4) is fixedly connected to the bottom of the eyeball (1-1). An eye surface (1-2) for simulating the upper and lower eyelids is provided on the upper support.

3. The eye pose recognition method based on a neural network according to claim 2, characterized in that: The upper and lower support seats are provided with vertically arranged limiting grooves on their sides. A limiting rod (1-3) is fixedly connected to the side of the eyeball (1-1). The limiting rod (1-3) is fitted with the limiting groove with a clearance. The limiting rod (1-3) limits the eyeball (1-1) so that the eyeball (1-1) mimics the human eyeball and has only two degrees of freedom in the direction of rotation.

4. The eye pose recognition method based on a neural network according to claim 2, characterized in that: In step S2, an image of the eyeball (1-1) is captured and the current posture angle of the eyeball (1-1) is obtained. During the capture, the gyroscope (1-4) is connected to the host computer to read the posture angle of the eyeball (1-1) in real time, and the camera is turned on to capture an image of the eyeball (1-1).

5. The eye pose recognition method based on a neural network according to claim 1, characterized in that: The enhancement of the posture image data in step S3 is to take pictures of the eyeball (1-1) rotating and translating in the x, y, and z axes.

6. The eye pose recognition method based on a neural network according to claim 1, characterized in that: The process of processing the attitude image data in step S3 is as follows: to obtain the attitude image data, it is necessary to extract the feature regions of the attitude image data, convert the attitude image data from the RGB color gamut to the HSV color gamut, perform threshold segmentation to extract the feature regions, then perform dilation and erosion operations to smooth the edges of the regions, finally convert them into different grayscale values ​​according to different colors to distinguish different features, normalize them into grayscale images, crop out the feature regions and center them to reduce the amount of data, and obtain a standard image of the attitude image data.

7. The eye pose recognition method based on a neural network according to claim 1, characterized in that: In step S4, the neural network model is constructed as follows from front to back: the original image with 1 channel * length 128 * width 128 is processed by convolutional layers and pooling to a channel count of 64 * length 64 * width 64, followed by two residual layers with a channel count of 64 * length 64 * width 64, then convolutional layers and pooling to a channel count of 128 * length 32 * width 32, followed by two residual layers with a channel count of 128 * length 32 * width 32, finally flattened and processed by two fully connected layers with a channel count of 256 * length 32 * width 32 to a channel count of 512, and then two pose outputs, finally obtaining the nonlinear relationship between the image feature information of the pose image data and the human eye pose.

8. The eye pose recognition method based on a neural network according to claim 1, characterized in that: The training process of the neural network model in step S5 is as follows: During each training session, the dataset is randomly shuffled and divided into a training set and a validation set in a 4:1 ratio; the model processes 64 data points per training batch, with a maximum epoch of 50 training batches; the initial learning rate is controlled at start_learning_rate = 0.001; the learning speed is dynamically adjusted using a cosine annealing algorithm; and the training is terminated if the neural network model training effect is not significantly improved after 8 training batches.

9. The eye pose recognition method based on a neural network according to claim 8, characterized in that: The verification process of the neural network model in step S5 is as follows: the trained neural network model is applied to the real-time eye model, and the real data of the eye model is compared with the data predicted by the neural network model to verify the error range.

10. The eye pose recognition method based on a neural network according to claim 1, characterized in that: In step S6, the process of matching the eyeball model with the real human eye features is as follows: First, semantic segmentation is performed on the intraoperative human eye image using UNet. Then, an ellipse fitting algorithm is used to fit the boundary of the segmented eyeball region to determine the geometric parameters of the iris. Color thresholding is used to segment the upper and lower eyelids, iris, and pupil regions. The geometric parameters of the iris are used to register with the iris region in the eyeball model to obtain the actual ratio between the real patient's eye and the eyeball model. Finally, by extracting and matching the dense feature points of the feature regions in the eyeball model pose image and the intraoperative image into multiple sets of feature point pairs, the feature regions in the eyeball model pose image are transferred to the real intraoperative human eye image, realizing feature matching with the real human eye image, enabling the neural network model to directly process the real human eye image.

Images