Fundus imaging method and device based on three-dimensional eyeball tracking and automatic focusing

Abstract

The invention discloses a fundus imaging method and device based on three-dimensional eyeball tracking and automatic focusing, and the method comprises the steps: calculating the wavefront aberration through an image collected by a Shack-Hartmann wavefront sensor, obtaining a defocusing amount, carrying out the fitting, and constructing a movement curve of a stepping motor; a human eye pupil image collected by the binocular vision system is positioned, and pupil center coordinates are obtained; performing three-dimensional eyeball tracking coordinate conversion processing according to the pupil center coordinate to obtain a three-dimensional coordinate of a human eye eyeball; and controlling the binocular camera to perform automatic focusing according to the three-dimensional coordinates of the eyeballs of the human eyes, and controlling the binocular camera to perform mobile shooting according to the moving curve of the stepping motor to obtain a human eye fundus image. According to the invention, the definition of fundus imaging can be improved through automatic focusing between the camera and the pupil of the human eye and real-time three-dimensional tracking of the eyeball. The fundus imaging method and device based on three-dimensional eyeball tracking and automatic focusing can be widely applied to the technical field of fundus imaging.

Description

Technical Field

[0001] The present invention relates to the technical field of fundus imaging, and in particular to a fundus imaging method and device based on three-dimensional eye tracking and autofocus. Background Art

[0002] The state of retinal vasculature serves as a pathological basis for fundus health and common diseases. The human eye is a complex optical system, with non-uniform refractive indices within its various refractive media, and individual variations in surface curvature and thickness. Even a normal human eye inevitably exhibits certain aberrations, which severely impact fundus imaging quality. Designing a fundus camera for high-resolution fundus imaging requires not only correcting for these aberrations but also accounting for individual variability. Therefore, finely focusing the image is a critical step in fundus photography. Lack of focus often results in poor image quality. Autofocus allows for faster and more accurate acquisition of high-quality fundus images. To achieve autofocus on the fundus, compensating for these aberrations while ensuring universal applicability, relevant fundus imaging technologies include autofocus and three-dimensional eye tracking. Currently, fundus camera focusing methods primarily rely on contrast detection and phase detection. Focusing methods include depth-in-focus (DFF) and depth-from-focus (DFD). However, existing adaptive autofocus methods often use a Shack-Hartmann wavefront sensor as the wavefront detector and incorporate a wavefront corrector into the optical path design. These methods require calculating the diopter or fitting the wavefront and compensating for the diopter. These methods are highly complex in terms of optical path design, have a limited range of precise adjustment, and require high computational effort, resulting in slow focusing speeds. Three-dimensional eye tracking methods for fundus cameras include manual and fully automatic methods. Traditional fundus cameras primarily use manual centering, which is limited by subjective judgment and cumbersome to operate. Fully automatic fundus camera eye tracking methods primarily include triangulation and offset displacement. However, triangulation is significantly affected by environmental factors, and external light entering the system can interfere with the scattered bright spots formed by the external eye illumination source, leading to calculation errors. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a fundus imaging method and device based on three-dimensional eye tracking and automatic focusing, which can improve the clarity of fundus imaging of the human eye through automatic focusing between the binocular camera and the human eye pupil and real-time three-dimensional tracking of the eyeball.

[0004] The first technical solution adopted by the present invention is: a fundus imaging method based on three-dimensional eye tracking and autofocus, comprising the following steps:

[0005] The wavefront aberration is calculated from the images collected by the Shack-Hartmann wavefront sensor, the defocus amount is obtained and fitted, and the movement curve of the stepper motor is constructed;

[0006] Position the pupil image of the human eye collected by the binocular vision system and obtain the coordinates of the pupil center;

[0007] Perform three-dimensional eye tracking coordinate conversion processing according to the pupil center coordinates to obtain the three-dimensional coordinates of the human eye;

[0008] The binocular camera is controlled to automatically focus according to the three-dimensional coordinates of the human eyeball, and the moving curve of the stepping motor is combined to control the binocular camera to move and shoot, thereby obtaining the fundus image of the human eye.

[0009] Furthermore, the step of calculating the wavefront aberration using the image collected by the Shack-Hartmann wavefront sensor, obtaining the defocus amount and performing fitting, and constructing the movement curve of the stepper motor specifically includes:

[0010] The fundus of the human eye is illuminated by a light source to form a secondary light source, which is detected by a Shack-Hartmann wavefront sensor to obtain an image collected by the Shack-Hartmann wavefront sensor;

[0011] Setting a light speckle pattern of a reference wavefront, obtaining a measurement wavefront corresponding to an image acquired by a Shack-Hartmann wavefront sensor and performing segmentation processing to obtain a light speckle pattern of the measurement wavefront;

[0012] The centroid coordinates of the light speckle array pattern of the reference wavefront and the light speckle array pattern of the measurement wavefront are obtained by the centroid method and calculated to obtain the centroid offset;

[0013] The center-of-mass offset is expressed by expanding the Zernike polynomial to obtain the wavefront aberration between the reference wavefront and the measured wavefront.

[0014] Considering the power of the Shack-Hartmann wavefront sensor wavefront correction, the wavefront aberration between the reference wavefront and the measured wavefront is converted to obtain the defocus value;

[0015] The defocus amount and the movement value of the stepper motor are subjected to curve fitting processing to obtain the movement curve of the stepper motor.

[0016] Furthermore, the step of locating the pupil image of the human eye collected by the binocular vision system and obtaining the pupil center coordinates specifically includes:

[0017] Perform coarse positioning on the human pupil image collected by the binocular vision system to obtain the preliminary pupil area;

[0018] The preliminary pupil area is finely positioned to obtain the pupil center coordinates.

[0019] Furthermore, the step of roughly locating the pupil image of the human eye collected by the binocular vision system to obtain a preliminary pupil area specifically includes:

[0020] Downsampling the human eye pupil image collected by the binocular vision system to obtain a downsampled image;

[0021] Perform Gaussian smoothing and grayscale normalization on the downsampled image to obtain the preprocessed image;

[0022] The pre-processed image is extracted and processed through dynamic threshold binarization and connected domain analysis to obtain the centroid of the connected domain;

[0023] The centroid of the connected domain is used as the center of the rectangular frame, and is positioned and matched with the human pupil image collected by the binocular vision system to obtain a preliminary pupil area.

[0024] Furthermore, the step of performing fine positioning processing on the preliminary pupil area to obtain pupil center coordinates specifically includes:

[0025] Perform Gaussian smoothing and grayscale normalization on the preliminary pupil area to obtain the preprocessed pupil area;

[0026] The pupil area after preprocessing is processed by binarization and image subtraction to remove light spots and obtain the preliminary pupil connected area;

[0027] Perform image binarization and template matching on the preliminary pupil connected domain to obtain the pupil connected domain;

[0028] The edges of the pupil connected domain are reconstructed using the Sobel operator edge detection algorithm to obtain the pupil center coordinates.

[0029] Furthermore, the step of performing three-dimensional eye tracking coordinate conversion processing based on the pupil center coordinates to obtain the three-dimensional coordinates of the human eye specifically includes:

[0030] Obtain the horizontal distance between the pupil center coordinates and the binocular camera, and the vertical distance between the pupil center coordinates and the center point of the human eye pupil image collected by the binocular vision system;

[0031] Determine the optical center coordinates of the binocular camera based on the horizontal distance and vertical distance combined with the working distance, tilt angle and focal length of the binocular camera;

[0032] The equation of the straight line between the optical center coordinates of the binocular camera and the pupil center coordinates is determined and solved according to the similar triangle theorem to obtain the three-dimensional coordinates of the human eyeball.

[0033] The second technical solution adopted by the present invention is: a fundus imaging device based on three-dimensional eye tracking and autofocus, including a fundus imaging system, an autofocus system and a binocular positioning system, wherein:

[0034] The fundus imaging system includes an illumination system and an imaging system. The fundus imaging system is used to image the retina of the human eye and obtain a fundus image of the human eye;

[0035] The autofocus system is used to calculate the defocus amount of the retinal image acquired in the imaging system;

[0036] The binocular positioning system is used for three-dimensional eye tracking.

[0037] Furthermore, the imaging system includes an eyepiece objective lens, a focusing lens group, an imaging lens group, a beam splitter prism and a charge coupled device.

[0038] The beneficial effects of the method and device of the present invention are as follows: the present invention calculates the wavefront aberration through the image collected by the Shack-Hartmann wavefront sensor, obtains the defocus amount and performs fitting, constructs the movement curve of the stepper motor, and realizes the automatic focus by using the Shack-Hartmann wavefront sensor. Since the defocus amount is calculated and the relationship curve between the defocus amount and the movement of the stepper motor is used to guide the motor movement during the focusing process, only the defocus amount is calculated, and no additional correction system aberration is required. Only the stereo correction aberration of the single lens needs to be calculated. This greatly simplifies the calculation and improves the speed of automatic focus. Then, the human eye pupil image collected by the binocular vision system is positioned, the pupil center coordinates are obtained, and a method of coarse and fine positioning is implemented to achieve better The pupil that is disturbed by various interferences can be located at the center, and the interfering pupil light spot can be removed. The three-dimensional eye tracking coordinate conversion processing is further performed according to the pupil center coordinates to obtain the three-dimensional coordinates of the human eye. There is no need to calibrate the camera, and the origin of the world coordinate system is set to the intersection of the main optical axes of the two camera lenses. It is only necessary to control the installation tilt angle of the binocular camera so that the intersection of the main optical axes of the left and right camera lenses is set to the focal plane of the fundus camera to achieve centering focus. Finally, the binocular camera is controlled to automatically focus according to the three-dimensional coordinates of the human eye, and the binocular camera is controlled to move and shoot in combination with the moving curve of the stepping motor to obtain the fundus image of the human eye, thereby improving the imaging accuracy of the fundus image of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of the steps of a fundus imaging method based on three-dimensional eye tracking and autofocus of the present invention;

[0040] Figure 2 This is a structural diagram of a fundus imaging device based on three-dimensional eye tracking and autofocus according to the present invention;

[0041] Figure 3is a schematic diagram of the Shack-Hartmann wavefront detection principle provided by a specific embodiment of the present invention;

[0042] Figure 4 1 is a schematic diagram of a curve showing the relationship between the defocus degree of the Hartmann system and the stepping motor provided by a specific embodiment of the present invention;

[0043] Figure 5 This is a stereoscopic schematic diagram of a convergent binocular reconstruction plane coordinate experiment provided by a specific embodiment of the present invention;

[0044] Figure 6 This is a stereoscopic schematic diagram of a single-camera plane coordinate reconstruction experiment provided by a specific embodiment of the present invention;

[0045] Figure 7 1 is a schematic diagram of a processing flow for extracting pupil center coordinates provided by a specific embodiment of the present invention;

[0046] Figure 8 1 is a schematic diagram of the effect of coarse pupil positioning provided by a specific embodiment of the present invention;

[0047] Figure 9 Schematic diagram of the effect of pupil fine positioning provided by a specific embodiment of the present invention;

[0048] Figure 10 It is a flowchart of a method for reconstructing the three-dimensional coordinates of an eyeball and a camera provided by a specific embodiment of the present invention.

[0049] Reference numerals: 1. Intersection of the optical axes of the left and right cameras; 2. Camera lens; 3. Position of the center point of the eyeball to be tested imaged on the target surface; 4. Camera target surface; 5. Eyeball to be tested. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] There are some deficiencies in the related technology, such as:

[0052] 1) Contrast detection is susceptible to noise, resulting in low robustness. Focusing speed cannot meet requirements and it is difficult to achieve real-time performance. Contrast detection is also susceptible to noise interference, which can cause local extremes in the evaluation curve and lead to focus failure.

[0053] 2) Split-image focusing requires high costs and increases the volume. In addition, the split-image focusing optical path structure requires additional optical path design, which is relatively costly.

[0054] 3) Adaptive focus is subject to high-diopter human eyes, which requires a large amount of calculation and low accuracy in residual aberration calculation, which can easily lead to slow focusing speed and reduced focusing accuracy. In addition, the adaptive focus optical path design is very complex and requires a complete reconstruction of the human eye aberration wavefront, resulting in a large amount of calculation and the need for additional correction of system aberration.

[0055] 4) When using the projection method to focus, the missing or deformed reflected image of the light spot will affect the focusing result. That is, the projection method relies on the reflected image of the light source on the ocular surface. When the cornea produces tears or there are long eyelashes to interfere, the circular reflected image will be deformed and missing.

[0056] 5) The calculation speed of using neural networks to locate the pupil is slow and cannot meet the needs of real-time eye tracking. Deep learning requires manual labeling of the light spot position and pupil position as training data for training the neural network, and the training and calculation processes are complex.

[0057] 6) The triangulation method is greatly affected by environmental factors. External light entering the system may interfere with the scattered bright spots formed by the external eye lighting source. In addition, the triangulation method requires obtaining the reflected image on the ocular surface or cornea to calculate the offset position of the eyeball. External light can easily affect the quality of the reflected image, resulting in errors in the calculation.

[0058] Based on this, the embodiments of the present invention use a Shack-Hartmann wavefront sensor to measure the defocus wavefront, calculate the wave aberration between the defocus wavefront and the calibration wavefront, achieve automatic focusing, and use a converging binocular vision system to achieve real-time three-dimensional tracking of the eyeball. When the clarity is adjusted to the best, the camera is controlled to take pictures, and one-click fully automatic photography of the fundus camera is achieved. This solves the problem of manually being unable to determine the working distance between the camera and the human eye, the problem of the system being unable to quickly focus when facing human eyes with different refractive powers, and the problem of the complex structure of the fully automatic fundus camera. It also provides a new fully automatic focusing method for the currently mainstream fully automatic fundus cameras, which helps to develop automated detection equipment with a smaller size, a simpler system structure, high precision, and high speed.

[0059] Reference Figure 1 The present invention provides a fundus imaging method based on three-dimensional eye tracking and autofocus, the method comprising the following steps:

[0060] S1. Calculate the wavefront aberration using images collected by the Shack-Hartmann wavefront sensor, obtain the defocus amount and perform fitting to construct the movement curve of the stepper motor;

[0061] Specifically, the fixation plate is designed to illuminate the fundus, creating a smaller focused spot. This spot forms a secondary light source at the fundus, passing through the eyeball and the imaging system, and then reflected by a dichroic prism to the Shack-Hartmann wavefront sensor above. The Shack-Hartmann wavefront sensor consists of a microlens array and a camera. When the image received by the fundus imaging system is clear, the convex lens above forms a plane wave that is detected by the Shack-Hartmann wavefront sensor. When the image received by the fundus imaging system is blurred, the light wavefront exiting the convex lens is distorted and detected by the Shack-Hartmann wavefront sensor. To compensate the distorted wavefront to a plane wave, the wavefront aberration between the two must be calculated, and the relationship between the wavefront aberration and the focusing lens group must be fitted to achieve autofocus of the system.

[0062] S11, irradiating the fundus of the human eye with a light source to form a secondary light source, detecting the light with a Shack-Hartmann wavefront sensor, and obtaining an image captured by the Shack-Hartmann wavefront sensor;

[0063] S12, setting a light speckle pattern of a reference wavefront, obtaining a measurement wavefront corresponding to an image acquired by a Shack-Hartmann wavefront sensor and performing segmentation processing to obtain a light speckle pattern of the measurement wavefront;

[0064] In this embodiment, the image acquired by the Shack-Hartmann wavefront sensor is as follows: Figure 3 As shown, the sensor splits the wavefront into an array of light spots.

[0065] S13, obtaining the centroid coordinates of the light spot array pattern of the reference wavefront and the centroid coordinates of the light spot array pattern of the measurement wavefront by a centroid method and performing calculations to obtain a centroid offset;

[0066] In this embodiment, the wavefront aberration is calculated using the image collected by the Shack-Hartmann wavefront sensor. It is necessary to first calibrate the centroid coordinates of the standard wavefront and use the centroid method to obtain the specific centroid coordinates of each light spot. The centroid method formula is as follows:

[0067]

[0068] In the above formula, x c Indicates the x-coordinate of the center of mass, y c represents the y coordinate of the center of mass, w represents the width of the image, and h represents the height of the image.

[0069] Then, the spot array diagram of the detection wavefront is collected, and the offsets of the spot mass center of the detection wavefront and the spot mass center of the standard wavefront on the x-axis and y-axis are calculated respectively. Figure 3 The wavefront slope geometry of the Shack-Hartmann wavefront sensor shown in can be obtained as:

[0070]

[0071] In the above formula, f is the focal length of the system, δx and δy represent the offset in the x-direction and y-direction respectively, (x1, y1) represents the coordinates of the centroid of the measured wavefront, and (x0, y0) represents the coordinates of the centroid of the reference wavefront.

[0072] S14, expanding the mass center offset using a Zernike polynomial to obtain a wavefront aberration between the reference wavefront and the measured wavefront;

[0073] In this embodiment, the wave surface is expressed using zernike to obtain:

[0074]

[0075] In the above formula, c j represents the j-th Zernike coefficient.

[0076] Among them, the Zernike polynomials are shown in Table 1.

[0077] Table 1 Zernike polynomial table

[0078]

[0079]

[0080] The Zernike coefficient can be calculated by expressing the wavefront using Zernike polynomials. Considering the wavefront aberration of the pupil with radius R, the spherical aberration is expressed by Zernike polynomials. The expression is:

[0081]

[0082] In the above formula, and are the coefficients of the second-order Zernike polynomials, representing defocus, tilt astigmatism, and 45° astigmatism, respectively.

[0083] in:

[0084]

[0085] The wavefront aberration between the detection wavefront and the standard wavefront can be expressed as:

[0086]

[0087] S15, considering the power of the wavefront correction of the Shack-Hartmann wavefront sensor, converting the wavefront aberration between the reference wavefront and the measurement wavefront to obtain the defocus amount;

[0088] In this embodiment, under the paraxial condition, the power of wavefront correction can be expressed as:

[0089]

[0090] In the above formula, R is the pupil radius.

[0091] From the expression of the power of wavefront correction, we can know that is a constant, It depends on the value of the cosine term, that is, the minimum value is 0, that is, when θ=φ, 1-cos2(θ-φ)=0, so the expression of spherical aberration is:

[0092]

[0093] In the above formula, S represents the defocus amount.

[0094] S16 , performing curve fitting processing on the defocus amount and the movement value of the stepping motor to obtain a movement curve of the stepping motor.

[0095] In this embodiment, the focus curve is fitted by the relationship between the defocus amount S and the movement of the stepping motor. By moving the focusing lens group, the wavefront images of different defocus positions of the system are collected. By recording the step movement value of the motor controlling the focusing lens group and collecting the image on the current Shack-Hartmann wavefront sensor and calculating the defocus amount S at this time, the defocus amount S is curve-fitted with the motor step value as shown in the following example: Figure 4 As shown, the relationship curve expression between the defocus amount S and the movement of the stepper motor is obtained:

[0096] T=6.9455*10 3 *S

[0097] In the above formula, T represents the movement curve of the stepper motor.

[0098] S2. Positioning the pupil image of the human eye collected by the binocular vision system to obtain the pupil center coordinates;

[0099] First of all, it should be noted that the experimental optical device diagram of the convergent binocular vision system is as follows Figure 5 and Figure 6 The experimental setup mainly consists of five parts, including the intersection of the optical axes of the left and right cameras 1, the camera lens 2, the position of the center of the eye to be tested on the target surface 3, the camera target surface 4 and the eye to be tested 5.

[0100] S21, performing coarse positioning on the human pupil image collected by the binocular vision system to obtain a preliminary pupil area;

[0101] Specifically, the human pupil image collected by the binocular vision system is downsampled to obtain a downsampled image; the downsampled image is Gaussian smoothed and grayscale normalized to obtain a preprocessed image; the preprocessed image is extracted through dynamic threshold binarization and connected domain analysis to obtain the centroid of the connected domain; the centroid of the connected domain is used as the center of the rectangular frame to perform positioning matching with the human pupil image collected by the binocular vision system to obtain a preliminary pupil area.

[0102] In this embodiment, the process of extracting the pupil center is as follows: Figure 7 As shown. For coarse positioning, the image is first downsampled to 1 / 25, smoothed by Gaussian and grayscale normalized, and then the pupil area is extracted by dynamic threshold binarization and connected domain analysis. The centroid of this connected domain is extracted and the center of the rectangular box is used to locate the approximate pupil area in the original image, namely the ROI (ROI / region of interest) area. The effect is as follows Figure 8 shown.

[0103] S22. Perform fine positioning processing on the preliminary pupil area to obtain the pupil center coordinates.

[0104] Specifically, the preliminary pupil area is subjected to Gaussian smoothing and grayscale normalization to obtain the preprocessed pupil area; the preprocessed pupil area is subjected to light spot removal through binarization and image subtraction to obtain the preliminary pupil connected domain; the preliminary pupil connected domain is subjected to image binarization and template matching to obtain the pupil connected domain; the edges of the pupil connected domain are reconstructed through the Sobel operator edge detection algorithm to obtain the pupil center coordinates.

[0105] In this embodiment, fine positioning is then performed, and the area is Gaussian smoothed and normalized. Since the binocular lighting beads will leave light spots in the pupil, the image is binarized and image subtraction, closing operations and other operations are performed to remove the light spots. Then the image is binarized and template matching is performed to find the pupil connected domain. The Sobel operator edge detection is used to derive the X direction. When the eyes are not fully open, the pupil outline blocked by the eyelid above the pupil is a slant line, which will become an isolated point after processing. At this time, the method of removing small connected domains can be used to obtain the edge of the pupil, and the least squares ellipse fitting can be used to reconstruct the pupil edge. The center of the fitted ellipse is the center of the pupil in this picture. The effect is as follows. Figure 9 shown.

[0106] S3, performing three-dimensional eye tracking coordinate conversion processing according to the pupil center coordinates to obtain the three-dimensional coordinates of the human eye;

[0107] Specifically, after obtaining the pupil center coordinates, the convergent binocular vision system is used to reconstruct the three-dimensional coordinates of the eye center and the camera. Figure 10 shown.

[0108] S31, obtaining the horizontal distance between the pupil center coordinate and the binocular camera and the vertical distance between the pupil center coordinate and the center point of the image collected by the Shack-Hartmann wavefront sensor;

[0109] S32. Determine the optical center coordinates of the binocular camera based on the horizontal distance and the vertical distance in combination with the working distance, tilt angle, and focal length of the binocular camera;

[0110] S33. Determine the straight line equation between the optical center coordinates of the binocular camera and the pupil center coordinates and solve it according to the similar triangle theorem to obtain the three-dimensional coordinates of the human eyeball.

[0111] In this embodiment, let x1 be the distance from the pupil center to the horizontal direction in the image, and a be the camera zoom ratio, as expressed by:

[0112]

[0113] Solve the vertical distance from the pupil center to the image center. Combined with the working distance OO R , camera tilt angle θ, focal length f According to the geometric relationship, the center point O of the right camera image is obtained R and the optical center O of the right camera gL The coordinates of O L and O gL Similarly. And because there are:

[0114]

[0115] The center of the eyeball can be solved to be the center point P of the left camera image L The coordinates of P R The coordinates of can be obtained similarly.

[0116] Solve the equation of the line connecting the optical centers of the left and right cameras and the center of the eyeball from the known points. The expression is:

[0117]

[0118] Then combine them to get the plane coordinates of point P, let:

[0119]

[0120] That is

[0121]

[0122] According to the similar triangle theorem, we have:

[0123]

[0124] You can find:

[0125]

[0126] In summary, the reconstructed three-dimensional coordinates of the eyeball are:

[0127]

[0128] The coordinates of point P have been obtained, and point O is known to be the origin coordinate. For the fundus camera, the intersection of the two optical axes of the binocular camera is also the optimal working distance of the fundus camera. When this device is deployed on the fundus camera, it is only necessary to calculate the vector The processed value is fed back to the motor for movement according to the relevant motor protocol, so that the fundus camera can realize real-time centering and focusing of the eyeball.

[0129] S4. Control the binocular camera to perform autofocus according to the three-dimensional coordinates of the human eyeball, and control the binocular camera to move and shoot in combination with the movement curve of the stepping motor to obtain a fundus image of the human eye.

[0130] In summary, the embodiments of the present invention have the following differences compared to the prior art:

[0131] 1) A method for fully automatic real-time three-dimensional tracking of the eyeball and automatic focusing of the fundus to obtain clear fundus images is proposed.

[0132] 2) The autofocus method designed based on the Shack-Hartmann wavefront sensor cleverly utilizes the relationship between symmetrical conjugate points in the optical path design. By simply detecting the defocused wavefront of the convex lens, it can be connected to the focus of the system to achieve automatic focusing. Compared with the existing focusing methods, it meets the requirements of high precision, high speed, and low interference, while achieving a smaller, simpler, and lower-cost optical path structure design. There are two mainstream focusing methods on the market, contrast detection and phase detection. Contrast focus obtains the current image, calculates the quality evaluation coefficient of the current image to determine the clarity of the picture, and thus determines the precise focus position. It has high requirements for the image and clarity evaluation function. When the amount of information is too much, it will lead to a decrease in focus accuracy or focus failure. The phase detection method requires an additional actual optical path, which is relatively costly, has high requirements for the accuracy of image detection, and increases the system volume. The autofocus implemented in this embodiment of the present invention using Zernike polynomials and a Shack-Hartmann wavefront sensor is highly universal because it calculates the defocus amount S and uses a curve representing the relationship between defocus amount S and stepper motor movement to guide motor movement during the focusing process. The Zernike fitting and wavefront reconstruction process does not reconstruct the entire wavefront, but only calculates the defocus amount S. This eliminates the need for additional correction for system aberrations, requiring only the calculation of the stereoscopic correction aberrations of a single lens. This significantly simplifies the calculation and improves autofocus speed.

[0133] 3) Automatic centering method based on reconstructing three-dimensional coordinates using converging binocular: The existing centering and focusing method of fundus cameras requires the design of an additional optical path to obtain the reflected image of the cornea or ocular surface. The converging binocular has a simple structure and does not require a new optical path. Three-dimensional eye tracking can be achieved by simply using cameras placed at a certain angle, reducing design costs. Some converging binocular systems used in other ophthalmic instruments can also reconstruct three-dimensional coordinates. They determine the optical center of one of the cameras as the origin, but they need to calibrate the camera first to obtain parameters such as internal and external parameters, and then use the projection matrix formula and the least squares method to obtain the three-dimensional coordinates of the eyeball. The three-dimensional eye tracking method of this patent does not require camera calibration, and sets the origin of the world coordinate system to the intersection of the main optical axes of the two camera lenses. Centering and focusing can be achieved by simply controlling the installation tilt angle of the binocular camera so that the intersection of the main optical axes of the left and right camera lenses is set to the focal plane of the fundus camera.

[0134] 4) Image processing-based method for accurately locating the pupil center to combat interference from partial eye occlusion and long eyelashes: This method is more robust in identifying the pupil center than traditional image processing. Traditional image processing rarely considers situations such as eyelash occlusion and partial eye occlusion. The computational speed of feature point recognition and feature matching is insufficient to meet the real-time centering and eyeball tracking required by convergent binocular systems. The improved radial symmetry transformation algorithm achieves high accuracy under the influence of light spots and partial eyelash occlusion of the pupil, but still falls short of the required accuracy under partial eye occlusion and long eyelash occlusion. To achieve fast computational speed and high-precision pupil positioning in situations such as eyelash occlusion and partial eye occlusion, this method uses: a coarse and fine positioning method to better locate the pupil center despite various interferences; Sobel edge extraction is performed on the extracted pupil connected domain, small connected domains are removed, and ellipse fitting is performed on the remaining points to reproduce the complete pupil contour under partial eye occlusion; threshold segmentation and image subtraction are used to remove interfering pupil spots.

[0135] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0136] 1) The embodiment of the present invention uses a Shack-Hartmann wavefront sensor to detect the defocused wavefront. The Shack-Hartmann wavefront sensor has very high sensitivity, can simultaneously measure the visual aberration of the entire pupil within milliseconds, is less affected by scattering, is relatively sensitive to the optical path difference detection of the wavefront, can achieve high-precision detection at the um level, and the speed of wavefront reconstruction is at the ms level, and the measurement speed is fast. In addition, this design uses a dichroic prism to create a conjugate point connected to the image plane. Only by detecting the wavefront information of a single lens can it be determined whether the image plane is defocused, and then the focusing lens group is connected to focus by fitting the defocus amount. This method does not require a large and complex structure and has certain reference value for ophthalmic measurement automation.

[0137] 2) The embodiment of the present invention utilizes image processing on the captured image to obtain the center position of the pupil on the image, and then combines the camera magnification and the geometric relationship between the positions of the original components within the system to reconstruct the three-dimensional coordinates of the eyeball and the camera. The structure of the present invention is simple. Compared with the traditional convergent binocular system that uses image stitching to achieve focus, the present invention derives the specific world three-dimensional coordinates of the binocular camera and the eyeball on the basis of achieving the focusing function, thereby broadening the applicable scenarios of the convergent binocular. At the same time, based on the image processing technology for extracting the center point of the pupil, the use of Sobel operator edge extraction and morphological processing methods can still quickly and accurately extract the pupil center when the eyes are not fully open or blocked by long eyelashes, with high robustness and real-time performance.

[0138] A fundus imaging device based on three-dimensional eye tracking and autofocus includes a fundus imaging system, an autofocus system, and a binocular positioning system, wherein:

[0139] The fundus imaging system includes an illumination system and an imaging system. The fundus imaging system is used to image the retina of the human eye and obtain a fundus image of the human eye;

[0140] The autofocus system is used to calculate the defocus amount of the retinal image acquired in the imaging system;

[0141] The binocular positioning system is used for three-dimensional eye tracking.

[0142] Furthermore, it should be noted that the imaging system includes an eyepiece objective lens, a focusing lens group, an imaging lens group, a beam splitter prism and a charge coupled device.

[0143] Specifically, in this embodiment, Figure 2As shown, the fundus imaging device of an embodiment of the present invention is divided into three parts: a fundus photography system, an autofocus system and a binocular positioning system. The fundus photography system is generally divided into illumination and imaging, and is used to image the fundus retina. The system adopts single-point illumination, near-infrared observation, and white light shooting to image the fundus retina. The illumination and imaging share the same set of eyepieces and are connected by a spectroscope. The system contains an internal fixation design and adopts internal focusing as a refractive compensation method. The imaging system consists of an eyepiece objective lens, a focusing lens group, an imaging lens group, a spectroscopic prism and a charge-coupled device (CCD). The eyepiece objective forms the primary image of the retina in the imaging system and serves as the illumination source for the pupil. The focusing lens assembly compensates for human refractive error in the imaging system, allowing for focus adjustment for eyes with varying diopters. The imaging lens assembly performs a secondary image of the primary image formed by the eyepiece objective, effectively eliminating residual aberrations from the primary image. The beam splitter prism connects the Shack-Hartmann wavefront sensor to the CCD, ensuring consistent image quality between the reflected and transmitted images and defocus conjugation. The CCD captures the retinal image and converts it into a digital signal for subsequent processing and analysis. The illumination system, consisting of an LED light source, a homogenizer, and the eyepiece objective, illuminates the retina, which is then captured by the imaging system to capture the image. Light passes through the homogenizer and is polarized by the analyzer. It then passes through the eyepiece objective to form a spot at the upper edge of the pupil, eliminating stray light reflected from the center of the cornea. Light reflected by the eyepiece objective is blocked by a linear polarizer, while light scattered by the retina passes through the imaging system to form an image on the camera. The autofocus system calculates the defocus of the retinal image captured by the imaging system and then connects the focusing lens group to compensate for the defocus. The system is located above the dichroic prism in front of the image plane. A square prism is used as a beam splitter at the image plane to ensure that reflection and transmission have the same aberration characteristics. The reflected focused light spot passes through the convex lens above to form a plane wave, which is detected by the Shack-Hartmann wavefront sensor. The distortion of the wavefront after the convex lens is defocused is used to achieve automatic focusing of the imaging system. The binocular positioning system is used to achieve three-dimensional eye tracking. The system includes two short-focus cameras and two LED lights. The cameras are located on either side of the fundus photography system and illuminated by infrared light. The center of the human pupil is captured in a converging manner, and three-dimensional eye tracking is achieved by utilizing geometric relationships.

[0144] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A fundus imaging method based on three-dimensional eye tracking and autofocus, characterized in that: The following steps are involved: The wavefront aberration is calculated through the images collected by the Shack-Hartmann wavefront sensor, the defocus amount is obtained and fitted, and the movement curve of the stepper motor is constructed; Position the pupil image of the human eye collected by the binocular vision system to obtain the coordinates of the pupil center; Perform three-dimensional eye tracking coordinate conversion processing according to the pupil center coordinates to obtain the three-dimensional coordinates of the human eye; The binocular camera is controlled to perform autofocus according to the three-dimensional coordinates of the human eyeball, and is controlled to perform mobile shooting in combination with the moving curve of the stepping motor to obtain the fundus image of the human eye.

2. A fundus imaging method based on three-dimensional eye tracking and autofocus according to claim 1, characterized in that: The step of calculating the wavefront aberration through the image collected by the Shack-Hartmann wavefront sensor, obtaining the defocus amount and performing fitting, and constructing the movement curve of the stepper motor specifically includes: The fundus of the human eye is illuminated by a light source to form a secondary light source, which is detected by a Shack-Hartmann wavefront sensor to obtain an image collected by the Shack-Hartmann wavefront sensor; A light spot array pattern of a reference wavefront is set, and a measurement wavefront corresponding to an image collected by a Shack-Hartmann wavefront sensor is obtained and segmented to obtain a light spot array pattern of the measurement wavefront; The centroid coordinates of the light spot array pattern of the reference wavefront and the centroid coordinates of the light spot array pattern of the measurement wavefront are obtained by the centroid method and calculated to obtain the centroid offset; The center of mass offset is expressed by expanding the Zernike polynomial to obtain the wavefront aberration between the reference wavefront and the measured wavefront; Considering the power of the wavefront correction of the Shack-Hartmann wavefront sensor, the wavefront aberration between the reference wavefront and the measured wavefront is converted to obtain the defocus value; The defocus amount and the movement value of the stepper motor are subjected to curve fitting processing to obtain the movement curve of the stepper motor.

3. The fundus imaging method based on three-dimensional eye tracking and automatic focusing according to claim 2, characterized in that: The step of positioning the pupil image of the human eye collected by the binocular vision system to obtain the pupil center coordinates specifically includes: Roughly locate the pupil image of the human eye collected by the binocular vision system to obtain the preliminary pupil area; The preliminary pupil area is finely positioned to obtain the pupil center coordinates.

4. The fundus imaging method based on three-dimensional eye tracking and autofocus according to claim 3, characterized in that: The step of roughly locating the pupil image of the human eye collected by the binocular vision system to obtain a preliminary pupil area specifically includes: Downsampling the pupil image of a human eye collected by the binocular vision system to obtain a downsampled image; Perform Gaussian smoothing and grayscale normalization on the downsampled image to obtain a preprocessed image; The pre-processed image is extracted through dynamic threshold binarization and connected domain analysis to obtain the centroid of the connected domain; The centroid of the connected domain is used as the center of the rectangular frame, and is positioned and matched with the pupil image of the human eye collected by the binocular vision system to obtain a preliminary pupil area.

5. The fundus imaging method based on three-dimensional eye tracking and autofocus according to claim 4, characterized in that: The step of performing fine positioning processing on the preliminary pupil area to obtain the pupil center coordinates specifically includes: Perform Gaussian smoothing and grayscale normalization on the preliminary pupil area to obtain the preprocessed pupil area; The pupil area after preprocessing is processed by binarization and image subtraction to remove light spots and obtain a preliminary pupil connected domain. Perform image binarization and template matching on the preliminary pupil connected domain to obtain the pupil connected domain; The edge of the pupil connected domain is reconstructed using the Sobel operator edge detection algorithm to obtain the pupil center coordinates.

6. The fundus imaging method based on three-dimensional eye tracking and autofocus according to claim 5, characterized in that: The step of performing three-dimensional eye tracking coordinate conversion processing according to the pupil center coordinates to obtain the three-dimensional coordinates of the human eye specifically includes: Obtain the horizontal distance between the pupil center coordinates and the binocular camera and the vertical distance between the pupil center coordinates and the center point of the human eye pupil image collected by the binocular vision system; Determine the optical center coordinates of the binocular camera based on the horizontal distance and the vertical distance combined with the working distance, tilt angle and focal length of the binocular camera; The equation of the straight line between the optical center coordinates of the binocular camera and the pupil center coordinates is determined and solved according to the similar triangle theorem to obtain the three-dimensional coordinates of the human eyeball.

7. A fundus imaging device based on three-dimensional eye tracking and autofocus, characterized in that: It includes fundus imaging system, autofocus system and binocular positioning system, including: The fundus imaging system includes an illumination system and an imaging system. The fundus imaging system is used to image the retina of the human eye to obtain a fundus image of the human eye; The autofocus system is used to calculate the defocus amount of the retinal picture acquired in the imaging system; The binocular positioning system is used for three-dimensional eye tracking.

8. The fundus imaging device based on three-dimensional eye tracking and autofocus according to claim 7, characterized in that: The imaging system comprises an eyepiece objective lens, a focusing lens group, an imaging lens group, a beam splitter prism and a charge coupled device.

Images