A head-mounted optical visual aid line-of-sight tracking method

Abstract

The present application relates to the technical field of human-computer interaction, and provides a line-of-sight tracking method for a head-mounted optical visual aid, comprising the following steps: firstly, pre-processing the collected image; secondly, judging the region where the visual aid is located; thirdly, solving the problems of occlusion and deformation of the pupil to obtain the pupil contour; fourthly, modeling the pupil and the eyeball according to the pupil contour and solving the parameters; and finally, realizing line-of-sight tracking through coordinate system conversion and single-point calibration. The present application can be used for line-of-sight tracking of a user in the case of wearing a head-mounted optical visual aid, can quickly and accurately complete the line-of-sight tracking task of the head-mounted optical visual aid, has good accuracy and robustness, and has positive significance for the attention mechanism research of low-vision people.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and more particularly to a gaze tracking method for head-mounted optical assistive devices. Background Technology

[0002] Optical visual aids are devices that enhance vision for visually impaired individuals by magnifying or adjusting their field of vision, expanding their field of vision or making images clearer. They are portable, easy to operate, and inexpensive, making them the preferred choice for people with low vision. The main form of optical visual aid is the eyeglass-style aid, most of which have specially made lenses with additional prisms, spherical lenses, and other devices to enhance the user's field of vision. Users need to wear the aid normally, just like wearing eyeglasses. Due to the special lenses, users often experience partial pupil obstruction when wearing the optical aid, which poses significant challenges to eye tracking and subsequent research.

[0003] Currently available eye-tracking methods are only suitable for situations where the pupil is not significantly obstructed, or where the pupil is wearing normal glasses. In both of these cases, accuracy is relatively high. However, if these methods are applied to people wearing optical assistive devices, they are mostly unusable, and even in the few cases where they are applicable, significant errors occur. This presents challenges for future research.

[0004] Therefore, considering the special characteristics of visually impaired individuals wearing head-mounted optical aids, the aid area can be identified first, followed by the judgment and processing of the magnified portion of the occluded area. Then, the pupil contour can be obtained through ellipse fitting, and the pupil and eyeball can be modeled based on the pupil contour. Finally, gaze tracking can be achieved through coordinate system transformation and single-point calibration. Summary of the Invention

[0005] In response to the technical problems mentioned in the background section, this invention provides a gaze tracking method for head-mounted optical aids. This invention is primarily applied to pupil recognition, eye modeling, and gaze point estimation when using head-mounted optical aids. It can fit and model pupils obscured by the optical aid, solving the problem of pupil obstruction when the user wears optical devices, improving the accuracy of gaze tracking under obstruction, and reducing time and economic costs. This invention can quickly and accurately complete gaze tracking tasks, providing significant convenience for research fields such as attention mechanism detection under head-mounted optical aids.

[0006] The technical means employed in this invention are as follows:

[0007] A gaze tracking method for head-mounted optical assistive devices includes the following steps:

[0008] Image acquisition and preprocessing; facial images are acquired using a camera, the human eye region is located using a feature point detection method, and the location of the visual aid is located using a cascaded classifier;

[0009] Pupil region detection; based on the human eye region and the position of the visual aid, predict the possible pupil region and determine the pupil region using automatic threshold segmentation;

[0010] Pupil region processing; determine whether the pupil is obstructed, determine whether the pupil is enlarged, establish an amplification distortion correction model using the magnified pupil in the visual aid region and the unobstructed pupil, and map the distorted magnified pupil region to the unobstructed pupil region.

[0011] Pupil fitting: Based on the current pupil area, an ellipse is fitted to obtain the fitted pupil edge, and the parameters are adjusted by pupil symmetry transformation under normal lenses to obtain pupil data;

[0012] Eye reconstruction and gaze tracking: The pupil and eyeball are reconstructed as a disk tangent to a rotating sphere; where the disk represents the pupil, the rotating sphere represents the eyeball, and the gaze is a vector from the center of the sphere to the center of the circle; the parameters of the disk and the rotating sphere are calculated, and gaze tracking is completed using coordinate system transformation and single-point calibration.

[0013] Furthermore, in the image acquisition and preprocessing process, the image is preprocessed by image grayscale conversion and image denoising, and then the human eye region is located by feature point detection method and the location of the vision aid is located by cascade classifier.

[0014] Furthermore, the image grayscale conversion adopts a weighted average method. During the processing, grayscale conversion can be completed by weighting the average with different weights and normalizing the grayscale values ​​to 0-255.

[0015] Furthermore, the eye image contains Gaussian noise and salt-and-pepper noise, so a Gaussian filter and a median filter are used for denoising; the window size of the Gaussian filter is set to 5×5, and the window size of the median filter is set to 7×7.

[0016] Furthermore, the determination of whether the pupil is obstructed is divided into three cases: the pupil is not obstructed by the visual aid, the pupil is completely obstructed by the visual aid, and the pupil is partially obstructed by the visual aid.

[0017] Furthermore, determining whether the pupil is obstructed includes the following steps:

[0018] The number of points that make up the contour is used to filter the contours in the image; if the number of contours retained is 0, the image needs to be re-acquired; if the number of contours retained is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the conditions.

[0019] Since the relative position between the camera and the head remains basically unchanged, when there are enough contours to meet the requirements, the pupil area is determined by threshold segmentation, and the pupil edge points are estimated by polar coordinate transformation, thereby determining whether there is an area of ​​the pupil that is obscured.

[0020] For the found pupil region, the region of interest of the pupil is obtained with the pupil center as the parameter; the region of interest is transformed by polar coordinates to obtain a polar coordinate radial derivative map, and the maximum value of each column of pixels in the polar coordinate radial derivative map may be the pupil edge point;

[0021] Since the cornea is reflective, a reflection threshold is set to remove columns in the polar coordinate radial derivative plot and polar coordinate plot where the maximum value is greater than the threshold. Based on the grayscale difference between the eyelid, iris and pupil, the part of the pupil obscured by the eyelid in the polar coordinate transformation plot is removed.

[0022] After obtaining the pupil edge points, the ellipse formed by the pupil edge points is considered to be the pupil range that best meets the requirements. The pupil occlusion can be determined by judging whether the edge points are a relatively complete ellipse; if there is occlusion, the empty area of ​​the ellipse is the occluded area.

[0023] Furthermore, determining whether the pupil is dilated includes the following steps:

[0024] First, the presence of magnified occluded pupil information is detected using a grayscale histogram. The magnified pupil information is usually displayed as a cut-out black pupil portion on the visual aid. The magnified occluded pupil area can be obtained by detecting significant changes in grayscale. After detecting the existence of the magnified area, the cut-out pupil images displayed on the visual aid are merged according to the spacing of the visual aid to remove the gaps in the cut-out black pupil images and the black areas are merged sequentially.

[0025] Secondly, since the magnification of the image by the visual aid remains basically constant, the merged pupil image is proportionally reduced according to the magnification to achieve merging with the unobstructed pupil area.

[0026] The image that has undergone merging and shrinking is offset along the edge until it is connected to the edge of the unobstructed pupil image.

[0027] Furthermore, the pupil fitting process includes the following steps:

[0028] Starting from the coarsely located pupil center, and selecting a fixed step size, four neighboring points are determined from this starting point. For each pixel, given the ratio of the major and minor axes of the ellipse and the rotation angle of the ellipse, the minimum residual of each point is obtained. The residuals of all five points are compared. If the residual of the starting point is the smallest, the step size is halved, and the steps are repeated until the step size becomes one pixel. If the residual of the starting point is not the minimum value, the neighborhood point with the smallest residual is selected as the starting point, and the above steps are repeated to obtain the ellipse that best meets the conditions as the pupil.

[0029] Furthermore, the eye reconstruction and gaze tracking include the following steps:

[0030] Using a pinhole camera model to perform full perspective projection on the image, the possible projection space of the pupil is regarded as a cone with the camera focus as the vertex and the pupil circle as the base, and the pupil image is the intersection of the cone and the image plane.

[0031] Based on the known pupil image, the pupil circle must meet the following requirements: First, the center of the correct pupil circle should be on the line connecting the center of the predicted pupil circle and the camera focus; second, the distance from the center of the circle to the center of the sphere should be the radius of the rotating sphere calculated in the above steps; and finally, there should be a proportional relationship between the parameters of the correct pupil circle and the predicted pupil circle.

[0032] Based on the above requirements, the radius, center, and normal vector of the pupil circle are calculated.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] This invention first preprocesses the acquired images, then determines the location of the visual aid, subsequently addresses the issues of pupil occlusion and distortion to obtain the pupil contour, and models and solves for parameters of the pupil and eyeball based on the pupil contour. Finally, it achieves gaze tracking through coordinate system transformation and single-point calibration. All the main steps of this invention bring beneficial effects.

[0035] First, pupil occlusion is determined by contour filtering and polar coordinate radial derivative plot, which ensures both the accuracy of the parameters and the recognition speed.

[0036] Secondly, by processing the magnified occluded area and merging it with the unoccluded area, higher accuracy can be achieved in subsequent pupil tracking.

[0037] Third, the common ellipse fitting method has been improved by selecting an iterative search method that can find the pupil ellipse that best fits the fitting conditions in a very short time.

[0038] Fourth, the pupil is modeled as a disk tangent to the rotating sphere, which reduces the amount of computation while ensuring accuracy.

[0039] This invention can be used for eye tracking when users are wearing head-mounted optical aids. It can quickly and accurately complete eye tracking tasks with head-mounted optical aids, and has good accuracy and robustness. It is of positive significance for research on the attention mechanisms of visually impaired people. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0042] Figure 2 This is a simplified schematic diagram of an optical vision aid according to the present invention.

[0043] Figure 3 This is a schematic diagram of pupil tracking under occlusion according to the present invention.

[0044] Figure 4 This is a structural diagram of the eye modeling of the present invention.

[0045] Figure 5 This is a schematic diagram illustrating the solution and modeling parameters for this invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] like Figure 1-5 As shown, the present invention provides a gaze tracking method for head-mounted optical aids, comprising the following steps:

[0049] Image acquisition and preprocessing: Facial images are acquired using a camera, the eye region is located using feature point detection methods, and the location of the visual aid is determined using a cascaded classifier. Specifically, individuals with visual impairments are asked to sit in front of a screen, above which a camera is placed to acquire images, including pupil and facial data. To ensure sufficient detail in the acquired images for subsequent pupil data analysis, additional auxiliary light sources are added around the camera to illuminate the face. When wearing optical aids, glare from the mirrors makes pupil localization more difficult; therefore, a specific polarizing filter is first added in front of the camera during image acquisition to pre-filter glare, significantly reducing the number of glare spots.

[0050] After eliminating reflections in the image, preprocessing is required to improve the accuracy of pupil tracking. Preprocessing steps include image grayscale conversion and image denoising. After preprocessing, the human eye region can be located using feature point detection methods. The specific steps are as follows:

[0051] Image grayscale conversion: Since the images captured by the camera are color images and color image processing is relatively slow, color images can be converted into grayscale images, which have a faster processing speed. Although grayscale images lose some color information, from the perspective of pupil tracking requirements, the information contained in grayscale images is consistent with that of their corresponding color images. Image grayscale conversion can be achieved using a weighted average method. During processing, different weights are used to apply a weighted average, which completes the grayscale conversion and normalizes the grayscale values ​​to 0-255.

[0052] Image Denoising: Eye images are easily contaminated by noise during acquisition and transmission due to environmental factors and sensor quality. Therefore, noise analysis is necessary. Eye images mainly contain Gaussian noise and salt-and-pepper noise, which can be reduced using filters. For each noise type, Gaussian filters and median filters can be used. The window size for the Gaussian filter is set to 5×5, and the window size for the median filter is set to 7×7. These settings effectively reduce Gaussian and salt-and-pepper noise.

[0053] Furthermore, pupil region detection is performed. Based on the human eye region and the location of the assistive device, the possible pupil region is predicted, and automatic threshold segmentation is used to determine the pupil region. Optical assistive devices typically have special lenses with extended vision, and the eye under these special lenses may be obstructed. Since there is a relatively clear boundary between the assistive device and the normal lens, this invention uses a cascaded classifier to detect the assistive device. The cascaded classifier is mainly used for target detection; it filters out regions that do not meet the criteria by chaining together a series of simple classifiers. By performing feature extraction and cascaded filtering on the facial image, the approximate range of the assistive device can be obtained.

[0054] Further processing of the pupil region involves determining whether the pupil is obstructed and whether it is magnified. A magnification distortion correction model is established using the magnified pupil in the visual aid region and the unobstructed pupil. The distorted magnified pupil region is then mapped to the unobstructed pupil region.

[0055] The preprocessed image is filtered to select the contours, and the pupil region is determined by threshold segmentation to determine whether the pupil is occluded.

[0056] After obtaining the position of the assistive device, there are three possible scenarios regarding the position of the pupil relative to the assistive device: the pupil is not obscured by the assistive device, the pupil is completely obscured by the assistive device, and the pupil is partially obscured by the assistive device. In the first scenario, pupil tracking can be performed normally. In the second and third scenarios, it is necessary to determine whether there is any offset on the assistive device such as the lens. If so, affine transformation and ellipse fitting are required to obtain the pupil region.

[0057] After image preprocessing, the next step is to extract the pupil contour and determine if it is occluded. Since there are many contours in the image, a reasonable filtering method is needed to find the true pupil contour. Experiments show that the edge point data of the pupil contour is usually between 100 and 300, so the number of points constituting the contour can be used as a preliminary filter for contours in the image. If the final number of retained contours is 0, the image needs to be re-acquired; if the number of retained contours is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the criteria.

[0058] Since the relative position of the camera and the head remains basically unchanged, when there are enough contours to meet the requirements, the pupil region can be determined by threshold segmentation, and the pupil edge points can be estimated by polar coordinate transformation, thereby determining whether there is an obscured area in the pupil.

[0059] For the identified pupil region, the region of interest (ROI) is obtained using the pupil center as a parameter. Analysis of the radial derivative of this region reveals that the pupil edge points are part of the radial edge. A polar coordinate transformation of the ROI yields a polar coordinate radial derivative plot; the maximum value of each column in this plot likely represents the pupil edge point. Due to the reflectivity of the cornea, specular reflection may occur. Therefore, a reflection threshold can be set to remove columns in the polar coordinate radial derivative plot and polar coordinate plot where the maximum value exceeds the threshold. Based on the grayscale differences between the eyelid, iris, and pupil, the portion of the pupil obscured by the eyelid in the polar coordinate transformation plot can be removed. After obtaining the pupil edge points, the ellipse formed by these points is considered the most suitable pupil range. The determination of pupil occlusion is completed by judging whether the edge points form a relatively complete ellipse. If occlusion exists, the empty area of ​​the ellipse is the occluded area.

[0060] After obtaining the occluded area, determine whether there is an area that obscures the magnified pupil. If so, the occluded pupil area needs to be restored through affine transformation and merged with the unoccluded pupil area.

[0061] After completing the pupil occlusion detection, the occluded area of ​​the pupil can be obtained. For this area, it's necessary to consider whether the occluded portion is magnified and thus visible due to the imaging principle of the visual aid. Due to the imaging principle of the visual aid, the occluded pupil portion is not completely invisible. The occluded portion is usually displayed in the visual aid on the special lens due to light refraction. The displayed occluded portion is an image that has been magnified to a certain extent and is shifted and segmented. The relationship between the two is an affine transformation, which can be merged with the unoccluded area after processing to improve the accuracy of pupil tracking. The specific steps are as follows:

[0062] Based on the identification of the assistive device area, the first step is to detect the presence of magnified occluded pupil information using a grayscale histogram. This magnified pupil information is typically displayed as a segmented black pupil on the assistive device. Significant changes in grayscale can identify the magnified occluded pupil area. After detecting the magnified area, the segmented pupil images displayed on the assistive device need to be merged according to the spacing between the assistive devices. Due to variations in assistive device settings, this step requires consideration of the assistive device's imaging principle and relevant parameters of the head-mounted device, such as the fixed spacing between assistive devices, to remove gaps in the segmented black pupil images and sequentially merge the black areas. Secondly, since the magnification of the image by the assistive device is essentially constant, the next step is to proportionally reduce the merged pupil image according to the magnification before merging it with the unoccluded pupil area.

[0063] Experiments show that the pupil image displayed on a visual aid often shifts to some extent relative to the normal image, and the direction of this shift is basically fixed, i.e., along the edge of the frame. Based on this conclusion, the image after merging and shrinking can be shifted along the edge until it connects with the edge of the unobstructed pupil image. To verify the accuracy of this step, the pupil parameters before and after the shift can be estimated using the least squares method. If they are close, the processing step is reasonable. Through these steps, the magnified obstructed pupil area can be processed and merged with the unobstructed pupil part, facilitating subsequent pupil tracking.

[0064] Furthermore, pupil fitting is performed. An ellipse is fitted based on the current pupil area to obtain the fitted pupil edge. Parameters are then adjusted using pupil symmetry transformation under normal lenses to obtain pupil data. By performing ellipse fitting on the current pupil area to obtain the fitted pupil edge, and adjusting parameters using pupil symmetry transformation under normal lenses, highly accurate pupil data is obtained.

[0065] After processing and merging the magnified pupil occlusion area, it is necessary to obtain complete pupil data based on the known pupil area and complete pupil tracking. Here, the ellipse fitting method is used to complete pupil fitting and tracking.

[0066] To quickly obtain the most suitable ellipse, this invention employs an iterative search method, ending when the optimal ellipse is found. First, the coarsely located pupil center is used as the starting point, and a suitable step size is selected. Four neighboring points are determined from this starting point. For each pixel, given the ratio of the major and minor axes of the ellipse and the ellipse's rotation angle, the minimum residual for each point can be obtained. The residuals of all five points are compared. If the residual at the starting point is the smallest, the step size is halved, and the above steps are repeated until the step size becomes one pixel. If the residual at the starting point is not the minimum, the neighborhood point with the smallest residual is selected as the starting point, and the above steps are repeated. Following these steps, the most suitable ellipse can be obtained as the pupil.

[0067] To verify the accuracy of the pupil obtained by ellipse fitting, this invention performs a symmetric transformation on the eye under normal lenses and compares the pupil obtained by the symmetric transformation with that obtained by ellipse fitting. If the error is within a reasonable range, the fitting is considered successful. Symmetric transformation is a local operator for gradient images, used to describe the symmetry of points on the image. When performing symmetric transformation for pupil tracking, different symmetry factors are usually selected. The symmetry factor can be determined by the physiological characteristics of the face, generally one-tenth of the face width. The parameters of the pupil can be fitted through symmetric transformation. The parameters obtained by symmetric transformation are compared with those obtained by ellipse fitting. If the difference is small, the result of ellipse fitting is considered to have higher accuracy. If there is a significant difference, the pupil data is adjusted by combining the parameters of both to achieve higher accuracy pupil tracking.

[0068] Finally, eye reconstruction and gaze tracking are performed. The pupil and eyeball are reconstructed as a disk tangent to a rotating sphere; the disk represents the pupil, the rotating sphere represents the eyeball, and the gaze is a vector from the center of the sphere to the center of the disk. The parameters of the disk and the rotating sphere are calculated, and gaze tracking is completed using coordinate system transformation and single-point calibration. After the above steps, the contour information of the pupil can be obtained. In order to perform gaze tracking, the eye needs to be reconstructed in three dimensions. Since we only consider the direction and position of the pupil, it is not necessary to reconstruct a complete model of the eye; it is only necessary to simulate the pupil and its range of motion. This invention models the relationship between the pupil and the eyeball as a disk tangent to a rotating sphere, where the disk represents the pupil, the rotating sphere represents the eyeball, and the gaze is a vector from the center of the sphere to the center of the disk.

[0069] To determine the relevant parameters of the disk and the rotating sphere, this invention uses a pinhole camera model to perform full perspective projection on the image. The possible projection space of the pupil can be regarded as a cone with the camera focal point as the vertex and the pupil circle as the base. The pupil image is the intersection of the cone and the image plane. Based on the known pupil image, the cone can be reconstructed, and the circular intersection of the cone is the pupil circle.

[0070] The radius, center coordinates, and normal vector of the pupil circle in space can be obtained by solving the 3D model of the cone. However, since the size of the pupil circle cannot be determined by the back projection from 2D to 3D, the radius of the pupil circle can be arbitrarily determined in this step. The specific parameters of the pupil circle can be determined after the radius of the rotating sphere is calculated.

[0071] When back-projecting a pupil image, given any pupil radius, it is possible to find two pupil circles in three-dimensional space with the same radius, different normal vectors, and different centers. Therefore, in order to correctly find the center of the rotating sphere, it is necessary to perform a disambiguation operation by projecting the pupil circle and the pupil circle normal vector. After the disambiguation operation, the normal vectors of the two pupil circles are parallel to each other, and the line connecting the projection centers of the two pupil circles is parallel to the normal vector. This means that after projection, it is only necessary to select any one of the two pupil circles' normal vectors for projection.

[0072] For each frame of the pupil image, the normal vector projection can be obtained by following the steps described above. Theoretically, the intersection of all normal vector projections is the projection of the center of the sphere of revolution. However, due to potential numerical, discretization, or measurement errors in the vectors, these lines are unlikely to intersect at the same point. Therefore, the point closest to each projection normal vector can be found using the least squares method; this point is the projection of the center of the sphere of revolution. Back-projecting this coordinate system yields the coordinates of the sphere's center in three-dimensional space.

[0073] After calculating the coordinates of the sphere's center, the radius of the rotating sphere needs to be calculated. Theoretically, the line connecting the center of the pupil circle and the center of the sphere is the radius of the sphere. However, since the pupil size cannot be determined, the pupil circle itself cannot be determined either. Therefore, the pupil circle generated in the previous step with an arbitrary radius needs to be corrected here. According to the imaging principle of the line of sight, the center of the correct pupil circle should be on the line connecting the camera focus and the center of the currently generated pupil circle, and their normal vectors are the same. Therefore, we consider drawing a straight line from the center of the rotating sphere along the normal vector direction and finding its intersection with any possible line from the center of the circle. This operation can be performed for each pupil image. Similarly, to avoid discretization or measurement bias, this step also needs to use the least squares method to find the point on the line connecting the camera intersection point and the center of the circle that is closest to all lines emanating from the center of the sphere. This point is on the rotating sphere, so the distance from this point to the center of the sphere is the radius of the rotating sphere.

[0074] After determining the radius of the rotating sphere, the next step is to estimate the pupil. Based on the above analysis, it can be found that, given the pupil image, the pupil circle must meet the following requirements: First, the center of the correct pupil circle should lie on the line connecting the center of the predicted pupil circle and the camera focus; second, the distance from the center of the circle to the center of the sphere should be equal to the radius of the rotating sphere calculated in the previous step; finally, there should be a proportional relationship between the parameters of the correct pupil circle and the predicted pupil circle. According to these constraints, the radius, center, and normal vector of the pupil circle can be calculated.

[0075] After obtaining the relevant parameters of the rotating sphere and the pupil circle, the line connecting the center of the sphere and the center of the pupil circle represents the direction of gaze. At this point, gaze tracking can be completed through coordinate system transformation. Considering individual differences, single-point calibration is required before gaze tracking. A bright spot will appear in the center of the screen, and participants need to focus on this bright spot while keeping their eyes open as much as possible. This step lasts 3-5 seconds. The system will collect human eye data and fine-tune the model. The adjusted model can then achieve relatively accurate gaze tracking.

[0076] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0077] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gaze tracking method for head-mounted optical assistive devices, characterized in that, Includes the following steps: Image acquisition and preprocessing; facial images are acquired using a camera, the human eye region is located using a feature point detection method, and the location of the visual aid is located using a cascaded classifier; Pupil area detection; Based on the human eye region and the location of the visual aid, the possible pupil region is predicted, and the pupil region is determined using automatic threshold segmentation. Pupil area processing; Determine whether the pupil is obstructed and whether the pupil is enlarged. Establish an amplification distortion correction model using the magnified pupil in the visual aid area and the unobstructed pupil. Map the distorted magnified pupil area to the unobstructed pupil area. Pupil fitting: Based on the current pupil area, an ellipse is fitted to obtain the fitted pupil edge, and the parameters are adjusted by pupil symmetry transformation under normal lenses to obtain pupil data; Eye reconstruction and gaze tracking; reconstructing the pupil and eyeball as a disk tangent to the rotating sphere; The disk represents the pupil, the rotating sphere represents the eyeball, and the line of sight is the vector from the center of the sphere to the center of the circle. The parameters of the disk and the rotating sphere are calculated, and the line of sight is tracked using coordinate system transformation and single-point calibration.

2. The gaze tracking method for head-mounted optical aids according to claim 1, characterized in that, In the image acquisition and preprocessing process, the image is preprocessed by image grayscale conversion and image denoising, and then the human eye region is located by feature point detection method and the location of the vision aid is located by cascade classifier.

3. The eye-tracking method for head-mounted optical aids according to claim 2, characterized in that, The image grayscale conversion uses a weighted average method. During the processing, different weights are used to perform weighted averaging, which can complete the grayscale conversion and normalize the grayscale values ​​to 0-255.

4. The eye-tracking method for head-mounted optical aids according to claim 2, characterized in that, The eye image contains Gaussian noise and salt-and-pepper noise, so a Gaussian filter and a median filter are used for denoising; the window size of the Gaussian filter is set to 5×5, and the window size of the median filter is set to 7×7.

5. The gaze tracking method for head-mounted optical aids according to claim 1, characterized in that, The determination of whether the pupil is obstructed is divided into three cases: the pupil is not obstructed by the visual aid, the pupil is completely obstructed by the visual aid, and the pupil is partially obstructed by the visual aid.

6. A gaze tracking method for head-mounted optical aids according to claim 1 or 5, characterized in that, The process of determining whether the pupil is obstructed includes the following steps: The number of points that make up the contour is used to filter the contours in the image; if the number of contours retained is 0, the image needs to be re-acquired; if the number of contours retained is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the conditions. Since the relative position between the camera and the head remains basically unchanged, when there are enough contours to meet the requirements, the pupil area is determined by threshold segmentation, and the pupil edge points are estimated by polar coordinate transformation, thereby determining whether there is an area of ​​the pupil that is obscured. For the found pupil region, the region of interest of the pupil is obtained with the pupil center as the parameter; the region of interest is transformed by polar coordinates to obtain a polar coordinate radial derivative map, and the maximum value of each column of pixels in the polar coordinate radial derivative map may be the pupil edge point; Since the cornea is reflective, a reflection threshold is set to remove columns in the polar coordinate radial derivative plot and polar coordinate plot where the maximum value is greater than the threshold. Based on the grayscale difference between the eyelid, iris and pupil, the part of the pupil obscured by the eyelid in the polar coordinate transformation plot is removed. After obtaining the pupil edge points, the ellipse formed by the pupil edge points is considered to be the pupil range that best meets the requirements. The pupil occlusion can be determined by judging whether the edge points are a relatively complete ellipse; if there is occlusion, the empty area of ​​the ellipse is the occluded area.

7. The gaze tracking method for head-mounted optical aids according to claim 1, characterized in that, The process of determining whether the pupil is dilated includes the following steps: First, the presence of magnified occluded pupil information is detected using a grayscale histogram. The magnified pupil information is usually displayed as a cut-out black pupil portion on the visual aid. The magnified occluded pupil area can be obtained by detecting significant changes in grayscale. After detecting the existence of the magnified area, the cut-out pupil images displayed on the visual aid are merged according to the spacing of the visual aid to remove the gaps in the cut-out black pupil images and the black areas are merged sequentially. Secondly, since the magnification of the image by the visual aid remains basically constant, the merged pupil image is proportionally reduced according to the magnification to achieve merging with the unobstructed pupil area. The image that has undergone merging and shrinking is offset along the edge until it is connected to the edge of the unobstructed pupil image.

8. The gaze tracking method for head-mounted optical aids according to claim 1, characterized in that, The pupil fitting process includes the following steps: Starting from the coarsely located pupil center, and selecting a fixed step size, four neighboring points are determined from this starting point. For each pixel, given the ratio of the major and minor axes of the ellipse and the rotation angle of the ellipse, the minimum residual of each point is obtained. The residuals of all five points are compared. If the residual of the starting point is the smallest, the step size is halved, and the steps are repeated until the step size becomes one pixel. If the residual of the starting point is not the minimum value, the neighborhood point with the smallest residual is selected as the starting point, and the above steps are repeated to obtain the ellipse that best meets the conditions as the pupil.

9. A gaze tracking method for head-mounted optical aids according to claim 1, characterized in that, The eye reconstruction and gaze tracking include the following steps: Using a pinhole camera model to perform full perspective projection on the image, the possible projection space of the pupil is regarded as a cone with the camera focus as the vertex and the pupil circle as the base, and the pupil image is the intersection of the cone and the image plane. Based on the known pupil image, the pupil circle must meet the following requirements: First, the center of the correct pupil circle should be on the line connecting the center of the predicted pupil circle and the camera focus; second, the distance from the center of the circle to the center of the sphere should be the radius of the rotating sphere calculated in the above steps; and finally, there should be a proportional relationship between the parameters of the correct pupil circle and the predicted pupil circle. Based on the above requirements, the radius, center, and normal vector of the pupil circle are calculated.

Images