Sight tracking method for head-mounted optical typoscope

Abstract

The invention relates to the technical field of man-machine interaction. The invention provides a sight tracking method for a head-mounted optical typoscope, and the method comprises the following steps: firstly, carrying out the preprocessing of a collected image, secondly, judging the region where the typoscope is located, then solving the problems of shielding and deformation of pupils to obtain the contour of the pupils, carrying out the modeling and parameter solving of the pupils and eyeballs according to the contour of the pupils, and carrying out the tracking of the sight of the head-mounted optical typoscope. And finally, realizing sight tracking through coordinate system conversion and single-point calibration. The method can be used for tracking the sight of the user under the condition that the user wears the optical typoscope on the head, the sight tracking task of the head-mounted optical typoscope can be rapidly and accurately completed, and the method has good accuracy and robustness and has positive significance in attention mechanism research and the like of low-vision people.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a sight tracking method for a head-mounted optical visual aid. Background Art

[0002] An optical magnifier is a device that helps visually impaired people enhance their vision by magnifying or adjusting the field of vision. It can expand the user's field of vision or make images clearer. Optical magnifiers are easy to carry, simple to operate, and inexpensive, making them the preferred magnifiers for people with low vision. The main form of optical magnifiers is eyeglass-type magnifiers, most of which have specially made lenses with additional prisms, spherical mirrors, and other devices that enhance the user's field of vision. Users need to wear magnifiers normally like glasses. Due to the presence of special lenses, users usually have the problem of partially blocked pupils when wearing optical magnifiers on their heads. In this case, line of sight tracking and subsequent research are very difficult.

[0003] The common gaze tracking methods on the market are only applicable to the cases of not wearing glasses or wearing normal glasses. In the above two cases, the pupils do not have a large occlusion problem, so the accuracy is relatively high. However, if the current gaze tracking methods are applied to the gaze tracking of people wearing optical vision aids, they are unusable in most cases, and there will be large errors in some cases. This poses a challenge to subsequent research.

[0004] Therefore, considering the particularity of visually impaired people when wearing optical aids, we can first identify the aid area, then judge and process the enlarged part of the occluded area, and then obtain the pupil contour through ellipse fitting, and model the pupil and eyeball according to the pupil contour. Finally, line of sight tracking is achieved through coordinate system transformation and single-point calibration. Summary of the invention

[0005] According to the technical problems mentioned in the above background technology, a line of sight tracking method for head-mounted optical visual aids is provided. The present invention is mainly used for pupil recognition, eye modeling and gaze point estimation when wearing a head-mounted optical visual aid. It can fit and model the pupil blocked by the optical visual aid, solve the problem of pupil being blocked when the user wears an optical device, improve the accuracy of line of sight tracking under occlusion, and reduce time and economic costs. The present invention can quickly and accurately complete the line of sight tracking task, and provides great convenience for research fields such as attention mechanism detection under head-mounted optical visual aids. .

[0006] The technical means adopted by the present invention are as follows:

[0007] A sight tracking method for a head-mounted optical visual aid comprises the following steps:

[0008] Image acquisition and preprocessing: collect facial images through the camera, locate the human eye area using the feature point detection method, and locate the position of the visual aid using the cascade classifier;

[0009] Pupil area detection: predict the possible pupil area based on the human eye area and the position of the visual aid, and determine the pupil area using automatic threshold segmentation;

[0010] Pupil area processing: determine whether the pupil is blocked, determine whether the pupil is enlarged, use the enlarged pupil in the visual aid area and the unblocked pupil to establish an enlargement distortion correction model, and map the distorted enlarged pupil area to the unblocked pupil area;

[0011] Pupil fitting: performing ellipse fitting according to the current pupil area to obtain the fitted pupil edge, and adjusting the parameters through pupil symmetry transformation under normal lenses to obtain pupil data;

[0012] Eye reconstruction and line of sight tracking; reconstruct the pupil and eyeball into a disk tangent to the rotating sphere; the disk is the pupil, the rotating sphere is the eyeball, and the line of sight is the vector from the center of the sphere to the center of the circle; calculate the parameters of the disk and the rotating sphere, and use coordinate system transformation and single-point calibration to complete line of sight tracking.

[0013] Furthermore, during the image acquisition and preprocessing process, the image is preprocessed by graying and denoising, and then the human eye area is located by a feature point detection method and the position of the visual aid is located by a cascade classifier.

[0014] Furthermore, the image grayscale is converted using a weighted average method. During the processing, grayscale conversion can be completed by weighted averaging with different weights and the grayscale value can be normalized to 0-255.

[0015] Furthermore, the eye image contains Gaussian noise and salt and pepper noise, so a Gaussian filter and a median filter are used for denoising; the window size of the Gaussian filter is set to 5×5, and the window size of the median filter is set to 7×7.

[0016] Furthermore, the determination of whether the pupil is blocked is divided into three situations: the pupil is not blocked by the visual aid, the pupil is completely blocked by the visual aid, and the pupil is partially blocked by the visual aid.

[0017] Further, the determining whether the pupil is blocked comprises the following steps:

[0018] The contours in the image are filtered using the number of points that make up the contours; if the number of contours that are finally retained is 0, the image needs to be recaptured; if the number of contours that are retained is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the conditions;

[0019] Since the relative position of the camera and the head remains basically unchanged, when there are contours that meet the requirements, the pupil area is determined by the threshold segmentation method, and the pupil edge points are estimated by polar coordinate transformation to determine whether there is an obstructed area in the pupil;

[0020] For the pupil area found, the pupil center is used as a parameter to obtain the pupil region of interest; polar coordinate transformation is performed on the region of interest to obtain a polar coordinate radial derivative map, wherein the maximum value of each column of pixels in the polar coordinate radial derivative map may be the pupil edge point;

[0021] Since the cornea is reflective, a reflection threshold is set to remove columns whose maximum values ​​are greater than the threshold in the polar coordinate radial derivative map and the polar coordinate map. The part where the eyelid blocks the pupil in the polar coordinate transformation map is removed based on the grayscale difference between the eyelid, iris and pupil.

[0022] After obtaining the pupil edge points, the ellipse formed by the pupil edge points is considered to be the pupil range that best meets the requirements. The pupil occlusion can be judged by judging whether the edge points are a relatively complete ellipse; if there is occlusion, the empty area of ​​the ellipse is the occluded area.

[0023] Further, the determining whether the pupil is dilated comprises the following steps:

[0024] First, a grayscale histogram is used to detect whether there is pupil information of an enlarged occluded portion. The enlarged pupil information is usually displayed as a cut black pupil portion on the visual aid. The enlarged occluded pupil area can be obtained by detecting a significant change in grayscale. After the enlarged area is detected, the cut pupil images displayed on the visual aid are merged according to the spacing of the visual aid to remove the gaps in the cut black pupil images and sequentially merge the black areas.

[0025] Secondly, since the magnification of the image by the visual aid is basically unchanged, the merged pupil image is proportionally reduced according to the magnification to achieve merging with the unobstructed pupil area.

[0026] The image after the merged and reduced steps is offset along the edge until the image is connected to the edge of the unobstructed pupil image.

[0027] Furthermore, the pupil fitting process comprises the following steps:

[0028] The roughly located pupil center is taken as the starting point, and a fixed step size is selected to determine the four neighborhood points based on this starting point. For each pixel point, the minimum residual of each point is obtained under the premise of knowing the ratio of the major and minor axes of the ellipse and the rotation angle of the ellipse. The residuals of all five points are compared. If the residual of the starting point is the smallest, the step size is halved, and the steps are repeated until the step size becomes one pixel. If the residual of the starting point is not the minimum value, the neighborhood point with the smallest residual is selected as the starting point, and the above steps are repeated to obtain the most qualified ellipse as the pupil.

[0029] Furthermore, the eye reconstruction and gaze tracking comprises the following steps:

[0030] The pinhole camera model is used to perform full perspective projection on the image. The possible projection space of the pupil is regarded as a cone with the camera focus as the vertex and the pupil circle as the base. The pupil image is the intersection of the cone and the image plane.

[0031] Based on the known pupil image, the pupil circle meets the following requirements: first, the center of the correct pupil circle should be on the line connecting the center of the inferred pupil circle and the camera focus; second, the distance between the center of the circle and the center of the sphere should be the radius of the rotating sphere calculated in the above steps; finally, the parameters of the correct pupil circle and the inferred pupil circle should be proportional;

[0032] According to the above requirements, the radius, center and normal vector of the pupil circle are calculated.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] The present invention first pre-processes the collected images, then determines the area where the visual aid is located, then solves the problems of pupil occlusion and deformation to obtain the pupil contour, models the pupil and eyeball and solves the parameters according to the pupil contour, and finally realizes line of sight tracking through coordinate system conversion and single-point calibration. The main steps of the present invention all bring beneficial effects.

[0035] First, pupil occlusion is determined by contour screening and polar coordinate radial derivative graph, which ensures both parameter accuracy and recognition speed;

[0036] Second, the enlarged occluded area is processed and merged with the unoccluded area, which can achieve higher accuracy in subsequent pupil tracking;

[0037] Thirdly, the common ellipse fitting method is improved, and an iterative search method is selected, which can find the pupil ellipse that best meets the fitting conditions in a very short time;

[0038] Fourthly, the pupil is modeled as a disk tangent to the rotating sphere, which reduces the amount of calculation while ensuring accuracy.

[0039] The present invention can be used for the user's line of sight tracking when wearing a head-mounted optical vision aid, and can quickly and accurately complete the line of sight tracking task of the head-mounted optical vision aid, with good accuracy and robustness, and has positive significance for the study of the attention mechanism of visually impaired people. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0042] Figure 2 The figure is a simplified schematic diagram of an optical vision aid of the present invention.

[0043] Figure 3 This is a schematic diagram of pupil tracking under occlusion according to the present invention.

[0044] Figure 4 This is a structural diagram of the eye modeling of the present invention.

[0045] Figure 5 This is a schematic diagram of modeling parameters for solving the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] like Figure 1-5 As shown, the present invention provides a sight tracking method for a head-mounted optical visual aid, comprising the following steps:

[0049] Image acquisition and preprocessing: collect facial images through the camera, locate the human eye area using the feature point detection method, and locate the position of the visual aid using the cascade classifier. Specifically, people with visual field impairment are required to sit in front of the screen. A camera responsible for collecting images is placed above the screen. The camera can collect an overall image containing pupil data and facial data. In order to ensure that the collected image has sufficient details and facilitate the subsequent judgment of the pupil data, additional auxiliary light sources need to be added around the camera to illuminate the face. After people with visual field impairment wear optical visual aids, the reflective occlusion of the mirror will make it more difficult to locate the pupil. Therefore, when collecting images, it is necessary to first add a specific polarizer in front of the camera to perform preliminary filtering of the reflection. This step can greatly reduce the number of reflective spots.

[0050] After removing the reflections on the image, in order to improve the accuracy of pupil tracking, the image needs to be preprocessed. The preprocessing steps include image grayscale and image denoising. After the preprocessing is completed, the human eye area can be located through the feature point detection method. The specific steps are as follows:

[0051] Image grayscale: Since the images captured by the camera are color images and the processing speed of color images is slow, the color images can be converted into grayscale images with faster processing speed. Although the grayscale images lose some color information, from the perspective of pupil tracking, the information contained in the grayscale image and the corresponding color image is consistent. The image grayscale can be completed by weighted averaging. During the processing, the grayscale conversion can be completed by weighted averaging with different weights and the grayscale value is normalized to 0-255.

[0052] Image denoising: Eye images are easily contaminated by noise during acquisition and transmission due to the influence of the environment and sensor quality, so noise analysis is required. Eye images mainly contain Gaussian noise and salt and pepper noise, which can be reduced by filters. According to the noise type, the filter can use Gaussian filter and median filter. The window size of the Gaussian filter is set to 5×5, and the window size of the median filter is set to 7×7. The above settings can effectively reduce Gaussian noise and salt and pepper noise.

[0053] Furthermore, pupil area detection; based on the human eye area and the position of the visual aid, the possible pupil area is predicted, and the pupil area is determined by automatic threshold segmentation; optical visual aids usually have special lenses with extended visual aids installed, and the eyes under the special lenses may be blocked. Since there is a relatively obvious boundary between the visual aid and the normal lens, the present invention uses a cascade classifier to realize the detection of the visual aid. The cascade classifier is mainly used for target detection. It connects a series of simple classifiers in series to gradually filter out areas that do not meet the conditions. The approximate range of the visual aid can be obtained by feature extraction and cascade screening of facial images.

[0054] Furthermore, pupil area processing is performed to determine whether the pupil is blocked, to determine whether the pupil is dilated, and to establish a magnification distortion correction model using the enlarged pupil in the visual aid area and the unblocked pupil, and to map the distorted enlarged pupil area to the unblocked pupil area.

[0055] The preprocessed image is filtered to screen the contour, the pupil area is determined by threshold segmentation, and it is judged whether the pupil is blocked.

[0056] After obtaining the position of the visual aid, there are three situations for the position of the pupil and the visual aid, namely, the pupil is not blocked by the visual aid, the pupil is completely blocked by the visual aid, and the pupil is partially blocked by the visual aid. In the first case, it is only necessary to perform normal pupil tracking. In the second and third cases, it is necessary to determine whether there is an offset on auxiliary equipment such as lenses. If so, it is necessary to perform affine transformation and ellipse fitting to obtain the pupil area.

[0057] After preprocessing the image, the next step is to extract the pupil contour and determine whether it is blocked. Since there are many contours on the image, a reasonable filtering method needs to be set to find the true pupil contour. Experiments show that the edge point data that make up the pupil contour is usually between 100 and 300, so the number of points that make up the contour can be used to filter the contours in the image. If the number of contours retained is 0, the image needs to be recaptured; if the number of contours retained is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the conditions.

[0058] Since the relative position of the camera and the head remains basically unchanged, when there are enough contours, the pupil area can be determined by the threshold segmentation method, and the pupil edge points can be estimated by polar coordinate transformation to determine whether there is an obstructed area of ​​the pupil.

[0059] For the pupil area found, the pupil center is used as a parameter to obtain the pupil region of interest. By analyzing the radial derivative of the area, it is found that the pupil edge point is part of the radial edge. The polar coordinate transformation of the region of interest can obtain the polar coordinate radial derivative map. The maximum value of each column of pixels in the map may be the pupil edge point. Since the cornea is reflective and may produce specular reflection, a reflection threshold can be set to eliminate the columns in the polar coordinate radial derivative map and the polar coordinate map whose maximum value is greater than the threshold. According to the grayscale difference between the eyelid, iris and pupil, the part of the pupil blocked by the eyelid in the polar coordinate transformation map can be eliminated. After obtaining the pupil edge point, the ellipse formed by the pupil edge points is considered to be the most satisfactory pupil range. The pupil occlusion can be judged by judging whether the edge point is a relatively complete ellipse. If there is occlusion, the empty area of ​​the ellipse is the occluded area.

[0060] After obtaining the occluded area, determine whether there is an area where the occluded pupil part is enlarged. If so, the occluded pupil area needs to be restored through affine transformation and merged with the unoccluded pupil area.

[0061] After completing the pupil occlusion detection, the occluded area of ​​the pupil can be obtained. For this area, it is necessary to consider whether the occluded part is magnified and visible due to the principle of aided vision. Due to the imaging principle of the aid, the occluded part of the pupil is not completely invisible. The occluded part is usually displayed in the aid on the special lens due to the refraction of light. At this time, the displayed occluded part is an image that is offset and cut after a certain degree of magnification based on the actual occluded part. The relationship between the two is an affine transformation, which can be merged with the unobstructed area after processing to improve the accuracy of pupil tracking. The specific steps are as follows:

[0062] On the basis of identifying the area of ​​the visual aid, firstly, the grayscale histogram can be used to detect whether there is pupil information of the enlarged occluded part. The enlarged pupil information is usually displayed as a cut black pupil part on the visual aid. The enlarged and displayed occluded pupil area can be obtained by detecting the significant change of grayscale; after the existence of the enlarged area is detected, the cut part of the pupil image displayed on the visual aid needs to be merged according to the spacing of the visual aid. Due to the different settings of the visual aid, this step needs to be combined with the imaging principle of the visual aid and the relevant parameters of the head-mounted device, such as the fixed spacing between the visual aids, to remove the gaps in the cut black pupil image and sequentially merge the black areas; secondly, since the magnification of the image by the visual aid is basically unchanged, the next step needs to proportionally reduce the merged pupil image according to the magnification before it can be merged with the unobstructed pupil area.

[0063] Experiments show that the pupil image displayed on the visual aid often has a certain degree of deviation from the normal image, and the direction of the deviation is basically fixed, that is, along the edge of the frame. Based on this conclusion, the image that has been merged and reduced can be offset along the edge until the image is connected to the edge of the unobstructed pupil image; in order to verify the accuracy of the processing of this step, the pupil parameters before and after the deviation can be estimated by the least squares method. If they are close, it means that the processing steps are reasonable. Through the above steps, the enlarged occluded pupil area can be processed and merged with the unoccluded pupil part, which is convenient for subsequent pupil tracking.

[0064] Further, pupil fitting: Ellipse fitting is performed according to the current pupil area to obtain the pupil edge after fitting, and the parameters are adjusted through the pupil symmetry transformation under normal lenses to obtain pupil data. Ellipse fitting is performed according to the current pupil area to obtain the pupil edge after fitting, and the parameters are adjusted through the pupil symmetry transformation under normal lenses to obtain pupil data with higher accuracy.

[0065] After completing the processing and merging of the enlarged pupil occlusion area, it is necessary to obtain complete pupil data based on the known pupil area and complete pupil tracking. Here, the ellipse fitting method is used to complete the pupil fitting and tracking.

[0066] In order to obtain the most suitable ellipse as quickly as possible, the invention adopts an iterative search method, and the method ends when the best ellipse is matched. First, the roughly located pupil center is used as the starting point, and a suitable step size is selected to determine the four neighborhood points based on this starting point; for each pixel point, the minimum residual of each point can be obtained under the premise of the known ratio of the major and minor axes of the ellipse and the rotation angle of the ellipse, and the residuals of all five points are compared. If the residual of the starting point is the smallest, the step size is halved, and the above steps are repeated until the step size becomes one pixel; if the residual of the starting point is not the minimum value, the field point with the smallest residual is selected as the starting point, and the above steps are repeated. According to the above steps, the most suitable ellipse can be obtained as the pupil.

[0067] In order to verify the correctness of the pupil obtained by ellipse fitting, the present invention performs a symmetric transformation on the eyes under normal lenses, and compares the pupil obtained by the symmetric transformation with the pupil obtained by ellipse fitting. If the error is within a reasonable range, the fitting is considered successful. Symmetric transformation is a local operator for gradient images, which is used to describe the symmetry of each point on the image. When performing symmetric transformation for pupil tracking, it is usually necessary to select different symmetry factors. The symmetry factor can be determined by the physiological characteristics of the face, which is generally one tenth of the width of the face. The parameters of the pupil can be fitted through symmetric transformation, and the parameters obtained by symmetric transformation are compared with the parameters obtained by ellipse fitting. When the gap is small, it is considered that the result of ellipse fitting has a higher accuracy. If there is a significant difference, the pupil data is adjusted in combination with the parameters of the two to complete pupil tracking with higher accuracy.

[0068] Finally, the eye reconstruction and line of sight tracking are performed; the pupil and the eyeball are reconstructed into a disk tangent to the rotating sphere; the disk is the pupil, the rotating sphere is the eyeball, and the line of sight is the vector from the center of the sphere to the center of the circle; the parameters of the disk and the rotating sphere are calculated, and the line of sight tracking is completed using coordinate system conversion and single-point calibration. After the above steps, the contour information of the pupil can be obtained. In order to perform line of sight tracking, the eye needs to be reconstructed in three dimensions. Since we only consider the direction and position of the pupil, there is no need to reconstruct the complete model of the eye. Only the pupil and its range of motion need to be simulated. The present invention models the relationship between the pupil and the eyeball as a disk tangent to the rotating sphere, wherein the disk is the pupil, the rotating sphere is the eyeball, and the line of sight is the vector from the center of the sphere to the center of the circle.

[0069] In order to determine the relevant parameters of the disk and the rotating sphere, the present invention uses a pinhole camera model to perform full perspective projection on the image. The possible projection space of the pupil can be regarded as a cone with the camera focus as the vertex and the pupil circle as the base. The pupil image is the intersection of the cone and the image plane. Based on the known pupil image, the cone can be reconstructed, and the circular intersection of the cone is the pupil circle.

[0070] The radius, center coordinates, and normal vector of the pupil circle in space can be obtained through 3D solution in the cone. However, since the size of the pupil circle cannot be determined by the back projection from 2D to 3D, in this step, it is only necessary to determine the radius of the pupil circle at will. After the radius of the rotating sphere is obtained by subsequent calculation, the specific parameters of the pupil circle can be determined.

[0071] When back-projecting the pupil image, given an arbitrary pupil radius, two pupil circles with the same radius, different normal vectors, and different centers can be found in three-dimensional space. Therefore, in order to correctly find the center of the rotating sphere, it is necessary to perform a disambiguation operation by projecting the pupil circle and the pupil circle normal vector. After the disambiguation operation, the normal vectors of the two pupil circles are parallel to each other, and the line connecting the projection centers of the two pupil circles is parallel to the normal vector. This means that after projection, only one of the two pupil circles needs to be selected for normal vector projection operation.

[0072] For each frame of pupil image, the normal vector projection can be obtained according to the above steps. Theoretically, the intersection of all normal vector projections is the projection of the center of the rotating sphere. Since there may be numerical, discretization or measurement errors in the vector, these lines are unlikely to intersect at the same point. Therefore, the point closest to each projection normal vector can be found by the least squares method. This point is the projection of the center of the rotating sphere. The coordinates can be back-projected to obtain the coordinates of the center of the sphere in three-dimensional space.

[0073] After the calculation of the coordinates of the center of the sphere is completed, the radius of the rotating sphere needs to be calculated. Theoretically, the line connecting the center of the pupil circle and the center of the sphere is the radius of the sphere. However, since the size of the pupil cannot be determined, the pupil circle cannot be determined naturally. Therefore, the pupil circle generated according to an arbitrary radius in the previous step needs to be corrected here. According to the imaging principle of line of sight, the center of the correct pupil circle should be on the line connecting the camera focus and the center of the currently generated pupil circle, and the normal vectors of the two are the same. Therefore, consider drawing a straight line from the center of the rotating sphere along the direction of the normal vector and finding the intersection with the straight line where the center of the circle may be. The above operation can be performed for each pupil image. Similarly, in order to avoid discretization or measurement deviations, this step also needs to use the least squares method to find the point on the line connecting the camera intersection point and the center of the circle that is closest to all straight lines emitted from the center of the sphere. This point is on the rotating sphere, so the distance from this point to the center of the sphere is the radius of the rotating sphere;

[0074] After determining the radius of the rotating sphere, the next step is to estimate the pupil. Based on the above analysis, it can be found that based on the known pupil image, the pupil circle meets the following requirements: first, the center of the correct pupil circle should be on the line connecting the center of the inferred pupil circle and the camera focus; second, the distance from the center of the circle to the center of the sphere should be the radius of the rotating sphere calculated in the above steps; finally, the parameters of the correct pupil circle and the inferred pupil circle should be proportional; according to the above constraints, the radius, center and normal vector of the pupil circle can be calculated.

[0075] After obtaining the relevant parameters of the rotating ball and the pupil circle, the line connecting the center of the ball and the center of the circle is the line of sight. At this time, the subsequent line of sight tracking can be completed through the conversion of the coordinate system. Taking into account individual differences, single-point calibration is required before line of sight tracking. A bright spot will appear in the center of the screen. Participants need to look at the bright spot and try to keep their eyes open. This step lasts for 3-5 seconds. The background will collect human eye data and fine-tune the model. The adjusted model can achieve more accurate line of sight tracking.

[0076] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0077] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0079] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0080] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sight tracking method for a head-mounted optical visual aid, characterized in that: The following steps are involved: Image acquisition and preprocessing: collect facial images through the camera, locate the human eye area using the feature point detection method, and locate the position of the visual aid using the cascade classifier; Pupil area detection; According to the human eye area and the position of the visual aid, the possible pupil area is predicted, and the pupil area is determined by automatic threshold segmentation; Pupil area processing; Determine whether the pupil is blocked, determine whether the pupil is enlarged, establish an enlargement distortion correction model using the enlarged pupil in the visual aid area and the unblocked pupil, and map the distorted enlarged pupil area to the unblocked pupil area; Pupil fitting: performing ellipse fitting according to the current pupil area to obtain the fitted pupil edge, and adjusting the parameters through pupil symmetry transformation under normal lenses to obtain pupil data; Eye reconstruction and gaze tracking; reconstructing the pupil and eyeball as a disk tangent to the rotating sphere; The disk is the pupil, the rotating ball is the eyeball, and the line of sight is the vector from the center of the ball to the center of the circle. The parameters of the disk and the rotating ball are calculated, and the line of sight tracking is completed using coordinate system conversion and single-point calibration.

2. A sight tracking method for a head-mounted optical visual aid according to claim 1, characterized in that: In the image acquisition and preprocessing process, the image is preprocessed by graying and denoising, and then the human eye area is located by a feature point detection method and the position of the visual aid is located by a cascade classifier.

3. A sight tracking method for a head-mounted optical visual aid according to claim 2, characterized in that: The image grayscale is converted by weighted averaging. During the processing, grayscale conversion can be completed by weighted averaging with different weights and the grayscale value can be normalized to 0-255.

4. The sight tracking method for a head-mounted optical visual aid according to claim 2, characterized in that: The eye image contains Gaussian noise and salt and pepper noise, so Gaussian filter and median filter are used for denoising; the window size of the Gaussian filter is set to 5×5, and the window size of the median filter is set to 7×7.

5. The sight tracking method for a head-mounted optical visual aid according to claim 1, characterized in that: The determination of whether the pupil is blocked is divided into three situations: the pupil is not blocked by the visual aid, the pupil is completely blocked by the visual aid, and the pupil is partially blocked by the visual aid.

6. A sight tracking method for a head-mounted optical visual aid according to claim 1 or 5, characterized in that: Determining whether the pupil is blocked comprises the following steps: The contours in the image are filtered using the number of points that make up the contours; if the number of contours that are finally retained is 0, the image needs to be recaptured; if the number of contours that are retained is not 0, pupil detection and parameter estimation are performed on the retained contours that meet the conditions; Since the relative position of the camera and the head remains basically unchanged, when there are contours that meet the requirements, the pupil area is determined by the threshold segmentation method, and the pupil edge points are estimated by polar coordinate transformation to determine whether there is an obstructed area in the pupil; For the pupil area found, the pupil center is used as a parameter to obtain the pupil region of interest; polar coordinate transformation is performed on the region of interest to obtain a polar coordinate radial derivative map, wherein the maximum value of each column of pixels in the polar coordinate radial derivative map may be the pupil edge point; Since the cornea is reflective, a reflection threshold is set to remove columns whose maximum values ​​are greater than the threshold in the polar coordinate radial derivative map and the polar coordinate map. The part where the eyelid blocks the pupil in the polar coordinate transformation map is removed based on the grayscale difference between the eyelid, iris and pupil. After obtaining the pupil edge points, the ellipse formed by the pupil edge points is considered to be the pupil range that best meets the requirements. The pupil occlusion can be judged by judging whether the edge points are a relatively complete ellipse; if there is occlusion, the empty area of ​​the ellipse is the occluded area.

7. The sight tracking method for a head-mounted optical visual aid according to claim 1, characterized in that: Determining whether the pupil is dilated comprises the following steps: First, a grayscale histogram is used to detect whether there is pupil information of an enlarged occluded portion. The enlarged pupil information is usually displayed as a cut black pupil portion on the visual aid. The enlarged occluded pupil area can be obtained by detecting a significant change in grayscale. After the enlarged area is detected, the cut pupil images displayed on the visual aid are merged according to the spacing of the visual aid to remove the gaps in the cut black pupil images and sequentially merge the black areas. Secondly, since the magnification of the image by the visual aid is basically unchanged, the merged pupil image is proportionally reduced according to the magnification to achieve merging with the unobstructed pupil area. The image after the merged and reduced steps is offset along the edge until the image is connected to the edge of the unobstructed pupil image.

8. The sight tracking method for a head-mounted optical visual aid according to claim 1, characterized in that: The pupil fitting process comprises the following steps: The roughly located pupil center is taken as the starting point, and a fixed step size is selected to determine the four neighborhood points based on this starting point. For each pixel point, the minimum residual of each point is obtained under the premise of knowing the ratio of the major and minor axes of the ellipse and the rotation angle of the ellipse. The residuals of all five points are compared. If the residual of the starting point is the smallest, the step size is halved, and the steps are repeated until the step size becomes one pixel. If the residual of the starting point is not the minimum value, the neighborhood point with the smallest residual is selected as the starting point, and the above steps are repeated to obtain the most qualified ellipse as the pupil.

9. The sight tracking method for a head-mounted optical visual aid according to claim 1, characterized in that: The eye reconstruction and gaze tracking comprises the following steps: The pinhole camera model is used to perform full perspective projection on the image. The possible projection space of the pupil is regarded as a cone with the camera focus as the vertex and the pupil circle as the base. The pupil image is the intersection of the cone and the image plane. Based on the known pupil image, the pupil circle meets the following requirements: first, the center of the correct pupil circle should be on the line connecting the center of the inferred pupil circle and the camera focus; second, the distance between the center of the circle and the center of the sphere should be the radius of the rotating sphere calculated in the above steps; finally, the parameters of the correct pupil circle and the inferred pupil circle should be proportional; According to the above requirements, the radius, center and normal vector of the pupil circle are calculated.

Images