Sight line estimation method based on calibration and pupil contour and related device

Abstract

The invention discloses a calibration and pupil contour-based sight line estimation method and a related device. The method comprises the following steps of: obtaining a calibrated pupil image frame sequence; performing pupil recognition on each frame of image to obtain a pupil contour, and obtaining an eyeball center and a virtual optical axis and a pupil center corresponding to each frame of image based on the pupil contours of multiple frames of images; performing corneal refraction correction on the virtual optical axis to obtain an actual optical axis of the eyeball corresponding to each frame of image; obtaining a calibration object image frame sequence, and recording position information of a fixation point; constructing a mapping relation based on the position information of the corresponding fixation point and the actual optical axis of the eyeball; and acquiring a pupil image frame sequence of the user, and obtaining position information of the fixation point based on the mapping relation. Personal calibration can be performed by using simple personal calibration data, the calibration procedure is effectively simplified, and the calibration method is low in cost and high in convenience; corneal refraction effect correction is carried out on the pupil optical axis, and the sight line estimation precision is effectively improved.

Description

Technical Field

[0001] The present application belongs to the technical field of line of sight estimation algorithms, and specifically relates to a line of sight estimation method based on calibration and pupil contour and related devices. Background Art

[0002] Existing head-mounted eye trackers mainly estimate line of sight based on two types of eye movement features: pupil-Purkinje spot vector or pupil center. However, in actual usage scenarios, since users will rotate their eyeballs significantly when observing things, the Purkinje spot will fall on the sclera instead of the cornea, making it difficult to identify, resulting in the failure of the line of sight estimation method based on the pupil-Purkinje spot vector. At the same time, due to the convenience of head-mounted eye trackers, users can move freely when using the eye tracker, which will cause the head-mounted eye tracker to slip during the movement. The line of sight estimation method based on the pupil center uses a polynomial regression method to find the mapping relationship between the pupil center and the gaze point, and the slippage of the head-mounted eye tracker will destroy the mapping relationship, resulting in a decrease in the accuracy of line of sight estimation.

[0003] In order to solve the above problems, a line of sight estimation method based on pupil contour is proposed in the prior art, which uses the pupil contour to calculate the three-dimensional pupil normal vector of the human eye. However, the existing line of sight estimation method based on pupil contour does not consider the influence of the corneal refraction effect, resulting in low estimation accuracy. At the same time, before formally performing line of sight estimation, personal calibration is performed by looking at dozens of designated position calibration points, and the calibration process is cumbersome and redundant. Summary of the invention

[0004] The purpose of the present application is to provide a line of sight estimation method and related devices based on calibration and pupil contour, so as to solve the technical problems that the line of sight estimation method in the prior art does not take into account the influence of corneal refraction effect, resulting in low estimation accuracy; at the same time, before formally performing line of sight estimation, personal calibration is performed by gazing at dozens of designated position calibration points, and the calibration process is cumbersome and redundant.

[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a sight line estimation method based on calibration and pupil contour, comprising:

[0006] Acquire a calibration pupil image frame sequence, wherein the calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object that moves arbitrarily on the calibration wall during the calibration process, wherein the calibration object is a finger;

[0007] Performing pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtaining an eyeball center and a virtual optical axis and pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images;

[0008] Based on the virtual optical axis and the pupil center, performing corneal refraction correction on the virtual optical axis to obtain the actual optical axis of the eyeball corresponding to each frame of image;

[0009] Acquire a calibration object image frame sequence, and record the position information of the gaze point in each frame of the image, wherein the calibration object image frame sequence includes a motion image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object;

[0010] Based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, a mapping relationship is constructed;

[0011] A pupil image frame sequence of the user is obtained, the actual optical axis of the eyeball corresponding to each frame of the image is calculated, and the position information of the gaze point is obtained based on the mapping relationship.

[0012] In one or more embodiments, the step of performing pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtaining an eyeball center and a virtual optical axis and a pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images comprises:

[0013] Performing pupil recognition on each frame of the calibrated pupil image frame sequence based on a pupil detection algorithm to obtain a pupil contour;

[0014] Back-projecting the pupil contour to obtain a three-dimensional pupil circle, and calculating a normal vector of the three-dimensional pupil circle;

[0015] Calculate the intersection of the normal vectors of the three-dimensional pupil circle corresponding to multiple frames of images to obtain the coordinates of the center of the eyeball;

[0016] Based on the eyeball center, construct an eyeball model;

[0017] Back-projecting the center of the pupil contour and combining it with the normal vector direction of the three-dimensional pupil circle to obtain a modified three-dimensional pupil circle tangent to the eyeball model;

[0018] The normal vector and center coordinates of the modified three-dimensional pupil circle are calculated to obtain the virtual optical axis and pupil center of the eyeball.

[0019] In one or more embodiments, the step of performing corneal refraction correction on the virtual optical axis based on the virtual optical axis and the pupil center to obtain the actual optical axis of the eyeball corresponding to each frame of image includes:

[0020] Constructing a virtual optical axis coordinate system based on the optical center of the eye camera, the virtual optical axis and the pupil center;

[0021] Based on the geometric optical analysis of human eyes, the relative rotation angle between the actual optical axis and the virtual optical axis in the virtual optical axis coordinate system is calculated;

[0022] Based on the relative rotation angle, the virtual optical axis is corrected for corneal refraction to obtain the actual optical axis.

[0023] In one or more embodiments, the origin of the virtual optical axis coordinate system is the optical center o of the eye camera, the z-axis direction is op′, and the y-axis direction is op′×l′, where p′ is the pupil center and l′ is the virtual optical axis;

[0024] The step of calculating the relative rotation angle between the actual optical axis and the virtual optical axis in the virtual optical axis coordinate system based on the human eye geometric optical analysis is specifically as follows:

[0025] Based on the following formula, the relative rotation angle α between the actual optical axis and the virtual optical axis when the y-axis of the virtual optical axis coordinate system is used as the rotation axis is calculated:

[0026] α=12.07·arctan(0.0139·θ)

[0027]

[0028] The step of performing corneal refraction correction on the virtual optical axis based on the relative rotation angle to obtain the actual optical axis is specifically as follows:

[0029] l=R y (α)l′

[0030] In the formula, R y (α) represents the rotation matrix of the virtual optical axis coordinate system rotating around the y-axis by an angle α, and l is the actual optical axis.

[0031] In one or more embodiments, the step of constructing a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball includes:

[0032] Construct polynomial regression equations;

[0033] The corresponding position information of the gaze point and the actual optical axis of the eyeball are substituted into the polynomial regression equation and solved to obtain a mapping relationship.

[0034] In one or more embodiments, the polynomial regression equation is as follows: F=AB;

[0035] Where, the coordinates of the gaze point F = (x, y, z) T , A is the solution matrix, Spherical coordinates of the actual optical axis

[0036] In one or more embodiments, the position information of the gaze point also includes the depth of the calibration object, and the step of constructing a mapping relationship based on the corresponding position information of the gaze point and the actual optical axis of the eyeball includes:

[0037] Constructing a transformation matrix for transforming the actual optical axis into the scene camera coordinate system, and transforming the actual optical axis based on the transformation matrix to obtain the scene visual axis;

[0038] Constructing a functional relationship model of the scene visual axis, the center of the human cornea and the position information of the gaze point;

[0039] An initial value of the center of the human cornea is set, and the transformation matrix and the center of the human cornea are iterated based on the position information of the corresponding gaze point and the actual optical axis of the eyeball. In each iteration, the transformation matrix is ​​solved based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, and the center of the human cornea is updated based on the transformation matrix;

[0040] After the iteration is completed, the target person's corneal center and the target transformation matrix are output;

[0041] Substitute the target person's corneal center and the target transformation matrix into the functional relationship model to obtain a mapping relationship.

[0042] In one or more embodiments, the functional relationship model is as follows: F=C+ωg, where the scene visual axis g=Rl, R is the transformation matrix, l is the actual optical axis, C is the center of the human cornea, ω is the depth of the calibration object, and F is the coordinate of the calibration object;

[0043] In one or more embodiments, the step of solving the target transformation matrix based on the position information of the corresponding gaze point and the actual optical axis of the eyeball in each iteration, and updating the center of the human cornea based on the target transformation matrix includes:

[0044] The optical axis data matrix M is constructed based on the following formula:

[0045] M=[M1;M2;…;M i ](i=1,2,…,N), Where N is the number of actual optical axis data, l i is the actual optical axis of the i-th group;

[0046] The gaze vector set W is constructed based on the following formula:

[0047] W=[W1;W2;…W N ](i=1,2…N), Among them, C is the center of the human cornea, F i C is the gaze vector of the i-th group;

[0048] Substitute the optical axis data matrix M and gaze vector set W into the following formula Get the row vector expansion r of the target transformation matrix:

[0049] A temporary matrix R′ is constructed based on the row vector expansion r, and a singular value decomposition is performed on the temporary matrix R′ based on the following formula: R′=USV T ;

[0050] The conversion matrix is ​​calculated based on the following formula: R = UV T ;

[0051] Update the human cornea center based on the following formula: C = (∑ i I-R i (Rl i ) T ) -1 (∑ i (I-Rl i (Rl i ) T )F i ), where I is the unit matrix.

[0052] To achieve the above-mentioned purpose, the second aspect of the present application provides a sight line estimation device based on calibration and pupil contour, comprising:

[0053] A first acquisition module is used to acquire a calibration pupil image frame sequence, wherein the calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object that moves arbitrarily on the calibration wall during the calibration process, wherein the calibration object is a finger;

[0054] A virtual optical axis calculation module is used to perform pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtain an eyeball center and a virtual optical axis and pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images;

[0055] An actual optical axis calculation module, used to perform corneal refraction correction on the virtual optical axis based on the virtual optical axis and the pupil center, so as to obtain the actual optical axis of the eyeball corresponding to each frame of image;

[0056] A second acquisition module is used to acquire a calibration object image frame sequence and record the position information of the gaze point in each frame of the image, wherein the calibration object image frame sequence includes a motion image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object;

[0057] A mapping relationship building module, used to build a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball;

[0058] The sight estimation module is used to obtain the user's pupil image frame sequence, calculate the actual optical axis of the eyeball corresponding to each frame of the image, and obtain the position information of the gaze point based on the mapping relationship.

[0059] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0060] at least one processor; and

[0061] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the sight line estimation method based on calibration and pupil contour as described in any of the above embodiments.

[0062] To achieve the above objectives, the fourth aspect of the present application provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the line of sight estimation method based on calibration and pupil contour as described in any of the above embodiments.

[0063] Different from the prior art, the beneficial effects of this application are:

[0064] The visual estimation method of the present application can use the user's simple personal calibration data for personal calibration. The user only needs to look at the finger on the calibration wall. The finger can be the user's own finger or someone else's finger. By analyzing the movement video of the pupil and the movement video of the finger during the user's calibration process, rapid calibration can be achieved, which effectively simplifies the calibration procedure. The calibration method is low-cost and highly convenient.

[0065] The visual estimation method of the present application corrects the corneal refraction effect on the user's pupil optical axis, which can effectively improve the accuracy of line of sight estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a flowchart of an implementation method of a sight line estimation method based on calibration and pupil contour of the present application;

[0068] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an implementation method corresponding to S200;

[0069] Figure 3 yes Figure 1 A schematic flow chart of an implementation method corresponding to S300;

[0070] Figure 4 yes Figure 1 A schematic flow chart of an implementation method corresponding to S500;

[0071] Figure 5 yes Figure 1 A schematic flow chart of another implementation method corresponding to S500;

[0072] Figure 6 It is a structural schematic diagram of an embodiment of a sight line estimation device based on calibration and pupil contour of the present application;

[0073] Figure 7 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0075] In order to solve the problems existing in the existing line of sight estimation methods, the applicant has developed a new line of sight estimation method, which can realize personal calibration based on a simple marking method. The calibration method is low-cost and convenient, and combines the corneal refraction effect to effectively improve the line of sight estimation accuracy.

[0076] Specifically, see Figure 1 , Figure 1 It is a flowchart of an implementation method of the line of sight estimation method based on calibration and pupil contour of the present application.

[0077] like Figure 1 As shown, the method includes:

[0078] S100: Obtain a calibrated pupil image frame sequence.

[0079] The calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object that moves arbitrarily on the calibration wall during the calibration process.

[0080] The calibration object may be a finger.

[0081] Before using the head-mounted eye tracker, users need to perform personal calibration to obtain the mapping relationship between human eye features and gaze points. In the calibration process of this application, users only need to look at the calibration object that moves arbitrarily on the marking wall. During the marking process, the built-in eye camera in the eye tracker can collect the user's pupil movement video, that is, calibrate the pupil image frame sequence.

[0082] Specifically, in one embodiment, the calibration object may be the user's right index finger or the right index finger of another person, so as to facilitate the subsequent identification of the right index finger tip and obtain the motion coordinates of the calibration object.

[0083] In one embodiment, the calibration wall may be any solid-color wall, and the user may stand at a designated position at a certain distance from the calibration wall to perform calibration.

[0084] S200, performing pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and based on the pupil contours of multiple frames of images, obtaining an eyeball center and a virtual optical axis and pupil center of the eyeball corresponding to each frame of the image.

[0085] After obtaining the user's calibrated pupil image frame sequence, pupil recognition can be performed on each frame to obtain the user's pupil contour in the frame. Since the pupil is a three-dimensional structure, the pupil contour is elliptical. Based on the pupil contours of multiple frames of images, the coordinates of the user's eyeball center can be calculated.

[0086] Specifically, see Figure 2 , Figure 2 yes Figure 1 A flow chart of an implementation method corresponding to S200 in FIG.

[0087] like Figure 2 As shown, the method for calculating the coordinates of the eye center is:

[0088] S201 . Perform pupil recognition on each frame of the calibrated pupil image frame sequence based on a pupil detection algorithm to obtain a pupil contour.

[0089] First, the pupil contour in the image is identified based on a pupil detection algorithm. The algorithm may adopt any image recognition algorithm commonly used in the art, such as a convolutional neural network model, etc., which will not be described in detail here.

[0090] S202, back-projecting the pupil contour to obtain a three-dimensional pupil circle, and calculating a normal vector of the three-dimensional pupil circle.

[0091] The elliptical pupil contour can be back-projected to obtain a three-dimensional pupil circle, and the normal vector of the three-dimensional pupil circle can be calculated.

[0092] S203, calculating the intersection of the normal vectors of the three-dimensional pupil circles corresponding to the multiple frames of images to obtain the coordinates of the center of the eyeball.

[0093] Since the normal vector of the three-dimensional pupil circle of each frame image passes through the center of the eyeball, the intersection of the normal vectors of the three-dimensional pupil circles of multiple frames of images can be used as the coordinates of the center of the eyeball.

[0094] Since the pupil of the human eye is a circular disk tangent to the eyeball, after obtaining the center of the eyeball, the pupil contour can be re-projected to reconstruct a three-dimensional pupil circle, and the virtual optical axis and pupil center can be obtained accordingly.

[0095] Specifically, see Figure 2 , the calculation methods of the virtual optical axis and pupil center include:

[0096] S204: construct an eyeball model based on the eyeball center.

[0097] After obtaining the coordinates of the center of the eyeball, a spherical eyeball model can be constructed.

[0098] S205 , back-projecting the center of the pupil contour and combining it with the normal vector direction of the three-dimensional pupil circle to obtain a modified three-dimensional pupil circle tangent to the eyeball model.

[0099] Based on the eyeball model constructed above, the pupil contour can be re-back-projected, and the three-dimensional pupil circle obtained by the back-projection is made tangent to the eyeball model, that is, the three-dimensional pupil circle is corrected.

[0100] S206, calculating the normal vector and center coordinates of the corrected three-dimensional pupil circle to obtain the virtual optical axis and pupil center of the eyeball.

[0101] S300, performing corneal refraction correction on the virtual optical axis to obtain the actual optical axis of the eyeball corresponding to each frame of image.

[0102] In the above S200, we obtain the coordinates of the virtual optical axis and the pupil center of each frame of the pupil by analyzing the pupil image, but the virtual optical axis does not take into account the influence of the corneal refraction effect. Therefore, it is necessary to perform corneal refraction correction on the virtual optical axis to obtain the corrected actual optical axis.

[0103] Specifically, see Figure 3 , Figure 3 yes Figure 1 A flow chart of an implementation method corresponding to S300 in FIG.

[0104] like Figure 4 As shown, the calculation method of the actual optical axis includes:

[0105] S301. Construct a virtual optical axis coordinate system based on the optical center of the eye camera, the virtual optical axis and the pupil center.

[0106] First, a virtual optical axis coordinate system can be constructed, in which the relationship between the virtual optical axis and the actual optical axis can be positioned for relative rotation based on optical principles.

[0107] In one embodiment, the origin of the virtual optical axis coordinate system may be the optical center o of the eye camera, the z-axis direction may be op', and the y-axis direction may be op'×l', where p' is the pupil center and l' is the virtual optical axis. It is understandable that the x-axis direction may be obtained based on the right-hand rule.

[0108] S302 . Calculate the relative rotation angle between the actual optical axis and the virtual optical axis in the virtual optical axis coordinate system based on human eye geometric optical analysis.

[0109] In the virtual optical axis coordinate system constructed in the example of S301, the relationship between the actual optical axis and the virtual optical axis can be defined as: relative rotation with the y-axis as the rotation axis, so the virtual optical axis can be corrected after the relative rotation angle is calculated.

[0110] Specifically, the relative rotation angle α between the actual optical axis and the virtual optical axis when the y-axis of the virtual optical axis coordinate system is used as the rotation axis can be calculated based on the following formula:

[0111] α=12.07·arctan(0.0139·θ)

[0112]

[0113] Where p' is the pupil center, l' is the virtual optical axis, and o is the optical center of the eye camera.

[0114] S303: Based on the relative rotation angle, the virtual optical axis is corrected by corneal refraction to obtain the actual optical axis. Specifically, after obtaining the relative rotation angle, the virtual optical axis can be corrected based on the following formula:

[0115] l=R y (α)l′

[0116] In the formula, R y (α) represents the rotation matrix of the virtual optical axis coordinate system around the y-axis with an angle α, and l is the actual optical axis.

[0117] S400: Obtain a calibration object image frame sequence, and record the position information of the gaze point in each frame of the image.

[0118] The calibration object image frame sequence includes a moving image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object.

[0119] Specifically, in one implementation, the coordinates of the calibration object may be the coordinates of the fingertips of the fingers of the right hand.

[0120] In the above S100 to S300, we obtain the actual optical axis of the pupil of each frame image based on the analysis of the pupil image, and need to build a model based on the correlation between the actual optical axis and the actual gaze point of the user, so that the user's gaze point can be estimated based on the actual optical axis of the user's pupil in the subsequent line of sight estimation. Therefore, it is first necessary to obtain the motion trajectory of the calibration object during the calibration process, that is, the motion trajectory of the fingertip.

[0121] During the marking process, the eye camera built into the eye tracker collects the user's pupil movement video while the scene camera built into the eye tracker also synchronously collects the user's gaze scene video, that is, the finger movement video. Therefore, a sequence of calibration object image frames can be obtained. In each frame of the image, the coordinates of the calibration object can be obtained based on the algorithm, and the position information of the gaze point can be obtained.

[0122] In one embodiment, when the calibration object is the right index finger, the coordinates of the user's right index fingertip in the gaze scene video can be identified based on the Mediapipe hand recognition algorithm as the coordinates of the calibration object. In other embodiments, when other calibration objects are used, the calibration object can also be identified and the coordinates recorded based on the recognition algorithm, and the effect of this embodiment can be achieved.

[0123] S500: construct a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball.

[0124] Since the calibration object image frame sequence and the calibration pupil image frame sequence correspond to each other frame by frame, the corresponding position information of the gaze point and the actual optical axis of the eyeball can be obtained. Based on the corresponding information, a mapping relationship between the two can be constructed.

[0125] In one embodiment, when the application scenario of the eye tracker is fixed-distance line of sight estimation, such as eye-tracking typing, the depth of the calibration object can be ignored and the mapping relationship can be directly constructed.

[0126] Specifically, see Figure 4 , Figure 4 yes Figure 1 A flow chart of an implementation method corresponding to S500 in FIG.

[0127] like Figure 4 As shown, the method for constructing the mapping relationship includes:

[0128] S501a. Construct a polynomial regression equation.

[0129] S502a, substituting the position information of the corresponding gaze point and the actual optical axis of the eyeball into and solving the polynomial regression equation to obtain a mapping relationship.

[0130] In one embodiment, the actual optical axis can be converted from a rectangular coordinate system to a spherical coordinate system to obtain the spherical coordinates of the actual optical axis: Then construct the following polynomial regression equation:

[0131] F=AB

[0132] Where, the coordinates of the gaze point F = (x, y, z) T , A is the solution matrix,

[0133]

[0134] It can be understood that in the above formula, A is a 3x7 matrix, and each element in the A matrix is ​​an unknown number. By substituting the known corresponding actual optical axis and the coordinates of the calibration object into the above formula, A can be solved and then the mapping relationship can be obtained.

[0135] In other implementations, other conventional regression fitting methods may also be used, all of which can achieve the effect of this implementation.

[0136] In another embodiment, when the application scenario of the eye tracker is line of sight estimation at a non-fixed distance, the scene camera can also record the depth of the calibration object when collecting the motion video of the calibration object. At this time, the position information of the gaze point also includes the depth of the calibration object. By combining the depth and coordinates of the calibration object, a mapping relationship can be constructed.

[0137] Specifically, see Figure 5 , Figure 5 yes Figure 1 A flowchart of another implementation method corresponding to S500 in FIG.

[0138] like Figure 5 As shown, the steps of constructing a mapping relationship by combining the depth and coordinates of the calibration object include:

[0139] S501b, constructing a transformation matrix for transforming the actual optical axis into the scene camera coordinate system, and transforming the actual optical axis based on the transformation matrix to obtain the scene visual axis.

[0140] First, the actual optical axis needs to be transformed into the same coordinate system as the calibration object, that is, the scene camera coordinate system; in this embodiment, the transformation of the actual optical axis is achieved through a learnable transformation matrix.

[0141] In one embodiment, the calculation formula of the scene visual axis is as follows: g=Rl, R is the transformation matrix, and l is the actual optical axis.

[0142] S502b, constructing a functional relationship model of the scene visual axis, the center of the human cornea and the position information of the gaze point.

[0143] According to optical analysis, the position of the gaze point can be calculated from the center of the human cornea, the depth of the gaze point and the scene visual axis, as shown in the following formula: F=C+ωg, where C is the center of the human cornea, ω is the depth of the calibration object, and F is the coordinate of the calibration object.

[0144] Therefore, a functional relationship model can be constructed based on the above formula.

[0145] S503b. Set the initial value of the center of the human cornea, iterate the transformation matrix and the center of the human cornea based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, and in each iteration, solve the transformation matrix based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, and update the center of the human cornea based on the transformation matrix.

[0146] S504b: After the iteration is completed, the target person's corneal center and the target transformation matrix are output.

[0147] Using the known corresponding gaze point position information and the actual optical axis of the eyeball, the above functional relationship model can be iterated. In each iteration, the gaze vector can be calculated based on the coordinates of the human cornea center and the calibration object output by the previous iteration. Based on the gaze vector and the actual optical axis, the transformation matrix can be solved. The human cornea center can be further updated using the transformation matrix. After the iteration is completed, the final human cornea center and target transformation matrix can be output.

[0148] The number of iterations can be preset based on actual needs.

[0149] In one embodiment, the initial value of the center of the human cornea can be set to C=(0,0,0) T .

[0150] Specifically, in one embodiment, the iterative method may be specifically as follows:

[0151] The optical axis data matrix M is constructed based on the following formula:

[0152] M=[M1;M2;…;M i ](i=1,2,…,N), Where N is the number of actual optical axis data, l i is the actual optical axis of the i-th group;

[0153] The gaze vector set W is constructed based on the following formula:

[0154] W=[W1;W2;…W N ](i=1,2…N), Among them, C is the center of the human cornea, F i C is the gaze vector of the i-th group;

[0155] Substitute the optical axis data matrix M and the gaze vector set W into the following formula Get the row vector expansion r of the target transformation matrix:

[0156] A temporary matrix R′ is constructed based on the row vector expansion r, and a singular value decomposition is performed on the temporary matrix R′ based on the following formula: R′=USV T ;

[0157] The conversion matrix is ​​calculated based on the following formula: R = UV T ;

[0158] Update the human cornea center based on the following formula: C = (∑ i I-R i (Rl i ) T ) -1 (∑ i (I-Rl i (Rl i ) T )F i ).

[0159] S505b, substituting the target person's corneal center and the target transformation matrix into the functional relationship model to obtain a mapping relationship.

[0160] By substituting the target human cornea center and the target transformation matrix obtained in the above iterative process into the functional relationship model, the functional relationship between the actual optical axis and the coordinates and depth of the calibration object, that is, the mapping relationship, can be obtained.

[0161] S600: Obtain a pupil image frame sequence of the user, calculate the actual optical axis of the eyeball corresponding to each frame of the image, and obtain the position information of the gaze point based on the mapping relationship.

[0162] In the above S100 to S500, personal calibration is completed and a customized mapping relationship is obtained. After that, the formal visual estimation operation can be started. By collecting the user's pupil image and performing similar analysis as the above S100 to S300, the actual optical axis of the user's pupil can be obtained.

[0163] By substituting the actual optical axis of the user into the mapping relationship obtained in S500, the position information of the calibration object can be obtained, thereby achieving visual estimation.

[0164] On the one hand, the visual estimation method of the above-mentioned embodiments can utilize the user's simple personal calibration data for personal calibration, which effectively simplifies the calibration procedure. The calibration method is low-cost and highly convenient. On the other hand, the user's pupil optical axis is corrected for the corneal refraction effect, which can effectively improve the accuracy of line of sight estimation.

[0165] This application also provides a sight line estimation device based on calibration and pupil contour, see Figure 6 , Figure 6 It is a structural schematic diagram of an implementation of a sight line estimation device based on calibration and pupil contour of the present application.

[0166] like Figure 6 As shown, the sight line estimation device includes: a first acquisition module 21, a virtual optical axis calculation module 22, an actual optical axis calculation module 23, a second acquisition module 24, a mapping relationship construction module 25 and a sight line estimation module 26.

[0167] The first acquisition module 21 is used to acquire a calibration pupil image frame sequence, wherein the calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object moving arbitrarily on the calibration wall during the calibration process;

[0168] The virtual optical axis calculation module 22 is used to perform pupil recognition on each frame of the calibrated pupil image frame sequence to obtain the pupil contour, and obtain the eyeball center and the virtual optical axis and pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images;

[0169] The actual optical axis calculation module 23 is used to perform corneal refraction correction on the virtual optical axis to obtain the actual optical axis of the eyeball corresponding to each frame of image;

[0170] The second acquisition module 24 is used to acquire a calibration object image frame sequence and record the position information of the gaze point in each frame of the image. The calibration object image frame sequence includes a motion image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object.

[0171] The mapping relationship building module 25 is used to build a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball;

[0172] The sight estimation module 26 is used to obtain a sequence of pupil image frames of the user, calculate the actual optical axis of the eyeball corresponding to each frame of the image, and obtain the position information of the gaze point based on the mapping relationship.

[0173] As above Figures 1 to 5 , a sight line estimation method according to an embodiment of this specification is described. The details mentioned in the above description of the method embodiment are also applicable to the sight line estimation device of the embodiment of this specification. The above sight line estimation device can be implemented by hardware, or by software, or by a combination of hardware and software.

[0174] This application also provides an electronic device, see Figure 7 , Figure 7 Schematic diagram of the structure of an electronic device of the present application. Figure 7As shown, the electronic device 30 may include at least one processor 31, a memory 32 (e.g., a non-volatile memory), a memory 33, and a communication interface 34, and the at least one processor 31, the memory 32, the memory 33, and the communication interface 34 are connected together via a bus 35. At least one processor 31 executes at least one computer-readable instruction stored or encoded in the memory 32.

[0175] It should be understood that the computer executable instructions stored in the memory 32, when executed, cause at least one processor 31 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 5 Describes the various operations and functions.

[0176] In the embodiments of the present specification, the electronic device 30 may include, but is not limited to, personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, and the like.

[0177] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of this specification. Figure 1-Figure 5 Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.

[0178] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0179] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0180] Those skilled in the art should understand that the various embodiments disclosed above can be modified and altered in various ways without departing from the essence of the invention. Therefore, the protection scope of this specification should be defined by the appended claims.

[0181] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or some components in multiple independent devices may be implemented together.

[0182] In the above embodiments, the hardware unit or module can be realized by mechanical or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. The hardware unit or processor can also include programmable logic or circuit (such as general-purpose processor or other programmable processor), which can be temporarily set by software to complete the corresponding operation. Specific implementation (mechanical method or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0183] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0184] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles corresponding to the present disclosure may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for estimating sight lines based on calibration and pupil contour, characterized in that: include: Acquire a calibration pupil image frame sequence, wherein the calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object that moves arbitrarily on the calibration wall during the calibration process, wherein the calibration object is a finger; Performing pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtaining an eyeball center and a virtual optical axis and a pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images; Perform corneal refraction correction on the virtual optical axis to obtain the actual optical axis of the eyeball corresponding to each frame of image; Acquire a calibration object image frame sequence, and record the position information of the gaze point in each frame of the image, wherein the calibration object image frame sequence includes a motion image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object; Based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, a mapping relationship is constructed; A pupil image frame sequence of the user is obtained, the actual optical axis of the eyeball corresponding to each frame of the image is calculated, and the position information of the gaze point is obtained based on the mapping relationship.

2. The line of sight estimation method according to claim 1, characterized in that: The step of performing pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtaining an eyeball center and a virtual optical axis and a pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images comprises: Performing pupil recognition on each frame of the calibrated pupil image frame sequence based on a pupil detection algorithm to obtain a pupil contour; Back-projecting the pupil contour to obtain a three-dimensional pupil circle, and calculating a normal vector of the three-dimensional pupil circle; Calculate the intersection of the normal vectors of the three-dimensional pupil circle corresponding to multiple frames of images to obtain the coordinates of the center of the eyeball; Based on the eyeball center, construct an eyeball model; Back-projecting the center of the pupil contour and combining it with the normal vector direction of the three-dimensional pupil circle to obtain a modified three-dimensional pupil circle tangent to the eyeball model; The normal vector and center coordinates of the modified three-dimensional pupil circle are calculated to obtain the virtual optical axis and pupil center of the eyeball.

3. The line of sight estimation method according to claim 1, characterized in that: The step of performing corneal refraction correction on the virtual optical axis based on the virtual optical axis and the pupil center to obtain the actual optical axis of the eyeball corresponding to each frame of image comprises: Constructing a virtual optical axis coordinate system based on the optical center of the eye camera, the virtual optical axis and the pupil center; Based on the geometric optical analysis of human eyes, the relative rotation angle between the actual optical axis and the virtual optical axis in the virtual optical axis coordinate system is calculated; Based on the relative rotation angle, the virtual optical axis is corrected for corneal refraction to obtain the actual optical axis.

4. The line of sight estimation method according to claim 3, characterized in that: The origin of the virtual optical axis coordinate system is the optical center o of the eye camera, the z-axis direction is op′, and the y-axis direction is op′×l′, where p′ is the pupil center and l′ is the virtual optical axis; The step of calculating the relative rotation angle between the actual optical axis and the virtual optical axis in the virtual optical axis coordinate system based on the human eye geometric optical analysis is specifically as follows: Based on the following formula, the relative rotation angle α between the actual optical axis and the virtual optical axis when the y-axis of the virtual optical axis coordinate system is used as the rotation axis is calculated: α=12.07·arctan(0.0139·θ) The step of performing corneal refraction correction on the virtual optical axis based on the relative rotation angle to obtain the actual optical axis is specifically as follows: l＝R y (α)l ′ In the formula, R y (α) represents the rotation matrix of the virtual optical axis coordinate system rotating around the y-axis by an angle α, and l is the actual optical axis.

5. The line of sight estimation method according to claim 1, characterized in that: The step of constructing a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball comprises: Construct polynomial regression equations; The corresponding position information of the gaze point and the actual optical axis of the eyeball are substituted into the polynomial regression equation and solved to obtain a mapping relationship.

6. The line of sight estimation method according to claim 5, characterized in that: The polynomial regression equation is as follows: F = AB; Where, the coordinates of the gaze point F = (x, y, z) T , A is the solution matrix, Spherical coordinates of the actual optical axis 7. The sight line estimation method according to claim 1, characterized in that: The position information of the gaze point also includes the depth of the calibration object. The step of constructing a mapping relationship based on the corresponding position information of the gaze point and the actual optical axis of the eyeball includes: Constructing a transformation matrix for transforming the actual optical axis into the scene camera coordinate system, and transforming the actual optical axis based on the transformation matrix to obtain the scene visual axis; Constructing a functional relationship model of the scene visual axis, the center of the human cornea and the position information of the gaze point; An initial value of the center of the human cornea is set, and the transformation matrix and the center of the human cornea are iterated based on the position information of the corresponding gaze point and the actual optical axis of the eyeball. In each iteration, the transformation matrix is ​​solved based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, and the center of the human cornea is updated based on the transformation matrix; After the iteration is completed, the target person's corneal center and the target transformation matrix are output; Substitute the target person's corneal center and the target transformation matrix into the functional relationship model to obtain a mapping relationship.

8. The sight line estimation method according to claim 7, characterized in that: The functional relationship model is as follows: F=C+ωg, where the scene visual axis g=Rl, R is the transformation matrix, l is the actual optical axis, C is the center of the human cornea, ω is the depth of the calibration object, and F is the coordinate of the calibration object.

9. The sight line estimation method according to claim 8, characterized in that: The step of solving the transformation matrix in each iteration based on the position information of the corresponding gaze point and the actual optical axis of the eyeball, and updating the center of the human cornea based on the transformation matrix includes: The optical axis data matrix M is constructed based on the following formula: Where N is the number of actual optical axis data, l i is the actual optical axis of the i-th group; The gaze vector set W is constructed based on the following formula: W=[W1;W2;…W N ](i=1,2…N), Among them, C is the center of the human cornea, F i C is the gaze vector of the i-th group; Substitute the optical axis data matrix M and gaze vector set W into the following formula Get the row vector expansion r of the target transformation matrix: A temporary matrix R′ is constructed based on the row vector expansion r, and a singular value decomposition is performed on the temporary matrix R′ based on the following formula: R′=USV T ; The conversion matrix is ​​calculated based on the following formula: R = UV T ; Update the human cornea center based on the following formula: C = (∑ i I-R i (Rl i ) T ) -1 (∑ i (I-Rl i (Rl i ) T )F i ), where I is the unit matrix.

10. A sight line estimation device based on calibration and pupil contour, characterized in that: include: A first acquisition module is used to acquire a calibration pupil image frame sequence, wherein the calibration pupil image frame sequence includes a moving image of the pupil when the user gazes at a calibration object that moves arbitrarily on the calibration wall during the calibration process, wherein the calibration object is a finger; A virtual optical axis calculation module is used to perform pupil recognition on each frame of the calibrated pupil image frame sequence to obtain a pupil contour, and obtain an eyeball center and a virtual optical axis and pupil center of the eyeball corresponding to each frame of the image based on the pupil contours of multiple frames of images; An actual optical axis calculation module, used to perform corneal refraction correction on the virtual optical axis to obtain the actual optical axis of the eyeball corresponding to each frame of image; A second acquisition module is used to acquire a calibration object image frame sequence and record the position information of the gaze point in each frame of the image, wherein the calibration object image frame sequence includes a motion image of the calibration object during the calibration process, and the position information of the gaze point includes the coordinates of the calibration object; A mapping relationship building module, used to build a mapping relationship based on the position information of the corresponding gaze point and the actual optical axis of the eyeball; The sight estimation module is used to obtain the user's pupil image frame sequence, calculate the actual optical axis of the eyeball corresponding to each frame of the image, and obtain the position information of the gaze point based on the mapping relationship.

11. An electronic device, comprising: at least one processor; as well as A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to perform the sight line estimation method based on calibration and pupil contour as described in any one of claims 1 to 9.

12. A machine-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the sight line estimation method based on calibration and pupil contour as described in any one of claims 1 to 9.

Images