Eye tracking method

Abstract

An eye tracking method, comprising: acquiring coordinate information of a target point and eyeball coordinate information of a simulated eyeball; on the basis of the coordinate information of the target point and the eyeball coordinate information, determining a first target angle at which a first rotating component drives the simulated eyeball to rotate when the simulated eyeball tracks the target point, and a second target angle at which a second rotating component drives the first rotating component and the simulated eyeball to synchronously rotate; and on the basis of the first target angle, the second target angle and motion parameter information, determining angle error information of rotation of the simulated eyeball, and controlling the simulated eyeball to rotate, so that the simulated eyeball tracks the target point.

Description

Eye tracking methods

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number: 202311641298.X and application date of November 30, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present disclosure relates to the technical field of visual equipment, and in particular to an eye tracking method, apparatus, and eye tracking equipment. Background Art

[0004] With the rapid development of fields such as optical instruments and intelligent robotics, the demand for devices that simulate eye movements is increasing. These devices can be used for quality inspection of optical instruments such as eye trackers and ophthalmic optical biometers, and can also be used as components in intelligent robotics. Devices that simulate eye movements are widely used in eye tracking, which measures the movement of the eyes, focusing on where the eyes are looking (the "fixation point" or "gaze point").

[0005] Currently, the motion control accuracy of the equipment used to simulate eye movements in eye tracking research is low and the simulation effect is poor.

[0006] Summary of the Invention

[0007] The present disclosure aims to solve at least one of the technical problems existing in the prior art. To this end, the present disclosure provides an eye tracking method, apparatus, and device that can effectively improve the motion control accuracy of eye tracking.

[0008] In a first aspect, the present disclosure provides an eye tracking method, the method being applied to at least one simulated eye having a first rotating component and a second rotating component to track a target point, wherein the rotation axis of the first rotating component and the rotation axis of the second rotating component are non-parallel and the two rotation axes are located on two non-parallel planes, and a line connecting the pupil and the center of the simulated eye forms a first angle with an axial direction of the first rotating component; the method comprising:

[0009] Obtaining the coordinate information of the target point and the eye coordinate information of the simulated eyeball;

[0010] determining, based on the coordinate information of the target point and the coordinate information of the eyeball, a first target angle to which the simulated eyeball should be rotated by the first rotating component when the simulated eyeball tracks the target point, and a second target angle to which the first rotating component and the simulated eyeball should be rotated when the second rotating component drives the first rotating component and the simulated eyeball to rotate synchronously;

[0011] determining angle error information of the simulated eyeball rotation based on the first target angle, the second target angle, and motion parameter information of the simulated eyeball;

[0012] Based on the first target angle, the second target angle and the angle error information, the simulated eyeball is controlled to rotate so that the simulated eyeball tracks the target point.

[0013] According to the eye tracking method disclosed in the present invention, by acquiring the coordinate information of the target point and the coordinate information of the eyeball, the rotation action of the simulated eyeball looking at the target point is decomposed into the rotation of the first rotating component and the second rotating component. According to the required rotation angles of the two rotating components and the corresponding angle error information, the motion of the simulated eyeball is controlled, which can effectively improve the motion control accuracy of eye tracking.

[0014] According to one embodiment of the present disclosure, determining, based on the coordinate information of the target point and the eyeball coordinate information, a first target angle to which the simulated eyeball should rotate when the simulated eyeball tracks the target point by the first rotating component, and a second target angle to which the second rotating component should rotate when the first rotating component and the simulated eyeball rotate synchronously, specifically includes:

[0015] determining a target pitch angle required for the simulated eyeball to track the target point based on the coordinate information of the target point and the eyeball coordinate information;

[0016] determining the first target angle based on the target pitch angle and the first included angle;

[0017] The second target angle is determined based on the coordinate information of the target point, the first target angle, and the eye coordinate information.

[0018] According to one embodiment of the present disclosure, determining the second target angle based on the coordinate information of the target point, the first target angle, and the eye coordinate information includes:

[0019] determining, based on the first target angle and the movement trajectory of the pupil, a projection displacement deviation of the pupil on a horizontal plane before and after the first rotating component rotates;

[0020] determining a second compensation angle based on the projection displacement deviation and the distance of the projection relative to the center of the eyeball, wherein the second compensation angle compensates for the angular deviation of the pupil in the horizontal plane;

[0021] determining a target azimuth angle of the simulated eyeball on a horizontal plane based on the coordinate information of the target point and the eyeball coordinate information;

[0022] The second target angle is determined based on the second compensation angle and the target azimuth angle.

[0023] According to one embodiment of the present disclosure, determining the angle error information of the simulated eye rotation based on the first target angle, the second target angle, and the motion parameter information of the simulated eye includes:

[0024] Inputting the first target angle, the second target angle, and the motion parameter information of the simulated eyeball into an error prediction model, and obtaining the angle error information of the simulated eyeball output by the error prediction model;

[0025] The error prediction model is obtained by training based on the training sample set of the simulated eyeball, and the training sample set includes a sample target rotation angle, template motion parameter information and a sample actual rotation angle.

[0026] According to one embodiment of the present disclosure, determining the angle error information of the simulated eye rotation based on the first target angle, the second target angle, and the motion parameter information of the simulated eye includes:

[0027] Based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball, solving according to the fitting function corresponding to the error prediction model to obtain the angle error information of the simulated eyeball;

[0028] The error prediction model is trained in the cloud, and the fitting function is obtained by interpolating and fitting the trained error prediction model.

[0029] According to one embodiment of the present disclosure, controlling the simulated eye movement based on the first target angle, the second target angle, and the angle error information includes:

[0030] Determining, based on the angle error information, a first predicted angle corresponding to the first target angle and a second predicted angle corresponding to the second target angle;

[0031] Establishing an optimization objective function based on the first target angle, the second target angle, the first predicted angle, and the second predicted angle in a target time domain with a minimum cumulative angle error;

[0032] Solve the optimization objective function and control the simulated eye movement.

[0033] According to one embodiment of the present disclosure, obtaining coordinate information of a target point includes:

[0034] receiving a first input from a user;

[0035] In response to the first input, coordinate information of the target point is obtained.

[0036] According to one embodiment of the present disclosure, obtaining coordinate information of a target point includes:

[0037] Acquire an image of the target point;

[0038] The image of the target point is recognized to obtain coordinate information of the target point.

[0039] In a second aspect, the present disclosure provides an eye tracking device, the device acting on at least one simulated eyeball having a first rotating component and a second rotating component to track a target point, wherein the rotation axis of the first rotating component and the rotation axis of the second rotating component are non-parallel and the two rotation axes are located on two non-parallel planes, and a line connecting the pupil and the center of the simulated eyeball forms a first angle with the axial direction of the first rotating component; the device comprises:

[0040] An acquisition module, configured to acquire the coordinate information of the target point and the eye coordinate information of the simulated eye;

[0041] a first processing module, configured to determine, based on the coordinate information of the target point and the eyeball coordinate information, a first target angle to which the simulated eyeball should be rotated by the first rotating component when the simulated eyeball tracks the target point, and a second target angle to which the first rotating component and the simulated eyeball should be rotated when the second rotating component drives the first rotating component and the simulated eyeball to rotate synchronously;

[0042] a second processing module, configured to determine angle error information of the simulated eyeball rotation based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball;

[0043] The third processing module is used to control the rotation of the simulated eyeball based on the first target angle, the second target angle and the angle error information, so that the simulated eyeball tracks the target point.

[0044] According to the eye tracking device disclosed in the present invention, by acquiring the coordinate information of the target point and the coordinate information of the eyeball, the rotation action of the simulated eyeball looking at the target point is decomposed into the rotation of the first rotating component and the second rotating component. According to the required rotation angles of the two rotating components and the corresponding angle error information, the motion of the simulated eyeball is controlled, which can effectively improve the motion control accuracy of eye tracking.

[0045] In a third aspect, the present disclosure provides an eye tracking device, comprising:

[0046] At least one simulated eyeball, the simulated eyeball having a first rotating part and a second rotating part;

[0047] The rotation axis of the first rotating component is not parallel to the rotation axis of the second rotating component, and the two rotation axes are located in two non-parallel planes, and the line connecting the pupil and the center of the simulated eyeball forms a first angle with the axial direction of the first rotating component;

[0048] A controller is electrically connected to the first rotating component and the second rotating component, and the controller controls the simulated eye movement to track the target point based on the eye tracking method described in the first aspect.

[0049] In a fourth aspect, the present disclosure provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the eye tracking method as described in the first aspect above when executing the computer program.

[0050] In a fifth aspect, the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye tracking method as described in the first aspect above.

[0051] In a sixth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the eye tracking method as described in the first aspect above.

[0052] Additional aspects and advantages of the present disclosure will be given in part in the description that follows and, in part, will be obvious from the description that follows, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] FIG1 is a flow chart of an eye tracking method according to an embodiment of the present disclosure;

[0055] FIG2 is a schematic diagram of the structure of an eye tracking device according to an embodiment of the present disclosure;

[0056] FIG3 is a second structural diagram of an eye tracking device provided by an embodiment of the present disclosure;

[0057] FIG4 is a third structural diagram of an eye tracking device provided in an embodiment of the present disclosure;

[0058] FIG5 is a schematic diagram of simulating eye movement to generate deviation according to an embodiment of the present disclosure;

[0059] FIG6 is a second schematic diagram of simulating eye movement to generate deviation according to an embodiment of the present disclosure;

[0060] FIG7 is a second flow chart of the eye tracking method provided by an embodiment of the present disclosure;

[0061] FIG8 is a third flow chart of the eye tracking method provided by an embodiment of the present disclosure;

[0062] FIG9 is a fourth flow chart of the eye tracking method provided in an embodiment of the present disclosure;

[0063] FIG10 is a schematic diagram of the structure of an eye tracking device provided by an embodiment of the present disclosure;

[0064] FIG11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0065] Reference numerals:

[0066] Simulated eyeball 210 , eyeball structure 211 , artificial eye patch 212 , bracket 220 , first rotating component 230 , second rotating component 240 , display screen 300 . DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present disclosure.

[0068] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects related to each other are in an "or" relationship.

[0069] The eye tracking method, eye tracking apparatus, eye tracking device, electronic device, and readable storage medium provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0070] The eye tracking method and eye tracking device of the embodiments of the present disclosure act on at least one simulated eyeball 210 having a first rotating component 230 and a second rotating component 240 to track a target point. The rotation axis of the first rotating component 230 and the rotation axis of the second rotating component 240 are not parallel and the two rotation axes are located on two non-parallel planes. The line connecting the pupil and the center of the simulated eyeball 210 forms a first angle with the axial direction of the first rotating component 230.

[0071] In some embodiments, the first rotating component 230 is connected to the simulated eyeball 210, and the first rotating component 230 drives the simulated eyeball 210 to rotate; the second rotating component 240 is connected to the first rotating component 230, and the second rotating component 240 drives the first rotating component 230 and the simulated eyeball 210 to rotate synchronously.

[0072] The first rotating component 230 is used to adjust the pitch angle of the simulated eyeball 210, and the second rotating component 240 is used to adjust the horizontal azimuth angle of the simulated eyeball 210. The rotation axis of the first rotating component 230 is not parallel to the rotation axis of the second rotating component 240, and the two rotation axes are on two non-parallel planes. The first rotating component 230 and the second rotating component 240 can realize the full-range rotation of the simulated eyeball 210.

[0073] It should be noted that the line of sight of the simulated eyeball 210 is the direction of the line connecting the pupil and the center of the eyeball of the simulated eyeball 210. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230 (the first angle is not 0° or 180°). When the first rotating component 230 drives the simulated eyeball 210 to move around the axial direction of the first rotating component 230, the position of the pupil of the simulated eyeball 210 in the vertical direction changes, and the angle between the line of sight and the horizontal direction changes accordingly, thereby realizing pitch angle adjustment.

[0074] It can be understood that if the line connecting the pupil and the center of the eyeball is on the same straight line as the axis of the first rotating component 230, when the first rotating component 230 drives the simulated eyeball 210 to move around the axis of the first rotating component 230, the pupil of the simulated eyeball 210 is on the rotation axis, and the position of the pupil in the vertical direction will not change, and the pitch angle adjustment cannot be achieved.

[0075] In actual implementation, the axial setting angle of the first rotating component 230 can be adjusted according to the angle range of the pitch angle required to adjust the simulated eyeball 210. The first rotating component 230 can be set in a horizontal direction or in an inclined direction with a certain angle to the horizontal direction.

[0076] For example, the axis of the first rotating component 230 is horizontal, and the line connecting the pupil and the center of the eyeball forms a first angle with the axis of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 210 to rotate, the movement trajectory of the pupil on the simulated eyeball 210 is located on the vertical plane, thereby realizing the pitch angle adjustment of the line of sight of the simulated eyeball 210.

[0077] For another example, the axial direction of the first rotating component 230 is inclined at a certain angle to the horizontal direction, and the axial direction of the first rotating component 230 is not completely vertical. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 210 to rotate, the movement trajectory of the pupil on the simulated eyeball 210 is on a non-horizontal plane, and the pitch angle adjustment can also be achieved.

[0078] The eye tracking method provided by the embodiment of the present disclosure is used to control the simulated eyeball 210 to gaze at a target point.

[0079] The executor of the eye tracking method may be an electronic device or a functional module or functional entity in the electronic device that can implement the eye tracking method. The electronic devices mentioned in the embodiments of the present disclosure include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The eye tracking method provided by the embodiments of the present disclosure is described below using an electronic device as an example of the executor.

[0080] As shown in FIG1 , the eye tracking method includes steps 110 to 140 .

[0081] Step 110 : Acquire the coordinate information of the target point and the eye coordinate information of the simulated eye 210 .

[0082] In this step, a spatial rectangular coordinate system may be established to obtain the coordinate information of the target point and the eye coordinate information of the simulated eye 210 according to the relative positional relationship between the simulated eye 210 and the target point.

[0083] For example, as shown in FIG. 2 , the simulated eyeball 210 is gazing at a target point on the display screen 300 .

[0084] The eye centers of the two simulated eyeballs 210 are d and e respectively, and the vertical mapping points of d and e on the display screen 300 are a and b respectively. ad=be=h, where h is the visual depth, that is, the distance between the eye centers of the simulated eyeballs 210 and the display screen 300, and de=IPD, where IPD is the pupil distance.

[0085] The midpoint of the line connecting points a and b is the screen center point O of the display screen 300 , and point c is the target point on the display screen 300 .

[0086] A spatial rectangular coordinate system can be established with the midpoint of the line connecting the two simulated eyeballs 210 (the midpoint of the line connecting points d and e) as the coordinate origin, the line connecting the two simulated eyeballs 210 as the x-axis, the line connecting the midpoint of the line connecting the two simulated eyeballs 210 to the center point of the display screen 300 as the y-axis, and the vertical direction as the z-axis.

[0087] According to the relative position relationship between point c and the center point O of the screen, the coordinate information (x0, h, z0) of the target point of point c is obtained, and the eye coordinate information of the two simulated eyeballs 210 are (1 / 2IPD, 0, 0) and (-1 / 2IPD, 0, 0) respectively.

[0088] In this embodiment, a spatial rectangular coordinate system can be established at a certain position of the simulated eyeball 210, or at any position in space to obtain the coordinate information of the target point and the eye coordinate information of the simulated eyeball 210.

[0089] Step 120: Based on the coordinate information of the target point and the eyeball coordinate information, determine the first target angle to which the first rotating component 230 should drive the simulated eyeball 210 to rotate when the simulated eyeball 210 looks at the target point, and the second target angle to which the second rotating component 240 should drive the first rotating component 230 and the simulated eyeball 210 to rotate synchronously.

[0090] In this embodiment, the simulated eyeball 210 gazing at the target point means that the eyeball center, pupil and target point of the simulated eyeball 210 are located in the same straight line, that is, the target point is located in the direction of the line of sight of the simulated eyeball 210 .

[0091] In this step, based on the coordinate information of the target point and the eye coordinate information of the simulated eyeball 210, the position deviation between the line of sight of the simulated eyeball 210 and the target point is determined, and the line of sight of the simulated eyeball 210 is adjusted to face the target point by rotating the first rotating component 230 and the second rotating component 240.

[0092] In actual implementation, the first target angle and the second target angle can be calculated based on the difference between the current eye coordinate information of the simulated eye 210 and the corresponding eye coordinate information when the simulated eye 210 rotates to the gaze target point.

[0093] It can be understood that when the first rotating component 230 drives the simulated eyeball 210 to rotate to the first target angle, and the second rotating component 240 drives the simulated eyeball 210 and the first rotating component 230 to rotate synchronously to the second target angle, the eyeball center, pupil and target point of the simulated eyeball 210 can be adjusted to the same straight line, that is, the line of sight of the simulated eyeball 210 is facing the target point.

[0094] For example, the first rotating component 230 drives the simulated eyeball 210 to rotate in the vertical direction, and the second rotating component 240 drives the simulated eyeball 210 to rotate in the horizontal direction.

[0095] In this embodiment, the first target angle P is calculated based on the coordinate information of the target point and the eye coordinate information. c , the second target angle is Q c , rotate the first rotating member 230 by P c , the second rotating member 240 rotates Q c Afterwards, the sight line direction of the simulated eyeball 210 can be adjusted to face the target point.

[0096] It should be noted that the eyeball coordinate information of the two simulated eyeballs 210 simulating two eyes is different. According to the eyeball coordinate information of each simulated eyeball 210 , the first target angle and the second target angle corresponding to each are calculated.

[0097] Step 130 : Determine angle error information of the rotation of the simulated eyeball 210 based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 .

[0098] The motion parameter information of the simulated eyeball 210 refers to parameter information such as the speed, acceleration, and rotation range of the motor when the simulated eyeball 210 rotates from the current position to the gaze target point.

[0099] It can be understood that the actual process of controlling the first rotating component 230 and the second rotating component 240 to drive the simulated eyeball 210 to move and simulating the rotation of the human eye is affected by factors such as the rotation speed of the rotating components and the inertia of the simulated eyeball 210. There is an error between the rotation angle of the final stop position of the simulated eyeball 210 and the set rotation angle.

[0100] In this embodiment, the angle error information of the rotation of the simulated eyeball 210 is used to represent the error between the rotation angle corresponding to the final stop position after the first rotating component 230 and the second rotating component 240 drive the simulated eyeball 210 to move, and the first target angle and the second target angle calculated based on the coordinate information.

[0101] It should be noted that the error in the rotation of the simulated eyeball 210 is not only related to the motion parameter information of the simulated eyeball 210 itself, but also to the target angle to which the simulated eyeball 210 needs to rotate when gazing at the target point.

[0102] In this step, the angle error information can be predicted by a neural network model, or calculated by a polynomial obtained by fitting historical data.

[0103] Step 140: Based on the first target angle, the second target angle and the angle error information, control the simulated eyeball 210 to rotate so that the simulated eyeball 210 looks at the target point.

[0104] In this step, based on the first target angle and the second target angle calculated based on the positional relationship between the simulated eyeball 210 and the target point, the control process of the rotation of the simulated eyeball 210 is corrected and optimized through the corresponding angle error information, so that the rotation angle of the final stop position of the simulated eyeball 210 is close to or equal to the first target angle and the second target angle calculated based on the coordinate information, so that the simulated eyeball 210 can accurately focus on the target point.

[0105] In actual implementation, the simulated eyeball 210 rotation control based on the first target angle, the second target angle and the angle error information can be achieved through control methods such as model predictive control (MPC) or proportional integral derivative control (PID).

[0106] It is understandable that when there are two simulated eyeballs 210, the eye coordinate information of the two simulated eyeballs 210 is different, the first target angle and the second target angle corresponding to the two simulated eyeballs 210 are also different, and the movement control of the two simulated eyeballs 210 is also independent.

[0107] According to the eye tracking method provided by the embodiment of the present disclosure, by obtaining the coordinate information of the target point and the eye coordinate information, the rotation movement of the simulated eye 210 looking at the target point is decomposed into the rotation of the first rotating component 230 and the second rotating component 240. According to the required rotation angles of the two rotating components and the corresponding angle error information, the motion control accuracy of the simulated eye 210 is effectively improved.

[0108] In some embodiments, step 120, based on the coordinate information of the target point and the eyeball coordinate information, determines a first target angle to which the first rotating component 230 should drive the simulated eyeball 210 to rotate when the simulated eyeball 210 gazes at the target point, and a second target angle to which the second rotating component 240 should drive the first rotating component 230 and the simulated eyeball 210 to rotate synchronously. Specifically includes:

[0109] Determine the target pitch angle required for the simulated eyeball 210 to track the target point based on the coordinate information of the target point and the eyeball coordinate information;

[0110] determining a first target angle based on the target pitch angle and the first included angle;

[0111] A second target angle is determined based on the coordinate information of the target point, the first target angle, and the eyeball coordinate information.

[0112] In this step, based on the eyeball coordinate information, the target pitch angle required for each simulated eyeball 210 to look at the target point can be calculated through the right-angled triangle algorithm of the pitch angle. The target pitch angle is used to represent the angle between the line of sight of the pupil and the horizontal plane when the simulated eyeball 210 looks at the target point.

[0113] Based on the fact that the angle between the pupil's sight line direction and the horizontal plane is equal to the target pitch angle, trigonometric operations can be performed to calculate the vertical coordinate information of the pupil's target position when the simulated eyeball 210 rotates to the target point of gaze.

[0114] It should be noted that there is a first angle between the axial direction of the first rotating component 230 and the line of sight emission direction of the simulated eyeball 210. The movement of the pupil position caused by the rotation of the first rotating component 230 is related to the first angle, that is, when calculating the first target angle, it is necessary to combine the first angle for calculation.

[0115] For example, there is a first angle of 45° between the first rotating component 230 and the line of sight direction of the simulated eyeball 210. When the angle range of the target pitch angle of the simulated eyeball 210 is -45°≤β≤45°, after conversion through the rectangular coordinate system and trigonometric functions, the first target angle can be obtained as -45°≤β≤45°, that is, when the first rotating component 230 rotates 360 degrees, it can drive the simulated eyeball 210 to adjust the pitch angle within the angle range of -45°≤β≤45°.

[0116] In actual implementation, the first rotating component 230 drives the simulated eyeball 210 to move axially around the first rotating component 230, driving the pupil to move from the current position to the target position of the target point of gaze, and the first target angle is calculated based on the current position and the target position combined with the first angle.

[0117] It should be noted that the specific calculation and derivation of the target pitch angle using the eye coordinate information and the target point coordinate information is derived by technicians in this field based on the calculation principle of the angle between the line of sight and the horizontal plane, and will not be repeated here.

[0118] It can be understood that when the two simulated eyeballs 210 are placed side by side, corresponding to the left eye and the right eye of the human eye respectively, the eye coordinate information of the two simulated eyeballs 210 is different, and the target pitch angles corresponding to the two simulated eyeballs 210 are different, as shown in Figure 3, which are β-left and β-right respectively. The first target angle calculated based on the first angle is also different.

[0119] It can be understood that after the rotation angle of the first rotating component 230 is fixed, the rotation of the second rotating component 240 will not change the pitch angle of the simulated eyeball 210. After the first target angle is calculated, the second target angle is calculated by combining the coordinate information of the target point and the eye coordinate information of the simulated eyeball 210.

[0120] In some embodiments, determining the second target angle based on the coordinate information of the target point, the first target angle, and the eye coordinate information includes:

[0121] Determine, based on the first target angle and the movement trajectory of the pupil, a projection displacement deviation of the pupil on the horizontal plane before and after the first rotating component 230 rotates;

[0122] Determining a second compensation angle based on the projection displacement deviation and the distance between the projection and the center of the eyeball, the second compensation angle compensating for the angular deviation of the pupil in the horizontal plane;

[0123] Determine the target azimuth angle of the simulated eyeball 210 on the horizontal plane based on the coordinate information of the target point and the eyeball coordinate information;

[0124] A second target angle is determined based on the second compensation angle and the target azimuth angle.

[0125] It should be noted that when the first rotating component 230 rotates to the first target angle, the vertical projection of the movement trajectory of the pupil of the simulated eyeball 210 to the horizontal plane will produce a certain displacement (i.e., projection displacement deviation), resulting in an angular offset of the pupil in the horizontal azimuth. The second compensation angle is calculated to compensate for the angular offset of the pupil in the horizontal plane caused by the first rotating component 230 rotating to the first target angle.

[0126] For example, as shown in FIG5 , the circle R1 is a motion trajectory simulating a pupil of the eyeball 210 rotating one circle, and r is the radius of the plane where the pupil motion trajectory is located.

[0127] In this embodiment, the first rotating component 230 rotates the first target angle β, and the pupil moves from position K1 to position K2. According to the trigonometric function relationship within the circle, it can be calculated that the projection displacement deviation a1=r×cosβ generated by the pupil rotating the first target angle on the horizontal plane caused by the first rotating component 230 can be obtained.

[0128] As shown in FIG6 , circle R2 is a vertical projection of the simulated eyeball 210 and can be regarded as a projection of the cross section of the second rotating component 240 . The angular offset generated when the pupil moves from position K1 to position K2 is α1, that is, the second compensation angle is α1, wherein the distance between the projection and the center of the eyeball can be the distance between the plane where the actual movement trajectory of the pupil is located and the center of the eyeball.

[0129] In this embodiment, the included angle can be calculated based on the projection displacement deviation a1 and the distance h1 between the projection and the eye center through the trigonometric function relationship of the vertical projection. The second compensation angle α1 can be calculated by drawing an auxiliary dotted line.

[0130] A specific embodiment for calculating the second compensation angle α1 is introduced below.

[0131] As shown in FIG4 , the axes of the first rotating parts 230 of the two simulated eyeballs 210 form a second angle of 90°, and the angle α between the plane perpendicular to the axis of the first rotating part 230 and the display screen 300 directly in front of the simulated eyeballs 210 is 45°.

[0132] The simulated eyeball 210 is offset and the angle formed by the maximum cross section of the simulated eyeball 210 and the display screen 300 directly in front can be calculated based on the second angle, and the angle α between the auxiliary dotted line and the maximum cross section of the sphere as shown in Figure 6 can be calculated based on this angle.

[0133] The sum of the included angle α, the second compensation angle α1 and the angle γ is 90 degrees, and the second compensation angle α1 = 90° - α - γ.

[0134] When the axes of the first rotating components 230 of the two simulated eyeballs 210 form a second angle of 90°, the angle α between the plane perpendicular to the axis of the first rotating component 230 and the display screen 300 directly in front of the simulated eyeballs 210 is 45°.

[0135] In this embodiment, the first rotating member 230 rotates to a first target angle P c After that, the second compensation angle of the second rotating component 240 in the horizontal direction is α1=45°-arctan(cos p c ).

[0136] It can be understood that the offset placement of the two simulated eyeballs 210 can leave interference-free rotation space for structures such as the simulated eyeball 210 and the first rotating component 230. When the second rotating component 240 rotates, mechanical interference between the simulated eyeball 210 and the first rotating component 230 can be avoided, and the rotation of the two first rotating components 230 will not affect each other.

[0137] It should be noted that after the first rotating component 230 rotates to the first target angle, the angle between the line of sight of the simulated eyeball 210 and the horizontal plane reaches the target pitch angle. After the second rotating component 240 rotates to the second compensation angle for compensation, the pupil returns to its initial horizontal position, that is, the vertical coordinate of the pupil changes (forming a target pitch angle with the horizontal plane), but the horizontal coordinate remains unchanged. Based on the coordinate information of this initial position and the coordinate information of the target point, the target azimuth angle can be calculated to complete all adjustments to the horizontal azimuth angle of the simulated eyeball 210.

[0138] It can be understood that the target azimuth angle is calculated based on the coordinate information of the pupil at the initial position and the target position of the gaze target point. The calculation of the target azimuth angle is independent of the second compensation angle. The second compensation angle can also be calculated first, and there is no limit on the calculation order.

[0139] In this embodiment, when the simulated eyeball 210 is gazing at the target point, the eyeball center, pupil and target point are on the same straight line, and the target azimuth angle α2 can be determined according to the coordinate information of the target point and the original eyeball coordinate information.

[0140] In this embodiment, the second target angle of the second rotating component 240 is equal to the sum of the second compensation angle and the target azimuth angle, that is, the second target angle Q c =α1+α2, when the first rotating component 230 rotates the first target angle and the second rotating component 240 rotates the second target angle, the line of sight of the pupil faces the target point, which simulates the eyeball 210 gazing at the target point.

[0141] In some embodiments, determining the angle error information of the rotation of the simulated eyeball 210 based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 includes:

[0142] Inputting the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 into the error prediction model, and obtaining the angle error information of the simulated eyeball 210 output by the error prediction model;

[0143] The error prediction model is obtained by training based on a training sample set of the simulated eyeball 210 , and the training sample set includes a sample target rotation angle, template motion parameter information, and a sample actual rotation angle.

[0144] In this embodiment, a training sample set is obtained through multiple tests, and the error prediction model is trained using the training sample set. The trained error prediction model can predict the angle error generated by the simulated eyeball 210 during the rotation process.

[0145] For the data of the training sample set, the single test process is: control the first rotating part 230 and the second rotating part 240 of each simulated eyeball 210 to move to the position of the target point to be tracked. The data collected during the test includes the sample target rotation angle, the template motion parameter information (speed, acceleration, etc.) and the actual rotation angle of the sample.

[0146] The sample actual rotation angle is the actual rotation angle when the first rotating component 230 and the second rotating component 240 stop, and can be calculated by reading the motor feedback signals of the first rotating component 230 and the second rotating component 240 .

[0147] In actual implementation, training sample data within a certain rotation speed range (100° / s-720° / s) and a certain acceleration time range (10ms-100ms) can be obtained and integrated to obtain a training sample set.

[0148] 60% of the data in the training sample set can be used as a training set, 30% of the data as a test set, and 10% of the data as a validation set. An error prediction model is established through a back propagation (BP) neural network for training.

[0149] In this embodiment, the trained model can predict the rotation angle error that will occur based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 that currently needs to be rotated, and output the angle error information.

[0150] For example, as shown in FIG8 , the simulated eyeball 210 is controlled to start moving, the camera obtains the spatial position coordinates of the target point to be tracked, and calculates the rotation angle required for the simulated eyeball 210 to look toward the tracked target point, ie, the sample target rotation angle.

[0151] According to the sample target rotation angle, control parameters are issued to control the first rotating component 230 and the second rotating component 240 to drive the simulated eyeball 210 to rotate, obtain the error between the actual rotation angle and the theoretical angle (i.e., the sample target rotation angle), and continuously look at new target points to obtain multiple batches of training sample data until there are no new target points and the movement ends, and the training sample set is sorted out.

[0152] In some embodiments, determining the angle error information of the rotation of the simulated eyeball 210 based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 includes:

[0153] Based on the first target angle, the second target angle and the motion parameter information, solving according to the fitting function corresponding to the error prediction model to obtain the angle error information of the simulated eyeball 210;

[0154] The error prediction model is trained in the cloud, and the fitting function is obtained by interpolating and fitting the trained error prediction model.

[0155] In this embodiment, the error prediction model cannot be implemented directly on the physical device. After obtaining the training sample set, the data is uploaded to the cloud, and the powerful computing power of the cloud is used to train the neural network model. After the training is completed, the input and output parameters of the obtained error prediction model are interpolated and fitted to obtain the algorithm implemented in the controller, that is, the fitting function corresponding to the error prediction model.

[0156] For example, the specific fitting function can be as follows:

[0157] Y=A0+A1gX1+A2gX2+A3gX3

[0158] Among them, Y is the angle error information, X1, X2, and X3 correspond to the target angle, rotation speed, and acceleration of the rotary axis motor, respectively, and A0, A1, A2, and A3 are function fitting coefficients.

[0159] In this embodiment, the fitting function obtained by fitting the difference can be written into the controller, and the angle error information corresponding to the first target angle and the second target angle is calculated through the above fitting function.

[0160] In some embodiments, controlling the simulated eyeball 210 to rotate based on the first target angle, the second target angle, and the angle error information includes:

[0161] Determining, based on the angle error information, a first predicted angle corresponding to the first target angle and a second predicted angle corresponding to the second target angle;

[0162] An optimization objective function is established based on the first target angle, the second target angle, the first predicted angle, and the second predicted angle, with the cumulative angle error minimized within the target time domain;

[0163] Solve the optimization objective function and control the rotation of the simulated eyeball 210.

[0164] In this embodiment, the rotation of the simulated eyeball 210 is controlled by a model predictive control algorithm. According to the angular error information corresponding to the first target angle and the second target angle, the actual rotation angle, i.e., the first predicted angle and the second predicted angle, can be predicted. An optimization problem is formulated based on the target angle and the predicted angle, an optimization objective function is established, and the optimization variables that meet the constraints and the optimization objective function are calculated. This can minimize the cumulative angular error of the rotation angle within the future target time domain.

[0165] In actual implementation, based on the two target angles and the two predicted angles, a loss function regarding the expected rotation and the predicted rotation can be obtained. The loss function can be the square of the difference between the future output of the model and the expected output in the model predictive control algorithm. The optimization objective function is established through the loss function, and the control quantity is solved by the quadratic programming method to minimize the optimization objective function. The solved control quantity can be directly used to simulate the rotation control of the eyeball 210.

[0166] For example, to minimize the cumulative angle error, the optimization objective function is established as follows:

[0167] Among them, N is the optimization time domain, M is the control time domain, q i is the output tracking weighting coefficient, r j is the input weighting coefficient, u is the control quantity, y N (k+i) is the future output of the model, that is, the predicted angle under the future control variables (target angle, rotation speed, acceleration); y r (k+i) is the desired output, that is, the ideal target angle to be achieved.

[0168] By designing N, M, q i 、r j The control effect is adjusted by the size of the control parameter. As the control time increases, the control parameters are continuously adjusted. The controller adopts the model predictive control algorithm to implement real-time error compensation control strategy to improve the motion control accuracy of eye tracking.

[0169] Take the implementation of eye tracking methods in eye tracking devices as an example.

[0170] As shown in FIG9 , the movement starts and the system dynamic model of the eye tracking device is obtained. The system dynamic model is a model used to describe the movement relationship of the eye tracking device and can be a trained error prediction model.

[0171] The optimization problem is set according to the target angle and the predicted angle, the optimization objective function is established, and the optimization variables that meet the constraints and the optimization objective function are calculated. The minimum cumulative angle error of the rotation angle can be achieved in the future target time domain.

[0172] The optimal control parameters in the target time domain are obtained through quadratic programming, and control feedback is obtained. The model predictive control algorithm is used to implement real-time error compensation control. After the control output in all time domains is completed, the movement ends.

[0173] As shown in FIG7 , during actual use of the eye tracking device, the camera obtains the spatial position coordinates of the point to be tracked (ie, the target point), and calculates the rotation axis coordinates of the simulated eye 210 when gazing at the point through spatial posture decomposition.

[0174] The coordinates of the rotation axis to be rotated are subtracted from the currently fed-back coordinates of the rotation axis to obtain control rotation parameters, namely, a first target angle corresponding to the first rotation component 230 and a second target angle corresponding to the second rotation component 240 .

[0175] The first target angle, second target angle, acceleration, maximum speed and other parameters are used as input, and the error estimation result is obtained through the error prediction model. The first predicted angle and the second predicted angle can also be calculated.

[0176] Through the model predictive control algorithm, the control parameters that minimize the cumulative angle error in the future time domain are obtained to control the motor of the rotating axis, effectively improving the motion control accuracy of eye tracking.

[0177] It should be noted that the coordinate information of the target point can be input by the user or obtained through autonomous identification.

[0178] In some embodiments, step 110, obtaining coordinate information of the target point, includes:

[0179] receiving a first input from a user;

[0180] In response to the first input, coordinate information of the target point is obtained.

[0181] In this embodiment, the user inputs coordinate information of one or more target points.

[0182] In actual implementation, the first input corresponding to the coordinate information of the target point can be a touch operation, including but not limited to a click operation, a slide operation, and a press operation, etc.; it can also be a physical button input or a voice input.

[0183] Of course, in other embodiments, the first input corresponding to the coordinate information of the target point may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present disclosure.

[0184] In some embodiments, step 110, obtaining coordinate information of the target point, includes:

[0185] Obtain image information of the target point;

[0186] Identify the image of the target point and obtain the coordinate information of the target point.

[0187] In this embodiment, the coordinate information of the target point can be autonomously identified based on the image information of the target point.

[0188] In actual implementation, the coordinate information of the target point can be autonomously identified based on the image sequence of the target point, thereby realizing full automation of eye tracking and improving the eye movement efficiency of eye tracking.

[0189] The eye tracking method provided in the embodiment of the present disclosure can be performed by an eye tracking device. In the embodiment of the present disclosure, the eye tracking device provided in the embodiment of the present disclosure is described by taking the eye tracking device performing the eye tracking method as an example.

[0190] The disclosed embodiment also provides an eye tracking device.

[0191] As shown in FIG10 , the eye tracking device includes:

[0192] An acquisition module 1010 is used to acquire coordinate information of a target point and eye coordinate information of a simulated eye 210;

[0193] The first processing module 1020 is configured to determine, based on the coordinate information of the target point and the eyeball coordinate information, a first target angle to which the simulated eyeball 210 should be rotated by the first rotating component 230 when the simulated eyeball 210 is gazing at the target point, and a second target angle to which the first rotating component 230 and the simulated eyeball 210 should be rotated when the second rotating component 240 drives the first rotating component 230 and the simulated eyeball 210 to rotate synchronously;

[0194] The second processing module 1030 is configured to determine angle error information of the rotation of the simulated eyeball 210 based on the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210;

[0195] The third processing module 1040 is configured to control the simulated eyeball 210 to rotate based on the first target angle, the second target angle, and the angle error information, so that the simulated eyeball 210 focuses on the target point.

[0196] According to the eye tracking device provided by the embodiment of the present disclosure, by acquiring the coordinate information of the target point and the eye coordinate information, the rotation movement of the simulated eye 210 looking at the target point is decomposed into the rotation of the first rotating component 230 and the second rotating component 240. According to the required rotation angles of the two rotating components and the corresponding angle error information, the motion control of the simulated eye 210 is controlled, which can effectively improve the motion control accuracy of eye tracking.

[0197] In some embodiments, the first processing module 1020 is configured to determine a target pitch angle that the simulated eye 210 needs to rotate to track the target point based on the coordinate information of the target point and the eye coordinate information;

[0198] determining a first target angle based on the target pitch angle and the first included angle;

[0199] A second target angle is determined based on the coordinate information of the target point, the first target angle, and the eyeball coordinate information.

[0200] In some embodiments, the first processing module 1020 is configured to determine a projection displacement deviation of the pupil on a horizontal plane before and after the first rotating component 230 rotates based on the first target angle and the movement trajectory of the pupil;

[0201] Determining a second compensation angle based on the projection displacement deviation and the distance between the projection and the center of the eyeball, the second compensation angle compensating for the angular deviation of the pupil in the horizontal plane;

[0202] Determine the target azimuth angle of the simulated eyeball 210 on the horizontal plane based on the coordinate information of the target point and the eyeball coordinate information;

[0203] A second target angle is determined based on the second compensation angle and the target azimuth angle.

[0204] In some embodiments, the second processing module 1030 is configured to input the first target angle, the second target angle, and the motion parameter information of the simulated eyeball 210 into the error prediction model to obtain the angle error information of the simulated eyeball 210 output by the error prediction model;

[0205] The error prediction model is obtained by training based on a training sample set of the simulated eyeball 210 , and the training sample set includes a sample target rotation angle, template motion parameter information, and a sample actual rotation angle.

[0206] In some embodiments, the second processing module 1030 is configured to obtain angle error information of the simulated eyeball 210 by solving a fitting function corresponding to the error prediction model based on the first target angle, the second target angle, and the motion parameter information;

[0207] The error prediction model is trained in the cloud, and the fitting function is obtained by interpolating and fitting the trained error prediction model.

[0208] In some embodiments, the third processing module 1040 is configured to determine a first predicted angle corresponding to the first target angle and a second predicted angle corresponding to the second target angle based on the angle error information;

[0209] An optimization objective function is established based on the first target angle, the second target angle, the first predicted angle, and the second predicted angle, with the cumulative angle error minimized within the target time domain;

[0210] Solve the optimization objective function and control the rotation of the simulated eyeball 210.

[0211] In some embodiments, the acquisition module 1010 is configured to receive a first input from a user;

[0212] In response to the first input, coordinate information of the target point is obtained.

[0213] In some embodiments, the acquisition module 1010 is configured to acquire an image of a target point;

[0214] Identify the image of the target point and obtain the coordinate information of the target point.

[0215] The eye tracking device in the embodiment of the present disclosure can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present disclosure does not specifically limit it.

[0216] The eye tracking device in the embodiments of the present disclosure may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present disclosure.

[0217] The eye tracking device provided in the embodiments of the present disclosure can implement each process implemented in the method embodiments of Figures 1 to 9. To avoid repetition, they are not described here.

[0218] The embodiment of the present disclosure also provides an eye tracking device.

[0219] The eye tracking device of the embodiment of the present disclosure is a device that can simulate human eye movement, and can be used for quality inspection of optical instruments such as eye trackers and ophthalmic optical biometers, and can also be used as a component of intelligent robots.

[0220] The eye tracking device includes: at least one simulated eyeball 210 and a controller, wherein the simulated eyeball 210 has a first rotating component 230 and a second rotating component 240 .

[0221] The simulated eyeball 210 may be a component of an eye tracking device for tracking gaze, and the first rotating component 230 and the second rotating component 240 may be components of the eye tracking device for driving the simulated eyeball 210 to rotate.

[0222] In some embodiments, the first rotating component 230 is connected to the simulated eyeball 210, and the first rotating component 230 drives the simulated eyeball 210 to rotate; the second rotating component 240 is connected to the first rotating component 230, and the second rotating component 240 drives the first rotating component 230 and the simulated eyeball 210 to rotate synchronously.

[0223] The rotation axis of the first rotating component 230 and the rotation axis of the second rotating component 240 are not parallel and are located on two non-parallel planes. The line connecting the pupil and the center of the simulated eyeball 210 forms a first angle with the axial direction of the first rotating component 230 .

[0224] The first rotating component 230 is used to adjust the pitch angle of the simulated eyeball 210, and the second rotating component 240 is used to adjust the horizontal azimuth angle of the simulated eyeball 210. The rotation axis of the first rotating component 230 and the rotation axis of the second rotating component 240 are not parallel and the two rotation axes are on two non-parallel planes. The first rotating component 230 and the second rotating component 240 can realize the full-range rotation of the simulated eyeball 210.

[0225] It should be noted that the line of sight of the simulated eyeball 210 is the direction of the line connecting the pupil and the center of the eyeball of the simulated eyeball 210. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 210 to move around the axial direction of the first rotating component 230, the position of the pupil of the simulated eyeball 210 in the vertical direction changes, and the angle between the line of sight and the horizontal direction changes accordingly, thereby realizing pitch angle adjustment.

[0226] It can be understood that if the line connecting the pupil and the center of the eyeball is on the same straight line as the axis of the first rotating component 230, when the first rotating component 230 drives the simulated eyeball 210 to move around the axis of the first rotating component 230, the pupil of the simulated eyeball 210 is on the rotation axis, and the position of the pupil in the vertical direction will not change, and the pitch angle adjustment cannot be achieved.

[0227] In actual implementation, the axial setting angle of the first rotating component 230 can be adjusted according to the angle range of the pitch angle required to adjust the simulated eyeball 210. The first rotating component 230 can be set in a horizontal direction or in an inclined direction with a certain angle to the horizontal direction.

[0228] For example, the axis of the first rotating component 230 is horizontal, and the line connecting the pupil and the center of the eyeball forms a first angle with the axis of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 210 to rotate, the movement trajectory of the pupil on the simulated eyeball 210 is located on the vertical plane, thereby realizing the pitch angle adjustment of the line of sight of the simulated eyeball 210.

[0229] For another example, the axial direction of the first rotating component 230 is inclined at a certain angle to the horizontal direction, and the axial direction of the first rotating component 230 is not completely vertical. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 210 to rotate, the movement trajectory of the pupil on the simulated eyeball 210 is on a non-horizontal plane, and the pitch angle adjustment can also be achieved.

[0230] The controller is electrically connected to the first rotating component 230 and the second rotating component 240 . The controller can control the simulated eyeball 210 to rotate based on the above-mentioned eye tracking method to track the target point.

[0231] According to the eye tracking device provided by the embodiment of the present disclosure, by acquiring the coordinate information of the target point and the eye coordinate information, the rotation action of the simulated eye 210 looking at the target point is decomposed into the rotation of the first rotating component 230 and the second rotating component 240. According to the required rotation angles of the two rotating components and the corresponding angle error information, the motion of the simulated eye 210 is controlled, which can effectively improve the motion control accuracy of the eye tracking device.

[0232] In some embodiments, the simulated eyeball 210 is not a single spherical structure. The simulated eyeball 210 includes an eyeball structure 211 and a fake eye patch 212. The sphere of the eyeball structure 211 is provided with a cross-section, and the fake eye patch 212 covers the cross-section. The radius of the fake eye patch 212 is smaller than the radius of the eyeball structure 211, and the pupil of the simulated eyeball 210 is located on the fake eye patch 212.

[0233] In some embodiments, the eye tracking device includes two simulated eyeballs 210 , and the axes of the first rotating components 230 of the two simulated eyeballs 210 form a second angle, which is not 0° or 180°.

[0234] In this embodiment, the two simulated eyeballs 210 are offset and placed to leave a rotation space without interference for the simulated eyeball 210 and the first rotating component 230 and other structures. When the second rotating component 240 rotates, mechanical interference between the simulated eyeball 210 and the first rotating component 230 and other structures can be avoided, and the rotation of the two first rotating components 230 does not affect each other.

[0235] In some embodiments, as shown in Figure 11, the embodiment of the present disclosure also provides an electronic device 1100, including a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the program is executed by the processor 1101, each process of the above-mentioned eye tracking method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0236] It should be noted that the electronic devices in the embodiments of the present disclosure include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0237] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processes of the above-mentioned eye tracking method embodiment are implemented and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0238] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0239] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned eye tracking method when executed by a processor.

[0240] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0241] An embodiment of the present disclosure further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned eye tracking method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0242] It should be understood that the chip mentioned in the embodiments of the present disclosure can also be called a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0243] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0244] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0245] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are protected by the present disclosure.

[0246] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0247] Although the embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. An eye tracking method, wherein: The method acts on at least one simulated eyeball having a first rotating component and a second rotating component to track a target point, wherein the rotation axis of the first rotating component is not parallel to the rotation axis of the second rotating component and the two rotation axes are located on two non-parallel planes, and a line connecting the pupil and the center of the simulated eyeball forms a first angle with the axial direction of the first rotating component; the method comprises: Obtaining coordinate information of the target point and eyeball coordinate information of the simulated eyeball; Based on the coordinate information of the target point and the coordinate information of the eyeball, determining a first target angle to which the first rotating component should drive the simulated eyeball to rotate when the simulated eyeball tracks the target point, and a second target angle to which the second rotating component should drive the first rotating component and the simulated eyeball to rotate synchronously; Determining angle error information of the simulated eyeball rotation based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball; Based on the first target angle, the second target angle and the angle error information, the simulated eyeball is controlled to rotate so that the simulated eyeball tracks the target point.

2. The eye tracking method according to claim 1, wherein: The determining, based on the coordinate information of the target point and the eyeball coordinate information, of a first target angle to which the simulated eyeball should be rotated by the first rotating component when the simulated eyeball tracks the target point, and a second target angle to which the second rotating component should be rotated when the first rotating component and the simulated eyeball are synchronously rotated, specifically includes: Based on the coordinate information of the target point and the eyeball coordinate information, determining a target pitch angle required for the simulated eyeball to rotate when tracking the target point; Determining the first target angle based on the target pitch angle and the first angle; The second target angle is determined based on the coordinate information of the target point, the first target angle, and the eyeball coordinate information.

3. The eye tracking method according to claim 2, wherein: The determining the second target angle based on the coordinate information of the target point, the first target angle and the eye coordinate information includes: Determining, based on the first target angle and the movement trajectory of the pupil, a projection displacement deviation of the pupil on a horizontal plane before and after the first rotating component rotates; Determining a second compensation angle based on the projection displacement deviation and the distance of the projection relative to the center of the eyeball, the second compensation angle compensating for the angular deviation of the pupil in the horizontal plane; Determining the target azimuth of the simulated eyeball on the horizontal plane based on the coordinate information of the target point and the coordinate information of the eyeball; The second target angle is determined based on the second compensation angle and the target azimuth angle.

4. The eye tracking method according to any one of claims 1 to 3, wherein: The step of determining the angle error information of the simulated eyeball rotation based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball comprises: Inputting the first target angle, the second target angle and the motion parameter information of the simulated eyeball into an error prediction model to obtain the angle error information of the simulated eyeball output by the error prediction model; The error prediction model is obtained by training based on the training sample set of the simulated eyeball, and the training sample set includes a sample target rotation angle, template motion parameter information and a sample actual rotation angle.

5. The eye tracking method according to claim 4, wherein: The step of determining the angle error information of the simulated eyeball rotation based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball comprises: Based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball, solving according to the fitting function corresponding to the error prediction model to obtain the angle error information of the simulated eyeball; The error prediction model is trained in the cloud, and the fitting function is obtained by interpolating and fitting the trained error prediction model.

6. The eye tracking method according to any one of claims 1 to 5, wherein: The controlling the simulated eyeball rotation based on the first target angle, the second target angle and the angle error information comprises: Based on the angle error information, determining a first predicted angle corresponding to the first target angle and a second predicted angle corresponding to the second target angle; According to the first target angle, the second target angle, the first predicted angle and the second predicted angle, in the target time domain Within, the optimization objective function is established with the minimum cumulative angle error; Solve the optimization objective function and control the simulated eye movement.

7. The eye tracking method according to any one of claims 1 to 6, wherein: The step of obtaining the coordinate information of the target point includes: receiving a first input from a user; In response to the first input, coordinate information of the target point is obtained.

8. The eye tracking method according to any one of claims 1 to 6, wherein: The step of obtaining the coordinate information of the target point includes: Acquire an image of the target point; The image of the target point is recognized to obtain the coordinate information of the target point.

9. An eye tracking device, wherein: The device acts on at least one simulated eyeball having a first rotating component and a second rotating component to track a target point, the rotation axis of the first rotating component is not parallel to the rotation axis of the second rotating component and the two rotation axes are located on two non-parallel planes, and the line connecting the pupil and the center of the simulated eyeball forms a first angle with the axial direction of the first rotating component; the device comprises: An acquisition module, used for acquiring the coordinate information of the target point and the eye coordinate information of the simulated eyeball; A first processing module is used to determine, based on the coordinate information of the target point and the eyeball coordinate information, a first target angle to which the first rotating component should drive the simulated eyeball to rotate when the simulated eyeball tracks the target point, and a second target angle to which the second rotating component should drive the first rotating component and the simulated eyeball to rotate synchronously when the second rotating component drives the first rotating component and the simulated eyeball to rotate synchronously; A second processing module, configured to determine angle error information of the simulated eyeball rotation based on the first target angle, the second target angle and the motion parameter information of the simulated eyeball; The third processing module is used to control the rotation of the simulated eyeball based on the first target angle, the second target angle and the angle error information, so that the simulated eyeball tracks the target point.

10. An eye tracking device, wherein: include: At least one simulated eyeball, the simulated eyeball having a first rotating part and a second rotating part; The rotation axis of the first rotating component is not parallel to the rotation axis of the second rotating component and the two rotation axes are located on two non-parallel planes, and the line connecting the pupil of the simulated eyeball and the center of the eyeball forms a first angle with the axial direction of the first rotating component; A controller, wherein the controller is electrically connected to the first rotating component and the second rotating component, and the controller controls the simulated eyeball to rotate to track a target point based on the eye tracking method according to any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the eye tracking method as described in any one of claims 1-8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the eye tracking method according to any one of claims 1 to 8 is implemented.

13. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the eye tracking method according to any one of claims 1 to 8 is implemented.

Images