Eye movement tracking method, intelligent glasses, mobile terminal and eye movement tracking system

Abstract

The invention relates to the technical field of man-machine interaction, and discloses an eye movement tracking method, intelligent glasses, a mobile terminal and an eye movement tracking system.The method comprises the steps that current head movement data of a wearer is collected and sent to the mobile terminal; receiving the current gazing direction of the wearing user fed back by the mobile terminal, and tracking the eye movement of the wearing user according to the current gazing direction; wherein the current gazing direction is obtained through the mobile terminal according to the current head movement data and a current eyeball image of the wearing user, and the current eyeball image is obtained by shooting eyeballs of the wearing user through a camera. Since the mobile terminal with the camera arranged outside is utilized to complete eyeball collection and fixation direction determination of the wearing user, an independent camera and an independent computing power unit do not need to be arranged in the intelligent glasses, and the size of the intelligent glasses is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to an eye movement tracking method, smart glasses, a mobile terminal and an eye movement tracking system. BACKGROUND

[0002] At present, with the development of science and technology, smart glasses (such as Augmented Reality (AR) glasses) gradually enter people's lives.

[0003] When implementing human-computer interaction, the AR glasses can generally collect the gaze direction of a wearing user, implement eye movement tracking by using the gaze direction, and complete human-computer interaction. When obtaining the gaze direction of the wearing user, the existing AR glasses generally implement by using an eye movement tracking module on the AR glasses, and thus a separate camera and a computing unit need to be arranged on the smart glasses, resulting in a large size of the smart glasses. SUMMARY

[0004] The main purpose of the present application is to provide an eye movement tracking method, smart glasses, a mobile terminal and an eye movement tracking system, and aims to solve the technical problem that a separate camera and a computing unit need to be arranged on the smart glasses in order to collect the gaze direction of a wearing user, resulting in a large size of the smart glasses.

[0005] To achieve the above purpose, the present application provides an eye movement tracking method, which is applied to smart glasses, the smart glasses are used in cooperation with a mobile terminal provided with a camera, and the method comprises the following steps. Collecting current head movement data of a wearing user and sending the current head movement data to the mobile terminal; Receiving a current gaze direction of the wearing user fed back by the mobile terminal, and performing eye movement tracking on the wearing user according to the current gaze direction; The current gaze direction is obtained by the mobile terminal according to the current head movement data and a current eye image of the wearing user, and the current eye image is obtained by photographing the eyeball of the wearing user by the camera.

[0006] In an embodiment, the step of collecting the current head movement data of the wearing user comprises the following steps. Collecting a current head angular velocity and a current head acceleration of the wearing user; Determining an attitude angle change rate and an angular velocity change rate according to the current head angular velocity and the current head acceleration by using a preset state equation; Performing attitude angle estimation based on the attitude angle change rate and the angular velocity change rate by using a Kalman filter, and obtaining a head attitude angle of the wearing user; The head pose angle is taken as current head movement data of the wearing user.

[0007] In an embodiment, the step of determining the attitude angle rate of change and the angular velocity rate of change according to the current head angular velocity and the current head acceleration by the preset state equation comprises: obtaining a gravity acceleration and a first adaptive gain coefficient, and determining the attitude angle rate of change according to the current head angular velocity, the first adaptive gain coefficient, the current head acceleration and the gravity acceleration by a first preset state equation; discretizing the attitude angle rate of change to obtain an attitude angle prediction value; obtaining a second adaptive gain coefficient, and obtaining the angular velocity rate of change according to the second adaptive gain coefficient and the attitude angle prediction value by a second preset state equation.

[0008] In addition, to achieve the above object, the application further provides an eye movement tracking method, which is applied to a mobile terminal provided with a camera, and is used in cooperation with smart glasses. The method comprises the following steps: receiving current head movement data of a wearing user sent by the smart glasses, the current head movement data being obtained by the smart glasses; photographing eyeballs of the wearing user by the camera to obtain a current eyeball image; determining a current gaze direction of the wearing user according to the current head movement data and the current eyeball image, and feeding back the current gaze direction to the smart glasses, so that the smart glasses track eye movement of the wearing user according to the current gaze direction.

[0009] In an embodiment, the step of photographing eyeballs of the wearing user by the camera to obtain a current eyeball image comprises: photographing a face of the wearing user by the camera to obtain a current face image of the wearing user; detecting face key points of the current face image to obtain key point coordinates of an eye region in the current face image; cropping a region of interest of the eye region in the current face image according to the key point coordinates to obtain a current eyeball image.

[0010] In an embodiment, the step of determining a current gaze direction of the wearing user according to the current head movement data and the current eyeball image comprises: determining iris ellipse parameters of the wearing user according to the current eyeball image; generate a gaze direction vector of the wearing user based on the iris ellipse parameter; fuse the current head movement data and the gaze direction vector to obtain a current gaze direction of the wearing user.

[0011] In an embodiment, the step of fusing the current head movement data and the gaze direction vector to obtain a current gaze direction of the wearing user comprises: fuse the current head movement data and the gaze direction vector; determine a current iris position of the wearing user according to the current eye image by using a Kalman filter, and perform gaze point coordinate estimation according to a preset observation matrix, a fusion result, the current head movement data and the current iris position to obtain a gaze point coordinate of the wearing user; use the gaze point coordinate as the current gaze direction of the wearing user.

[0012] In addition, to achieve the above object, the present application further provides an intelligent glasses, which comprises a first memory, a first processor and an eye movement tracking program stored in the first memory and executable on the first processor, and the eye movement tracking program realizes the steps of the eye movement tracking method when executed by the first processor.

[0013] In addition, to achieve the above object, the present application further provides a mobile terminal, which comprises a camera, a second memory, a second processor and an eye movement tracking program stored in the second memory and executable on the second processor, and the eye movement tracking program realizes the steps of the eye movement tracking method when executed by the second processor.

[0014] In addition, to achieve the above object, the present application further provides an eye movement tracking system, which comprises the intelligent glasses and the mobile terminal.

[0015] The present application provides an eye movement tracking method, intelligent glasses, mobile terminal and eye movement tracking system, the method is applied to the intelligent glasses, the intelligent glasses is used in cooperation with the mobile terminal provided with a camera, and the method comprises the following steps: collecting current head movement data of a wearing user, and sending the current head movement data to the mobile terminal; receiving a current gaze direction of the wearing user fed back by the mobile terminal, and performing eye movement tracking on the wearing user according to the current gaze direction; wherein the current gaze direction is obtained by the mobile terminal according to the current head movement data and a current eye image of the wearing user, and the current eye image is obtained by photographing eyeballs of the wearing user by the camera.

[0016] In the application, the smart glasses can be used in cooperation with a mobile terminal provided with a camera. In actual use, the smart glasses can first collect current head movement data of a wearing user, and send the current head movement data to the mobile terminal. The mobile terminal can capture the eyeballs of the wearing user through the camera to obtain a current eyeball image. Then, the current gaze direction of the wearing user is determined according to the current head movement data and the current eyeball image, and the current gaze direction is fed back to the smart glasses. The smart glasses perform eye movement tracking according to the current gaze direction. Since the application uses the mobile terminal provided with the camera to complete the collection of the eyeballs of the wearing user and the determination of the gaze direction, it is not necessary to set an independent camera and a computing unit in the smart glasses, thereby reducing the size of the smart glasses. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative work.

[0019] Figure 1 The hardware running environment of the smart glasses involved in the embodiments of the present application is shown in the structural schematic diagram. Figure 2 The flowchart of the eye movement tracking method according to the first embodiment of the present application is shown in the flowchart. Figure 3 The flowchart of the eye movement tracking method according to the second embodiment of the present application is shown in the flowchart. Figure 4 The flowchart of the eye movement tracking method according to the third embodiment of the present application is shown in the flowchart. Figure 5 The flowchart of the eye movement tracking method according to the fourth embodiment of the present application is shown in the flowchart. Figure 6 The hardware running environment of the mobile terminal involved in the embodiments of the present application is shown in the structural schematic diagram.

[0020] The implementation of the present application, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0022] Reference Figure 1, Figure 1 This is a schematic diagram of the smart glasses structure of the hardware operating environment involved in the embodiments of this application.

[0023] like Figure 1 As shown, the smart glasses may include: a first processor 1001, such as a central processing unit (CPU), a first communication bus 1002, a first user interface 1003, a first network interface 1004, and a first memory 1005. The first communication bus 1002 is used to enable communication between these components. The first user interface 1003 may be connected to a display screen. Optionally, the first user interface 1003 may include a standard wired interface or a wireless interface; in this application, the wired interface of the first user interface 1003 may be a USB interface. The first network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The first memory 1005 may be a high-speed random access memory (RAM) or a non-volatile memory (NVM), such as a disk storage device. Optionally, the first memory 1005 may also be a storage device independent of the aforementioned first processor 1001.

[0024] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on smart glasses and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0025] like Figure 1 As shown, the first memory 1005, which is identified as a computer storage medium, may include a first operating system, a first network communication module, a first user interface module, and an eye-tracking program.

[0026] exist Figure 1 In the smart glasses shown, the first network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the first user interface 1003 is mainly used to connect to user devices (such as mobile terminals equipped with cameras); the smart glasses call the eye-tracking program stored in the first memory 1005 through the first processor 1001 and execute the steps of the eye-tracking method provided in the first and second embodiments of this application.

[0027] It should be noted that, with the development of technology, smart glasses (such as augmented reality (AR) glasses) are gradually entering people's lives.

[0028] When implementing human-computer interaction, the AR glasses can generally collect the gaze direction of the wearer, implement eye movement tracking by using the gaze direction, and complete human-computer interaction. When obtaining the gaze direction of the wearer, the existing AR glasses generally use an eye movement tracking module on the AR glasses, and thus a separate camera and a computing unit need to be arranged on the smart glasses, resulting in a large size of the smart glasses.

[0029] Therefore, to solve the above defects, the embodiment provides an eye movement tracking method. In the embodiment, the smart glasses can be used in cooperation with a mobile terminal provided with a camera. In actual use, the smart glasses can first collect current head movement data of the wearer, and send the current head movement data to the mobile terminal. The mobile terminal can capture the eyeballs of the wearer by using the camera to obtain a current eyeball image. Then, the current gaze direction of the wearer is determined according to the current head movement data and the current eyeball image, and the current gaze direction is fed back to the smart glasses. The smart glasses perform eye movement tracking according to the current gaze direction. Since the embodiment uses the mobile terminal provided with the camera to collect the eyeballs of the wearer and determine the gaze direction, a separate camera and a computing unit do not need to be arranged in the smart glasses, thereby reducing the size of the smart glasses.

[0030] For the convenience of understanding, the following will be combined with Figures 2 to 6 The eye movement tracking method provided by the embodiment of the application is specifically introduced.

[0031] Reference Figure 2 , Figure 2 is a flowchart of the first embodiment of the eye movement tracking method of the application. As shown in Figure 2 , in the embodiment, the specific method includes: Step S10: Collecting current head movement data of the wearer, and sending the current head movement data to the mobile terminal.

[0032] It can be understood that the method of the embodiment can be applied to the above-mentioned smart glasses. The smart glasses can be any device with data processing, program running and eye movement tracking functions, such as AR glasses, and the embodiment does not limit this. The execution subject of the method embodiment can use the above-mentioned smart glasses to describe the embodiment and the following embodiments.

[0033] It should also be understood that the smart glasses described above in the embodiment can be used in cooperation with a mobile terminal provided with a camera, that is, the smart glasses can be connected in communication with the mobile terminal for data transmission. The connection mode can be Bluetooth connection, Wi-Fi connection, etc., which is not limited in the embodiment. Moreover, the mobile terminal can be any electronic terminal provided with a camera, such as a mobile phone, a tablet, etc., which is also not limited in the embodiment.

[0034] It should be understood that the wearing user can be a user wearing the smart glasses described above. The current head motion data can be the motion state data of the head of the wearing user at the current time, which can include angular velocity and linear acceleration. In the embodiment, the current head motion state data can be obtained by acquisition. Specifically, in the embodiment, an acquisition component for acquiring the current head motion state data can be provided in the smart glasses, such as a 6-axis inertial measurement unit (IMU), etc., which is described in the embodiment by using the IMU. Furthermore, the angular velocity of the head of the wearing user can be acquired by an angular velocity meter in the IMU, and the linear acceleration of the head of the wearing user can be acquired by an accelerometer in the IMU.

[0035] In actual use, when the wearing user wears the smart glasses, the IMU in the smart glasses starts to work, and the angular velocity and acceleration of the head can be acquired at a certain acquisition frequency (such as 100 Hz, etc.) to obtain the current head motion data, and the current head motion data is sent to the mobile terminal.

[0036] It should be further emphasized that, in order to facilitate the cooperation between the smart glasses and the mobile terminal, the smart glasses in the embodiment can support Huffman coding compression when sending the current head motion data to the mobile terminal, that is, the current head motion data is compressed by Huffman coding and then transmitted to the mobile terminal. At the same time, the space-time synchronization algorithm based on the bidirectional precision time protocol (PTP) can be used between the mobile terminal and the smart glasses to reduce errors.

[0037] Step S50: receiving the current gaze direction of the wearing user fed back by the mobile terminal, and performing eye movement tracking on the wearing user according to the current gaze direction.

[0038] The current gaze direction is obtained by the mobile terminal according to the current head motion data and the current eye image of the wearing user, and the current eye image is obtained by photographing the eyeball of the wearing user by the camera.

[0039] It should be noted that the current eye image can be an image of the eye of the user wearing the smart glasses at the current time. The current gaze direction can be the gaze direction of the eye of the user wearing the smart glasses at the current time.

[0040] It should be emphasized that while the smart glasses collect the current head movement data of the user wearing the smart glasses, the mobile terminal can also capture the eye of the user wearing the smart glasses through the camera to obtain the current eye image. Then, the mobile terminal can obtain the current gaze direction of the user wearing the smart glasses in combination with the current eye image obtained by capturing after receiving the current head movement data. After obtaining the current gaze direction, the mobile terminal can feed back the current gaze direction to the smart glasses, and the smart glasses can track the eye movement of the user wearing the smart glasses according to the current gaze direction.

[0041] Therefore, since the embodiment uses the mobile terminal with an external camera to complete the collection of the eye of the user wearing the smart glasses and the determination of the gaze direction, it is not necessary to set a separate camera and computing unit in the smart glasses, thereby reducing the size of the smart glasses.

[0042] Further, the mobile terminal in the embodiment can be provided with an application (Application, APP) matched with the smart glasses. Before use, the user wearing the smart glasses can open the APP, and then establish a communication connection between the APP and the smart glasses. After the connection is completed, in order to ensure that the eye of the user wearing the smart glasses can be captured, the mobile terminal can be fixed, for example, fixed on a mobile phone holder.

[0043] It should be further emphasized that since the existing mobile phone can generally be provided with a front camera and a rear camera, in the embodiment, the front camera can be used as the camera for capturing, or the rear camera can be used as the camera for capturing. When using the rear camera, the rear main camera in the rear camera can be used, or the wide-angle camera in the rear camera can be used, which is not limited in the embodiment.

[0044] Further, in order to ensure that it can be automatically detected whether the user wearing the smart glasses, in the embodiment, the smart glasses can also be provided with a detection component. Before the step S10, it further includes: Step S01: collecting a current detection distance by the detection component, and when the current detection distance reaches a preset detection distance threshold, performing the step of collecting the current head movement data of the user wearing the smart glasses.

[0045] It should be noted that the detection component can be any device for detecting the distance between the detection component and an obstacle, such as an infrared proximity sensor, which is not limited in the embodiment. In the embodiment, the collection component can be arranged on the side of the smart glasses that is in contact with the user wearing the smart glasses when the smart glasses are worn.

[0046] Further, in actual use, the smart glasses can first detect the distance between the smart glasses and the obstacle in real time using the detection component as the current detection distance, and then determine whether the current detection distance reaches a preset distance threshold, which can be set according to actual conditions. If not, it can be explained that the user does not wear the smart glasses at this time, and there is no need to wake up other (for example, the above-mentioned collection device) to work, thereby reducing the energy consumption. If so, it can be explained that the user wears the smart glasses at this time, and the collection component is woken up to collect the motion data, and at the same time, a wake-up instruction can be sent to the mobile terminal to make the mobile terminal start the camera to take a picture.

[0047] Further, in order to ensure that a clear eye image can be taken, in the embodiment, after the current detection distance reaches the preset detection distance threshold, the step of collecting the current head motion data of the wearing user is performed, and then the method further comprises: Step S02: transmitting the generated wake-up instruction to the mobile terminal to make the mobile terminal start the camera to take a picture, and taking a picture of the eye of the wearing user when the initial image taken by the camera meets the preset quality requirement.

[0048] It should be further noted that the wake-up instruction can be an instruction for waking up the mobile terminal to take a picture. The initial image can be an image taken by the camera at initialization. The preset quality requirement can be to determine whether the initial image contains an eye, and whether the eye meets the preset clarity requirement.

[0049] In actual use, when the smart glasses detect that the user wears the smart glasses, a wake-up instruction can be generated to the mobile terminal. After the mobile terminal receives the wake-up instruction, it can start taking a picture through the camera to obtain an initial image, and perform image recognition on the initial image to determine whether the initial image contains an eye; If not, a prompt is issued to the wearing user to adjust the position. If so, it can be determined whether the clarity of the initial image meets the preset clarity requirement. If not, the user can be prompted to focus, or the mobile terminal can automatically focus. If so, the wearing user can be prompted that the detection is completed and can start using.

[0050] In the embodiment, the smart glasses can be used in cooperation with a mobile terminal provided with a camera. In actual use, the smart glasses can first collect current head movement data of a wearing user and send the current head movement data to the mobile terminal. The mobile terminal can capture the eyeballs of the wearing user through the camera to obtain a current eyeball image. Then, the current gaze direction of the wearing user is determined according to the current head movement data and the current eyeball image. The current gaze direction is fed back to the smart glasses, and the smart glasses perform eye movement tracking according to the current gaze direction. Since the embodiment uses the mobile terminal provided with the camera to collect the eyeballs of the wearing user and determine the gaze direction, it is not necessary to provide a separate camera and a computing unit in the smart glasses, thereby reducing the size of the smart glasses.

[0051] Meanwhile, since the embodiment adopts the eye movement separation design, the functions of capturing the eyeballs and determining the gaze direction are moved to the external mobile terminal to perform, thereby reducing the power consumption of the smart glasses and prolonging the use time of the smart glasses.

[0052] Reference Figure 3 , Figure 3 FIG. 2 is a flowchart of a second embodiment of the eye movement tracking method of the present application. Based on the first embodiment, the second embodiment of the eye movement tracking method of the present application is proposed.

[0053] It is considered that the sensitivity and accuracy required in eye movement tracking are high, and therefore, if the current gaze direction is obtained only according to the current eyeball image, the data accuracy can be poor. Therefore, in the embodiment, the current gaze direction can be determined by analyzing the current head movement data collected by the IMU. In order to remove the interference of noise in collection and ensure that the user can obtain a smooth and natural interactive experience when using the smart glasses, in the embodiment, the step of collecting the current head movement data of the wearing user includes the following steps. Step S11: collecting current head angular velocity and current head acceleration of the wearing user.

[0054] It should be noted that the current head angular velocity can be a parameter reflecting the speed of head rotation in a three-dimensional space. In the embodiment, the current head angular velocity can be collected by an angular velocity sensor in the IMU. In order to facilitate subsequent description, the current head angular velocity is denoted as in the embodiment. The current head angular velocity can include components of x-axis, y-axis and z-axis, which can be denoted as .

[0055] It should be further noted that the current head acceleration described above can be the linear acceleration of the head of the user wearing in the three-dimensional space, and can include acceleration components of the head in each direction. In the embodiment, the current head acceleration described above can be collected by the accelerometer in the IMU, and for the convenience of subsequent description, the current head acceleration described above is denoted as a in the embodiment, which can include components of x-axis, y-axis and z-axis, and can be denoted as ax, ay and az respectively. .

[0056] Step S12: determining the attitude angle change rate and the angular velocity change rate according to the current head angular velocity and the current head acceleration through a preset state equation.

[0057] It can be understood that the preset state equation described above can be a mathematical equation that is preset to describe the relationship between the attitude angle change rate and the head angular velocity, acceleration and the like in the head motion state.

[0058] The attitude angle change rate described above can be the rate of change of the head attitude angle with time, that is, the change amount of the attitude angle per unit time. In the embodiment, the head attitude angle can be used to determine the direction and attitude of the head in space, and the attitude angle change rate can reflect the speed of dynamic adjustment of the head attitude. For the convenience of subsequent description, the head attitude angle described above is denoted as in the embodiment, and the attitude angle change rate is denoted as . The head attitude angle may include a pitch angle, a roll angle and a yaw angle. The pitch angle can be an angle of rotation of the head around a horizontal axis (usually a left-right axis).

[0059] For example, when the head looks up at the sky or looks down at the ground, the pitch angle changes; the roll angle can be an angle of rotation of the head around a longitudinal axis (usually a front-back axis), for example, when the head tilts to the left shoulder or the right shoulder, the roll angle changes; the yaw angle can be an angle of rotation of the head around a vertical axis (usually an axis perpendicular to the ground), for example, when the head turns to the left or the right to look at the side, the yaw angle changes.

[0060] The angular velocity change rate described above can be the rate of change of the head angular velocity with time, that is, the change amount of the angular velocity per unit time. In the embodiment, the angular velocity change rate can describe the acceleration or deceleration of the head rotation motion, and for the convenience of subsequent description, the angular velocity change rate described above is denoted as in the embodiment. .

[0061] Further, in order to obtain the above-mentioned attitude angle change rate and the angular velocity change rate, in the embodiment, the step of determining the attitude angle change rate and the angular velocity change rate according to the current head angular velocity and the current head acceleration by using the preset state equation comprises: Step S121: obtaining the gravity acceleration and the first adaptive gain coefficient, and determining the attitude angle change rate according to the current head angular velocity, the first adaptive gain coefficient, the current head acceleration and the gravity acceleration by using the first preset state equation.

[0062] It should be understood that the above-mentioned gravity acceleration can be the acceleration of the object on the earth surface under the action of gravity, which can be denoted as g. The above-mentioned first adaptive gain coefficient can be a parameter for automatic adjustment, which can be denoted as K in the embodiment. The above-mentioned first adaptive gain coefficient can be determined according to experience or experiment, and can be adjusted in real time subsequently. The above-mentioned first preset state equation can be a preset equation for describing the relationship between the attitude angle change rate and the current head angular velocity, the current head acceleration, the gravity acceleration and the first adaptive gain coefficient.

[0063] In the embodiment, if the attitude angle change rate is denoted as The above-mentioned first preset state equation can be:

[0064] In actual use, the smart glasses can bring the acquired current head angular velocity and the current head acceleration into the above-mentioned first preset state equation, so as to obtain the attitude angle change rate.

[0065] Step S122: discretizing the attitude angle change rate to obtain an attitude angle prediction value.

[0066] In order to facilitate numerical calculation and processing of data, after obtaining the attitude angle change rate, the attitude angle change rate can be discretized, for example, the Euler method can be used for discretization according to the sampling frequency (100 Hz), and the above-mentioned attitude angle prediction value can be obtained by combining the prediction update equation of the attitude angle. The attitude angle prediction value is the prediction value of the attitude angle of the head of the wearing user at the current time.

[0067] Specifically, the prediction update equation is:

[0068] wherein, is the sampling time, which can be obtained according to the sampling frequency, i.e. 1 / 100 s, is the attitude angle prediction value at k-1 time (i.e. the last time), is a change rate of the attitude angle at k-1 time (i.e. at the last time), is a predicted value of the attitude angle at k time (i.e. at the current time).

[0069] In actual use, the smart glasses can obtain the change rate of the attitude angle at the current time according to the first preset state equation by using the current head angular velocity at the current time and the current head acceleration, and then predict the predicted value of the attitude angle at the next time according to the change rate of the attitude angle by using the prediction update equation. The prediction update equation can convert the continuous change rate of the attitude angle into the discrete predicted value of the attitude angle.

[0070] Step S123: obtaining a second adaptive gain coefficient, and obtaining the change rate of the angular velocity according to the second adaptive gain coefficient and the predicted value of the attitude angle by using a second preset state equation.

[0071] It can be understood that the second adaptive gain coefficient can be a parameter for automatic adjustment, which can be denoted as The second adaptive gain coefficient can be obtained according to experience or experiment, and can be adjusted in real time subsequently.

[0072] The second preset state equation can be a predefined equation for describing the relationship between the change rate of the angular velocity and the predicted value of the attitude angle.

[0073] In the embodiment, if the change rate of the angular velocity is denoted as The second preset state equation can be: ; In actual use, after the smart glasses obtain the predicted value of the attitude angle, the predicted value of the attitude angle can be brought into in the second preset state equation to obtain the change rate of the angular velocity in combination with the second adaptive gain coefficient.

[0074] Step S13: performing attitude angle estimation based on the change rate of the attitude angle and the change rate of the angular velocity by using a Kalman filter to obtain the head attitude angle of the wearing user; Step S14: taking the head attitude angle as the current head motion data of the wearing user.

[0075] It should be understood that the Kalman filter can be a recursive algorithm for estimating the state of a dynamic system, and can perform optimal estimation on the state of the system. In the above steps, the Kalman filter can be used to estimate the head attitude angle.

[0076] Before the estimation is actually performed by the Kalman filter, the smart glasses can also initialize the Kalman filter, specifically, the initial head posture angle and the covariance matrix can be set, and the initial head posture angle can be set according to the initial output or experience value. The covariance matrix can reflect the degree of uncertainty of the initialized head posture angle, and can be set as a large diagonal matrix, which can be set according to the actual situation, and the embodiment is not limited in this regard.

[0077] In actual use, the smart glasses can first predict the attitude angle rate and the angular velocity rate at the current time according to the preset state equation, the angular velocity rate at the last time and the attitude angle rate at the last time. The specific prediction process and the process of determining the attitude angle rate by the first preset state equation and determining the angular velocity rate by the second preset state equation will not be repeated in the embodiment.

[0078] After obtaining the attitude angle rate at the current time and the angular velocity rate at the current time, the head posture angle estimated at the last time is updated according to the attitude angle rate at the current time, and the estimated head posture angle at the current time is obtained, and the estimated covariance matrix is updated.

[0079] Then, the current head angular velocity and the current head acceleration collected at the current time are substituted into the observation equation to obtain the measurement information about the head posture, and then the Kalman gain is used to obtain a more accurate head posture angle of the wearing user by comprehensively using the estimated head posture angle and the actual measurement information, and the estimated covariance matrix is updated. Further, the obtained head posture angle of the wearing user can be used as the current head motion data of the wearing user.

[0080] Since the Kalman filter can be used in the embodiment, the measured data can be smoothed to remove noise. At the same time, since the Kalman filter is a recursive algorithm, it can estimate and update the attitude angle in real time, and then the optimal head posture angle at the current time can be quickly calculated according to the state estimation value at the last time and the measurement value at the current time, which can meet the requirement of real-time tracking of the head posture in the embodiment.

[0081] In addition, to achieve the above-mentioned purpose, the embodiment of the application also provides an eye movement tracking method, which is described with reference to Figure 4 , Figure 4 The flowchart of the third embodiment of the eye movement tracking method of the application is shown in the figure, and the third embodiment of the eye movement tracking method of the application is proposed based on the above-mentioned embodiments.

[0082] In the embodiment, the method comprises the following steps: Step S20: receiving the current head movement data of the wearing user sent by the smart glasses, the current head movement data being obtained by the smart glasses collecting the wearing user.

[0083] It can be understood that the method of the embodiment can be applied to the mobile terminal provided with the camera as described above, and the mobile terminal can be any device with data processing, program running and eye movement tracking functions, such as a mobile phone, a computer, etc. The embodiment is not limited thereto. The execution subject of the method embodiment can use the mobile terminal to describe the embodiment and each of the following embodiments.

[0084] It can also be understood that the step S20 as described above in the embodiment can be after the step S10. In actual use, the mobile terminal in the embodiment can be in communication connection with the smart glasses, and the current head movement data sent by the smart glasses can be obtained. The specific process can be referred to the above embodiments, and the embodiment is not described in detail.

[0085] Step S30: capturing the eyeball of the wearing user through the camera to obtain a current eyeball image.

[0086] It is emphasized that while receiving the current head movement data as described above, the mobile terminal in the embodiment can also capture the eyeball of the wearing user through the camera to obtain a current eyeball image.

[0087] Further, in order to obtain a clear current eyeball image, in the embodiment, the step of capturing the eyeball of the wearing user through the camera to obtain a current eyeball image includes: Step S31: capturing the face of the wearing user through the camera to obtain a current face image of the wearing user.

[0088] It is noted that the current face image can be an image of the face of the wearing user at the current time. When the mobile terminal captures, the whole face of the wearing user can be captured first to obtain the current face image of the wearing user.

[0089] Step S32: performing face key point detection on the current face image to obtain key point coordinates of an eye region in the current face image.

[0090] It should be noted that in the embodiment, a lightweight AI model can be arranged in the mobile terminal, specifically, a neural processing unit (Neural Processing Unit, NPU) can be arranged in the mobile terminal, and the AI model is deployed in the NPU, and the AI model can be any model used for key point detection. In the embodiment, the point in the eye region can be used as a key point. In actual use, after obtaining the current face image, the mobile terminal can perform face key point detection through the model to obtain the key point coordinates in the eye region.

[0091] Step S33: Region of interest cropping is performed on the eye region in the current face image according to the key point coordinates, and a current eyeball image is obtained.

[0092] It can be understood that the current eyeball image can be an image of the eyeball of the user. After obtaining the key point coordinates of the eye region, the eye region can be cropped as a region of interest according to the key point coordinates, and the current eyeball image is obtained.

[0093] It can also be understood that in the embodiment, the size of the region of interest collected when performing region of interest cropping can be 256x256 pixels, and can also be other pixels, which is not limited in the embodiment.

[0094] Step S40: Determine the current gaze direction of the user wearing the smart glasses according to the current head movement data and the current eyeball image, and feed back the current gaze direction to the smart glasses, so that the smart glasses perform eye movement tracking on the user wearing the smart glasses according to the current gaze direction.

[0095] It can be understood that the above step S40 in the embodiment can be performed before the above step S50. After obtaining the current eyeball image, the current gaze direction of the user wearing the smart glasses can be obtained by fusing the received current head movement data and the cropped current eyeball image, and the current gaze direction is sent to the smart glasses, so that the smart glasses can perform eye movement tracking according to the current gaze direction.

[0096] Therefore, since the embodiment uses a mobile terminal externally provided with a camera to complete the collection of the eyeball of the user wearing the smart glasses and the determination of the gaze direction, it is not necessary to arrange a separate camera and a computing unit in the smart glasses, thereby reducing the size of the smart glasses.

[0097] Reference Figure 5 , Figure 5 The flowchart of the fourth embodiment of the eye movement tracking method of the present application is shown in the figure. Based on the above embodiments, the fourth embodiment of the eye movement tracking method of the present application is proposed.

[0098] To achieve the fusion of the current head movement data and the current eye image, so as to determine the current gaze direction. As shown in Figure 5 In the embodiment, the step of determining the current gaze direction of the wearing user according to the current head movement data and the current eye image includes: Step S41: determining the iris ellipse parameter of the wearing user according to the current eye image.

[0099] It should be noted that the iris ellipse parameter can be a geometric feature of the iris region of the wearing user's eye, including the major axis, minor axis and rotation angle of the ellipse, etc. Specifically, the iris ellipse parameter can be obtained by ellipse fitting according to the size and position of the iris in the current eye image.

[0100] Step S42: generating the line-of-sight direction vector of the wearing user based on the iris ellipse parameter.

[0101] It can be understood that the line-of-sight direction vector can be a vector from the optical center of the wearing user's eye to the target point being gazed at.

[0102] In actual use, when the mobile terminal obtains the iris ellipse parameter, the optical center of the eye can be determined according to the current eye image, and the line-of-sight direction vector can be obtained according to the optical center and the rotation angle in the iris ellipse parameter. For the convenience of subsequent description, the line-of-sight direction vector is denoted as v_eye in the embodiment.

[0103] Step S43: fusing the current head movement data and the line-of-sight direction vector to obtain the current gaze direction of the wearing user.

[0104] After obtaining the line-of-sight direction vector, it can be fused with the current head movement data to obtain the current gaze direction of the wearing user's eye.

[0105] Specifically, the step of fusing the current head movement data and the line-of-sight direction vector to obtain the current gaze direction of the wearing user includes: Step S431: fusing the current head movement data and the line-of-sight direction vector.

[0106] In actual use, the mobile terminal can first fuse the quaternion in the received current head movement data and the line-of-sight direction vector, and if the fusion result is denoted as v_gaze and the quaternion is denoted as q_head, then v_gaze=q_head×v_eye.

[0107] Step S432: determining the current iris position of the wearing user according to the current eye image by using the Kalman filter, and performing gaze point coordinate estimation according to a preset observation matrix, the fusion result, the current head movement data and the current iris position, to obtain the gaze point coordinate of the wearing user.

[0108] It should be understood that the current iris position described above can be a specific position of the iris in the eye coordinate system in the current eye image, and the iris center point coordinate can be represented. Specifically, the position of the iris in the current eye image can be obtained according to the key point detection result as the iris center point coordinate, and then the iris center point coordinate is converted into a position in the eye coordinate system, so as to obtain the current iris position. The current iris position can be denoted as .

[0109] In actual use, the coordinate point estimation can be completed by the following observation equation. If the current head movement data is denoted as , the observation equation is .

[0110] Wherein, the above is the preset observation matrix described above. The observation matrix can be a matrix for mapping the observation vector to the observation space. Specifically, the observation matrix can be defined according to the observation model, and the embodiment is not limited in this regard.

[0111] The gaze point coordinate described above can be the coordinate of the falling point of the line of sight of the wearing user in the three-dimensional space, and can be used to represent the current gaze direction of the wearing user. In the embodiment, it can be denoted as .

[0112] Step S433: taking the gaze point coordinate as the current gaze direction of the wearing user.

[0113] In actual use, the mobile terminal can first obtain the preset observation matrix set in advance, and then predict the gaze point coordinate at the current time by using the observation equation and the gaze point coordinate at the last time by the Kalman filter, and finally take the gaze point coordinate as the current gaze direction of the wearing user.

[0114] It should be emphasized that the specific implementation mode of the eye movement tracking method performed by the mobile terminal in the embodiment can be mutually referred to the specific implementation mode of the eye movement tracking method performed by the smart glasses, and the embodiment will not be described in detail.

[0115] In addition, referring to Figure 6 , Figure 6 is a mobile terminal structure schematic diagram of a hardware running environment involved in the embodiment scheme of the application.

[0116] As Figure 6As shown, the mobile terminal can include a second processor 2001, such as a Central Processing Unit (CPU), a second communication bus 2002, a second user interface 2003, a second network interface 2004, and a second memory 2005. The second communication bus 2002 is configured to realize the connection communication between the components. The second user interface 2003 can be connected with a display screen (Display), and optionally, the second user interface 2003 can include a standard wired interface, a wireless interface, and the wired interface of the second user interface 2003 can be a USB interface in the present application. The second network interface 2004 can optionally include a standard wired interface and a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The second memory 2005 can be a high-speed Random Access Memory (RAM), and can also be a stable Non-volatile Memory (NVM), such as a disk memory. The second memory 2005 can also be a storage device independent of the aforementioned second processor 2001.

[0117] Those skilled in the art can understand that Figure 6 The structure shown in the figure does not constitute a limitation on the mobile terminal, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0118] As Figure 6 As shown, the second memory 2005, which is a computer storage medium, can include a second operating system, a second network communication module, a second user interface module, and an eye movement tracking program.

[0119] In Figure 6 In the mobile terminal shown, the second network interface 2004 is mainly configured to connect to a background server and communicate data with the background server; the second user interface 2003 is mainly configured to connect to a user device (such as the smart glasses described above); the mobile terminal invokes the eye movement tracking program stored in the second memory 2005 by the second processor 2001, and executes the steps of the eye movement tracking method provided in the third and fourth embodiments of the present application. It should be noted that the mobile terminal in the present embodiment can also be provided with a camera, which can be connected with the second processor 2001 to take pictures.

[0120] In addition, the present application also provides an eye movement tracking system, which includes the smart glasses described above and the mobile terminal described above.

[0121] The specific implementation manners of the eye tracking system in this embodiment can refer to the descriptions of the above method embodiments, and this embodiment will not be described here.

[0122] It should be noted that in this document, the terms "comprise", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles, or systems that include a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles, or systems. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.

[0123] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (Read Only Memory image, ROM) / random access memory (Random Access Memory, RAM), a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0125] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An eye-tracking method, characterized in that, The method is applied to smart glasses, which are used in conjunction with a mobile terminal equipped with a camera. The method includes: Collect the current head movement data of the wearer and send the current head movement data to the mobile terminal; The system receives the current gaze direction of the user as fed back by the mobile terminal, and performs eye tracking on the user based on the current gaze direction. The current gaze direction is obtained by the mobile terminal based on the current head movement data and the current eye image of the wearer. The current eye image is obtained by the camera capturing the wearer's eyes.

2. The method as described in claim 1, characterized in that, The step of collecting the current head movement data of the wearer includes: Collect the wearer's current head angular velocity and current head acceleration; The attitude angle change rate and angular velocity change rate are determined by a preset state equation based on the current head angular velocity and the current head acceleration. The head posture angle of the wearer is obtained by using a Kalman filter to estimate the posture angle based on the rate of change of posture angle and the rate of change of angular velocity. The head posture angle is used as the current head movement data of the wearer.

3. The method as described in claim 2, characterized in that, The step of determining the rate of change of attitude angle and the rate of change of angular velocity based on the current head angular velocity and the current head acceleration using a preset state equation includes: The gravitational acceleration and the first adaptive gain coefficient are obtained, and the attitude angle change rate is determined by the first preset state equation based on the current head angular velocity, the first adaptive gain coefficient, the current head acceleration, and the gravitational acceleration. Discretize the rate of change of the attitude angle to obtain the predicted attitude angle value; Obtain the second adaptive gain coefficient, and then obtain the angular velocity change rate based on the second adaptive gain coefficient and the predicted attitude angle value through the second preset state equation.

4. An eye-tracking method, characterized in that, The method is applied to a mobile terminal equipped with a camera, the mobile terminal being used in conjunction with smart glasses, and the method includes: Receive the current head movement data of the user being worn, which is obtained by the smart glasses from the user being worn; The camera captures an image of the user's eyeballs to obtain a current eyeball image. The current gaze direction of the wearer is determined based on the current head movement data and the current eye image, and the current gaze direction is fed back to the smart glasses so that the smart glasses can perform eye tracking on the wearer based on the current gaze direction.

5. The method as described in claim 4, characterized in that, The step of capturing an image of the wearer's eyeball using the camera to obtain a current eyeball image includes: The camera captures a picture of the user's face to obtain a current facial image of the user. Facial key point detection is performed on the current facial image to obtain the coordinates of key points in the eye region within the current facial image; Based on the coordinates of the key points, the region of interest in the eye area of ​​the current facial image is cropped to obtain the current eyeball image.

6. The method as described in claim 4, characterized in that, The step of determining the current gaze direction of the wearer based on the current head motion data and the current eye image includes: The iris ellipse parameters of the wearer are determined based on the current eye image; Generate the gaze direction vector of the wearer based on the iris ellipse parameters; The current head motion data and the gaze direction vector are fused to obtain the current gaze direction of the wearer.

7. The method as described in claim 6, characterized in that, The step of fusing the current head motion data and the gaze direction vector to obtain the current gaze direction of the wearer includes: The current head motion data and the gaze direction vector are fused together; The current iris position of the wearer is determined by using a Kalman filter based on the current eye image, and the gaze point coordinates are estimated based on a preset observation matrix, fusion results, the current head motion data, and the current iris position to obtain the gaze point coordinates of the wearer. The coordinates of the gaze point are used as the current gaze direction of the wearer.

8. A type of smart glasses, characterized in that, The smart glasses include: a first memory, a first processor, and an eye-tracking program stored in the first memory and executable on the first processor, wherein the eye-tracking program, when executed by the first processor, implements the steps of the eye-tracking method as described in any one of claims 1 to 3.

9. A mobile terminal, characterized in that, The mobile terminal includes: a camera, a second memory, a second processor, and an eye-tracking program stored in the second memory and executable on the second processor, wherein the eye-tracking program, when executed by the second processor, implements the steps of the eye-tracking method as described in any one of claims 4 to 7.

10. An eye-tracking system, characterized in that, The system includes the smart glasses as described in claim 8 and the mobile terminal as described in claim 9.

Images