Gaze information determination method and device, equipment and medium

Abstract

The invention discloses a fixation information determination method, device, equipment and medium, and the method comprises the steps: obtaining a data set which comprises a plurality of groups of data pairs, each group of data pair comprises first data and second data which are obtained by different equipment, the first data are data which are determined through an optical recording method and are related to fixation information, and the second data are data which are related to fixation information; the second data is data which is determined by the micro electro mechanical system and is related to gaze information; determining a target object according to the data set; correcting initial gazing information determined by the micro electro mechanical system based on the target object, and determining corrected target gazing information; according to the method, the data pairs are formed according to different data acquired by different devices, the data set is formed according to the multiple groups of data pairs, then the target object is determined, the initial gaze information is corrected through the target object, the corrected target gaze information is determined, and the accuracy of determining the gaze information is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to a method, device, equipment and medium for determining gaze information. Background Art

[0002] With the rapid development of computer vision, artificial intelligence technology and digital technology, eye tracking technology has become a hot research field and has been widely used in the field of human-computer interaction.

[0003] The existing technology uses pre-deployed optical devices to confirm whether corneal reflection occurs in the corresponding direction, determines the approximate direction of eye rotation based on its orientation, and then obtains the required line of sight information.

[0004] However, the above scheme for determining line of sight information can only obtain eye tracking results with lower accuracy, because its eye tracking accuracy is directly restricted by the number of optical devices, and if too many optical devices are deployed, the overall signal refresh rate will be affected. Summary of the invention

[0005] The present invention provides a gaze information determination method, device, equipment and medium to solve the problem of inaccurate gaze information calculation in the prior art.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining gaze information, including:

[0007] Acquire a data set, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data is data related to gaze information determined by an optical recording method, and the second data is data related to gaze information determined by a micro-electromechanical system;

[0008] Determine a target object according to the data set;

[0009] Initial gaze information determined by the micro-electromechanical system is corrected based on the target object to determine corrected target gaze information.

[0010] Optionally, the first data includes gaze point information calculated using an optical recording method, the second data includes gaze information calculated by a micro-electromechanical system, the first data and the second data are data determined at the same time, the gaze information includes a line of sight direction and / or gaze point coordinates, and the target object includes a correction coefficient;

[0011] Determining the target object according to the data set includes:

[0012] For each data pair in the data set, determining the data deviation of the data pair on each coordinate axis;

[0013] For the data deviations on each coordinate axis, determining a deviation average of the data deviations;

[0014] The average value of the deviations corresponding to each coordinate axis is determined as a candidate correction coefficient;

[0015] determining whether the candidate correction coefficient satisfies a correction condition;

[0016] If so, a correction factor is determined based on the determined candidate correction factor.

[0017] Optionally, determining whether the candidate correction coefficient satisfies a correction condition includes:

[0018] Correcting second data at a subsequent time based on the candidate correction coefficient to obtain corrected target data, wherein the second data at the subsequent time is gaze information determined by the micro-electromechanical system after determining the candidate correction coefficient;

[0019] Determine a data difference between the target data and first data at a subsequent moment, where the first data at the subsequent moment is data acquired simultaneously with the second data at the subsequent moment;

[0020] If the data difference is less than the set threshold, the candidate correction coefficient meets the correction condition; otherwise, return to continue to obtain the next data set, determine the candidate correction coefficient of the next data set, and judge whether the candidate correction coefficient of the next data set meets the correction condition;

[0021] When determining the candidate correction coefficient corresponding to the next data set, the data deviation corresponding to the next data set is determined based on the corrected second data in the next data set and the corresponding first data, and the corrected second data is the data obtained by correcting the second data in the next data set based on the previous candidate correction coefficient.

[0022] Optionally, the first data includes first process data in a process of determining gaze information by an optical recording method, the second data includes second process data in a process of determining gaze information by the micro-electromechanical system, the target object includes an eye model, and determining the target object according to the data set includes:

[0023] modifying the eye model according to the first process data and the second process data;

[0024] If the residual of the corrected eye model is less than the set residual threshold, the corrected eye model is determined as the target object; otherwise, return to continue to obtain the next data set, correct the eye model based on the next data set, and determine whether the corrected eye model corresponding to the next data set is less than the set residual threshold.

[0025] Optionally, the first process data includes eye data measured and data calculated during the measurement process when gaze information is determined by an optical recording method, and the second process data includes eye data measured and data calculated during the measurement process when gaze information is determined by the micro-electromechanical system.

[0026] Optionally, correcting the initial gaze information determined by the micro-electromechanical system based on the target object to determine the corrected target gaze information includes:

[0027] determining initial gaze information by the micro-electromechanical system;

[0028] The initial gaze information is corrected based on the determined target object to obtain corrected target gaze information.

[0029] Optionally, in the process of determining the corrected target gaze information, the state of the optical device corresponding to the optical recording method is a closed state.

[0030] In a second aspect, an embodiment of the present invention provides a gaze information determination device, including:

[0031] an acquisition module, configured to acquire a data set, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data is data related to gaze information determined by an optical recording method, and the second data is data related to gaze information determined by a micro-electromechanical system;

[0032] A first determination module, used to determine a target object according to the data set;

[0033] The second determination module is used to correct the initial gaze information determined by the micro-electromechanical system based on the target object to determine the corrected target gaze information.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0035] An optical device and a micro-electromechanical system for determining gaze information using an optical recording method, at least one processor; and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The optical device and the micro-electromechanical system are respectively connected to the processor;

[0038] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the gaze information determination method provided by any embodiment of the present invention;

[0039] The micro-electromechanical system is used to transmit second data to the processor, and the optical device is used to transmit first data to the processor.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gaze information determination method described in any embodiment of the present invention when executed.

[0041] The technical solution of the embodiment of the present invention is to obtain a data set, wherein the data set includes multiple groups of data pairs, each group of data pairs includes first data and second data obtained by different devices, the first data is data related to gaze information determined by an optical recording method, and the second data is data related to gaze information determined by a micro-electromechanical system; according to the data set, a target object is determined; based on the target object, initial gaze information determined by the micro-electromechanical system is corrected to determine corrected target gaze information; the present invention forms data pairs according to different data obtained by different devices, forms a data set according to multiple groups of data pairs, and then determines the target object, corrects the initial gaze information by the target object, and determines the corrected target gaze information, thereby solving the problem of inaccurate calculation of gaze information in the prior art and effectively improving the accuracy of determining gaze information.

[0042] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flowchart of a method for determining gaze information provided in Embodiment 1 of the present invention;

[0045] Figure 2A A flowchart of a method for determining gaze information provided in Embodiment 2 of the present invention;

[0046] Figure 2BA flowchart of a method for determining gaze information provided in Embodiment 3 of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a device for determining gaze information provided by a fourth embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] Embodiment 1

[0052] Figure 1A flowchart of a gaze information determination method provided in the first embodiment of the present invention. This embodiment can be applied to eye tracking. The method can be executed by a gaze information determination device. The gaze information determination device can be implemented in the form of hardware and / or software. The gaze information determination device can be configured in an electronic device, such as a head-mounted device. The head-mounted device can be considered as a device worn on the user's head. Among them, the head-mounted device can be analytical glasses, and the electronic device of the present application can also be a virtual reality (VR) / augmented reality (AR) accessory. The VR / AR accessory can be considered as an accessory of the VR / AR device, which can realize the determination of gaze information. The determined gaze information can be subjected to eye tracking, also known as line of sight tracking. Eye tracking is a technology that estimates eye gaze information by measuring eye movement. The gaze information includes line of sight direction and / or gaze point. As Figure 1 As shown, the method includes:

[0053] S110. Acquire a data set, wherein the data set includes multiple groups of data pairs, each group of data pairs includes first data and second data acquired by different devices, the first data is data related to the gaze information determined by an optical recording method, and the second data is data related to the gaze information determined by a micro-electromechanical system.

[0054] Among them, a data set may refer to a set consisting of data; a data pair may consist of a pair of data, wherein a pair of data includes at least two data; different devices may refer to different devices for acquiring data; and different data may be acquired through different devices. In this embodiment, different data acquired through different devices may be divided into first data and second data.

[0055] The optical recording method can be based on the pupil-corneal reflection of an infrared camera and an infrared light source. The camera captures the position of the Purkinje spot reflected by the cornea and the position of the pupil, and based on the constructed 3D reconstructed eyeball model, the spatial line of sight direction is calculated to obtain the required line of sight information.

[0056] The optical recording method may include a pupil-corneal reflection method. The working principle of the pupil-corneal reflection method can be simply summarized as obtaining an eye image and estimating the line of sight and / or gaze point based on the eye image. The hardware of the pupil-corneal reflection method includes a light source and an image acquisition device. The light source is generally an infrared light source, such as a plurality of infrared light sources arranged in a set manner, such as a herringbone shape and / or a straight line shape. The image acquisition device may be an infrared camera device, an infrared sensor, a camera and / or a video camera, etc.

[0057] The present invention can acquire eye images through pupil corneal reflection method as follows: study the direction of light source, photograph the eyes with image acquisition equipment, and correspondingly photograph the reflection point of the light source on the cornea, i.e., the light spot, thereby acquiring an eye image with the light spot.

[0058] The method of the present invention for estimating line of sight and / or gaze point by pupil-corneal reflection method is as follows: as the eyeball rotates, the relative position relationship between the pupil center and the light spot changes accordingly, and the position change relationship reflected by a number of eye images with light spots is collected accordingly, and the line of sight and / or gaze point is estimated based on the position change relationship.

[0059] The micro-electromechanical system can confirm whether corneal reflection occurs in the corresponding direction through pre-deployed optical signal transmitters and optical signal receivers at multiple locations, determine the approximate direction of eyeball rotation based on its orientation, and then obtain the required line of sight information. The gaze information can be the line of sight direction or the gaze point. The micro-electromechanical system can include an optical signal transmitter (such as an infrared light source), an optical signal receiver (such as an optical sensor), and a MEMS micro-electromechanical system.

[0060] Specifically, in this embodiment, the first data can be determined by an optical recording method, the second data can be determined by a micro-electromechanical system, the first data and the second data can be combined into a data pair and stored in a data set, and multiple data pairs can be formed by acquiring the first data and the second data multiple times and stored in a data set. In this embodiment, the first data and the second data that form a data pair can be acquired at the same time, or the data whose acquisition time difference is less than a set threshold. The acquired data set can be a data set consisting of multiple first data and second data determined by the optical recording method and the micro-electromechanical system within a period of time; it can also be a data set consisting of N first data and second data determined by the optical recording method and the micro-electromechanical system.

[0061] It should be noted that the acquisition, storage, use and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations and are authorized by the user. The user may be the owner of the data. The data may include the first data, the second data and the target gaze information.

[0062] S120: Determine a target object according to the data set.

[0063] The target object may refer to data that can correct the gaze information.

[0064] In this embodiment, the target object can be calculated and determined based on multiple data pairs in the data set. For example, this step can calculate the deviation average value based on the deviation of the first data and the second data in each data pair in the data set, and then determine whether the deviation average value meets the correction condition. The preset condition can be a set threshold. If the correction condition is met, the deviation average value is determined as the target object. This step can also pre-establish an eye model, correct the eye model based on the first data and the second data in each data pair in the data set, and then determine whether the corrected eye model meets the preset condition. The preset condition can be a residual threshold. If the preset condition is met, the corrected eye model is determined as the target object. This embodiment does not limit the specific calculation method.

[0065] S130 , correcting the initial gaze information determined by the micro-electromechanical system based on the target object to determine corrected target gaze information.

[0066] Among them, the initial gaze information may refer to the gaze information determined by the microelectromechanical system after the target object is determined, such as the subsequent second data determined by the microelectromechanical system after the target object is determined by acquiring the data set; for example, if the data set acquires N data within a T time period, and the target object is determined based on a data set consisting of N first data and N second data, then the initial gaze information is the N+1th data determined by the microelectromechanical system. The target gaze information may refer to the gaze information after the initial gaze information is corrected according to the target object. For example, this step can correct the N+1th gaze information determined by the microelectromechanical system according to the target object, and determine the corrected N+1th gaze information as the target gaze information.

[0067] In this embodiment, the initial gaze information can be corrected by adding the determined target object to the initial gaze information determined by the micro-electromechanical system, thereby determining the corrected target gaze information to ensure the accuracy of the output target gaze information; in addition, the initial gaze information can be corrected by using the eye model corrected by the first data and the second data, thereby determining the corrected target gaze information to ensure the accuracy of the output target gaze information.

[0068] Exemplarily, this embodiment combines the optical recording method with the micro-electromechanical system. When the electronic device is just turned on, the optical recording method and the micro-electromechanical system are used to generate corresponding data, such as the first data and the second data. At the same time, during the startup process, the target object for correcting the micro-electromechanical system is determined, and finally the initial gaze information of the micro-electromechanical system is corrected by the target object to obtain the target gaze information. That is, the optical recording method is used in the initial stage of the electronic device startup to correct the gaze information of the micro-electromechanical system. When the correction reaches a certain accuracy, the optical recording method is stopped, and only the micro-electromechanical system is used to determine the target gaze information. The method provided in this embodiment not only improves the eye tracking precision and accuracy, but also reduces the power consumption of the entire system and reduces the amount of data transmission.

[0069] In the present embodiment, a data set is acquired, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data being data related to gaze information determined by an optical recording method, and the second data being data related to gaze information determined by a micro-electromechanical system; a target object is determined according to the data set; initial gaze information determined by the micro-electromechanical system is corrected based on the target object to determine corrected target gaze information; the present invention forms data pairs according to different data acquired by different devices, forms a data set according to a plurality of data pairs, and further determines the target object, corrects the initial gaze information by the target object, and determines the corrected target gaze information, thereby solving the problem of inaccurate calculation of gaze information in the prior art and effectively improving the accuracy of determining gaze information.

[0070] Optionally, determining corrected target gaze information based on the target object and initial gaze information determined by the micro-electromechanical system includes:

[0071] Initial gaze information is determined by the micro-electromechanical system; and the initial gaze information is corrected based on the determined target object to obtain corrected target gaze information.

[0072] Specifically, in this embodiment, the initial gaze information determined by the micro-electromechanical system is that after the target object is determined, a camera is not used to collect the user's eye image, but a low-power infrared diode is used to emit infrared light. After the infrared light is emitted, it is directed to the plane mirror of the scanner. At the same time, the scanner will control the infrared light to be directed to the user's eye, and then reflected from the corneal surface to the optical sensor. The optical sensor converts the light signal into an electrical signal, that is, an eye electrical signal; due to the different signal intensities fed back by the pupil area and the non-pupil area for the same stimulus, the pupil area will be significantly lower than the non-pupil area, and the eyeball parameters are determined from the generated signal intensity distribution map, including the user's pupil contour, pupil size, pupil diameter data, corneal data and / or iris data and other information, and finally the generated second data is identified, and the second data can be the eye gaze direction, or it can be a gaze point in a determined space; wherein, the eye gaze direction can be represented by a vector extending vertically outward from the center of the eye pupil.

[0073] Specifically, correcting the initial gaze information according to the target object can be understood as combining the deviation average value on the basis of the first data at a subsequent moment, or correcting the initial gaze information according to the corrected eye model to obtain the corrected target gaze information.

[0074] In the present embodiment, initial gaze information is determined by the micro-electromechanical system; the initial gaze information is corrected based on the determined target object to obtain corrected target gaze information, thereby improving the accuracy of determining the gaze information; further, after determining the target object and then determining the target gaze information by the micro-electromechanical system, eye electrical signals are used instead of eye images, and corresponding eye data are determined based on the signal strength in the eye electrical signals, thereby determining the initial gaze information, thereby avoiding the shortcomings of high power consumption and large data transmission volume of the eye tracking method using a camera, effectively reducing the power consumption of the entire system and reducing the data transmission volume.

[0075] Optionally, in the process of determining the corrected target gaze information, the state of the optical device corresponding to the optical recording method is a closed state.

[0076] Among them, the optical device, also known as the optical device, can be an infrared camera, an infrared light source and other devices, and its state can be closed or open. In this embodiment, in the process of determining the corrected target gaze information, it can be shown that the premise is that the target object is determined according to the data set; the initial gaze information determined by the micro-electromechanical system is corrected based on the target object, and the corrected target gaze information is determined to be more accurate; therefore, the target object can be used to correct the initial gaze information determined by the micro-electromechanical system in the future, and there is no need to use the data related to the gaze information determined by the optical recording method; therefore, the state of the corresponding optical device can be set to the closed state, which also saves the power consumption of the entire system and reduces the data transmission volume.

[0077] In this embodiment, in the process of determining the corrected target gaze information, the state of the optical device corresponding to the optical recording method is set to a closed state, which effectively reduces the power consumption of the entire system.

[0078] Embodiment 2

[0079] Figure 2A This is a flow chart of a method for determining gaze information provided by Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above embodiment. Figure 2A As shown, if the first data includes gaze information calculated using an optical recording method, the second data includes gaze information calculated by a micro-electromechanical system, the first data and the second data are data determined at the same time, and the target object includes a correction coefficient; the method of this embodiment may include the following steps:

[0080] S210a, obtaining a data set.

[0081] The data set includes multiple groups of data pairs, each group of data pairs includes first data and second data acquired by different devices, the first data includes gaze information calculated using an optical recording method, the second data includes gaze information calculated by a micro-electromechanical system, the first data and the second data are data determined at the same time, and the gaze information includes a line of sight direction and / or gaze point coordinates.

[0082] Among them, the gaze information can be information associated with the gaze. For example, the gaze information can refer to the coordinates of the gaze point in the plane space, that is, the gaze point coordinates. For example, a two-dimensional gaze point coordinate system can be pre-constructed, and then a plurality of different first data and second data can be determined in the pre-constructed two-dimensional coordinate system according to different devices. In addition, a three-dimensional gaze point coordinate system can also be established, and this embodiment does not limit this. The first data includes a plurality of gaze point coordinates calculated using an optical recording method, and the second data includes a plurality of gaze point coordinates calculated by a micro-electromechanical system; for another example, the gaze information can also refer to the sight direction determined in the three-dimensional space, the first data includes a plurality of sight directions calculated and determined using an optical recording method, and the second data includes a plurality of sight directions calculated and determined by a micro-electromechanical system; the first data and the second data are data determined at the same time, which can indicate that the two devices for determining the first data and the second data are turned on at the same time. The correction coefficient can refer to a coefficient that can correct the initial gaze information.

[0083] S220a. For each data pair in the data set, determine the data deviation of the data pair on each coordinate axis.

[0084] The data deviation on each coordinate axis can be understood as the deviation value of the first data and the second data in the data pair on each coordinate axis, such as the coordinate deviation of the gaze point coordinates, or the angle deviation of the line of sight direction. If the coordinate system is a three-dimensional coordinate system, each coordinate axis can be an x-axis, a y-axis, and a z-axis respectively; if the coordinate system is a two-dimensional coordinate system, each coordinate axis can be an x-axis and a y-axis respectively, and this embodiment does not limit this.

[0085] In one embodiment, if the data deviation is the coordinate deviation of the gaze point coordinates, this embodiment processes each data pair in the data set separately, and determines the coordinate deviation of each data pair respectively. Each coordinate axis has its own coordinate deviation. For each coordinate axis, the difference of the gaze point coordinates under the coordinate axis can be determined as the coordinate deviation of the coordinate axis. For example, for the x-axis, the difference between the gaze point coordinates of the first data on the x-axis and the gaze point coordinates of the second data on the x-axis is determined as the coordinate deviation of the first data and the second data on the x-axis. For the y-axis, the difference between the gaze point coordinates of the first data on the y-axis and the gaze point coordinates of the second data on the y-axis is determined as the coordinate deviation of the first data and the second data on the y-axis.

[0086] Exemplarily, the first data includes the gaze point coordinates A(X A ,Y A ); At the same time, the second data includes the gaze point coordinate A'(X A’ ,Y A’); The data set includes multiple corresponding data pairs A and A', B and B', C and C'... Calculate the difference θ of the fixation point coordinates in each data pair A (X A -X A’ , Y A -Y A’ ), θ B (X B -X B’ , Y B -Y B’ ), θ C (X C -X C’ , Y C -Y C’ )……. A time threshold may be pre-set, for example, T1 second, and N θ are generated within T1 second.

[0087] Among them, X A -X A’ The data deviation on the x-axis can be centered for the first set of data. A -Y A’ The data deviation on the y-axis can be centered for the first set of data. B -X B’ The data deviation on the x-axis can be centered for the second set of data. B -Y B’ The data deviation on the y-axis can be centered for the second set of data.

[0088] In another embodiment, if the data deviation is the angle deviation of the line of sight direction, this embodiment processes each data pair in the data set separately, and determines the angle deviation of each data pair respectively, and each coordinate axis has its own angle deviation. For example, for the x-axis, the difference between the angle of the line of sight direction of the first data on the x-axis and the angle of the line of sight direction of the second data on the x-axis is determined as the angle deviation of the first data and the second data on the x-axis, wherein the angle of the line of sight direction on the x-axis can be the angle between the x-axis and the line of sight direction in space. For the y-axis, the difference between the angle of the line of sight direction of the first data on the y-axis and the angle of the line of sight direction of the second data on the y-axis is determined as the angle deviation of the first data and the second data on the y-axis, wherein the angle of the line of sight direction on the y-axis can be the angle between the y-axis and the line of sight direction in space. For the z-axis, the difference between the angle of the line of sight direction of the first data on the z-axis and the angle of the line of sight direction of the second data on the z-axis is determined as the angle deviation of the first data and the second data on the z-axis, wherein the angle of the line of sight direction on the z-axis can be the angle between the z-axis and the line of sight direction in space.

[0089] Exemplarily, the first data includes the sight direction V calculated using the optical recording method 1 =( ∠ X1 ,∠Y 1，∠ Z 1 ),in ∠ X 1 ,∠Y 1，∠ Z 1 is the angle between the line of sight and the X, Y, and Z axes; at the same time, the second data includes the line of sight direction V calculated by the micro-electromechanical system 1’ =( ∠ X 1’ ,∠Y 1’ ,∠Z 1’ ); the data set includes multiple sets of corresponding data pairs V 1 ( ∠ X 1 ,∠Y 1，∠ Z 1 ) and V 1’ ( ∠ X 1’ ,∠Y 1’ ,∠Z 1’ ), V 2 ( ∠ X 2 ,∠Y 2，∠ Z 2 ) and V 2’ ( ∠ X 2’ ,∠Y 2’ ,∠Z 2’ ), V 3 ( ∠ X 3 ,∠Y 3，∠ Z 3 ) and V 3’ ( ∠ X 3’ ,∠Y 3’ ,∠Z 3’ )……. Calculate the difference θ between the line of sight direction and the XYZ axis in each data pair x1 ( ∠ X 1 -∠X 1’ ), θ Y1 ( ∠ Y 1 -∠Y 1’ ) ， θ Z1 ( ∠ Z 1 -∠Z 1’ ), θ x2 ( ∠ X 2 -∠X 2’ ), θ Y2 ( ∠ Y 2-∠Y 2’ ) ， θ Z2 ( ∠ Z 2 -∠Z 2’ )……. A time threshold can be set in advance, for example, T1 seconds. Within T1 seconds, N θ Xi， θ Yi， θ Zi .

[0090] in, ∠ X 1 -∠X 1’ The data deviation on the x-axis can be centered for the first set of data. ∠ Y 1 -∠Y 1’ The data deviation on the y-axis can be centered for the first set of data. ∠ Z 1 -∠Z 1’ The data deviation on the z-axis can be centered for the first set of data. ∠ X 2 -∠X 2’ The data deviation on the x-axis can be centered for the second set of data. ∠ Y 2 -∠Y 2’ The data deviation on the y-axis can be centered for the second set of data. ∠ Z 2 -∠Z 2’ The data deviation on the z-axis can be centered for the second set of data.

[0091] S230a. For the data deviations on each coordinate axis, determine the deviation average of each data deviation.

[0092] The deviation average value can be considered as the average value of the data deviation. The deviation average value of the data deviation can refer to calculating the deviation average value on each coordinate axis for the data deviation on each coordinate axis. Each coordinate axis determines its own deviation average value.

[0093] In this step, for each coordinate axis, the mean value of the data deviation under the coordinate axis can be determined as the deviation average value of the coordinate axis. The deviation average value of the x-axis can be an average value determined based on the data deviation of the x-axis. The deviation average value of the y-axis can be an average value determined based on the data deviation of the y-axis.

[0094] For example, taking coordinate deviation as an example, the average deviation of N θ is calculated:

[0095]

[0096] Here, X' may be the average deviation value of the x-axis, and Y' may be the average deviation value of the y-axis.

[0097] The method for calculating the angle deviation of the line of sight is the same as the method for calculating the coordinate deviation, which will not be described here.

[0098] S240a, determining the average value of the deviation corresponding to each coordinate axis as a candidate correction coefficient.

[0099] The candidate correction coefficient can be understood as a candidate correction coefficient. In this embodiment, the candidate correction coefficient of the coordinate deviation can be θ N =(X',Y'). Similarly, the candidate correction coefficient for the angle deviation of the line of sight direction can be θ N =(X',Y',Z').

[0100] S250a, determine whether the candidate correction coefficient meets the correction condition, if so, execute S260a.

[0101] The correction condition can be understood as a preset condition for determining whether the correction coefficient can accurately correct the gaze information. If the candidate correction coefficient meets the correction condition, the determined candidate correction coefficient is determined as the correction coefficient, and the initial gaze information is corrected by the correction coefficient to determine the corrected target gaze information.

[0102] This step does not limit the correction conditions. For example, the gaze information collected by the MEMS can be corrected by the candidate correction coefficient, and then the accuracy of the corrected gaze information by the MEMS is verified based on the gaze information determined by the optical recording method. If it is accurate, it can be considered that the correction conditions are met, otherwise it is considered that the candidate correction coefficient does not meet the correction conditions.

[0103] S260a. Determine a correction coefficient based on the determined candidate correction coefficient.

[0104] When the candidate correction coefficient meets the correction condition, this step may determine the candidate correction coefficient as the correction coefficient, or may determine the candidate correction coefficient as the correction coefficient after processing the candidate correction coefficient. The means of processing the candidate correction coefficient is not limited here.

[0105] In this embodiment, the data deviation of each data pair in the data set on each coordinate axis is determined; the deviation average of each data deviation is calculated; the deviation average corresponding to each coordinate axis is determined as a candidate correction coefficient; it is determined whether the candidate correction coefficient meets the correction condition; if the candidate correction coefficient meets the correction condition, the determined candidate correction coefficient is determined as the correction coefficient, the initial gaze information is corrected by the correction coefficient, and the corrected target gaze information is determined, thereby ensuring the accuracy of the target object used to correct the gaze information.

[0106] Optionally, determining whether the candidate correction coefficient satisfies a correction condition includes:

[0107] Correcting second data at a subsequent time based on the candidate correction coefficient to obtain corrected target data, wherein the second data at the subsequent time is gaze information determined by the micro-electromechanical system after determining the candidate correction coefficient;

[0108] Determine a data difference between the target data and first data at a subsequent moment, where the first data at the subsequent moment is data acquired simultaneously with the second data at the subsequent moment;

[0109] If the data difference is less than the set threshold, the candidate correction coefficient meets the correction condition; otherwise, return to continue to obtain the next data set, determine the candidate correction coefficient of the next data set, and judge whether the candidate correction coefficient of the next data set meets the correction condition;

[0110] When determining the candidate correction coefficient corresponding to the next data set, the data deviation corresponding to the next data set is determined based on the corrected second data in the next data set and the corresponding first data, and the corrected second data is the data obtained by correcting the second data in the next data set based on the previous candidate correction coefficient.

[0111] Among them, the second data at a subsequent moment can be the gaze information determined again by the micro-electromechanical system after acquiring the data set and determining the candidate correction coefficient; if the moment of acquiring the data set is moment T, then the subsequent moment can be moment T+1, and the gaze information, such as the gaze point coordinates and / or the line of sight direction, is determined by the micro-electromechanical system. In one embodiment, if the data set acquires the coordinates of N gaze points within a T time period, then the second data at a subsequent moment is the first gaze point coordinate within the subsequent T+1 moment determined by the micro-electromechanical system. Considering the continuity of time, the first gaze point coordinate within the subsequent T+1 moment can also be understood as the N+1th gaze point coordinate determined by the micro-electromechanical system. If the data set acquires N line of sight directions within a T time period, then the second data at a subsequent moment is the angle between the first line of sight direction at the subsequent T+1 moment determined by the micro-electromechanical system and the coordinate axis, which can also be understood as the angle between the N+1th line of sight direction determined by the micro-electromechanical system and the coordinate axis. The second data at a subsequent moment can be recorded as M M’ (X M’ ,Y M’ );

[0112] The target data may be the corrected second data at a subsequent time.

[0113] In this embodiment, for each coordinate axis, the sum of the corresponding candidate correction coefficient and the first data at the corresponding subsequent moment can be determined as the corrected target data.

[0114] For example, the target data may be M'(X M’ +X',Y M’ +Y'), where the x-axis coordinate of the target data is X M’ +X', that is, the candidate correction coefficient X' of the x-axis and the first data X of the subsequent time of the x-axis M’ The y-axis coordinate of the target data is Y M’ +Y', that is, the candidate correction coefficient Y' of the y-axis and the first data Y at the subsequent time of the x-axis M’ The sum of .

[0115] The first data at the subsequent moment may be data obtained simultaneously with the second data at the subsequent moment by the optical recording method, and may be understood as gaze information determined by the optical recording method at the T+1 moment at the same time, and the gaze information includes the sight direction and / or the gaze point coordinates. For example, if the data set obtains N gaze point coordinates within the T time period, then the first data at the subsequent moment is the first gaze point coordinate within the subsequent T+1 moment determined by the optical recording method, and may also be understood as the N+1th gaze point coordinate determined by the optical recording method. If the data set obtains N sight directions within the T time period, then the first data at the subsequent moment is the angle between the first sight direction at the subsequent T+1 moment determined by the optical recording method and the coordinate axis, and may also be understood as the angle between the N+1th sight direction determined by the optical recording method and the coordinate axis, which may be recorded as M. M (X M ,Y M ).

[0116] In this embodiment, the first data at the subsequent moment can be used to verify the accuracy of the target data. Therefore, in this embodiment, the data difference between the target data and the first data at the subsequent moment is determined. The data difference can be the difference in coordinates under the same coordinate axis, which is the same as the means of determining the data deviation mentioned above. The two are determined at different stages. For example, the data difference can be the difference in coordinates or the angle difference in the line of sight between the second data at the subsequent moment after correction and the first data at the subsequent moment under the same coordinate axis in the verification stage; and the data deviation can be the difference in coordinates or the angle difference in the line of sight between the first data and the second data in the data pair under the same coordinate axis in the candidate correction coefficient determination stage. For example, the difference between the gaze point coordinates of the x-axis of the target data and the gaze point coordinates of the x-axis of the first data at the subsequent moment is used as the data difference of the x-axis. The Y-axis is similar and will not be described here. The data difference may also be the difference in sight direction angles, such as the difference between the sight direction angle of the target data x-axis and the sight direction angle of the first data x-axis at a subsequent moment, as the angle difference of the x-axis, where the sight direction angle of the x-axis may be the angle between the x-axis and the sight direction in the spatial relationship. The same is true for the Y-axis and will not be described here.

[0117] In one embodiment, the second data at the subsequent moment is corrected by the candidate correction coefficient to obtain the corrected target data, and the data difference between the target data and the first data at the subsequent moment is determined, that is, M'(X M’ +X',Y M’ +Y') and M M (X M ,Y M ) to achieve the comparison between the target data and the first data at the subsequent moment.

[0118] The set threshold value may be a pre-set data difference threshold value, which may be denoted as K. The threshold value may be set by the user according to actual conditions, and this embodiment does not limit this. When the data difference value is less than the set threshold value, that is, M'(X M’ +X',Y M’ +Y') and M M (X M ,Y M ) is less than K, the target data can be considered accurate, and the candidate correction coefficient can be further determined to meet the correction condition. At this time, the optical device can be turned off, and the micro-electromechanical system can be used to determine the initial gaze information, and the target gaze information is used as the output of the final gaze information.

[0119] If the data difference is not less than the set threshold, it can be said that the accuracy of the target data does not meet the conditions and the correction is not accurate enough. The correction can be continued, that is, the step of determining the candidate correction coefficient is repeated to determine the candidate correction coefficient corresponding to the next data set. The next data set can be the data set of the next time threshold, that is, the data set composed of the data acquired within the next T1 second.

[0120] In this embodiment, when determining the data deviation of the next data set, the second data used is the corrected data, such as the data corrected based on the last candidate correction coefficient corresponding to the next data set.

[0121] In one embodiment, when the candidate correction coefficient determined by the current data set does not meet the correction condition, the next data set can be acquired. When the candidate correction coefficient is determined by the next data set, the second data of the microelectromechanical system needs to be corrected by the candidate correction coefficient of the current data set before the candidate correction coefficient of the next data set is determined, and so on, until there is a candidate correction coefficient that meets the correction condition, and the candidate correction coefficient that meets the condition is determined as the correction coefficient.

[0122] For example, a data set is acquired within T1s, A and A', B and B', C and C', etc. The candidate correction coefficient is determined as θ N=(X', Y'). When verifying the candidate correction coefficient, the target data after T1s and the first data at the subsequent time are obtained. The data difference between the target data after the candidate correction coefficient is corrected and the first data at the subsequent time is greater than the set threshold. Then continue to obtain the next data set, such as continuing to obtain the next data set within the subsequent T1s, E and E', F and F', G and G'... The candidate correction coefficient of the next data set is determined. When determining the candidate correction coefficient, the second data is first subjected to the candidate correction coefficient as θ N = (X', Y') correction, get the corrected second data, and then combine the first data to determine the candidate correction coefficient of the next data set. For example, first add the candidate correction coefficient to the coordinates in E', F' and G' to get θ N =(X',Y'), and then subtract the gaze point coordinates and / or the angle between the sight line and the coordinate axis corresponding to E, F and G to determine the candidate correction coefficient of the next data set. If the candidate correction coefficient of the next data set meets the correction condition, the candidate correction coefficient of the next data set is used as the correction coefficient.

[0123] The gaze information determination method of the present invention may be determined during the use of the electronic device, and the data set may be the gaze point coordinates and / or sight line directions at different times collected recursively over time.

[0124] In this embodiment, the second data at the subsequent moment is corrected by the candidate correction coefficient to obtain the corrected target data, and the data difference between the target data and the first data at the subsequent moment is determined. Based on the first data at the subsequent moment determined by the more accurate optical recording method, it is verified whether the correction condition is met. After determining the data difference between the target data and the first data at the subsequent moment, the size of the data difference and the set threshold is judged. If the data difference is less than the set threshold, the candidate correction coefficient meets the correction condition, and the corrected target gaze information is determined with the corrected target data, thereby ensuring the accuracy of the target object used to correct the gaze information; if not, return to continue to obtain the data set within the next time threshold, determine the candidate correction coefficient of the next data set, and again judge whether the candidate correction coefficient of the next data set meets the correction condition; and so on, until the data difference is less than the set threshold, the determined candidate correction coefficient is determined as the correction coefficient, which is used to correct the initial gaze information. This embodiment improves the accuracy of judging whether the correction is completed by continuously determining whether the candidate correction coefficient meets the correction condition.

[0125] Embodiment 3

[0126] Figure 2B This is a flow chart of a method for determining gaze information provided by Embodiment 3 of the present invention. The technical solution of this embodiment is further refined on the basis of the above embodiment. Figure 2BAs shown, if the first data includes first process data in a process of determining gaze information by an optical recording method, the second data includes second process data in a process of determining gaze information by the micro-electromechanical system, and the target object includes an eye model, the target object is determined according to the data set; the method of this embodiment may include the following steps:

[0127] S210b. Acquire a data set, wherein the data set includes multiple groups of data pairs, each group of data pairs includes first data and second data acquired by different devices, the first data includes first process data in a process of determining gaze information by an optical recording method, and the second data includes second process data in a process of determining gaze information by the micro-electromechanical system.

[0128] The process data may be data in the process of determining the gaze information, for example, a measurement value in the process of determining the gaze information and calculation data in the process of determining the gaze information. The first process data in the process of determining the gaze information by the optical recording method may be the first data, and the second process data in the process of determining the gaze information by the micro-electromechanical system may be the second data.

[0129] The specific content of the first process data is not limited, and includes but is not limited to the following: measurement values ​​of the eye in the actual space, such as corneal curvature center data, spatial pupil position, eye spot, eyeball position, eyeball orientation, eyeball refractive index, corneal curvature radius, incident light and reflected light. The calculation data in the first process data can be an intermediate calculation result in the process of calculating the measurement value of the eye, such as spatial vector-related data, and the spatial vector-related data includes the position information of the spatial point after the eyeball modeling is performed.

[0130] The second process data includes but is not limited to: data associated with spatial information, such as eyeball parameters, data associated with the micro-electromechanical system, and intermediate calculation amounts in the process of the micro-electromechanical system measuring eyeball parameters.

[0131] Different data may correspond to different acquisition methods, which are not limited here. This step may acquire the first data transmitted by the optical device and acquire the second data transmitted by the micro-electromechanical system.

[0132] S220b. Correct the eye model according to the first process data and the second process data.

[0133] The eye model can be a model abstracted from eyeball parameters; the eye model is corrected according to the first process data and the second process data, which can be understood as correcting some parameters of the eye model according to the first process data in the process of determining the gaze information through the optical recording method and the second process data in the process of determining the gaze information through the micro-electromechanical system. Some parameters may include refractive index, center of corneal curvature and pupil distance, two kappa angles, corneal curvature radius, etc., wherein the kappa angle is the angle between the pupil midline (optical axis) and the visual axis (the line connecting the gaze target and the fovea).

[0134] This step does not limit the correction means. For example, the degree of matching with the eye model is determined through the second process data, such as the degree of matching between the infrared emission point and the infrared receiving point in the second process data and the incident light and the reflected light in the eye model, so as to correct the eye model. During the correction process, the correction parameters are not limited, and the parameters in the model that affect the degree of matching can be corrected, such as the radius of corneal curvature.

[0135] Both the optical recording method and the micro-electromechanical system can determine the size, position and / or angle of the eye features in space. In this embodiment, the first process data and the second process data are mutually verified, and the eye model is adjusted based on the degree of matching between each and the eye model. The degree of matching between the adjusted eye model and the first process data of the optical recording method and the second process data of the micro-electromechanical system respectively meets the required requirements.

[0136] S230b, determine whether the residual of the corrected eye model is less than the set residual threshold, if so, execute S240b; if not, return to execute S210b.

[0137] Among them, the setting of the residual threshold can be understood as a threshold of the residual of the pre-set eye model, which can be set by the user according to the actual situation, and this embodiment does not limit this. If the residual of the corrected eye model is less than the set residual threshold, the corrected eye model is determined as the target object, and the accuracy of the eye model is improved by correcting the eye model, so that the corrected gaze information is more accurate; otherwise, it will return to continue to obtain the data set of the next time threshold, correct the eye model based on the process data in the next data set, and determine whether the corrected eye model corresponding to the next data set is less than the set residual threshold, and so on, repeat the above steps until the corrected eye model is less than the set residual threshold, and the accuracy of the eye model is improved by correcting the eye model, so that the corrected gaze information is more accurate.

[0138] The residual of the eye model can be represented by the residual of the eye parameters in the eye model, such as the residual of the spatial position of a structure in the eye, such as the residual of the spatial position of the pupil, the residual of the incident light and the reflected light.

[0139] S240b, determining the corrected eye model as the target object.

[0140] When the residual of the eye model is less than the set residual threshold, the eye model is considered accurate and the eye model is determined as the target object for subsequent micro-electromechanical system to determine the target gaze information based on the eye model.

[0141] In this embodiment, by abstracting the eyeball parameters into an eye model, some parameters of the eye model are corrected according to the first process data in the process of determining the gaze information by the optical recording method and the second process data in the process of determining the gaze information by the micro-electromechanical system, to obtain a corrected eye model, and the corrected eye model is compared with a set residual threshold. If the residual of the corrected eye model is less than the set residual threshold, the corrected eye model is determined as the target object, and the accuracy of the eye model is improved by correcting the eye model, so that the corrected gaze information is more accurate; otherwise, it returns to continue to obtain the data set of the next time threshold, corrects the eye model based on the process data in the next data set, and determines whether the corrected eye model corresponding to the next data set is less than the set residual threshold, and so on, until the corrected eye model is less than the set residual threshold, and then the accuracy of the eye model is improved by correcting the eye model, so that the corrected gaze information is more accurate.

[0142] Optionally, the first process data includes eye data measured and data calculated during the measurement process when gaze information is determined by an optical recording method, and the second process data includes eye data measured and data calculated during the measurement process when gaze information is determined by the micro-electromechanical system.

[0143] The eye data may be a measurement value in the process of determining the gaze information, and the eye data includes eyeball parameters, such as pupil data, corneal data, iris data, etc., such as pupil contour, pupil size, pupil diameter data and / or corneal diameter, etc.; the calculation data may be an intermediate calculation amount in the process of determining the gaze information; the eye data in the first process data may be determined according to the actual measurement method when the gaze information is determined by the optical recording method, and the calculation data in the first process data may be determined according to calculation when the gaze information is determined by the optical recording method. The eye data in the second process data may be determined according to the actual measurement method when the gaze information is determined by the micro-electromechanical system, and the calculation data in the second process data may be determined according to calculation when the gaze information is determined by the micro-electromechanical system.

[0144] In this embodiment, the eye model is corrected by using the eye data and the calculation data in the first process data and the second process data, thereby ensuring that the process of correcting the eye model is more streamlined.

[0145] Embodiment 4

[0146] Figure 3 FIG. 4 is a schematic diagram of a structure of a device for determining gaze information provided by Embodiment 4 of the present invention. Figure 3 As shown, the device comprises:

[0147] An acquisition module 310 is used to acquire a data set, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data is data related to the gaze information determined by an optical recording method, and the second data is data related to the gaze information determined by a micro-electromechanical system;

[0148] A first determination module 320, configured to determine a target object according to the data set;

[0149] The second determination module 330 is configured to correct the initial gaze information determined by the micro-electromechanical system based on the target object to determine corrected target gaze information.

[0150] Optionally, the first data includes gaze information calculated using an optical recording method, the second data includes gaze information calculated using a micro-electromechanical system, the first data and the second data are data determined at the same time, the gaze information includes a line of sight direction and / or gaze point coordinates, and the target object includes a correction coefficient;

[0151] The first determining module 320 includes:

[0152] A first determining unit, configured to determine, for each data pair in the data set, a data deviation of the data pair on each coordinate axis;

[0153] A second determining unit, configured to determine, for each data deviation on each coordinate axis, a deviation average of the data deviations;

[0154] A third determining unit is used to determine the average value of the deviations corresponding to each coordinate axis as a candidate correction coefficient;

[0155] The judgment unit is used to determine whether the candidate correction coefficient meets the correction condition; if so, determine the correction coefficient based on the determined candidate correction coefficient.

[0156] Optionally, the judging unit is specifically used for:

[0157] Correcting second data at a subsequent time based on the candidate correction coefficient to obtain corrected target data, wherein the second data at the subsequent time is gaze information determined by the micro-electromechanical system after determining the candidate correction coefficient;

[0158] Determine a data difference between the target data and first data at a subsequent moment, where the first data at the subsequent moment is data acquired simultaneously with the second data at the subsequent moment;

[0159] If the data difference is less than the set threshold, the candidate correction coefficient meets the correction condition; otherwise, return to continue to obtain the next data set, determine the candidate correction coefficient of the next data set, and judge whether the candidate correction coefficient of the next data set meets the correction condition;

[0160] When determining the candidate correction coefficient corresponding to the next data set, the data deviation corresponding to the next data set is determined based on the corrected second data in the next data set and the corresponding first data, and the corrected second data is the data obtained by correcting the second data in the next data set based on the previous candidate correction coefficient.

[0161] Optionally, the first data includes first process data in a process of determining gaze information by an optical recording method, the second data includes second process data in a process of determining gaze information by the micro-electromechanical system, the target object includes an eye model, and the first determination module 320 includes:

[0162] a correction unit, configured to correct the eye model according to the first process data and the second process data;

[0163] If the residual of the corrected eye model is less than the set residual threshold, the corrected eye model is determined as the target object; otherwise, return to continue to obtain the next data set, correct the eye model based on the next data set, and determine whether the corrected eye model corresponding to the next data set is less than the set residual threshold.

[0164] Optionally, the first process data includes eye data measured and data calculated during the measurement process when gaze information is determined by an optical recording method, and the second process data includes eye data measured and data calculated during the measurement process when gaze information is determined by the micro-electromechanical system.

[0165] Optionally, the second determining module 330 includes:

[0166] a fourth determining unit, configured to determine initial gaze information through the micro-electromechanical system;

[0167] The fifth determining unit is used to correct the initial gaze information based on the determined target object to obtain corrected target gaze information.

[0168] Optionally, in the process of determining the corrected target gaze information, the state of the optical device corresponding to the optical recording method is a closed state.

[0169] The gaze information determination device provided in the embodiment of the present invention can execute the gaze information determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] Embodiment 5

[0171] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0172] like Figure 4 As shown, the electronic device 10 includes an optical device (not output in the figure) and a micro-electromechanical system (not output in the figure) for determining gaze information using an optical recording method, at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the optical device and the micro-electromechanical system are connected to the processor 11 respectively; wherein the memory stores a computer program executable by the at least one processor 11, and the computer program is executed by the at least one processor 11 so that the at least one processor 11 can execute the method provided by the present invention; the micro-electromechanical system is used to transmit second data to the processor, and the optical device is used to transmit first data to the processor.

[0173] The processor 11 can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 12 or a computer program loaded from a storage unit 18 into a random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0174] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0175] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the gaze information determination method.

[0176] In some embodiments, the gaze information determination method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the gaze information determination method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the gaze information determination method in any other appropriate manner (e.g., by means of firmware).

[0177] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0179] In the context of the present invention, a computer-readable storage medium stores computer instructions, which are used to implement the gaze information determination method provided by the present invention when the processor executes. The computer-readable storage medium may be a tangible medium, which may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0180] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user may provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), blockchain network, and the Internet.

[0182] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0183] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0184] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining gaze information, characterized in that: include: Acquire a data set, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data is data related to gaze information determined by an optical recording method, and the second data is data related to gaze information determined by a micro-electromechanical system; Determine a target object according to the data set; Initial gaze information determined by the micro-electromechanical system is corrected based on the target object to determine corrected target gaze information.

2. The method according to claim 1, characterized in that: The first data includes gaze information calculated using an optical recording method, the second data includes gaze information calculated using a micro-electromechanical system, the first data and the second data are data determined at the same time, the gaze information includes a line of sight direction and / or gaze point coordinates, and the target object includes a correction coefficient; Determining the target object according to the data set includes: For each data pair in the data set, determining the data deviation of the data pair on each coordinate axis; For the data deviations on each coordinate axis, determining a deviation average of the data deviations; The average value of the deviations corresponding to each coordinate axis is determined as a candidate correction coefficient; determining whether the candidate correction coefficient satisfies a correction condition; If so, a correction factor is determined based on the determined candidate correction factor.

3. The method according to claim 2, characterized in that The determining whether the candidate correction coefficient satisfies a correction condition includes: Correcting second data at a subsequent time based on the candidate correction coefficient to obtain corrected target data, wherein the second data at the subsequent time is gaze information determined by the micro-electromechanical system after determining the candidate correction coefficient; Determine a data difference between the target data and first data at a subsequent moment, where the first data at the subsequent moment is data acquired simultaneously with the second data at the subsequent moment; If the data difference is less than the set threshold, the candidate correction coefficient meets the correction condition; otherwise, return to continue to obtain the next data set, determine the candidate correction coefficient of the next data set, and judge whether the candidate correction coefficient of the next data set meets the correction condition; When determining the candidate correction coefficient corresponding to the next data set, the data deviation corresponding to the next data set is determined based on the corrected second data in the next data set and the corresponding first data, and the corrected second data is the data obtained by correcting the second data in the next data set based on the previous candidate correction coefficient.

4. The method according to claim 1, characterized in that The first data includes first process data in a process of determining gaze information by an optical recording method, the second data includes second process data in a process of determining gaze information by the micro-electromechanical system, the target object includes an eye model, and determining the target object according to the data set includes: modifying the eye model according to the first process data and the second process data; If the residual of the corrected eye model is less than the set residual threshold, the corrected eye model is determined as the target object; otherwise, return to continue to obtain the next data set, correct the eye model based on the next data set, and determine whether the corrected eye model corresponding to the next data set is less than the set residual threshold.

5. The method according to claim 4, characterized in that The first process data includes the eye data measured and the calculated data during the measurement when the gaze information is determined by the optical recording method, and the second process data includes the eye data measured and the calculated data during the measurement when the gaze information is determined by the micro-electromechanical system.

6. The method according to any one of claims 1 to 5, characterized in that: Correcting initial gaze information determined by the micro-electromechanical system based on the target object to determine corrected target gaze information includes: determining initial gaze information by the micro-electromechanical system; The initial gaze information is corrected based on the determined target object to obtain corrected target gaze information.

7. The method according to claim 6, characterized in that In the process of determining the corrected target gaze information, the state of the optical device corresponding to the optical recording method is a closed state.

8. A gaze information determination device, characterized in that: include: an acquisition module, configured to acquire a data set, wherein the data set includes a plurality of data pairs, each data pair includes first data and second data acquired by different devices, the first data is data related to gaze information determined by an optical recording method, and the second data is data related to gaze information determined by a micro-electromechanical system; A first determination module, used to determine a target object according to the data set; The second determination module is used to correct the initial gaze information determined by the micro-electromechanical system based on the target object to determine the corrected target gaze information.

9. An electronic device, characterized in that: The electronic device comprises: An optical device and a micro-electromechanical system for determining gaze information using an optical recording method, at least one processor; and a memory communicatively coupled to the at least one processor; The optical device and the micro-electromechanical system are respectively connected to the processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the gaze information determination method according to any one of claims 1 to 7; The micro-electromechanical system is used to transmit second data to the processor, and the optical device is used to transmit first data to the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gaze information determination method according to any one of claims 1 to 7 when executed.

Images