Gaze estimation method, device, readable storage medium, and electronic device

Abstract

The present invention provides a gaze estimation method, device, readable storage medium, and electronic device, the method including: acquiring eye data and determining state and position information of a plurality of gaze feature points based on the eye data; obtaining a graph model by treating each of the gaze feature points as a node and establishing relationships between the nodes; determining feature information of the graph model based on the state and position information of each of the gaze feature points and assigning the feature information to the graph model to obtain a graph representation corresponding to the eye data; inputting the graph representation into a graph machine learning model, performing gaze estimation using the graph machine learning model, and outputting gaze data. The present invention utilizes a pre-trained graph machine learning model to calculate gaze data based on the graph representation of gaze feature data. This method is highly robust, more accurate, and does not require a calibration process.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to a Chinese patent application filed with the China Patent Office on February 16, 2023, bearing application number 202310120571.8 and entitled "Gaze estimation method, device, readable storage medium and electronic device," the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of computer vision, and in particular to a gaze estimation method, device, readable storage medium and electronic device. [Background technology]

[0003] Gaze estimation technology is widely applied in fields such as man-machine interaction, virtual reality, augmented reality, medical analysis, etc. Gaze tracking technology is used to estimate the user's gaze direction, and gaze estimation is generally achieved by a gaze estimation device.

[0004] Conventional gaze estimation methods generally require a gaze calibration process before providing gaze estimation capabilities, which affects the user experience, and generally require the relative position and orientation between the gaze estimation device and the user's head to be fixed during use. However, it is difficult for users to maintain the relative position and orientation between the gaze estimation device and their head for a long period of time, making it difficult to provide accurate gaze estimation capabilities. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above situation, there is a need to provide a gaze estimation method, device, readable storage medium, and electronic device to address the problem of inaccurate gaze estimation in the prior art. [Means for solving the problem]

[0006] The present invention discloses a gaze estimation method, acquiring eye data and determining state and position information of a plurality of gaze feature points based on the eye data, the gaze feature points including eye movement information and usable for calculating gaze data; each of the gaze feature points is a node, and a relationship between the nodes is established to obtain a graph model; determining feature information of the graph model based on state and position information of each of the gaze feature points, and assigning the feature information to the graph model to obtain a graph representation corresponding to the eye data; inputting the graph representation into a graph machine learning model, performing gaze estimation using the graph machine learning model, and outputting gaze data, wherein the graph machine learning model is pre-trained with a sample set, the sample set including a plurality of graph representation samples and corresponding gaze data samples.

[0007] Further, in the above-described gaze estimation method, the eye data is an eye image collected by a camera or data collected by a sensor device, When the eye data is an eye image collected by a camera, the plurality of gaze feature points include at least two necessary feature points, or at least one necessary feature point and at least one non-essential feature point, the necessary feature points include a pupil center point, a pupil ellipse focus, a pupil contour point, an iris feature, and an iris edge contour point, and the non-essential feature points include a glare center point and an eyelid key point; When the eye data is data collected by a sensor device, the sensor device includes a plurality of photoelectric sensors having a sparse spatial distribution, and the plurality of gaze feature points are predetermined reference points of the photoelectric sensors.

[0008] Furthermore, in the above-described gaze estimation method, the eye data is an eye image collected by a camera, and the plurality of gaze feature points are a plurality of feature points determined by performing feature extraction on the eye image using a feature extraction network.

[0009] Furthermore, in the above-described gaze estimation method, the feature information includes a node feature and / or an edge feature, and the node feature is including the state and / or position of the gaze feature points corresponding to the nodes; The edge feature is It includes the distance and / or vector between the line of sight feature points corresponding to the two nodes connected to the edge.

[0010] Furthermore, in the above gaze estimation method, the step of establishing a relationship between nodes includes: The method includes connecting the nodes with edges according to a predetermined rule based on the distribution pattern of each of the nodes.

[0011] Furthermore, in the above-described gaze estimation method, the eye data is an eye image collected by a camera, and the plurality of gaze feature points include a pupil center point and a plurality of glare center points around the pupil center point, and the step of connecting nodes with edges according to a predetermined rule based on the distribution form of each of the nodes includes: This involves connecting the nodes corresponding to the pupil centers and the nodes corresponding to the glare centers with undirected edges.

[0012] Furthermore, in the above gaze estimation method, the eye data is an eye image collected by a camera, and the plurality of gaze feature points are feature points determined by performing feature extraction on the eye image using a feature extraction network, and the step of connecting nodes with edges according to a predetermined rule based on the distribution form of each of the nodes comprises: It involves connecting adjacent feature points with undirected edges.

[0013] Furthermore, in the above-described gaze estimation method, the eye data is data collected by a sensor device, the sensor device includes a plurality of photoelectric sensors having a sparse spatial distribution, the plurality of gaze feature points are preset reference points of the photoelectric sensors, and the step of connecting nodes with edges according to a preset rule based on the distribution pattern of each of the nodes includes: It involves connecting adjacent nodes with undirected edges.

[0014] Furthermore, in the above-mentioned gaze estimation method, the process of training the graph machine learning model includes: Collecting {eye data samples, gaze data samples} sample examples, the eye data samples including eye data samples collected by an eye data collection device at a plurality of positions relative to the user's head; extracting each gaze feature point in the eye data sample to obtain a gaze feature point sample; generating a graph representation sample based on the gaze feature point sample, and establishing an example {graph representation sample, gaze data sample} sample based on the graph representation sample and the corresponding gaze data sample; and training the graph machine learning model using the {graph representation sample, gaze data sample} sample examples, where the input of the graph machine learning model is the graph representation sample and the output is the gaze data.

[0015] Furthermore, in the above-described gaze estimation method, the posture of the eye data collection device relative to the user's head is The eye data collection device is worn on the head of the user; the eye data collection device moves upward by a preset distance or rotates upward by a preset angle relative to a state in which the eye data collection device is attached to the user's head; the eye data collection device moves downward by a preset distance or rotates downward by a preset angle relative to a state in which the eye data collection device is attached to the user's head; the eye data collection device moves to the left by a preset distance or rotates to the left by a preset angle relative to a state in which the eye data collection device is worn on the user's head; The eye data collection device may move to the right by a preset distance or rotate to the right by a preset angle relative to a state in which the eye data collection device is attached to the user's head.

[0016] The present invention further discloses a gaze estimation device, a data acquisition module used to acquire eye data and determine state and position information of a plurality of gaze feature points based on the eye data, the gaze feature points including eye movement information and being usable for calculating gaze data; a graph model establishment module, which is used to establish a graph model by taking each of the gaze feature points as a node and establishing a relationship between the nodes; a graph representation establishment module, which is used to determine feature information of the graph model based on state and position information of each of the gaze feature points, and to assign the feature information to the graph model to obtain a graph representation corresponding to the eye data; and a gaze estimation module used to input the graph representation into a graph machine learning model, perform gaze estimation using the graph machine learning model, and output gaze data, wherein the graph machine learning model is pre-trained with a sample set, the sample set including a plurality of graph representation samples and corresponding gaze data samples.

[0017] The present invention further discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, realizes the gaze estimation method according to any one of the above aspects.

[0018] The present invention further discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor realizes the gaze estimation method described in any one of the above claims when executing the computer program. [Effects of the Invention]

[0019] The present invention provides a gaze estimation method based on a graph representation, which determines the state and position of gaze feature points based on eye data, constructs a graph representation based on the gaze feature points and the state and position of the gaze feature points, and uses a pre-trained graph machine learning model to calculate gaze data based on the graph representation of the gaze feature data, which is highly robust, has higher accuracy, and does not require a calibration process. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a flowchart of a gaze estimation method according to a first embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram of a pupil center point and six glare centers in an eye image. [Figure 3] 10 is a graph representation of gaze features in Example 2. [Figure 4] 1 is a schematic diagram of a photoelectric sensor device with sparse spatial distribution; [Figure 5] 13 is a graph representation of gaze features in Example 3. [Figure 6] FIG. 10 is a structural schematic diagram of a line-of-sight estimation device according to a fourth embodiment of the present invention. [Figure 7] 1 is a structural schematic diagram of an electronic device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, the embodiments of the present invention will be described in detail, and examples of the embodiments are shown in the drawings, in which the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are merely illustrative and are intended to explain the present invention, and should not be understood as limitations on the present invention.

[0022] These and other aspects of embodiments of the present invention will become apparent upon reference to the following description and drawings. While the description and drawings specifically disclose certain specific embodiments of embodiments of the present invention and illustrate certain ways of carrying out the principles of embodiments of the present invention, it is to be understood that the scope of embodiments of the present invention is not limited thereto. On the contrary, embodiments of the present invention include all alterations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0023] Example 1 Referring to FIG. 1, the gaze estimation method according to the first embodiment of the present invention includes steps S11 to S14.

[0024] In step S11, eye data is acquired, and state and position information of a plurality of gaze feature points is determined based on the eye data, and the gaze feature points include eye movement information and can be used to calculate gaze data.

[0025] The eye data is an image of the human eye collected by a camera, and may be, for example, a single image taken by one camera, multiple images (sequential images) taken by a single camera, multiple images of the same object taken by multiple cameras, or the position and readings of a photoelectric sensor with a sparse spatial distribution. The camera in this embodiment refers to any device that can capture and record an image, and its components generally include an imaging element, a darkroom, an imaging medium, and an imaging control structure, and the imaging medium is a CCD or CMOS. The photoelectric sensor with a sparse spatial distribution means that the photoelectric sensor has a sparse spatial distribution.

[0026] The eye data can determine a plurality of gaze feature points and state and position information of each feature point. If the eye data is an eye image collected by a camera, the plurality of gaze feature points include at least two required feature points, or at least one required feature point and at least one non-required feature point, where the required feature points include a pupil center point, a pupil ellipse focus point, a pupil contour point, an iris on-feature point, and an iris edge contour point, and the non-required feature points include a glare center point and an eyelid keypoint. If the eye data is eye data collected by a sensor device (where the sensor device includes a plurality of photoelectric sensors with sparse spatial distribution), the plurality of gaze feature points are predetermined reference points of the photoelectric sensors.

[0027] Furthermore, in another embodiment of the present invention, when the eye data is an eye image collected by a camera, the gaze feature points may be determined by performing feature extraction on the eye image using a feature extraction network. The feature extraction network HS-ResNet first generates a feature graph using conventional convolution, and the gaze feature points are feature points in the feature graph. The feature points in the feature graph may be the necessary and unnecessary feature points mentioned above, or may be points other than the necessary and unnecessary feature points.

[0028] The state of a gaze feature point refers to the existence state of this gaze feature point, such as whether it exists in an image, whether it has been successfully extracted by a feature extraction module, or the photoelectric sensor reading corresponding to the gaze feature point. The position of a gaze feature point refers to the two-dimensional coordinate of the gaze feature point in the image coordinate system or the three-dimensional coordinate of the gaze feature point in a physical coordinate system (e.g., any one camera coordinate system or any one photoelectric sensor coordinate system).

[0029] A set of gaze feature points is formed by a set of gaze feature points. For an image captured by a single camera, the data format of the set of gaze feature points is {[x0, y0], [x1, y1], ..., [x m , y m ]}, where [x m , y m ] is the coordinate of the gaze feature point numbered m in the image coordinate system.

[0030] For multiple images (sequence images) of the same object taken by the same camera or multiple images of the same object taken simultaneously by multiple cameras, the data format of the gaze feature point set is { [x 00 , y 00 ], [x 01 , y 01 ], ..., [x 0n , y 0n ]}, {[x 10 , y 10 ], [x 11 , y 11 ], ..., [x 1n , y 1n ]}, ..., {[x m0 , y m0 ], [x m1 , y m1 ], ..., [x mn , y mn ]}, or {[x 00 , y 00 ], [x 10 , y 10 ], ..., [x m0 , y m0 ]}, {[x 01 , y 01], [x 11 , y 11 ], ..., [x m1 , y m1 ]}, ..., {[x 0n , y 0n ], [x 1n , y 1n ], ..., [x mn , y mn ]}, where m is the feature point number, n is the image number, and [x mn , y mn ] represents the two-dimensional coordinates of the gaze feature point numbered m in the image coordinate system numbered n.

[0031] For multiple images (sequence images) of the same object taken by the same camera or multiple images of the same object taken simultaneously by multiple cameras, the data format of the gaze feature point set is {[x0, y0, z0], [x1, y1, z1], ..., [x n , y n , z n ]}, where [x n , y n , z n ] are the three-dimensional coordinates of the feature point numbered n in the physical coordinate system (for example, any one of the camera coordinate systems).

[0032] As can be understood, the two-dimensional coordinates of the gaze feature points in the image coordinate system in one or more images can be obtained by conventional image processing or a neural network model based on deep learning, and the three-dimensional coordinates of the gaze feature points can be calculated and obtained based on the two-dimensional coordinates in the multiple images by conventional multi-view geometric calculation or a neural network model based on deep learning, or can be directly calculated and obtained based on a single figure or multiple figures based on a neural network model based on deep learning.

[0033] If the eye data is collected by a photoelectric sensor device, the data format of the gaze feature point set is {[x0, y0, z0, s0], [x1, y1, z1, s1], ..., [x n , y n , z n , s n ]}, where [x n , y n , z n , s n ] represents the position and reading of a photoelectric sensor numbered n.

[0034] In step S12, each of the gaze feature points is taken as a node, and the relationship between the nodes is established to obtain a graph model.

[0035] In discrete mathematics, graphs are used to represent structures in which some relationship exists between objects. After mathematical abstraction, "objects" are called nodes or vertices, and the correlations between nodes are called edges. When drawing a graph, nodes are generally represented by a set of points or small circles, and edges in the graph are either straight or curved, and the edges of the graph may be directional or undirectional. Each gaze feature point is treated as a node, and relationships between the nodes are established to obtain a graph model. Here, when establishing relationships between nodes, edges may be connected between nodes according to preset rules based on the distribution pattern of each node.

[0036] In step S13, feature information of the graph model is determined based on the state and position information of each of the gaze feature points, and the feature information is assigned to the graph model to obtain a graph representation corresponding to the eye data.

[0037] The feature information includes node features and / or edge features, and the node features include the state and / or position of the gaze feature points corresponding to the nodes; The edge features include the distance and / or vector between the line-of-sight feature points corresponding to the two nodes connected to the edge.

[0038] In step S14, the graph representation is input into a graph machine learning model, gaze estimation is performed by the graph machine learning model, and gaze data is output, where the graph machine learning model is pre-trained with a sample set, and the sample set includes a plurality of graph representation samples and corresponding gaze data samples.

[0039] The graph machine learning model is trained in advance on a sample set, which includes a plurality of graph representation samples and corresponding gaze data samples. The training steps of the graph machine learning model are as follows: a) {eye data sample, gaze data sample} Collect sample examples, where the eye data sample is image data or a photoelectric sensor position and reading. The eye data sample includes eye data samples collected by an eye data collection device at multiple poses relative to the user's head. The eye data sample is an example (a description of the corresponding information recorded by a camera or photoelectric sensor), and the gaze data is a mark (related to gaze result information corresponding to the example).

[0040] Here, the posture of the eye data collection device relative to the head of the user is The eye data collection device is worn on the head of the user; the eye data collection device moves upward by a preset distance or rotates upward by a preset angle relative to a state in which the eye data collection device is attached to the user's head; the eye data collection device moves downward by a preset distance or rotates downward by a preset angle relative to a state in which the eye data collection device is attached to the user's head; the eye data collection device moves to the left by a preset distance or rotates to the left by a preset angle relative to a state in which the eye data collection device is worn on the user's head; The eye data collection device may move to the right by a preset distance or rotate to the right by a preset angle relative to a state in which the eye data collection device is attached to the user's head.

[0041] b) Generate {gaze feature point set sample, gaze data sample} sample instances: Based on {eye data sample, gaze data sample} sample instances, determine gaze feature points based on the eye data to obtain a gaze feature point set, and generate corresponding gaze data samples and {gaze feature point set sample, gaze data sample} sample instances.

[0042] c) Create a {graph representation sample, gaze data sample} sample example: Based on the {gaze feature point set sample, gaze data sample}, obtain a graph representation sample corresponding to the gaze feature point set sample based on the gaze feature point set sample and steps S12 and S13, and compose the graph representation sample and the corresponding gaze data sample into a {graph representation sample, gaze data sample} sample example.

[0043] d) Determine the graph machine learning model structure. The model input is a graph representation, and the model output is gaze data. The model structure consists of a multilayer graph neural network and a fully connected network.

[0044] e) Forward propagation calculation. From the {graph representation sample, gaze data sample} sample, take one lot of data and obtain a graph representation sample A and gaze data mark D. Input the graph representation sample A into the graph machine learning model, and first obtain a graph representation B through a multi-layer graph neural network, and then obtain the model output gaze data C through a fully connected network.

[0045] f) Forward propagation: A loss calculation is performed on the calculation result gaze data C and gaze data mark D to obtain a loss value L. Here, the loss function may be MAE or MSE.

[0046] g) Based on the loss value L, the graph machine learning model parameters are updated using gradient descent.

[0047] h) Repeat steps e to g to iteratively update the graph machine learning model parameters so that the loss value L decreases. If a preset training condition is met, the training is terminated. The preset condition includes, but is not limited to, that the loss value L converges, that the number of training iterations reaches a preset number, and that the training time length reaches a preset time length.

[0048] After the graph machine learning model is trained, the trained graph machine learning model can be used to perform gaze estimation on the graph representation obtained based on the current eye data.

[0049] The gaze estimation method in this embodiment can perform gaze estimation by combining data of multiple types of gaze features, which is highly robust and accurate. This method does not require a calibration process; the distribution of the user's eye data is included in the dataset for training the graph machine learning model. After training of the graph machine learning model is complete, the user can use the gaze estimation function without calibration. Furthermore, the dataset for training the gaze estimation model further includes eye and gaze data collected at different relative positions and orientations between the gaze estimation device and the user's head. Therefore, this method is not sensitive to changes in the relative position and orientation between the gaze estimation device and the user's head, making operation more flexible and convenient for the user and achieving accurate gaze estimation.

[0050] Example 2 In this embodiment, the gaze estimation method of the present invention will be described by taking as an example an example in which eye data is image data captured by a camera, and includes the following steps S21 to S24.

[0051] In S21, eye data is acquired by a camera to obtain an eye image, and then gaze feature points are extracted from the image to obtain a gaze feature point set {[x0, y0], [x1, y1], ..., [x6, y6]}, where [x m , y m ] is the coordinate of the gaze feature point number m in the image coordinate system. In this embodiment, the pupil center point and six glare center points are selected as gaze feature points, each numbered 0-6, as shown in Figure 2.

[0052] In S22, each gaze feature point is treated as a node, and the relationships between the nodes are established to obtain a graph model, as shown in Figure 3. The node corresponding to the pupil center point and the node corresponding to each glare center point are connected by undirected edges.

[0053] In step S23, feature information of the graph model is determined based on the state and position information of the pupil center point and the glare center point, and the feature information is assigned to the graph model to obtain a graph representation corresponding to the eye data. The feature information is normalized coordinates of the pupil center point and the glare center point in the image coordinate system.

[0054] In step S24, the graph representation is input to a graph machine learning model, gaze estimation is performed by the graph machine learning model, and gaze data is output. The graph machine learning model is pre-trained with a sample set, which includes a plurality of graph representation samples and corresponding gaze data samples. The training steps of the graph machine learning model are as follows:

[0055] a) {eye data sample, gaze data sample} sample examples are collected, and the eye data sample is image data. The eye data is an example (a description of the corresponding information recorded by the camera), and the gaze data is a mark (related to the gaze result information corresponding to the example). The user wears the gaze estimation device multiple times to collect {eye data sample, gaze data sample} sample examples in different wearing situations. The user wears the gaze estimation device correctly and collects data three times. The normally worn gaze estimation device is moved up a certain distance or rotated up a certain angle relative to the head and collected data twice. The normally worn gaze estimation device is moved down a certain distance or rotated down a certain angle relative to the head and collected data twice. The normally worn gaze estimation device is moved to the left a certain distance or a certain angle relative to the head and collected data once. The normally worn gaze estimation device is moved to the right a certain distance or a certain angle relative to the head and collected data once.

[0056] b) Generate {gaze feature point set sample, gaze data sample} sample examples: Based on the {eye data sample, gaze data sample} sample examples, determine gaze feature point set samples based on the eye data samples, and generate corresponding gaze data and {gaze feature point set sample, gaze data sample} sample examples.

[0057] c) Create a {graph representation sample, gaze data sample} sample example: According to {gaze feature point set sample, gaze data sample} and steps S22 and S23, obtain a graph representation sample corresponding to the gaze feature point set sample, and configure the graph representation sample and the corresponding gaze data sample into a {graph representation sample, gaze data sample} sample example.

[0058] d) Determine the graph machine learning model structure. The model input is a graph representation, and the model output is gaze data. The model structure consists of a multilayer graph neural network and a fully connected network.

[0059] e) Forward propagation calculation. From the {graph representation sample, gaze data sample} sample, take one lot of data and obtain a graph representation sample A and gaze data mark D. Input the graph representation sample A into the graph machine learning model, and first obtain a graph representation B through a multi-layer graph neural network, and then obtain the model output gaze data C through a fully connected network.

[0060] f) Forward propagation calculation result: Perform loss calculation on the gaze data C and gaze data mark D to obtain a loss value L. The loss function may be MAE (Mean Absolute Error) or MSE (Mean Squared Error). MAE calculation formula: JPEG2026507150000002.jpg47116where x i is the graph representation (model input), f is the graph machine learning model, and y i is the gaze data mark.

[0061] g) Based on the loss value L, the graph machine learning model parameters are updated using gradient descent.

[0062] h) Repeat step eg to iteratively update the graph machine learning model parameters so that the loss value L decreases. If a preset training condition is met, the training is terminated. The preset condition includes, but is not limited to, that the loss value L converges, that the number of training iterations reaches a preset number, and that the training time length reaches a preset time length.

[0063] Example 3 In this embodiment, the gaze estimation method of the present invention is described by taking the eye data collected by a photoelectric sensor with a discrete spatial distribution as an example, and the steps of the method are as follows:

[0064] In S31, eye data is acquired by a photoelectric sensor. A predetermined reference point of the photoelectric sensor is set as a gaze feature point, and a gaze feature point set {[x0, y0, z0, s0], [x1, y1, z1, s1], ..., [x6, y6, z6, s6]} is obtained, where [x n , y n , z n , s n ] represents the normalized coordinates in the physical coordinate system of the photoelectric sensor numbered n and the sensor reading. In this embodiment, the gaze feature points are numbered 0-6, respectively, as shown in FIG.

[0065] In S32, each gaze feature point is treated as a node, and the relationships between the nodes are established to obtain a graph model, as shown in Figure 5. The first to sixth nodes are each connected to the zeroth node by an edge, and the adjacent nodes between the first to sixth nodes are connected by an undirected edge.

[0066] In S33, characteristic information of the graph model is determined based on the state and position information of the photoelectric sensor, and the characteristic information is assigned to the graph model to obtain a graph representation corresponding to the eye data.

[0067] In step S34, the graph representation is input to a graph machine learning model, gaze estimation is performed by the graph machine learning model, and gaze is output. The graph machine learning model is pre-trained with a sample set, which includes a plurality of graph representation samples and corresponding gaze data samples. The training steps of the graph machine learning model are as follows: a) {eye data sample, gaze data sample} sample examples are collected, where the eye data is the position and reading of a photoelectric sensor. The eye data sample is an example (a description of the corresponding information recorded by the photoelectric sensor), and the gaze data is a mark (related to the gaze result information corresponding to the example). The user wears the gaze estimation device multiple times to collect {eye data sample, gaze data sample} sample examples under different user wearing conditions. The user normally wears the gaze estimation device and collects data three times. The normally worn gaze estimation device is moved up a certain distance or rotated up by a certain angle relative to the head and collected data twice. The normally worn gaze estimation device is moved down a certain distance or rotated down by a certain angle relative to the head and collected data twice. The normally worn gaze estimation device is moved to the left a certain distance or a certain angle relative to the head and collected data once. The normally worn gaze estimation device is moved to the right a certain distance or a certain angle relative to the head and collected data once.

[0068] b) Generate {gaze feature point set sample, gaze data sample} sample example: Based on the {eye data sample, gaze data sample} sample example, determine the gaze feature point set sample based on the eye data sample, and generate the corresponding gaze data sample and {gaze feature point set sample, gaze data sample} sample example.

[0069] c) Create a {graph representation sample, gaze data sample} sample example: According to {gaze feature point set sample, gaze data sample} and steps S32 and S33, obtain a graph representation sample corresponding to the gaze feature point set sample, and configure the graph representation sample and the corresponding gaze data sample into a {graph representation sample, gaze data sample} sample example.

[0070] d) Determine the graph machine learning model structure. The model input is a graph representation, and the model output is gaze data. The model structure consists of a multilayer graph neural network and a fully connected network.

[0071] e) Forward propagation calculation. From the {graph representation sample, gaze data sample} sample, take one lot of data and obtain a graph representation sample A and gaze data mark D. Input the graph representation sample A into the graph machine learning model, and first obtain a graph representation B through a multi-layer graph neural network, and then obtain the model output gaze data C through a fully connected network.

[0072] f) Forward propagation calculation result: Perform loss calculation on the gaze data C and gaze data mark D to obtain a loss value L. The loss function may be MAE (Mean Squared Error) or MSE (Mean Absolute Error). MAE calculation formula: JPEG2026507150000003.jpg47116where x i is the graph representation (model input), f is the graph machine learning model, and y i is the gaze data mark.

[0073] g) Based on the loss value L, the graph machine learning model parameters are updated using gradient descent.

[0074] h) Repeat step eg to iteratively update the graph machine learning model parameters so that the loss value L decreases. If a preset training condition is met, the training is terminated. The preset condition includes, but is not limited to, that the loss value L converges, that the number of training iterations reaches a preset number, and that the training time length reaches a preset time length.

[0075] Example 4 Referring to FIG. 6, a gaze estimation device according to a fourth embodiment of the present invention is shown. a data acquisition module 41, which is used to acquire eye data and determine state and position information of a plurality of gaze feature points based on the eye data, the gaze feature points including eye movement information and being usable for calculating gaze data; a graph model establishment module 42, which is used to establish a graph model by taking each of the gaze feature points as a node and establishing a relationship between the nodes; a graph representation establishment module 43, which is used to determine feature information of the graph model based on state and position information of each of the gaze feature points, and to assign the feature information to the graph model to obtain a graph representation corresponding to the eye data; and a gaze estimation module 44 used to input the graph representation into a graph machine learning model, perform gaze estimation by the graph machine learning model, and output gaze data, the graph machine learning model being pre-trained with a sample set, the sample set including a plurality of graph representation samples and corresponding gaze data samples.

[0076] The gaze estimation device according to the embodiment of the present invention has the same realization principle and resulting technical effects as the above-described method embodiment, and for the sake of simplicity, reference may be made to the corresponding content in the above-described method embodiment for parts not mentioned in the device embodiment.

[0077] Another aspect of the present invention further provides an electronic device. Referring to FIG. 7, an electronic device according to an embodiment of the present invention is shown, which includes a processor 10, a memory 20, and a computer program 30 stored in the memory and executable on the processor, and the processor 10 realizes the gaze estimation method as described above when executing the computer program 30.

[0078] Here, the electronic device may be, but is not limited to, a gaze estimation device, a wearable device, etc. In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, and is used to execute program code stored in the memory 20 or process data, etc.

[0079] Here, memory 20 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, or an optical disk. In some embodiments, memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, memory 20 may be an external storage device of the electronic device, such as an insertable hard disk, a smart memory card, a secure digital card, or a flash card. Furthermore, memory 20 may include both an internal storage unit and an external storage device of the electronic device. Memory 20 may be used not only to store application software and various data installed in the electronic device, but also to temporarily store data that has already been output or is to be output.

[0080] Optionally, the electronic device may further include a user interface, a network interface, a communication bus, etc. The user interface may include a display and an input unit (e.g., a keyboard), and the optional user interface may further include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (organic light-emitting diode) touch panel, etc. Here, the display may be appropriately referred to as a display screen or a display unit, and is used to display information processed in the electronic device and to display a visualized user interface. The network interface may optionally include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface), and is generally used to establish a communication connection between the device and other electronic devices. The communication bus is used to realize the connection communication between these components.

[0081] It should be noted that the structure shown in FIG. 7 does not constitute a limitation on the electronic device, and in other embodiments, the electronic device may include fewer or more components than those shown, or a combination of some components, or a different arrangement of components.

[0082] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, realizes the above-described gaze estimation method.

[0083] As will be appreciated by those skilled in the art, the logic and / or steps illustrated in flowcharts or otherwise described herein, e.g., ordered listings of executable instructions that may be used to implement logical functions, may be embodied in any computer-readable medium and used in or in conjunction with an instruction execution system, device (e.g., a computer-based system, a system including a processor, or other system capable of obtaining commands from a command execution system, device, and executing the commands). As used herein, a "computer-readable medium" may be any device that contains, stores, communicates, propagates, or transmits a program for use in conjunction with an instruction execution system, device, or device, or in conjunction with such an instruction execution system, device, or device.

[0084] More specific examples (a non-exhaustive list) of computer-readable media include an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable optical disk read-only memory (CD-ROM). Additionally, the computer-readable medium may be paper or other suitable medium on which the program may be printed, so that the program may be obtained electronically, for example by optically scanning the paper or other medium and subsequently edited, interpreted, or processed in any other suitable manner as needed, and then stored in computer memory.

[0085] It should be understood that each part of the present invention may be realized by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be realized by software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, when realized by hardware, as in the other embodiments, the hardware may be realized by any one or a combination of techniques known in the art, such as a discrete logic circuit having logic gate circuits for implementing logical functions of data signals, a dedicated integrated circuit having appropriate combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0086] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with this embodiment or example are included in at least one embodiment or example of the present invention. In the description of this specification, general expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples.

[0087] The above examples only show some embodiments of the present invention, and although the descriptions are more specific and detailed, they should not be understood as limiting the scope of the present invention patent. It should be noted that those skilled in the art can make further modifications and improvements without departing from the concept of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention patent shall be governed by the scope of the appended claims.

Claims

1. A gaze estimation method, acquiring eye data and determining state and position information of a plurality of gaze feature points based on the eye data, the gaze feature points including eye movement information and usable for calculating gaze data; each of the gaze feature points is a node, and a relationship between the nodes is established to obtain a graph model; determining feature information of the graph model based on state and position information of each of the gaze feature points, and assigning the feature information to the graph model to obtain a graph representation corresponding to the eye data; inputting the graph representation into a graph machine learning model, performing gaze estimation using the graph machine learning model, and outputting gaze data, wherein the graph machine learning model is pre-trained with a sample set, the sample set including a plurality of graph representation samples and corresponding gaze data samples.

2. the eye data is an eye image collected by a camera or data collected by a sensor device, When the eye data is an eye image collected by a camera, the plurality of gaze feature points include at least two necessary feature points, or at least one necessary feature point and at least one non-essential feature point, the necessary feature points include a pupil center point, a pupil ellipse focus, a pupil contour point, an iris feature, and an iris edge contour point, and the non-essential feature points include a glare center point and an eyelid key point; 2. The gaze estimation method according to claim 1, wherein, when the eye data is data collected by a sensor device, the sensor device includes a plurality of photoelectric sensors having a sparse spatial distribution, and the plurality of gaze feature points are preset reference points of the photoelectric sensors.

3. 2. The gaze estimation method according to claim 1, wherein the eye data is an eye image collected by a camera, and the plurality of gaze feature points are a plurality of feature points determined by performing feature extraction on the eye image using a feature extraction network.

4. The feature information includes node features and / or edge features, and the node features include: including the states and / or positions of gaze feature points corresponding to the nodes; The edge feature is The gaze estimation method according to claim 1 , further comprising: a distance and / or a vector between gaze feature points corresponding to two nodes connected to an edge.

5. Establishing relationships between nodes includes:

2. The gaze estimation method according to claim 1, further comprising connecting nodes with edges according to a predetermined rule based on the distribution pattern of each of the nodes.

6. The eye data is an eye image collected by a camera, and the plurality of gaze feature points include a pupil center point and a plurality of glare center points around the pupil center point, and the step of connecting nodes with edges according to a predetermined rule based on the distribution form of each of the nodes includes:

6. The gaze estimation method according to claim 5, further comprising connecting a node corresponding to a pupil center point and a node corresponding to a glare center point with an undirected edge.

7. The eye data is an eye image collected by a camera, and the plurality of gaze feature points are feature points determined by performing feature extraction on the eye image using a feature extraction network, and the step of connecting nodes with edges according to a predetermined rule based on the distribution form of each of the nodes includes: The gaze estimation method according to claim 5, further comprising connecting adjacent feature points with an undirected edge.

8. The eye data is data collected by a sensor device, the sensor device includes a plurality of photoelectric sensors having a sparse spatial distribution, the plurality of gaze feature points are preset reference points of the photoelectric sensors, and the step of connecting nodes with edges according to a preset rule based on the distribution pattern of each of the nodes includes: The gaze estimation method according to claim 5, further comprising connecting adjacent nodes with undirected edges.

9. The graph machine learning model training process includes: collecting sample examples {eye data samples, gaze data samples}, the eye data samples including eye data samples collected by an eye data collection device at a plurality of positions relative to the user's head; extracting each gaze feature point in the eye data sample to obtain a gaze feature point sample; generating a graph representation sample based on the gaze feature point sample, and establishing a {graph representation sample, gaze data sample} sample example based on the graph representation sample and the corresponding gaze data sample; and training the graph machine learning model using the {graph representation sample, gaze data sample} sample examples, wherein an input of the graph machine learning model is the graph representation sample and an output is the gaze data.

10. A gaze estimation device, a data acquisition module used to acquire eye data and determine state and position information of a plurality of gaze feature points based on the eye data, the gaze feature points including eye movement information and being usable for calculating gaze data; a graph model establishment module, which is used to establish a graph model by taking each of the gaze feature points as a node and establishing a relationship between the nodes; a graph representation establishment module, which is used to determine feature information of the graph model based on state and position information of each of the gaze feature points, and to assign the feature information to the graph model to obtain a graph representation corresponding to the eye data; and a gaze estimation module used to input the graph representation into a graph machine learning model, perform gaze estimation using the graph machine learning model, and output gaze data, wherein the graph machine learning model is trained in advance with a sample set, and the sample set includes a plurality of graph representation samples and corresponding gaze data samples.

11. 10. A computer-readable storage medium having a computer program stored therein, the computer-readable storage medium realizing the gaze estimation method according to any one of claims 1 to 9 when the program is executed by a processor.

12. 10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device realizing the gaze estimation method according to any one of claims 1 to 9 when the processor executes the computer program.

Images