Gaze point estimation method and device and computer readable storage medium
By combining the gaze point estimation method of user eye images and sensor signals, combined with multi-resolution feature extraction and gaze point tracking model, the accuracy and real-time problems of gaze point estimation in the prior art are solved, and the gaze point estimation is realized in complex environments Fast and accurate gaze positioning and tracking.
Patent Information
- Application Number
- CN202411167528.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
AI Technical Summary
The existing gaze point estimation technology has challenges in terms of accuracy and real-time. The 2D image-based method has high accuracy but poor real-time performance, while the 1D sensor signal-based method has strong real-time performance but low accuracy, making it difficult to achieve fast and accurate in complex environments. Positioning of the gaze point.
Combining user eye images and sensor signals, the multi-resolution feature extraction module, feature fusion module and gaze point prediction module are used for gaze point detection, and precise estimation is used for gaze point detection model, and fast through gaze point tracking model when needed. Positioning, real-time correction through sensor signals to eliminate cumulative errors.
It realizes accurate estimation and fast tracking of gaze points in different scenarios, improves the response speed and robustness of the algorithm, meets the requirements of accuracy and real-time, and is suitable for a variety of human-computer interaction applications.
Smart Images

Figure CN119025862A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to but are not limited to the field of human-computer interaction technology, and in particular to a gaze point estimation method, device, and computer-readable storage medium. Background Art
[0002] With the continuous advancement of science and technology, human-computer interaction technology has attracted more and more attention. Gaze point estimation technology is a key technology in the field of human-computer interaction, which aims to determine the specific area in the visual scene that the user is focusing on. Gaze point estimation technology tracks the user's gaze area and gains insight into the user's visual interests, thereby achieving efficient and natural interaction methods and high-quality user experience. The application field of gaze point estimation technology is very wide. For example, in virtual reality, gaze point estimation technology is used for display rendering, focusing computing resources on the part that the user is focusing on, thereby achieving a more realistic virtual scene; in human-computer interaction, gaze point estimation technology allows users to interact with the virtual environment by gazing, further enhancing the connection between users and devices; in addition, gaze point estimation technology also has potential applications in the fields of medicine, advertising, user research, etc., which can reveal users' interests and behaviors and provide a basis for decision-making in related fields. Summary of the invention
[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
[0004] The present disclosure provides a method for estimating a gaze point, including:
[0005] Acquire user eye images and sensor signals;
[0006] Determining a gaze point prediction method at a current moment according to a correspondence between a preset time and a gaze point prediction method, wherein the gaze point prediction method includes a first method for predicting according to the user eye image and a sensor signal and a second method for predicting according to the sensor signal;
[0007] When the determined gaze point prediction method is the first method, calling a gaze point detection model to estimate the gaze point position, the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module and a gaze point prediction module, the multi-resolution feature extraction module is configured to extract a feature map of the user eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map and the sensor signal at the current moment, and the gaze point prediction module is configured to perform gaze point prediction according to the fused features;
[0008] When the determined gaze point prediction method is the second method, a gaze point tracking model is called to estimate the gaze point position, and the gaze point tracking model is configured to predict the gaze point based on the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment.
[0009] An embodiment of the present disclosure further provides a gaze point estimation device, comprising: a processor and a memory storing a computer program that can be run on the processor, wherein the processor implements the steps of the gaze point estimation method as described above when executing the program.
[0010] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the gaze point estimation method as described in any one of the above items can be implemented.
[0011] The embodiments of the present disclosure further provide a computer program product, comprising instructions. When the computer program product is executed by a computer, the instructions execute the gaze point estimation method as described in any one of the above items.
[0012] The present disclosure also provides a gaze point estimation device, including: a camera, an optical sensor, a processor, and a memory, wherein:
[0013] The camera is configured to capture an eye image of the user;
[0014] The optical sensor is configured to collect brightness information of the user's eye area;
[0015] The memory is configured to store a gaze point detection model and a gaze point tracking model, wherein the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module and a gaze point prediction module, wherein the multi-resolution feature extraction module is configured to extract a feature map of the user's eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map and the sensor signal at the current moment, and the gaze point prediction module is configured to perform gaze point prediction according to the fused features; the gaze point tracking model is configured to perform gaze point prediction according to the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment;
[0016] The processor is configured to obtain a user eye image and a sensor signal, and determine a gaze point prediction method at a current moment according to a correspondence between a preset time and a gaze point prediction method, wherein the gaze point prediction method includes a first method for predicting according to the user eye image and the sensor signal and a second method for predicting according to the sensor signal; when the determined gaze point prediction method is the first method, a gaze point detection model is called to estimate the gaze point position; when the determined gaze point prediction method is the second method, a gaze point tracking model is called to estimate the gaze point position.
[0017] The gaze point estimation method, device and computer-readable storage medium of the embodiments of the present invention obtain user eye images and sensor signals, and use the user eye images and sensor signals to perform accurate gaze point estimation at some moments, while only using sensor signals for rapid positioning at other moments. This combines the real-time nature of the gaze point estimation method based on 1D sensor signals and the high precision of the gaze point estimation method based on 2D images, and can achieve fast and accurate gaze point tracking and positioning functions.
[0018] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. Other advantages of the present disclosure can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0020] Figure 1 The present invention is a flowchart of a method for estimating a gaze point according to an exemplary embodiment of the present invention.
[0021] Figure 2 The present invention is a flowchart of a gaze point estimation method according to an exemplary embodiment of the present invention.
[0022] Figure 3 A structural schematic diagram of a gaze point detection model according to an exemplary embodiment of the present disclosure.
[0023] Figure 4 A structural schematic diagram of a gaze tracking model according to an exemplary embodiment of the present disclosure.
[0024] Figure 5 The present invention is a flowchart of a gaze point estimation method that automatically adapts to different computing platforms according to an exemplary embodiment of the present invention.
[0025] Figure 6 and Figure 7Schematic diagram of the structures of two gaze point estimation devices according to exemplary embodiments of the present disclosure. DETAILED DESCRIPTION
[0026] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.
[0027] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in the present disclosure may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the attached claims.
[0028] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.
[0029] At present, gaze point estimation technology faces some challenges. For example, the visual behavior of users varies from person to person, and the location of the device worn and the lighting conditions may also affect the accuracy of the estimation results. In order to overcome these problems, researchers have proposed a variety of solutions. Depending on the input signal, the gaze point estimation method can be divided into a gaze point estimation method based on 2D images and a gaze point estimation method based on 1D sensor signals.
[0030] (1) Gaze point estimation method based on 2D images
[0031] The pupil-corneal reflection method based on the structure of the human eye is a traditional method for gaze point estimation. It estimates the gaze point through the reflection of light in the eyeball, especially the pupil and corneal reflection. The basic process of this method is divided into the registration process and the use process. When used for the first time, gaze point registration is required first. The user performs the registration operation between the human eye and the device to establish a mapping function f between the pupil position and the gaze point of the human eye. In daily use, this method is divided into two key steps. First, the position of the user's pupil is detected in the image. Then, the detected pupil position is mapped to the gaze point position on the screen using the previously established mapping function f.
[0032] This method has high positioning accuracy, making the estimation of the gaze point relatively accurate. However, this method also faces some specific problems. First, the precise design of the acquisition system is crucial, including the position of the camera and the position of the fill light to ensure that the pupil and corneal reflections of the eyeball can be accurately captured. Secondly, the stability of the worn device is an issue because the sliding or movement of the device may cause the accuracy of the algorithm's estimation to decrease. In general, the pupil-corneal reflection method based on the structure of the human eye is a method that can provide high-precision gaze point estimation under suitable conditions, but requires sophisticated system design and stable wearing to achieve the best results.
[0033] The deep learning method based on global features makes up for the problems of the traditional pupil-corneal reflection method to a certain extent with the help of the descriptive ability of the deep learning model. Specifically, the method includes two key parts: offline learning and online prediction. In the offline learning stage, developers collect eye images of different users and corresponding gaze point position marks to train the gaze point estimation model. Through the training of the deep learning model, the relationship between the eye image and the gaze point can be established, so that the model can predict the gaze point in daily use.
[0034] The advantage of this method is that it does not require complex alignment steps. Once the trained model is deployed on a mobile device, it can be directly used for gaze point prediction for any user. Compared with the traditional pupil-corneal reflection method, this method is more adaptable and does not require an additional sliding compensation algorithm. It should be noted that due to the physiological characteristics of the human eye, the eye image can only predict the statistical position of the gaze point. The relationship between the gaze point position of different users and the eye image is related to the position of the fovea (the most sensitive area of the retina) of the human eye. However, the position of the fovea cannot be directly obtained from the image. Therefore, compared with the pupil-corneal reflection method, the prediction accuracy may decrease to a certain extent. In addition, a large amount of labeled data is required for training, the model is highly complex, and the amount of calculation is large.
[0035] (2) Gaze point estimation method based on 1D sensor signal
[0036] The gaze point estimation method based on 1D sensor signals collects brightness information of the user's eye fixation area by designing a dedicated optical sensor, and feeds this information into a prediction model to estimate the user's gaze point position. The specific process of this method is similar to the deep learning method based on global features, and is divided into two key stages: offline training and online prediction. In the offline training stage, developers use the collected sensor data and the corresponding gaze point positions to train the prediction model for gaze point prediction in daily use. In actual use, in order to further improve the prediction accuracy, the prediction results of the offline model are fine-tuned or calibrated by online registration to improve the accuracy of gaze point estimation.
[0037] It should be noted that although the gaze point estimation method based on 1D sensor signals has advantages in speed and power consumption, it is usually less accurate than the gaze point estimation method based on 2D images.
[0038] In summary, in the field of gaze point estimation, both the gaze point estimation methods based on 2D images and 1D sensor signals have their own advantages and disadvantages. Among them, the gaze point estimation method based on 2D images can learn the abstract features of the scene through deep learning and other technologies to achieve high-precision estimation of the gaze point. At the same time, image processing technology can adapt to various complex scenes, including illumination changes, occlusion, etc., making the method more robust. However, the image processing process is relatively cumbersome and requires a lot of computing resources, resulting in poor real-time performance and is not suitable for some scenes that require rapid response. The gaze point estimation method based on 1D sensor signals has the characteristics of strong real-time performance and is suitable for scenes that require rapid positioning. The sensor data acquisition is relatively simple and does not require complex image processing processes. It is suitable for some resource-constrained environments. However, the sensor signal is easily interfered by noise, resulting in limited accuracy, especially in complex environments where cumulative errors are prone to occur.
[0039] The embodiments of the present disclosure provide a gaze point estimation method, which can make up for the shortcomings of the respective methods and give play to their advantages by jointly using 2D image information and 1D sensor signal information, so as to estimate the gaze point position more accurately, quickly and with low power consumption.
[0040] like Figure 1 As shown, the embodiment of the present disclosure provides a gaze point estimation method, including:
[0041] Step 101: Acquire a user's eye image and a sensor signal;
[0042] Step 102: determining a gaze point prediction method at the current moment according to a correspondence between a preset time and a gaze point prediction method, wherein the gaze point prediction method includes a first method for predicting according to a user eye image and a sensor signal and a second method for predicting according to the sensor signal;
[0043] Step 103: When the determined gaze point prediction method is the first method, calling a gaze point detection model to estimate the gaze point position, the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module and a gaze point prediction module, the multi-resolution feature extraction module is configured to extract a feature map of the user's eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map and the sensor signal at the current moment, and the gaze point prediction module is configured to perform gaze point prediction according to the fused features;
[0044] Step 104: When the determined gaze point prediction method is the second method, a gaze point tracking model is called to estimate the gaze point position, and the gaze point tracking model is configured to predict the gaze point based on the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment.
[0045] The gaze point estimation method based on 2D images can accurately predict the gaze point position, but it is difficult to quickly collect image data due to the limitations of the imaging principle. At the same time, although the gaze point estimation method based on 1D sensor signals can quickly obtain sensor data, it is difficult to provide accurate gaze point prediction results. The gaze point estimation method of the disclosed embodiment obtains the user's eye image and sensor signal, and uses the user's eye image and sensor signal to accurately estimate the gaze point at some moments, and only uses the sensor signal for rapid positioning at other moments, thereby combining the real-time nature of the gaze point estimation method based on 1D sensor signals and the high-precision characteristics of the gaze point estimation method based on 2D images, and can achieve fast and accurate gaze point tracking and positioning functions.
[0046] like Figure 2 As shown, when using the gaze point estimation method of the embodiment of the present disclosure, two models need to be established and trained in advance, namely a gaze point detection model based on multimodal information (sensor signals and user eye images), and a gaze point tracking model based on sensor signals and the prediction results of the previous moment.
[0047] First, after obtaining the user's eye image and sensor signal, the user's eye image and sensor signal can be input into the gaze point detection model, and the gaze point is initially predicted by the gaze point detection model based on multimodal information (that is, the gaze point position at the initial moment is obtained by the gaze point detection model). The model deeply analyzes the user's eye image and combines the environmental information provided by the sensor signal to accurately estimate the gaze point position at the current moment. This step aims to make full use of multimodal information and improve the accuracy of gaze point position estimation.
[0048] Next, a gaze tracking model based on sensor signals was introduced. This model predicts the gaze position at the current moment based on the sensor signals at the previous and next moments and the gaze position prediction results at the previous moment. This model is designed to achieve fast tracking of the gaze point while taking into account the dynamic changes of the gaze point. By making full use of the real-time nature of the sensor signals, the response speed of the algorithm is improved.
[0049] In some exemplary embodiments, in step 101, the acquisition frequency of the user's eye image and the sensor signal may be different. For example, the acquisition frequency of the sensor signal may be greater than the acquisition frequency of the user's eye image. For example, within a preset first time interval T, M sensor signals and 1 user eye image may be acquired, where T is greater than 0 and M is greater than 1. That is, within the preset first time interval T, a gaze point tracking model based on sensor signals may be used to perform M-1 times of rapid gaze point tracking, and a gaze point detection model based on multimodal information may be used to perform one precise gaze point positioning. By performing one precise gaze point positioning with a gaze point detection model based on multimodal information, the accumulated error generated when performing rapid gaze point tracking based on sensor signals may be eliminated.
[0050] In some exemplary embodiments, in step 102, determining the gaze point prediction method at the current moment according to the correspondence between the preset time and the gaze point prediction method includes:
[0051] Obtain an initial time t and a first time interval T, wherein t is greater than or equal to 0, T=MT', M is a natural number greater than 1, and T' is the time interval between two adjacent time moments;
[0052] When the current time is equal to (t+kT), the gaze point prediction method at the current time is determined to be the first method, and k is a natural number greater than or equal to 0;
[0053] When the current time is not equal to (t+kT), it is determined that the gaze point prediction method at the current time is the second method.
[0054] In the embodiment of the present disclosure, multiple moments for determining the gaze point prediction method include: t, t+T', t+2T', ..., t+MT', t+(M+1)T', t+(M+2)T', ..., t+2MT', t+(2M+1)T', t+(2M+2)T', ..., wherein t is the initial moment, and when the current moment is equal to (t+k MT') = (t+kT), the first method is used to predict the gaze point; when the current moment is not equal to (t+k MT') (for example, equal to t+T', t+(M-1)T', t+(M+1)T', etc.), the second method is used to predict the gaze point.
[0055] The gaze point estimation method of the disclosed embodiment uses the user's eye image and sensor signal to perform accurate gaze point estimation at the key time point (t+kT) (when k=0, the key time point is the initial moment, that is, at the initial moment, the gaze point prediction method for determining the current moment is the first method), and at the moments between the key time points, only the sensor signal is used for rapid positioning, thereby meeting the requirements of accuracy and real-time performance at the same time.
[0056] In the disclosed embodiment, when the current moment is not equal to (t+kT), the method for predicting the gaze point at the current moment is determined to be the second method. At this time, the gaze point tracking model is called to estimate the gaze point position, and the gaze point tracking model predicts the gaze point based on the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment. When the current moment is equal to (t+kT+T'), the previous moment is equal to (t+kT), so the gaze point coordinates at the previous moment are obtained by calling the first method of the gaze point detection model; when the current moment is equal to (t+kT+mT'), m is a natural number and 1<m<M, the previous moment is equal to (t+kT+(m-1)T'), so the gaze point coordinates at the previous moment are obtained by calling the second method of the gaze point tracking model.
[0057] In order to ensure the accuracy and stability of the gaze point prediction results, the present disclosure corrects the positioning results of the gaze point tracking model based on the sensor signal at regular intervals. Specifically, at every first time interval T, the gaze point detection model based on multimodal information (sensor signals and user eye images) is used to correct the gaze point position estimation result at the current moment to eliminate inaccuracies that may be caused by factors such as cumulative errors. This strategy effectively improves the robustness and reliability of gaze point prediction.
[0058] In summary, the disclosed embodiment achieves accurate estimation and rapid tracking of the gaze point by carefully designing a gaze point detection model based on multimodal information and a gaze point tracking model based on sensor information. The introduction of the correction step further enhances the robustness of the algorithm, making it more reliably applicable in different scenarios.
[0059] In some exemplary embodiments, the multi-resolution feature extraction module includes: a first convolutional network, a second convolutional network, and a third convolutional network, wherein:
[0060] The first convolution network includes n first convolution units connected in series, each first convolution unit outputs a first feature map, the input of the first first convolution unit is the user's eye image, the input of the i-th first convolution unit is the first feature map output by the (i-1)-th first convolution unit, i is between 2 and n, and n is greater than 1;
[0061] The second convolutional network includes n second convolutional units connected in series, each second convolutional unit outputs a second feature map, the input of the first second convolutional unit is the first feature map output by the nth first convolutional unit, the input of the ith second convolutional unit is the third feature map, and the third feature map is obtained by summing the corresponding position elements of the second feature map output by the (i-1)th second convolutional unit and the first feature map output by the (n-i+1)th first convolutional unit;
[0062] The third convolutional network includes (n-1) third convolutional units connected in series, each third convolutional unit outputs a fourth feature map, the input of the first third convolutional unit is the second feature map output by the nth second convolutional unit, the input of the i-th third convolutional unit is the fifth feature map, and the fifth feature map is obtained by summing the corresponding position elements of the fourth feature map output by the (i-1)th third convolutional unit and the second feature map output by the (n-i+1)th second convolutional unit.
[0063] In some exemplary embodiments, the feature fusion module includes: a feature dimensionality reduction (2D→1D) module, a first multi-layer perceptron, a first concatenation module, a second multi-layer perceptron, and a feature dimensionality increase (1D→2D) module, wherein:
[0064] The feature dimension reduction module is configured to sequentially concatenate the extracted feature maps (i.e., the fourth feature maps output by the (n-1)th third convolutional unit) row by row to obtain a one-dimensional feature vector;
[0065] A first multi-layer perceptron is configured to extract a first feature vector from the one-dimensional feature vector;
[0066] A first splicing module is configured to splice the first feature vector with the sensor signal at the current moment to obtain a fused feature vector;
[0067] A second multilayer perceptron is configured to extract a second feature vector from the fused feature vector;
[0068] The feature dimension upgrading module is configured to segmentally splice the second feature vector into a two-dimensional feature map.
[0069] In the disclosed embodiment, a multilayer perceptron (including the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron and the fourth multilayer perceptron described below) is composed of multiple neurons in a hierarchical structure, including an input layer, one or more hidden layers, and an output layer. The number of neurons in the input layer corresponds to the characteristic dimension of the input data, the number of hidden layers and the number of neurons can be set according to the complexity of the problem and the characteristics of the data, and the number of neurons in the output layer corresponds to the characteristic dimension of the output feature vector.
[0070] Exemplarily, the first feature vector extracted by the first multilayer perceptron from the one-dimensional feature vector may be a one-dimensional feature vector of 1*128, wherein 1*128 indicates that it contains 1 row and 128 columns of elements. However, this is not limited in the embodiments of the present disclosure.
[0071] Exemplarily, the second feature vector extracted by the second multilayer perceptron from the fused feature vector may be a one-dimensional feature vector of 1*64, where 1*64 indicates that it contains 1 row and 64 columns of elements. However, the embodiments of the present disclosure are not limited to this.
[0072] In the disclosed embodiment, row-by-row sequential splicing refers to connecting all elements in row order. For example, assuming that the extracted feature map is an 8*8 2D image pixel matrix, where 8*8 means that it contains 8 rows and 8 columns of elements, the first row to the eighth row of the feature map are connected end to end row by row to obtain a 1-dimensional feature vector of 1*64.
[0073] In the disclosed embodiment, segmented splicing refers to determining the length of each segment according to the length of the two-dimensional feature map to be generated, dividing the one-dimensional feature vector into multiple segments according to the length of each segment, arranging the multiple segments in columns, and obtaining a two-dimensional feature map, wherein the number of segments into which the one-dimensional feature vector is divided is the width of the two-dimensional feature map. For example, assuming that the one-dimensional feature vector is a 1D pixel feature vector of 1*64, and the length and width of the spliced two-dimensional feature map are 8*8, the one-dimensional feature vector is divided into 8 segments, each segment is regarded as a pixel row in the two-dimensional feature map, and the 8 segments are arranged in columns, so as to obtain an 8*8 two-dimensional feature map.
[0074] In some exemplary embodiments, the gaze point prediction module includes: a deconvolution module and a prediction module, wherein:
[0075] A deconvolution module is configured to use the two-dimensional feature map as an initial intensity map, perform a deconvolution operation on the initial intensity map, and obtain a fixation point intensity map, wherein the fixation point intensity map includes a plurality of fixation point positions and a probability prediction value of each fixation point position;
[0076] The prediction module is configured to take the gaze point position with the maximum probability prediction value as the predicted gaze point position.
[0077] Figure 3 The following is a schematic diagram of the structure of a gaze point detection model based on multimodal information (sensor signals and user eye images) according to an embodiment of the present disclosure. The arrows represent the flow of data, the blocks represent the calculation units, and ⊕ represents the sum of the corresponding position elements of the two input data and the output. It represents the output after the two input data are spliced together. In order to ensure that the data can be spliced effectively, some operations to adjust the feature size will be added. Before using the gaze point detection model to estimate the gaze point position at a given moment, it is necessary to jointly use the user's eye image and sensor signal to train the model. Figure 3 As shown, the computational units used in the gaze point detection model include:
[0078] Convolution unit: Use convolution operation to extract features from the user's eye image and output feature images. Each convolution unit specifically includes a convolution layer, a batch normalization layer, and a nonlinear layer.
[0079] Feature dimension reduction (2D→1D) module: The 2D feature images are concatenated row by row to obtain a 1D data feature vector.
[0080] Multilayer Perceptron: Performs multiple linear and nonlinear transformations on 1D data in sequence.
[0081] Feature dimension upgrading (1D→2D) module: segment the 1D data and splice it into a 2D image.
[0082] Gaze point prediction module: improves the resolution of the converted image, obtains the intensity map of the gaze point position, and performs gaze point prediction based on the obtained intensity map.
[0083] like Figure 3 As shown in Figure 1, the gaze point detection model consists of three parts: multi-resolution feature extraction module, feature fusion module and gaze point prediction module. The following is a detailed introduction to these three parts.
[0084] The multi-resolution feature extraction module mainly realizes the fusion of multi-resolution information to extract image features that are highly correlated with the gaze point prediction results. This process can be divided into three parts, corresponding to the three columns (three convolutional networks) in the figure. Each column uses a different strategy to optimize the effect of feature extraction.
[0085] The first column (the first convolutional network) uses a traditional multi-layer convolutional unit structure. Through convolution operations, it gradually increases the abstract understanding of image information, thereby capturing higher-level semantic information and providing richer features for gaze point prediction. The second column (the second convolutional network) starts from the lowest resolution, performs convolution operations, and fuses with adjacent high-resolution convolution results, so that abstract features can more fully compensate for specific features. The third column (the third convolutional network) uses the opposite fusion order of the second column, fusing features from the highest resolution to the lowest resolution. The purpose of this strategy is to ensure that the feature map of each resolution is integrated with the image features of other resolutions through different fusion directions to describe the image in a more comprehensive way. Through the operations of the above three columns, the feature maps of different resolutions are fully fused, so as to better extract the features most relevant to the prediction.
[0086] Since the human eye gaze area and the collected human eye image exist in different spaces, in order to achieve the mapping of these two spaces and fuse the one-dimensional features of the sensor, the feature fusion module takes a series of processing steps. First, the 2D feature map is stretched into a 1D feature vector through the 2D→1D module, and then the stretched 1D feature vector is transformed by the first multi-layer perceptron. Next, the transformed feature vector is spliced with the sensor signal through the first splicing module. The spliced feature vector is further transformed by the second multi-layer perceptron, and is restored to a 2D feature map after segmented splicing through the 1D→2D module. This 2D feature map is used as a low-resolution initial value (i.e., initial intensity map) of the gaze point estimation intensity map. The purpose of this design is to overcome the differences between different spaces and achieve effective fusion of the user's eye image features and sensor signals.
[0087] The fixation point prediction module uses deconvolution operation to refine the initial intensity map based on the initial intensity map, restores the high-resolution fixation point intensity map, and uses the least squares loss to optimize the model on the restored high-resolution fixation point intensity map.
[0088] In some exemplary embodiments, the gaze tracking model includes: a third multi-layer perceptron, a feature concatenation module, a fourth multi-layer perceptron, and a summation module connected in series, wherein:
[0089] a third multilayer perceptron configured to concatenate the sensor signal at the previous moment and the sensor signal at the current moment into a concatenated sensor signal, and extract a third feature vector from the concatenated sensor signal;
[0090] A feature concatenation module is configured to concatenate the third feature vector with the gaze point coordinates at the previous moment to obtain a concatenated feature vector;
[0091] A fourth multi-layer perceptron is configured to extract a fourth feature vector from the concatenated feature vector, where the fourth feature vector is a difference between the gaze point coordinates at the current moment and the gaze point coordinates at the previous moment;
[0092] The summing module is configured to perform a summing operation on the obtained difference and the gaze point coordinates at the previous moment to obtain the gaze point coordinates at the current moment.
[0093] Exemplarily, the third feature vector extracted by the third multilayer perceptron from the spliced sensor signal may be a one-dimensional feature vector of 1*32, where 1*32 indicates that it contains 1 row and 32 columns of elements. However, the embodiments of the present disclosure are not limited to this.
[0094] Exemplarily, the fourth feature vector extracted by the fourth multi-layer perceptron from the concatenated feature vector can be a one-dimensional feature vector of 1*2, where 1*2 means containing 1 row and 2 columns of elements, however, the disclosed embodiment is not limited to this. The 1 row and 2 columns of elements contained in the fourth feature vector extracted by the fourth multi-layer perceptron can be regarded as the difference between the gaze point coordinates (two-dimensional coordinates, i.e., XY axis coordinates) at the current moment and the gaze point coordinates at the previous moment.
[0095] After obtaining the gaze point position predicted at the previous moment (when the previous moment is equal to (t+kT), the gaze point position predicted at the previous moment is the gaze point position predicted according to the first method; when the previous moment is not equal to (t+kT), the gaze point position predicted at the previous moment is the gaze point position predicted according to the second method), the embodiment of the present disclosure uses a gaze point tracking model to predict the gaze point position at the current moment based on sensor signals at previous and subsequent moments combined with the gaze point position predicted at the previous moment. Figure 4 FIG. 1 is a schematic diagram of the structure of the gaze tracking model of the embodiment of the present disclosure. Figure 4 As shown, first, the sensor signals s before and after are used t-1 、s t , combined with the gaze point position g at the previous moment t-1 , predict the current gaze point position g t and the gaze point position g at the previous moment t-1 Then, through g t =g t-1 +d, get the current gaze point position. Let the abstract representation of this model be f s , then the current gaze point position can be expressed as follows:
[0096] g t =f s (s t-1 ,s t , g t-1 )+g t-1 ;
[0097] The disclosed embodiment utilizes the least squares loss to optimize the gaze tracking model.
[0098] Although the gaze point tracking model is simple and has a fast response speed, the gaze point tracking model can only make corrections based on the gaze point position prediction result of the previous moment to obtain the gaze point position prediction result of the current moment. As time goes by, the prediction error may gradually accumulate, resulting in a decrease in prediction accuracy. In order to solve this problem, the embodiment of the present disclosure uses the gaze point detection model based on multimodal information proposed above, and uses the gaze point detection model to correct the predicted gaze point position after a preset first time interval, thereby helping to ensure the accuracy and robustness of the gaze point prediction result in long-term operation, solves the problem of insufficient accuracy caused by long-term accumulated errors of the gaze point tracking model, and provides a more reliable gaze point estimation strategy for the system.
[0099] In some exemplary embodiments, the method further comprises:
[0100] Detect whether the current system resource usage exceeds the preset usage threshold;
[0101] When the current system resource usage is less than or equal to a preset usage threshold, shortening the first time interval T;
[0102] When the current system resource usage is greater than a preset usage threshold, the first time interval T is increased.
[0103] The gaze point estimation method of the embodiment of the present disclosure achieves high-precision gaze point prediction by regularly using the gaze point detection model to correct the gaze point position. In addition, the gaze point estimation method of the embodiment of the present disclosure also has adaptive performance, dynamically adjusting the call ratio of the gaze point detection model and the gaze point tracking model in the gaze point estimation process according to the platform computing power level, achieving a free trade-off between speed and accuracy, and better adapting to different computing power computing platforms.
[0104] For example, Figure 5As shown, in the initialization phase, the user sets the first time interval T (or the ratio r of the number of calls of the gaze point detection model per unit time to the total number of calls). During the operation of the system, the system can make decisions adaptively by querying the availability of computing resources. When computing resources are sufficient, the system will shorten the value of the first time interval T (or increase the value of r) accordingly, increase the frequency of fixation point correction, and thus improve the detection accuracy of the overall system. Conversely, when system resources are limited, the system will increase the value of the first time interval T (or reduce the value of r) and reduce the number of corrections to shorten the overall response time of the system and ensure the efficiency of the system operation. This adaptive strategy enables the system to flexibly respond to changes in computing resources in different environments, so that it can find a balance between performance and efficiency, so as to provide stable and efficient results under various conditions.
[0105] like Figure 6 As shown, the embodiment of the present disclosure further provides a gaze point estimation device, including a camera 601, an optical sensor 602, a memory 603 and a processor 604, wherein:
[0106] Camera 601, configured to capture user eye images;
[0107] The optical sensor 602 is configured to collect brightness information of the user's eye area;
[0108] The memory 603 is configured to store a gaze point detection model and a gaze point tracking model, wherein the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module and a gaze point prediction module, wherein the multi-resolution feature extraction module is configured to extract a feature map of the user's eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map and the sensor signal at the current moment, the gaze point prediction module is configured to perform gaze point prediction according to the fused features, and the gaze point tracking model is configured to perform gaze point prediction according to the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment;
[0109] Processor 604 is configured to obtain user eye images and sensor signals; determine a gaze point prediction method at a current moment according to a correspondence between a preset time and a gaze point prediction method, wherein the gaze point prediction method includes a first method for predicting based on the user eye image and the sensor signal and a second method for predicting based on the sensor signal; when the determined gaze point prediction method is the first method, call a gaze point detection model to estimate the gaze point position; when the determined gaze point prediction method is the second method, call a gaze point tracking model to estimate the gaze point position.
[0110] In the embodiment of the present disclosure, the optical sensor 602 may be an infrared sensor, however, the present disclosure is not limited to this.
[0111] When the gaze point estimation device is used in the field of virtual reality technology (VR), augmented reality (AR) or mixed reality technology (MR), the modules in the gaze point estimation device can be dispersed and arranged in specific areas in the head-mounted display as needed. Before the camera 601 collects the user's eye image and the optical sensor 602 collects the brightness information of the user's eye area, the fill light can be turned on, and the fill light will illuminate the human eye area, and then the camera 601 collects the user's eye image and / or the optical sensor 602 collects the brightness information of the user's eye area. The brightness information of the user's eye area collected by the optical sensor 602 is the brightness information of the preset designated position, and the preset designated position is the position most related to eye movement determined in the product design stage.
[0112] In some exemplary embodiments, the multi-resolution feature extraction module includes: a first convolutional network, a second convolutional network, and a third convolutional network, wherein:
[0113] The first convolution network includes n first convolution units connected in series, each first convolution unit outputs a first feature map, the input of the first first convolution unit is the user's eye image, the input of the i-th first convolution unit is the first feature map output by the (i-1)-th first convolution unit, and i is between 2 and n;
[0114] The second convolutional network includes n second convolutional units connected in series, each second convolutional unit outputs a second feature map, the input of the first second convolutional unit is the first feature map output by the nth first convolutional unit, the input of the ith second convolutional unit is the third feature map, and the third feature map is obtained by summing the corresponding position elements of the second feature map output by the (i-1)th second convolutional unit and the first feature map output by the (n-i+1)th first convolutional unit;
[0115] The third convolutional network includes (n-1) third convolutional units connected in series, each third convolutional unit outputs a fourth feature map, the input of the first third convolutional unit is the second feature map output by the nth second convolutional unit, the input of the i-th third convolutional unit is the fifth feature map, and the fifth feature map is obtained by summing the corresponding position elements of the fourth feature map output by the (i-1)th third convolutional unit and the second feature map output by the (n-i+1)th second convolutional unit.
[0116] In some exemplary embodiments, the feature fusion module includes: a feature dimension reduction module, a first multi-layer perceptron, a first concatenation module, a second multi-layer perceptron, and a feature dimension increase module, wherein:
[0117] The feature dimension reduction module is configured to sequentially concatenate the extracted feature maps (i.e., the fourth feature maps output by the (n-1)th third convolutional unit) row by row to obtain a one-dimensional feature vector;
[0118] A first multi-layer perceptron is configured to extract a first feature vector from the one-dimensional feature vector;
[0119] A first splicing module is configured to splice the first feature vector with the sensor signal at the current moment to obtain a fused feature vector;
[0120] A second multilayer perceptron is configured to extract a second feature vector from the fused feature vector;
[0121] The feature dimension upgrading module is configured to segmentally splice the second feature vector into a two-dimensional feature map.
[0122] In some exemplary embodiments, the gaze point prediction module includes: a deconvolution module and a prediction module, wherein:
[0123] A deconvolution module is configured to use the two-dimensional feature map as an initial intensity map, perform a deconvolution operation on the initial intensity map, and obtain a fixation point intensity map, wherein the fixation point intensity map includes a plurality of fixation point positions and a probability prediction value of each fixation point position;
[0124] The prediction module is configured to take the gaze point position with the maximum probability prediction value as the predicted gaze point position.
[0125] In some exemplary embodiments, the gaze tracking model includes: a third multi-layer perceptron, a feature concatenation module, a fourth multi-layer perceptron, and a summation module connected in series, wherein:
[0126] a third multilayer perceptron configured to splice the sensor signal at the previous moment and the sensor signal at the current moment into a spliced sensor signal, and extract a third feature vector from the spliced sensor signal;
[0127] A feature concatenation module is configured to concatenate the third feature vector with the gaze point coordinates at the previous moment to obtain a concatenated feature vector;
[0128] A fourth multi-layer perceptron is configured to extract a fourth feature vector from the concatenated feature vector, where the fourth feature vector is a difference between the gaze point coordinates at the current moment and the gaze point coordinates at the previous moment;
[0129] The summation module is configured to perform a summation operation on the extracted fourth eigenvector and the gaze point coordinates at the previous moment to obtain the gaze point coordinates at the current moment.
[0130] In some exemplary embodiments, the processor 604 determines the gaze point prediction method at the current moment according to the correspondence between the preset time and the gaze point prediction method, including:
[0131] Obtain an initial time t and a first time interval T, wherein t is greater than or equal to 0, T=MT', M is a natural number greater than 1, and T' is the time interval between two adjacent time moments;
[0132] When the current time is equal to t+kT, determining that the gaze point prediction method at the current time is the first method, and k is a natural number greater than or equal to 0;
[0133] When the current time is not equal to t+kT, it is determined that the gaze point prediction method at the current time is the second method.
[0134] In some exemplary embodiments, the processor 604 is further configured to:
[0135] Detect whether the current system resource usage exceeds the preset usage threshold;
[0136] When the current system resource usage is less than or equal to a preset usage threshold, shortening the first time interval T;
[0137] When the current system resource usage is greater than a preset usage threshold, the first time interval T is increased.
[0138] An embodiment of the present disclosure also provides a gaze point estimation device, which may include a processor and a memory storing a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the gaze point estimation method as described in any of the preceding items in the present disclosure are implemented.
[0139] like Figure 7As shown, in one example, a gaze point estimation device may include: a processor 710, a memory 720, a bus system 730 and a transceiver 740, wherein the processor 710, the memory 720 and the transceiver 740 are connected via the bus system 730, the memory 720 is used to store instructions, and the memory 720 is also used to store a gaze point detection model and a gaze point tracking model, the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module and a gaze point prediction module, the multi-resolution feature extraction module is configured to extract a feature map of the user's eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map and the sensor signal at the current moment, the gaze point prediction module is configured to perform gaze point prediction based on the fused features, and the gaze point tracking model is configured to perform gaze point prediction based on the fused features. The processor 710 is configured to predict the gaze point according to the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment. The processor 710 is used to execute the instructions stored in the memory 720 to control the transceiver 740 to obtain the user eye image and the sensor signal. The processor 710 determines the gaze point prediction method at the current moment according to the correspondence between the preset time and the gaze point prediction method. The gaze point prediction method includes a first method for predicting according to the user eye image and the sensor signal and a second method for predicting according to the sensor signal. When the determined gaze point prediction method is the first method, the gaze point detection model is called to estimate the gaze point position. When the determined gaze point prediction method is the second method, the gaze point tracking model is called to estimate the gaze point position.
[0140] It should be understood that the processor 710 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0141] The memory 720 may include a read-only memory and a random access memory, and provides instructions and data to the processor 710. A portion of the memory 720 may also include a nonvolatile random access memory. For example, the memory 720 may also store information on the device type.
[0142] In addition to the data bus, the bus system 730 may also include a power bus, a control bus, and a status signal bus. Figure 7 Various buses are labeled as bus system 730 .
[0143] In the implementation process, the processing performed by the gaze point estimation device can be completed by the hardware integrated logic circuit in the processor 710 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 720, and the processor 710 reads the information in the memory 720 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.
[0144] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores executable instructions. When the executable instructions are executed by the processor, the gaze point estimation method provided by any of the above embodiments of the present disclosure can be implemented. The gaze point estimation method can make up for the shortcomings of the respective methods and give play to their advantages by jointly using 2D image information and 1D sensor signal information, so as to estimate the gaze point position more accurately, quickly and with low power consumption. The gaze point estimation method driven by executing executable instructions to perform the gaze point estimation method is basically the same as the gaze point estimation method provided by the above embodiments of the present disclosure, and will not be repeated here.
[0145] In some possible implementations, various aspects of the gaze point estimation method provided by the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the gaze point estimation method according to various exemplary embodiments of the present disclosure described above in this specification. For example, the computer device may execute the gaze point estimation method recorded in the embodiments of the present disclosure.
[0146] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0147] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0148] It should be noted that the above embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to the contents specifically shown and described herein. Various modifications, substitutions or omissions may be made to the forms and details of implementation without departing from the scope of the present disclosure.
Claims
1. A method for estimating a gaze point, characterized in that: include: Acquire user eye images and sensor signals; Determining a gaze point prediction method at a current moment according to a correspondence between a preset time and a gaze point prediction method, the gaze point prediction method including a first method for prediction based on the user eye image and a sensor signal and a second method for prediction based on the sensor signal; When the determined gaze point prediction method is the first method, calling a gaze point detection model to estimate the gaze point position, the gaze point detection model including a multi-resolution feature extraction module, a feature fusion module, and a gaze point prediction module, the multi-resolution feature extraction module being configured to extract a feature map of the user's eye image, the feature fusion module being configured to perform feature fusion on the extracted feature map with a sensor signal at a current moment, and the gaze point prediction module being configured to perform gaze point prediction based on the fused features; When the determined gaze point prediction method is the second method, the gaze point tracking model is called to estimate the gaze point position, and the gaze point tracking model is configured to predict the gaze point based on the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment.
2. The method according to claim 1, characterized in that The multi-resolution feature extraction module includes: a first convolutional network, a second convolutional network and a third convolutional network, wherein: The first convolutional network includes n first convolutional units connected in series, each of which outputs a first feature map. The input of the first first convolutional unit is the user eye image, and the input of the i-th first convolutional unit is the first feature map output by the (i-1)-th first convolutional unit, where i is between 2 and n, and n is greater than 1. The second convolutional network includes n second convolutional units connected in series, each second convolutional unit outputs a second feature map, the input of the first second convolutional unit is the first feature map output by the nth first convolutional unit, the input of the i-th second convolutional unit is a third feature map, and the third feature map is obtained by summing the corresponding position elements of the second feature map output by the (i-1)th second convolutional unit and the first feature map output by the (n-i+1)th first convolutional unit; The third convolutional network includes (n-1) third convolutional units connected in series, each of the third convolutional units outputs a fourth feature map, the input of the first third convolutional unit is the second feature map output by the nth second convolutional unit, the input of the i-th third convolutional unit is the fifth feature map, and the fifth feature map is obtained by summing the fourth feature map output by the (i-1)th third convolutional unit and the second feature map output by the (n-i+1)th second convolutional unit.
3. The method according to claim 1, characterized in that The feature fusion module includes: a feature dimensionality reduction module, a first multi-layer perceptron, a first splicing module, a second multi-layer perceptron and a feature dimensionality increase module, wherein: The feature dimensionality reduction module is configured to sequentially concatenate the extracted feature maps row by row to obtain a one-dimensional feature vector; The first multi-layer perceptron is configured to extract a first feature vector from the one-dimensional feature vector; The first splicing module is configured to splice the first feature vector with the sensor signal at the current moment to obtain a fused feature vector; The second multi-layer perceptron is configured to extract a second feature vector from the fused feature vector; The feature dimension upgrading module is configured to segmentally splice the second feature vector into a two-dimensional feature map.
4. The method according to claim 3, characterized in that The gaze point prediction module includes: a deconvolution module and a prediction module, wherein: The deconvolution module is configured to use the two-dimensional feature map as an initial intensity map, perform a deconvolution operation on the initial intensity map, and obtain a fixation point intensity map, wherein the fixation point intensity map includes multiple fixation point positions and a probability prediction value of each fixation point position; The prediction module is configured to take the gaze point position with the maximum probability prediction value as the predicted gaze point position.
5. The method according to claim 1, wherein The gaze tracking model includes a third multi-layer perceptron, a feature splicing module, a fourth multi-layer perceptron, and a summation module connected in series, wherein: The third multilayer perceptron is configured to splice the sensor signal at the previous moment and the sensor signal at the current moment into a spliced sensor signal, and extract a third feature vector from the spliced sensor signal; The feature splicing module is configured to splice the third feature vector with the gaze point coordinates at the previous moment to obtain a spliced feature vector; The fourth multi-layer perceptron is configured to extract a fourth eigenvector from the concatenated eigenvector, where the fourth eigenvector is the difference between the gaze point coordinates at the current moment and the gaze point coordinates at the previous moment; The summing module is configured to perform a summing operation on the extracted fourth eigenvector and the gaze point coordinates at the previous moment to obtain the gaze point coordinates at the current moment.
6. The method according to claim 1, wherein The determining of the gaze point prediction method at the current moment according to the correspondence between the preset time and the gaze point prediction method includes: Obtain an initial time t and a first time interval T, where t is greater than or equal to 0, T=MT', M is a natural number greater than 1, and T' is the time interval between two adjacent time moments; When the current moment is equal to (t+kT), determining that the gaze point prediction method at the current moment is the first method, where k is a natural number greater than or equal to 0; When the current moment is not equal to (t+kT), the gaze point prediction method at the current moment is determined to be the second method.
7. The method according to claim 6, characterized in that The method further comprises: Detect whether the current system resource usage exceeds the pre-set usage threshold; When the current system resource usage is less than or equal to a preset usage threshold, shortening the first time interval T; When the current system resource usage is greater than a preset usage threshold, the first time interval T is increased.
8. A gaze point estimation device, characterized in that: include: A processor and a memory storing a computer program that can be run on the processor, wherein when the processor executes the computer program, the steps of the gaze point estimation method according to any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the gaze point estimation method according to any one of claims 1 to 7.
10. A computer program product, characterized in that The computer program product comprises instructions, which, when executed by a computer, perform the gaze point estimation method according to any one of claims 1 to 7.
11. A gaze point estimation device, characterized in that: include: Cameras, optical sensors, processors, and memory, including: The camera is configured to capture an image of the user's eyes; The optical sensor is configured to collect brightness information of the user's eye area; The memory is configured to store a gaze point detection model and a gaze point tracking model, wherein the gaze point detection model includes a multi-resolution feature extraction module, a feature fusion module, and a gaze point prediction module. The multi-resolution feature extraction module is configured to extract a feature map of the user's eye image, the feature fusion module is configured to perform feature fusion on the extracted feature map with the sensor signal at the current moment, and the gaze point prediction module is configured to perform gaze point prediction based on the fused features; the gaze point tracking model is configured to perform gaze point prediction based on the gaze point coordinates at the previous moment, the sensor signal at the previous moment, and the sensor signal at the current moment; The processor is configured to obtain a user eye image and a sensor signal, and determine a gaze point prediction method at a current moment based on a correspondence between a preset time and a gaze point prediction method, wherein the gaze point prediction method includes a first method for predicting based on the user eye image and the sensor signal and a second method for predicting based on the sensor signal; when the determined gaze point prediction method is the first method, a gaze point detection model is called to estimate the gaze point position; when the determined gaze point prediction method is the second method, a gaze point tracking model is called to estimate the gaze point position.
Citation Information
Patent Citations
Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof
CN101540090A
Eye-tracking enabled wearable devices
CN108700933A
Eyeball tracking method, device and storage medium
CN113366491A
Sight gaze point estimation method and device, electronic equipment and storage medium
CN117475499A
Free-viewpoint photorealistic view synthesis from casually captured video
US20200226816A1
