User gaze point estimation and precision evaluation method based on visual tracking

Abstract

The application discloses a user gaze point estimation method based on visual tracking, a user wears a head-mounted eye movement interaction device, a gaze point extraction module is used to acquire the gaze point coordinates of the user, a residual estimation module is used to calculate the residual of the eye movement offset of the user and the gaze point coordinates, and the obtained residual is sent to an offset adaptive module to update the gaze point coordinates, so that the final user gaze point estimation value is obtained. The application also discloses a method for evaluating the precision of the user gaze point estimation method, a user wears a head-mounted eye movement interaction device, gazes at precision test points displayed on a display interface in sequence, acquires the gaze point coordinates of the user, calculates and acquires the eye movement precision, and the eye movement precision values of all sub-regions are averaged to obtain the final evaluation value of the eye movement precision. The application can extract the adaptive individual difference eye movement offset, and has very important significance for improving the robustness and generalization ability of the eye movement algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital image processing, in particular to a user gaze point estimation and precision evaluation method based on visual tracking. BACKGROUND

[0002] Eye tracking, also known as visual tracking technology, is a technology that estimates the line of sight and gaze point coordinates by extracting eye movement related parameters. With the continuous development of eye tracking technology, the application scenarios of the technology in the fields of human-computer interaction, behavior analysis, etc. are also constantly enriched.

[0003] Since the eye tracking technology is processed and analyzed by collecting human eye movement physiological signals through a head-mounted eye movement device, the technology is strongly related to the individual physiological structure difference of each user's eye and the personalized use habit of the device. On the one hand, since the eye movement signal is a physiological signal of the human body, there are differences in physiological structure among different individuals, so the eye movement tracking algorithm needs to be able to adapt to the differences in individual physiological structure in order to effectively ensure the eye movement precision. On the other hand, the existing head-mounted eye movement device is based on an optical recording method, which uses an infrared light source to irradiate the eyeball and uses a high-speed eye camera to collect near-eye pictures. The image acquisition device is set in front of the eye, and there is a certain angle between the image acquisition device and the horizontal direction where the eye is located. Different subjects will have different use habits when using the same eye movement tracking device, which will directly affect the eye movement parameters collected by the eye movement device, thereby causing a large fluctuation in eye movement precision. In addition, for users wearing glasses, the wearing of glasses will change the pupillary distance and the eyeball may be partially blocked, which will also affect the precision deviation of the eye movement algorithm reasoning result. SUMMARY

[0004] In view of the problem that the existing eye tracking technology is strongly related to the individual physiological structure difference of each user's eye and the personalized use habit of the device, the present application discloses a user gaze point estimation and precision evaluation method based on visual tracking.

[0005] The present application discloses a user gaze point estimation method based on visual tracking, which specifically comprises:

[0006] The user wears a head-mounted eye movement interaction device, uses a gaze point extraction module to obtain the gaze point coordinates of the user, calculates the residual error between the eye movement offset of the user and the gaze point coordinates through a residual error estimation module, and then sends the obtained residual error to an offset adaptive module to update the gaze point coordinates, thereby obtaining the final user gaze point estimation value.

[0007] The gaze point extraction module is implemented by a deep learning artificial neural network composed of multiple deep convolutional neural layers and multiple dilated convolution layers, the user's binocular image collected by the head-mounted eye movement interaction device is taken as the input of the module, and the gaze point coordinate value of the user extracted by the module is taken as the output of the module.

[0008] The gaze point extraction module is implemented by a deep learning artificial neural network composed of multiple deep convolutional neural layers and multiple dilated convolution layers, the user's binocular image collected by the head-mounted eye movement interaction device is taken as the input of the module, and the gaze point coordinate value of the user extracted by the module is taken as the output of the module.

[0009] The sample data set is constructed by requiring a plurality of users to wear a head-mounted eye movement interaction device, the user gazes at a target anchor point in the display interface of the device, the target anchor point moves to each pixel position of each row and each column of the display interface in a snake-like traversal manner, the target anchor point changes more than three different moving speeds during the movement, the head-mounted eye movement interaction device collects the eye images of the user gazing at the target anchor point moving constantly, and one round of sample data extraction is completed after the target anchor point completes a snake-like traversal. In each round of sample data extraction, the near-eye high-speed camera carried on the head-mounted eye movement interaction device saves the user's binocular image and the position coordinate value of the target anchor point gazed at by the user at the moment when the target anchor point traverses to each pixel position on the display interface in a snake-like manner, and the user's binocular image and the position coordinate value of the target anchor point gazed at by the user are taken as the sample and label of the sample data set, so that the construction of the sample data set is completed.

[0010] The deep learning artificial neural network is built by first extracting the features of the left and right eye images of the user's binocular image by using a deep convolutional neural layer, the convolution kernel size of each convolution layer in the deep convolutional neural layer is 3*3, and the convolution step is 2. Three dilated convolution layers are stacked after the deep convolutional neural layer, the convolution kernel size of the first dilated convolution layer is 3*3, the expansion rate is (1, 2), the convolution kernel size of the second dilated convolution layer is 3*3, the expansion rate is (2, 3), the convolution kernel size of the third dilated convolution layer is 3*3, the expansion rate is (4, 5), and the convolution step of the three dilated convolution layers is 1. The final output of the dilated convolution layer is deactivated to control the parameter amount of the deep learning artificial neural network, ensure the real-time performance of the deep learning artificial neural network, use ReLU as the activation function, and normalize the parameters of the deep learning artificial neural network before the activation processing.

[0011] The training and testing of the built deep learning artificial neural network, the size and pixel distribution of the sample data set are standardized, the resolution of the user's two-eye picture of the sample data set is reduced to the set value, all pixel values of the user's two-eye picture of the sample data set are divided by 256, so that the pixel value distribution is between 0 and 1, thereby realizing the normalization of the pixel value, and then the standardization distribution processing is performed on all pixel value data of the user's two-eye picture of the sample data set with 0.5 as the mean and 0.5 as the variance. The data after the standardization distribution processing is converted into tensor data using the PyTorch framework as the input of the deep learning artificial neural network, the parameters of the network are updated using the stochastic gradient descent algorithm, the parameters of the network are optimized using the Adam function, and the sample data set is divided into a training set and a test set in a data amount ratio of 7:3 using the cross-validation method. The L1 norm loss function is used as the loss function of the network, and the Adam function is used as the optimizer when training the network. The best set of network parameters of the training result is taken as the final parameters of the deep learning artificial neural network trained, thereby completing the training of the deep learning artificial neural network.

[0012] The residual estimation module, after the user's gaze offset extraction identifier is extracted, the module realizes the extraction of the user's eye movement offset, and uses a first-order difference function to calculate the residual of the user's eye movement offset and the gaze point coordinates.

[0013] The residual estimation module, in the display interface of the head-mounted eye movement interaction device, a two-dimensional plane rectangular coordinate system is established, the offset extraction identifier is displayed at the center position of the display interface, and the position coordinates are (x0, y0). The offset extraction identifier is a static picture or an animation. The user wearing the head-mounted eye movement interaction device gazes at the offset extraction identifier in the display interface, and the real-time gaze point coordinates of the user extracted by the gaze point extraction module in the i-th display interface are (x gi , y gi ), which is the eye movement offset of the user.

[0014] The frame rate of the display interface of the head-mounted eye movement interaction device is 30fps, the display time of the above-mentioned offset extraction identifier is set to one second, and the residual of the user's eye movement offset and the gaze point coordinates is calculated using a first-order difference function [x d , y d ], and the calculation formula is:

[0015]

[0016] Where i is an integer from 0 to 29.

[0017] The eye movement offset of the user includes a user usage habit offset and an angle between an eye visual axis and an eye optical axis of the user, the user usage habit offset is a fixed value, and an estimation process of the angle between the eye visual axis and the eye optical axis includes that the eye visual axis is a line connecting a macula fovea of the eye to the offset extraction mark, and the eye optical axis is a line connecting a pupil center of the eye to a retina center of the eye. A position coordinate of the pupil center of the eye is represented by P, a position coordinate of a corneal curvature center of the eye is represented by C, a direction vector of the eye visual axis is represented by V, a direction vector of the pupil center of the eye to the offset extraction mark, that is, a direction vector of the eye visual axis, is represented by U, an angle between the eye visual axis and the eye optical axis is represented by e, and a direction vector of the eye optical axis is represented by W, and a calculation formula of the direction vector of the eye optical axis is as follows:

[0018]

[0019] wherein, the direction vector of the eye optical axis, and a calculation formula of the direction vector of the eye visual axis is as follows:

[0020]

[0021] wherein, (α, β) represents an offset correction of the direction vector of the eye optical axis, and the calculation formula of the direction vector of the eye visual axis is as follows: wherein T represents a position coordinate of the offset extraction mark, and a calculation formula of the angle e between the eye visual axis and the eye optical axis is e = arccos θ (U, V), so as to complete the estimation of the angle between the eye visual axis and the eye optical axis.

[0022] The offset adaptive module corrects the user gaze point coordinates obtained by the gaze point extraction module by using the residual error calculated by the residual error estimation module, and takes the corrected user gaze point coordinates as the final user gaze point estimation value.

[0023] The offset adaptive module is implemented by a deep learning artificial neural network, the deep learning artificial neural network includes a plurality of deep convolutional neural layers, a plurality of dilated convolutional layers, an offset prediction branch and a full connection layer, and the four parts are sequentially connected, wherein the deep convolutional neural layers and the dilated convolutional layers have the same structure as the deep convolutional neural layers and the dilated convolutional layers in the gaze point extraction module, and a loss function L used in the training process of the deep learning artificial neural network is as follows: 1new L 1new = L1+ λ|b|, wherein L1 is an L1 norm loss function used in the gaze point extraction module, λ|b| is a regularization term, used for adjusting the adaptive ability of the network, wherein λ is an adjustment coefficient, and b is the residual error extracted in the gaze point extraction module.

[0024] The residual of the eye movement offset of the user and the gaze point coordinates is calculated by the residual estimation module, and the obtained residual is sent to the offset adaptive module to update the gaze point coordinates, so that the final user gaze point estimation value is obtained.

[0025] The application further discloses a method for evaluating the precision of the user gaze point estimation method, which specifically comprises rendering a plurality of precision test points on a display interface of the head-mounted eye movement interaction device according to preset positions, and controlling the precision test points to be displayed and then hidden on the display interface in a certain time sequence, so that only one precision test point is displayed at each moment.

[0026] The user wears the head-mounted eye movement interaction device, gazes at the precision test points displayed on the display interface in turn, and the gaze point coordinates of the user are obtained by using the gaze point extraction module, so that the eye movement precision is calculated and obtained.

[0027] For each precision test point, a plurality of eye movement precision values are calculated and averaged to obtain the eye movement precision value of the precision test point. The display interface is divided into a plurality of sub-regions, and a plurality of precision test points are arranged on each sub-region. The eye movement precision is calculated and evaluated for different sub-regions. The eye movement precision values obtained for all the precision test points in each sub-region are averaged to obtain the eye movement precision value of the sub-region. The eye movement precision values of all the sub-regions are averaged to obtain the final evaluation value of the eye movement precision.

[0028] The eye movement precision is used to reflect the precision of the user gaze point extraction and the concentration degree of the user attention. The eye movement precision is obtained by calculating the angle deviation δ between the precision test point and the user gaze point coordinates obtained by the gaze point extraction module. The calculation formula of the angle deviation δ is as follows:

[0029]

[0030] Wherein, (x, y) represents the position coordinates of the precision test point, represents the user gaze point coordinates obtained by the gaze point extraction module, Z represents the virtual screen depth of the display interface of the head-mounted eye movement interaction device, and W and H represent the number of pixel points in the horizontal and vertical directions of the display interface, respectively.

[0031] The application has the following beneficial effects:

[0032] The present application can be applicable to different users in the experience of eye movement interaction process, free from the cumbersome process of multi-point calibration and long waiting, and also can free from the calibration page designed by the developer, saving the storage and computing resources of the eye movement device. The eye movement offset extraction in the present application extracts the eye movement offset of adaptive individual difference through the preset animation in the user gaze interaction scene, and the offset directly affects the accuracy of the gaze point inferred by the eye movement algorithm, so it has very important significance for improving the robustness and generalization ability of the eye movement algorithm.

[0033] The present application renders the offset extraction identifier in the VR eye movement device according to the preset target position information, extracts the offset residual by the user briefly gazing at the identifier, and feeds back the residual to the algorithm for inference compensation, effectively improves the eye movement accuracy, and enables different users to use the eye movement device more conveniently and efficiently. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The flowchart of the user gaze point estimation method based on visual tracking of the present application;

[0035] Figure 2 The schematic diagram of the offset extraction identifier of the present application;

[0036] Figure 3 The schematic diagram of the angle between the optical axis and the visual axis of the eye of the present application;

[0037] Figure 4 The flowchart of the precision evaluation of the user gaze point estimation method of the present application;

[0038] Figure 5 The schematic diagram of the offset adaptive module of the present application. DETAILED DESCRIPTION

[0039] In order to better understand the content of the present application, an embodiment is given here.

[0040] In view of the problem that the existing eye tracking technology is strongly related to the physiological structure of the eyes of each user and the personalized use habit of the device, the present application discloses a user gaze point estimation and precision evaluation method based on visual tracking.

[0041] Figure 1 The flowchart of the user gaze point estimation method based on visual tracking of the present application; Figure 2 The schematic diagram of the offset extraction identifier of the present application; Figure 3 The schematic diagram of the angle between the optical axis and the visual axis of the eye of the present application; Figure 4 The flowchart of the precision evaluation of the user gaze point estimation method of the present application; Figure 5 The schematic diagram of the offset adaptive module of the present application.

[0042] The application discloses a user gaze point estimation method based on visual tracking, and specifically comprises the following steps:

[0043] The user wears a head-mounted eye movement interaction device, and the gaze point extraction module is used to obtain the gaze point coordinates of the user; the residual estimation module is used to calculate the residual of the eye movement offset of the user and the gaze point coordinates; and the obtained residual is sent to the offset adaptive module to update the gaze point coordinates, so that the final user gaze point estimation value is obtained.

[0044] The gaze point extraction module considers that the extracted gaze point has use value in a real scene, and needs to ensure the real-time performance of gaze point extraction and good gaze point accuracy, so that the gaze point extraction process is realized by using a deep learning method, a deep learning artificial neural network composed of multiple deep convolutional neural layers and multiple dilated convolution layers is used to realize the gaze point extraction process, the user's binocular image collected by the head-mounted eye movement interaction device is used as the input of the module, and the output of the module is the extracted gaze point coordinate value of the user.

[0045] The gaze point extraction module is used to obtain the gaze point coordinates of the user, first, a sample data set is constructed, then a deep learning artificial neural network is built, the deep learning artificial neural network is trained and tested, the trained deep learning artificial neural network is used as a gaze point extraction model, and the gaze point extraction model is used to obtain the gaze point coordinates of the user.

[0046] The construction of the sample data set considers that the physiological characteristics of different users are different, and the personalized use habits of the head-mounted eye movement interaction device affect the cross-subject performance of the model, so that a plurality of users wear the head-mounted eye movement interaction device, the users gaze at a target anchor point in the display interface of the device, the target anchor point moves to each pixel position of the display interface in a snake-like traversal manner, the target anchor point changes more than three different moving speeds during the movement, the head-mounted eye movement interaction device collects the eye images of the user gazing at the target anchor point moving constantly, and after the target anchor point completes a snake-like traversal, a round of sample data extraction is completed. Each user needs to perform more than 10 rounds of sample data extraction. In each round of sample data extraction, the near-eye high-speed camera mounted on the head-mounted eye movement interaction device saves the binocular image of the user at this moment and the position coordinate value of the target anchor point gazed at by the user when the target anchor point traverses to each pixel position on the display interface in a snake-like manner. The resolution of the binocular image of the user is 640*400, and the position coordinate value is the x and y values in the two-dimensional plane rectangular coordinate system. The binocular image of the user and the position coordinate value of the target anchor point gazed at by the user are used as the sample and label of the sample data set, so that the construction of the sample data set is completed.

[0047] The built deep learning artificial neural network first extracts the features of the left and right eye images of the user's two eye pictures by using a deep convolutional neural layer, the convolution kernel size of each convolution layer in the deep convolutional neural layer is 3*3, and the convolution step is 2. Since the dilated convolution has a larger receptive field, it can improve the efficiency of feature extraction, so three layers of dilated convolution layers are stacked after the deep convolutional neural layer, the convolution kernel size of the first layer of dilated convolution layer is 3*3, the expansion rate is (1, 2), the convolution kernel size of the second layer of dilated convolution layer is 3*3, the expansion rate is (2, 3), the convolution kernel size of the third layer of dilated convolution layer is 3*3, the expansion rate is (4, 5), and the convolution step of the three layers of dilated convolution layers is 1. The final output of the dilated convolution layer is deactivated, and the corresponding dropout function value is 0.1, so that the parameter amount of the deep learning artificial neural network is controlled, ensuring the real-time performance of the deep learning artificial neural network, using ReLU as the activation function, and normalizing the parameters of the deep learning artificial neural network before activation processing.

[0048] The built deep learning artificial neural network is trained and tested, considering that too large parameter amount will affect the real-time processing speed of the model, and to improve the effective extraction of sample features by the convolutional neural network, the sample data set is standardized in size and pixel distribution, the resolution of the user's two eye pictures of the sample data set is reduced to a set value, for example, from 640*400 to 128*192, all pixel values of the user's two eye pictures of the sample data set are divided by 256, so that the pixel value distribution is between 0 and 1, thereby realizing the normalization of pixel value, and then the standardization distribution processing is performed on all pixel value data of the user's two eye pictures of the sample data set with 0.5 as the mean and 0.5 as the variance. The data after standardization distribution processing is converted into tensor data using the PyTorch framework as the input of the deep learning artificial neural network, the parameters of the network are updated using the stochastic gradient descent algorithm, the parameters of the network are optimized using the Adam function, the sample data set is divided into a training set and a test set in a data amount ratio of 7:3 using the cross-validation method, the L1 norm loss function is used as the loss function of the network, and when the network is trained, the training times epoch value is set to 64, and the Adam function is used as the optimizer. The initial learning rate is 1.0e-3, and it decreases by 1 / 10 every 35 epochs. The total training model is 100 epochs. By iteratively training the built deep learning artificial neural network, the best set of network parameters of the training result is taken as the final parameters obtained by training the deep learning artificial neural network, thereby completing the training of the deep learning artificial neural network.

[0049] The gaze point coordinates extracted by the gaze point extraction module will have a relatively consistent personalized difference offset due to the physiological differences of different users and different use habits of the head-mounted eye movement interaction device, which will have a greater impact on the accuracy of the gaze point coordinates. The residual estimation module extracts the eye movement offset of the user after the user gaze offset extraction mark is extracted, and calculates the residual of the eye movement offset of the user and the gaze point coordinates using a first-order difference function.

[0050] The residual estimation module establishes a two-dimensional plane rectangular coordinate system in the display interface of the head-mounted eye movement interaction device, displays the offset extraction mark at the center position of the display interface, and the position coordinates are (x0, y0). The offset extraction mark is a static picture or an animation. The user wearing the head-mounted eye movement interaction device gazes at the offset extraction mark in the display interface, and the real-time gaze point coordinates of the user extracted by the gaze point extraction module in the i-th display interface are (x gi , y gi ), which is the eye movement offset of the user.

[0051] The frame rate of the display interface of the head-mounted eye movement interaction device is 30 fps, the display time of the above-mentioned offset extraction mark is set to one second, and the residual of the eye movement offset of the user and the gaze point coordinates is calculated using a first-order difference function [x d , y d ], and the calculation formula is:

[0052]

[0053] Where i is an integer from 0 to 29.

[0054] The eye movement offset of the user includes the user use habit offset and the angle between the eye visual axis and the eye optical axis. The user use habit offset is a fixed value, and the estimation process of the angle between the eye visual axis and the eye optical axis includes that the eye visual axis is the line from the offset extraction mark to the fovea of the eye, and the eye optical axis is the line from the center of the pupil of the eye to the center of the retina of the eye. The position coordinates of the pupil of the eye are represented by P, the coordinates of the corneal curvature center of the eye are represented by C, the direction vector of the eye visual axis is represented by V, the direction vector from the center of the pupil of the eye to the offset extraction mark, i.e. the direction vector of the eye visual axis, is represented by U, the angle between the eye visual axis and the eye optical axis is represented by e, and the direction vector of the eye optical axis is represented by W. The calculation formula is:

[0055]

[0056] Wherein, The calculation formula of the direction vector of the eye visual axis is:

[0057]

[0058] Wherein, (alpha, beta) represents the offset correction of the direction vector of the eye optical axis, the calculation formula of the eye visual axis direction vector is: Wherein T represents the position coordinates of the offset extraction mark, the calculation formula of the included angle e between the eye optical axis and the eye visual axis is e = arccos theta (U, V), so as to complete the estimation of the included angle between the eye visual axis and the eye optical axis.

[0059] The offset adaptive module, as shown in Figure 5 The offset adaptive module corrects the user gaze point coordinates obtained by the gaze point extraction module by using the residual error calculated by the residual error estimation module, and takes the corrected user gaze point coordinates as the final user gaze point estimation value.

[0060] The offset adaptive module is realized by a deep learning artificial neural network, which includes a plurality of deep convolutional neural layers, a plurality of dilated convolutional layers, an offset prediction branch and a full connection layer, and the four parts are connected in sequence, wherein the deep convolutional neural layer and the dilated convolutional layer adopt the same structure as the deep convolutional neural layer and the dilated convolutional layer in the gaze point extraction module, and the loss function L 1new used in the training process of the deep learning artificial neural network is expressed as L 1new =L1+lambda|b|, wherein L1 is the L1 norm loss function used in the gaze point extraction module, lambda|b| is a regularization term for adjusting the adaptive ability of the network, wherein lambda is an adjustment coefficient, and b is the residual error extracted in the gaze point extraction module. When lambda is larger, the influence of the residual error on the model is greater, when lambda is smaller, the influence of the residual error on the model is smaller, when lambda is 0, the model will degenerate into the backbone model used in the gaze point extraction module, and experiments prove that taking lambda as 0.01 in the offset adaptive module can obtain the output with higher gaze point precision.

[0061] The residual error between the eye movement offset of the user and the gaze point coordinates is calculated by the residual error estimation module, and the obtained residual error is sent to the offset adaptive module to update the gaze point coordinates, so as to obtain the final user gaze point estimation value.

[0062] The application also discloses a method for evaluating the precision of the user gaze point estimation method, which specifically comprises rendering a plurality of precision test points on the display interface of the head-mounted eye movement interaction device according to a preset position, and controlling the precision test points to be displayed and hidden in a certain time sequence, so that only one precision test point is displayed at each moment.

[0063] The user wears a head-mounted eye movement interaction device, gazes at precision test points displayed on a display interface in turn, acquires gaze point coordinates of the user by using a gaze point extraction module, and calculates and acquires eye movement precision;

[0064] For each precision test point, a plurality of eye movement precision values are calculated and averaged to obtain an eye movement precision value of the precision test point. The display interface is divided into a plurality of sub-regions, a plurality of precision test points are set on each sub-region, eye movement precision calculation and evaluation are performed for different sub-regions, the eye movement precision values obtained for all precision test points in each sub-region are averaged to obtain an eye movement precision value of the sub-region, and the eye movement precision values of all sub-regions are averaged to obtain an evaluation value of the final eye movement precision.

[0065] Eye movement precision is used to reflect the precision of user gaze point extraction and the degree of concentration of user attention, and is obtained by calculating an angle deviation δ between a precision test point and user gaze point coordinates acquired by the gaze point extraction module. The smaller the angle deviation is, the higher the precision is. The calculation formula of the angle deviation δ is:

[0066]

[0067] wherein (x, y) represents position coordinates of the precision test point, represents user gaze point coordinates acquired by the gaze point extraction module, Z represents a virtual screen depth of the display interface of the head-mounted eye movement interaction device, and W and H respectively represent the number of pixel points in the horizontal and vertical directions of the display interface.

[0068] The above only describes the embodiments of the present application and is not used to limit the present application. The present application can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for estimating a user's gaze point based on visual tracking, characterized in that, It specifically includes: The user wears a head-mounted eye movement interaction device, obtains the gaze point coordinates of the user by using a gaze point extraction module, calculates the residual error between the eye movement offset of the user and the gaze point coordinates by using a residual error estimation module, and then sends the obtained residual error to an offset adaptive module to update the gaze point coordinates to obtain the final user gaze point estimation value. The residual error estimation module extracts the eye movement offset of the user after the user gaze offset extraction identification, and uses a first-order difference function to calculate the residual error between the eye movement offset of the user and the gaze point coordinates. The residual error estimation module establishes a two-dimensional plane rectangular coordinate system in the display interface of the head-mounted eye movement interaction device, displays the offset extraction mark at the center position of the display interface, and the position coordinates are (x0, y0). The offset extraction mark is a static picture or an animation. A user wearing the head-mounted eye movement interaction device gazes at the offset extraction mark in the display interface, and the gaze point extraction module extracts the real-time gaze point coordinates of the user in the i-th appearing display interface as (x gi , y gi ), which is the eye movement offset of the user. The frame rate of the display interface of the head-mounted eye movement interaction device is 30 fps, the time for displaying the extracted offset mark is set to one second, and the residual error between the eye movement offset of the user and the gaze point coordinates is calculated using a first-order difference function [x d , y d ], and the calculation formula is: Where i is an integer from 0 to 29. The offset adaptive module is realized by a deep learning artificial neural network, the deep learning artificial neural network comprises a plurality of deep convolutional neural layers, a plurality of dilated convolutional layers, an offset prediction branch and a full connection layer, and the four parts are sequentially connected, and a loss function L used in the training process of the deep learning artificial neural network is 1new The expression of L is L 1new =L1+λ|b|, wherein L1 is an L1 norm loss function used in the gaze point extraction module, λ|b| is a regularization term, used for adjusting the adaptive ability of the network, wherein λ is an adjustment coefficient, and b is a residual error extracted in the gaze point extraction module.

2. The visual tracking based user eye point estimation method of claim 1, wherein, It specifically includes: The gaze point extraction module is realized by a deep learning artificial neural network composed of multiple deep convolutional neural layers and multiple dilated convolutional layers, and the user's binocular image collected by the head-mounted eye movement interaction device is used as the input of the module, and the output of the module is the extracted user's gaze point coordinate value. The gaze point extraction module is realized by a deep learning artificial neural network composed of multiple deep convolutional neural layers and multiple dilated convolutional layers, and the user's binocular image collected by the head-mounted eye movement interaction device is used as the input of the module, and the output of the module is the extracted user's gaze point coordinate value.

3. The visual tracking based user eye point estimation method of claim 2, wherein, It specifically includes: The construction of the sample data set requires a plurality of users to wear a head-mounted eye movement interaction device, and the user gazes at the target anchor point in the display interface of the device that moves constantly, the target anchor point moves to each row and column pixel position of the display interface in turn through a snake-shaped traversal manner, the target anchor point changes more than three different moving speeds during the movement, the head-mounted eye movement interaction device collects the eye images of the user gazing at the constantly moving target anchor point, and after the target anchor point completes a snake-shaped traversal, a round of sample data extraction is completed; during each round of sample data extraction, the near-eye high-speed camera carried on the head-mounted eye movement interaction device saves the user's binocular image and the position coordinate value of the target anchor point gazed at by the user at the moment when the target anchor point is snake-shaped traversed to each pixel position on the display interface, and the user's binocular image and the position coordinate value of the target anchor point gazed at by the user are used as the sample and label of the sample data set, thereby completing the construction of the sample data set.

4. The visual tracking based user eye point estimation method of claim 2, wherein, It specifically includes: The deep learning artificial neural network is built, first, the features of the left and right eye images of the user's two eye images are extracted by using a deep convolutional neural layer, the convolution kernel size of each convolution layer in the deep convolutional neural layer is 3*3, and the convolution step is 2; three expansion convolution layers are stacked after the deep convolutional neural layer, the convolution kernel size of the first expansion convolution layer is 3*3, the expansion rate is (1, 2), the convolution kernel size of the second expansion convolution layer is 3*3, the expansion rate is (2, 3), the convolution kernel size of the third expansion convolution layer is 3*3, the expansion rate is (4, 5), and the convolution step of the three expansion convolution layers is 1; the final output of the expansion convolution layer is deactivated to control the parameter amount of the deep learning artificial neural network, ensure the real-time performance of the deep learning artificial neural network, use ReLU as the activation function, and normalize the parameters of the deep learning artificial neural network before the activation process.

5. The visual tracking based user eye point estimation method of claim 2, wherein, It specifically includes: The built deep learning artificial neural network is trained and tested, the sample data set is standardized in size and pixel distribution, the resolution of the user's two eye images of the sample data set is reduced to a set value, all pixel values of the user's two eye images of the sample data set are divided by 256 to make the pixel value distribution between 0 and 1, so as to realize the normalization of the pixel value, then the standardization distribution processing is performed on all pixel value data of the user's two eye images of the sample data set with 0.5 as the mean value and 0.5 as the variance; the data after the standardization distribution processing is converted into tensor data by using the PyTorch framework, which is used as the input of the deep learning artificial neural network, the parameters of the network are updated by using the stochastic gradient descent algorithm, the parameters of the network are optimized by using the Adam function, the sample data set is divided into a training set and a test set in a data amount ratio of 7:3 by using the cross-validation method, the L1 norm loss function is used as the loss function of the network, and the Adam function is used as the optimizer when the network is trained; the best set of network parameters of the training result is taken as the final parameters obtained by training the deep learning artificial neural network, so as to complete the training of the deep learning artificial neural network.

6. A method of evaluating the precision of the user gaze point estimation method of any one of claims 1 to 5, characterized in that, On the display interface of the head-mounted eye movement interaction device, a plurality of precision test points are rendered in a predetermined position in sequence, and the precision test points are controlled to be displayed in sequence on the display interface and then hidden in sequence according to a certain time sequence, and only one precision test point is displayed at each moment; The user wears the head-mounted eye movement interaction device, gazes at the precision test points displayed on the display interface in sequence, acquires the gaze point coordinates of the user by using a gaze point extraction module, and calculates and acquires the eye movement precision; For each precision test point, a plurality of eye movement precision values are calculated and averaged as the eye movement precision value of the precision test point; the display interface is divided into a plurality of sub-regions, a plurality of precision test points are respectively set on each sub-region, eye movement precision calculation and evaluation are respectively performed for different sub-regions, the eye movement precision values obtained for all the precision test points in each sub-region are averaged as the eye movement precision value of the sub-region, and the eye movement precision values of all the sub-regions are averaged to obtain the final evaluation value of the eye movement precision.

7. The method of claim 6, wherein, The eye movement precision is used to reflect the precision of the gaze point extraction and the concentration degree of the user's attention, and the eye movement precision is obtained by calculating an angle deviation δ between a precision test point and a user gaze point coordinate obtained by the gaze point extraction module, and a calculation formula of the angle deviation δ is: Wherein, (x, y) represents the position coordinates of the precision test point, The user gaze point coordinates obtained by the gaze point extraction module, Z represents the virtual screen depth of the display interface of the head-mounted eye movement interaction device, and W and H represent the number of pixel points in the horizontal and vertical directions of the display interface, respectively.

Images