Eye movement trajectory generation method, device, equipment and readable storage medium

Abstract

The application discloses an eye movement track generation method, device and equipment and a readable storage medium. The method comprises the following steps: acquiring line-of-sight landing point data collected based on an eye tracker and an experiment duration; performing denoising processing on the line-of-sight landing point data through a sampling window, and determining saccade data and fixation data from the data points obtained after the denoising processing; determining a first time factor through a start time, an end time and the experiment duration of a fixation point, and determining a second time factor through a duration and the experiment duration; and generating eye movement track information based on the position of the fixation point, the first time factor and the second time factor. The application adopts a new eye movement track representation method, which not only contains duration and fixation position information, but also increases more specific time point information, can comprehensively express dynamic changes of visual attention with time, and constructs a new eye movement track model based on a conditional generative adversarial network, so that the model can accurately predict eye movement tracks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an eye movement trajectory generation method and device, equipment and a readable storage medium. BACKGROUND

[0002] Visual saliency map is currently the main way to represent human visual attention, and in recent years, more and more models using full convolutional network architecture to generate visual saliency map. Among them, the generation of visual saliency map cannot be separated from the analysis of user eye movement trajectory. The traditional eye movement trajectory prediction model is generally divided into two types, PathGAN model and SaltiNet model.

[0003] Specifically, the PathGAN model is an eye movement trajectory prediction model based on a generative adversarial network (GAN) architecture. The core idea is to use an adversarial learning mechanism to make the generator and the discriminator compete with each other during training, thereby generating realistic eye movement trajectories. The generator is responsible for simulating human eye movement behavior and outputting a series of fixation points and saccade paths to form a complete eye movement trajectory; the discriminator is responsible for judging whether the generated trajectory is real. Through multiple rounds of training, the generator gradually learns how to generate more realistic eye movement trajectories. However, due to the lack of dependent variables, the eye movement trajectories generated by the PathGAN model are too single and cannot fully simulate the diversified visual behavior of real users in different environments and situations.

[0004] The SaltiNet model uses a saliency volume on a 360° image to predict eye movement trajectories. This model uses a deep convolutional neural network (DCNN) to output the saliency volume of the image, and the network is trained using stochastic gradient descent and cross-entropy loss. The saliency volume is an increase in the time dimension based on the saliency map, and the saliency map can be generated by adding and normalizing all time slices. In addition, a time-weighted saliency map can be obtained by weighting all time slices. At the same time, the saliency volume can obtain the eye movement trajectory by sampling the fixation points in the time slices. However, this model relies on a large number of time slice accumulation and normalization when generating the saliency volume, which significantly increases the computational complexity, especially when dealing with high-resolution 360° images, which may cause high computational cost and memory consumption. For devices with limited resources or real-time application scenarios, it may be difficult to meet performance requirements. In addition, the model may not be adaptable when dealing with atypical user behavior or complex environments. Since its training is mainly based on regular fixation patterns, when encountering special observation tasks or extreme situations (such as rapid switching of visual lines, long-time fixation, etc.), the generated eye movement trajectories may lack diversity and accuracy.

[0005] The prior art cannot solve the technical problem that the traditional visual saliency model cannot fully represent individualized eye movement differences. SUMMARY

[0006] The present application provides an eye movement trajectory generation method, device, equipment and readable storage medium, which can solve the technical problem that the traditional visual saliency model cannot fully represent individualized eye movement differences in the prior art.

[0007] One aspect of the present application provides an eye movement trajectory generation method, which comprises: acquiring gaze landing point data collected based on an eye tracker and an experimental duration, wherein the gaze landing point data comprises a plurality of data points; performing denoising processing on the gaze landing point data through a sampling window, and determining saccade data and fixation data from the data points obtained after the denoising processing; wherein the fixation data comprises a start time, an end time, a duration and a position of a fixation point; determining a first time factor through the start time, the end time and the experimental duration of the fixation point, and determining a second time factor through the duration and the experimental duration; and generating eye movement trajectory information based on the position of the fixation point, the first time factor and the second time factor.

[0008] Optionally, the method further comprises: preparing an observation interface picture data set and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface picture data set is a batch of images composed of the gaze landing point data, and the observation interface picture data uniquely corresponds to the eye movement trajectory information; introducing a preset distance algorithm in an initial generative adversarial network to obtain an intermediate prediction model; iteratively training the intermediate prediction model with the observation interface picture data as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and using the trained eye movement trajectory prediction model to infer the observation interface picture data.

[0009] Optionally, the denoising processing of the gaze landing point data through the sampling window comprises: determining the length of the sampling window, taking the first gaze landing point data as a starting unit, framing the gaze landing point data of the corresponding length through the sampling window, and calculating the position mean value of the framed gaze landing point data; moving the position of the sampling window by one position, calculating the mean value of the framed gaze landing point data after the movement, repeating the step until the last data enters the sampling window to calculate the position mean value, and ending the denoising operation.

[0010] Optionally, the saccade data and the fixation data are determined from the data points obtained after the de-noising processing, including: calculating the data point speed of each sampling window through the position information and the collection time of the data points; processing the data point speed of each sampling window by using the difference method to obtain the data point acceleration; determining whether the data point acceleration is greater than the acceleration threshold and the data point does not belong to an isolated point, and if so, determining the data point as a saccade start point; taking the saccade start point as a reference, determining whether there is a target data point within a preset time interval from the saccade start point, wherein the data point acceleration of the target data point is less than the deceleration threshold; if so, determining the target data point as a saccade end point; and determining the fixation data point through the time interval between the saccade start point and the saccade end point.

[0011] Optionally, the method further includes: determining whether the time interval between the saccade start point and the saccade end point is less than a preset threshold; if so, regarding the data points between the saccade start point and the saccade end point as the saccade data, and merging the saccade start point, the saccade end point and the saccade data; and if not, determining the data points between the saccade start point and the saccade end point as the fixation data points.

[0012] Optionally, the first time factor is determined through the start time, the end time and the experimental duration of the fixation point, including: determining whether the fixation point is the first fixation point; if so, determining the start time of the fixation point and the experimental duration as the first time factor; and if not, calculating the difference between the start time of the fixation point and the start time of the previous fixation point, and taking the ratio of the difference to the experimental duration as the first time factor.

[0013] Another aspect of the present application provides a device for generating eye movement trajectory, the device comprising: an acquisition module configured to acquire line-of-sight landing point data collected based on an eye tracker and an experimental duration, wherein the line-of-sight landing point data comprises a plurality of data points; a de-noising module configured to de-noise the line-of-sight landing point data through a sampling window, and determine saccade data and fixation data from the data points obtained after the de-noising processing; wherein the fixation data comprises the start time, the end time, the duration and the position of the fixation point; a determination module configured to determine a first time factor through the start time, the end time and the experimental duration of the fixation point, and determine a second time factor through the duration and the experimental duration; and a generation module configured to generate eye movement trajectory information based on the position of the fixation point, the first time factor and the second time factor.

[0014] Optionally, the device further comprises a prediction module, specifically configured to: prepare an observation interface picture data set and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface data set is a batch of images containing gaze landing point data, and the observation interface picture data and the eye movement trajectory information correspond to each other; introduce a preset distance algorithm in an initial generative adversarial network to obtain an intermediate prediction model; iteratively train the intermediate prediction model with the observation interface picture data set as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and use the trained eye movement trajectory prediction model to infer the observation interface picture data.

[0015] Still another aspect of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the eye movement trajectory generation method of any of the above embodiments when executing the computer program.

[0016] Still another aspect of the present application provides a computer storage medium having a computer program stored thereon, the computer program being executable on a processor to implement the eye movement trajectory generation method of any of the above embodiments. Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0017] The present application can comprehensively express the dynamic changes of visual attention over time by converting the gaze landing point data into eye movement trajectory information containing gaze position and time information, and can adapt to data sets of different experimental durations by introducing time point information, thereby ensuring the accuracy of eye movement trajectory representation in different scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the application. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0019] Figure 1 An optional flowchart of the eye movement trajectory generation method provided by the embodiment one of the present application is shown;

[0020] Figure 2 A structural block diagram of the eye movement trajectory generation device provided by the embodiment two of the present application is shown; and

[0021] Figure 3A block diagram of a computer device suitable for implementing the eye movement trajectory generation method according to the third embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0023] It should be noted that, in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0024] Embodiment One

[0025] The present embodiment provides an eye movement trajectory generation method, Figure 1 A flowchart of the eye movement trajectory generation method is shown, as shown in Figure 1 The eye movement trajectory generation method can include steps S101 to S104, wherein:

[0026] Step S101, acquiring line-of-sight landing point data and experimental duration based on an eye tracker, wherein the line-of-sight landing point data includes a plurality of data points;

[0027] The line-of-sight landing point data can be saliency data collected from an image by an eye tracker.

[0028] Step S102, denoising the line-of-sight landing point data through a sampling window, and determining saccade data and fixation data from the data points obtained after denoising; wherein the fixation data includes the start time, end time, duration and position of the fixation point;

[0029] The denoising operation can be implemented by a moving average filter.

[0030] The saccade data can include the duration of saccade, the pixel distance of saccade, the visual angle distance of saccade, and the dirty eye movement data rate; the fixation data can include the duration of each fixation, the position of fixation (using the median position of the visual line landing point when fixation), the start time of fixation, the end time of fixation, and the variance of the visual line landing point when fixation. The saccade data and the fixation data are saved in the form of a JSON file, facilitating the effective calling of subsequent eye movement trajectory generation operations.

[0031] The visual line landing point data is filtered by a moving average filter, which excludes invalid data interference on the generation of the eye movement trajectory, and selects saccade data and fixation data from the denoised data points, thereby improving the accuracy of the eye movement trajectory generation.

[0032] In step S103, a first time factor is determined by the start time of the fixation point, the end time and the experimental duration, and a second time factor is determined by the duration and the experimental duration;

[0033] The first time factor is the ratio of the fixation start time to the experimental duration, and the second time factor is the ratio of the fixation duration to the experimental duration. The fixation point is represented in multiple dimensions by different types of time point information, which can adapt to different experimental duration data sets and ensure the accuracy of the fixation order.

[0034] In step S104, the eye movement trajectory information is generated based on the position of the fixation point, the first time factor and the second time factor.

[0035] The eye movement trajectory is represented in the form of [x', y', t1, t2], where x' and y' are the relative positions of the fixation point in the image, t1 is the ratio of the fixation start time to the total experimental time (first time factor), and t2 is the fixation duration ratio (second time factor). By the position of the fixation point, combined with the multi-dimensional time point information, more comprehensive eye movement information is provided to ensure that the spatial information and time information of the landing point can be obtained simultaneously, thereby more accurately representing the dynamic changes of human visual attention.

[0036] Preferably, the method further comprises steps A1 to A4, wherein:

[0037] In step A1, an observation interface picture data set and eye movement trajectory information for eye movement trajectory prediction are prepared, wherein the observation interface picture data set is a batch of images containing visual line landing point data, and the observation interface picture data and the eye movement trajectory information correspond uniquely;

[0038] In step A2, a preset distance algorithm is introduced into the initial generative adversarial network to obtain an intermediate prediction model;

[0039] The initial generative adversarial network can be a cGAN, which is a variant of GAN, introducing a conditional variable y to condition the data generation process. The cGAN includes a generator and a discriminator. The generator uses a combination of VGG to extract spatial features and LSTM to process temporal sequence information to generate complete eye movement trajectories. The discriminator is used to determine whether the generated eye movement trajectory is consistent with the real trajectory. The loss function uses a generative adversarial loss (GAN Loss) and a trajectory feature matching loss to ensure the authenticity and diversity of the generated trajectory. The saliency dataset is used for supervised training, and the generation effect is improved by optimizing the adversarial process of the generator and the discriminator.

[0040] The preset distance algorithm can be the Jarodzka distance algorithm

[0041] Step A3, iteratively train the intermediate prediction model with the observation interface picture data set as input information and eye movement trajectory information as output information until the intermediate prediction model reaches a convergent state, obtaining an eye movement trajectory prediction model.

[0042] Step A4, using the trained eye movement trajectory prediction model to infer the observation interface picture data.

[0043] The eye movement trajectory prediction model is explained and described by the following example, which does not specifically limit the eye movement trajectory prediction model.

[0044] The model is trained using an RTX 3090 with 24G graphics memory, a CPU using an intel (R) Xeon (R) Gold 5218R with a frequency of 2.10GHz and 10 cores, and a memory of 64GB. The programming environment is python3.8, the deep learning framework is tensorflow2.5.0, the parallel computing architecture is cuda11.2, and the neural network acceleration library is cudnn. Here, opencv-python=4.5.5.60 is installed, which can use opencv in python to read, display, process and store images. At the same time, scipy=1.8.0 is installed to read mat type data. Panda=1.3.4 is installed to process structured data, numpy=1.19.5 is used for matrix operation. Pydot=1.4.2 is installed to print the model graph.

[0045] The final noise z adopts a normal distribution with a size of 110, a mean of 0, and a variance of 1. The image is added with noise of a normal distribution with a mean of 0 and a variance of 4, and the eye movement trajectory is added with noise of a normal distribution with a mean of 0 and a variance of 0.1. When the model selects the cGAN loss function, the one-sided label of the discriminator is smoothed, and the label smoothing coefficient is 0.1. Only the label of the real eye movement trajectory is smoothed, and the label of the generated eye movement trajectory is not processed. In this way, the discriminator can produce smooth predictions, and the generator's error behavior can be weakened. The learning rate of the generator is 1x10-4, and the learning rate of the discriminator is 8.3x10-6. The batchsize size is 32. The generator uses 10 3x3 convolution kernels with a step size of 2x2, and the activation function is linear, which is used to reduce the size of image features and reduce the complexity of the model. The output dimension of the generator's LSTM is 30, and the dropout size of the LSTM is 0.1, which is used to process sequences and obtain sequence outputs. The purpose of using noise and dropout measures is to improve the generalization performance of the model and prevent the model from overfitting. The discriminator is used to process the eye movement trajectory with an LSTM output dimension of 55. The convolution layer uses 5 3x3 convolution kernels with a step size of 2x2, and the activation function is linear. The output dimensions of the last two LSTM layers are 10 and 1, respectively. The nonlinear activation function uses leaky_relu, and the slope size of the part less than 0 is 0.3. The model uses the RMSprop optimizer. The epoch size is 24, and when the loss L^2 and cGAN loss are used for model training, the generator only uses the L^2 loss in the first epoch. In this way, the generator can obtain better performance faster and prevent the generator from being too weak in the early training stage, causing the training to be too slow. After 1 epoch, the generator uses the cGAN loss and the L^2 loss for joint training, and the weight of the L^2 loss is 1.

[0046] It should be particularly noted that in addition to using the generative adversarial network (cGAN), the following technical solutions can also be used to realize the prediction of the eye movement trajectory:

[0047] Reinforcement learning (RL): Reinforcement learning learns complex strategies through trial and error, and is suitable for generating eye movement trajectories. It considers trajectory prediction as a sequence decision problem. In this scheme, the state includes the current gaze position and image features, the action is to predict the position of the next fixation point, and the reward is based on the similarity score of the trajectory and the real data. RL can adaptively optimize the trajectory generation in complex environments, and is particularly suitable for dynamic scenarios or online prediction tasks.

[0048] Variational Autoencoder (VAE) + Recurrent Neural Network (RNN): Variational Autoencoder (VAE) is good at generating diverse data, combined with Recurrent Neural Network (RNN), it can generate eye movement trajectories with time sequence information. This scheme first uses VAE to extract and compress image features into latent vectors, and then generates time series of eye movement trajectories through RNN (such as LSTM or GRU).

[0049] Transformer model: Transformer performs well in processing long sequences and can effectively capture long-time dependencies, so it is more suitable for generating eye movement trajectories than RNN. By inputting the encoded feature sequence of the input image into the Transformer encoder, a complete trajectory sequence can be generated, and the gaze point and duration can be predicted using the self-attention mechanism. This model does not depend on sequence order and is particularly suitable for generating long-span eye movement trajectories, especially for predicting long-time fixation behavior.

[0050] Preferably, step S102 can include steps S1021 to S1022, wherein:

[0051] Step S1021, determine the length of the sampling window, take the first gaze point data as the starting unit, frame the gaze point data of the corresponding length through the sampling window, and calculate the position mean value of the framed gaze point data;

[0052] The length of the sampling window is set by the actual scene requirement, which is not limited here.

[0053] Step S1022, move the position of the sampling window one bit backward, and calculate the mean value of the framed gaze point data after moving, repeat the step until the last data enters the sampling window to calculate the position mean value, and the denoising operation ends.

[0054] Determine the length of the sampling frame, put the sampling data into the sampling frame, and calculate the mean value of the data in the sampling frame to get the filtered data. Then move the sampling data one bit backward without moving the sampling frame, and calculate the mean value of the data in the sampling frame. Filter the entire data in turn until the last data is filtered. When processing signals, moving average filtering is commonly used for denoising.

[0055] It should be particularly noted that before moving average filtering, it is necessary to determine whether the NaN proportion of the gaze point data is less than a threshold value, if so, the denoising process can be entered, otherwise, it means that the group of gaze point data is unusable, and the group of gaze point data is deleted. This step avoids additional processing of invalid data and improves the efficiency of eye movement trajectory generation.

[0056] Preferably, step S102 can also include steps B1 to B6, wherein:

[0057] Step B1, calculating the data point speed of each sampling window by the position information and the collection time of the data point;

[0058] The data point speed can be obtained by dividing the difference of the position coordinates of the data point by the difference of the collection time nodes of the data point. The average value of the data point speed in each sampling window is calculated.

[0059] Step B2, processing the data point speed of each sampling window by the difference method to obtain the data point acceleration;

[0060] The data point acceleration is determined by the ratio of the difference of the average value of the data point speed to the difference of the collection time nodes.

[0061] Step B3, judging whether the data point acceleration is greater than the acceleration threshold value and the data point does not belong to isolated points, if yes, determining the data point as the saccade start point;

[0062] The data point acceleration greater than the acceleration threshold value represents the start of saccade. In particular, since there is also a micro-saccade and other eye movement processes during fixation, the process is represented by isolated data points, and the acceleration of the data points may also be greater than the acceleration threshold value. The data points do not belong to saccade data and need to be removed.

[0063] Step B4, taking the saccade start point as a reference, judging whether there is a target data point within a preset time interval from the saccade start point, wherein the data point acceleration of the target data point is less than the deceleration threshold value;

[0064] The data point acceleration less than the deceleration threshold value represents the end of saccade. The preset time interval can be 25 gaze point time intervals.

[0065] Step B5, if yes, determining the target data point as the saccade end point;

[0066] If no, it is considered that the data points greater than the acceleration threshold value belong to fixation data.

[0067] Step B6, determining the fixation data point by the time interval between the saccade start point and the saccade end point.

[0068] Through the above operations, the data point at the start of each saccade and the data point at the end of each saccade can be obtained, and the data points between two saccades are fixation data points.

[0069] Preferably, step B6 can further include steps B61 to B63, wherein:

[0070] Step B61, judging whether the time interval between the saccade start point and the saccade end point is less than a preset threshold value;

[0071] Step B62, if yes, merging the data points between the saccade start point and the saccade end point as saccade data, and the saccade start point, the saccade end point and the saccade data.

[0072] Step B63, if no, determining the data points between the saccade start point and the saccade end point as fixation data points.

[0073] If the fixation time is less than the set sampling number (preset threshold), the fixation data is essentially saccade data, only the speed is reduced due to noise and other reasons, so the fixation and the saccade data points on both sides need to be merged.

[0074] Preferably, step S103 can include steps S1031 to S1033, wherein:

[0075] Step S1031, determining whether the fixation point is the first fixation point;

[0076] Step S1032, if yes, determining the first time factor as the ratio of the start time of the fixation point to the experimental duration;

[0077] Step S1033, if no, calculating the difference between the start time of the fixation point and the start time of the previous fixation point, and determining the first time factor as the ratio of the difference to the experimental duration.

[0078] For the first fixation point, the first time factor is obtained by dividing the start time of the fixation point by the total experimental time, and for other fixation points, the first time factor is obtained by dividing the difference between the start time of the current fixation point and the end time of the previous fixation point by the total experimental time, and the second time factor is obtained by dividing the fixation duration by the total experimental time.

[0079] The setting has the following advantages: first, generally, the last sigmoid function of the deep learning model normalizes the results to [0, 1], so the value of the fixation point representation is also in [0, 1], which simplifies the subsequent processing process.

[0080] Second, since there is no uniform standard for the experimental time of the observation interface picture data set, which leads to different sampling times of different data sets, therefore, the length of the eye movement trajectory generated by the same picture will also be different due to the experimental time. Based on this design, the model trained using the data set with a longer experimental time has a larger time range for predicting the eye movement trajectory, and the eye movement trajectory of the data set with a shorter experimental time only needs to be cut off in the corresponding time period, so that the generated trajectory has more individual differences and diversity.

[0081] Third, the setting of the first time factor t1 and the second time factor t2 ensures that the end time of a certain gaze is greater than the start time of the gaze, and the start time of the gaze point can be obtained by adding the t1 and t2 of the previous gaze point and the t1 of the gaze point, and the end time of the gaze point can also be obtained by adding the t1 and t2 of the previous gaze point and the t2 of the gaze point, ensuring the accuracy of mutual reasoning of data information.

[0082] The embodiment can comprehensively express the dynamic changes of visual attention over time by converting the gaze landing point data into eye movement trajectory information containing gaze position and time information, and can adapt to data sets of different experimental durations by introducing time point information, and ensure the accuracy of eye movement trajectory representation in different scenes.

[0083] Embodiment Two

[0084] Embodiment Two of the application also provides an eye movement trajectory generation device, which corresponds to the eye movement trajectory generation method provided in Embodiment One, and the corresponding technical features and technical effects will not be described in detail in this embodiment, and the relevant parts can be referred to in Embodiment One. Specifically, Figure 2 The structure block diagram of the eye movement trajectory generation device is shown. As Figure 2 The eye movement trajectory generation device 200 includes an acquisition module 201, a denoising module 202, a determination module 203 and a generation module 204, wherein:

[0085] The acquisition module 201 is configured to acquire gaze landing point data collected based on an eye tracker and an experimental duration, wherein the gaze landing point data includes a plurality of data points;

[0086] The denoising module 202 is connected with the acquisition module 201, and is configured to perform denoising processing on the gaze landing point data through a sampling window, and determine saccade data and gaze data from the data points obtained after the denoising processing; wherein the gaze data includes the start time, the end time, the duration and the position of the gaze point;

[0087] The determination module 203 is connected with the denoising module 202, and is configured to determine a first time factor through the start time, the end time and the experimental duration of the gaze point, and determine a second time factor through the duration and the experimental duration;

[0088] The generation module 204 is connected with the determination module 203, and is configured to generate eye movement trajectory information based on the position of the gaze point, the first time factor and the second time factor.

[0089] Optionally, the apparatus further comprises a prediction module, specifically configured to: prepare an observation interface picture data set and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface picture data set is a batch of images composed of gaze landing point data, and the observation interface picture data uniquely corresponds to the eye movement trajectory information; introduce a preset distance algorithm in an initial generative adversarial network to obtain an intermediate prediction model; iteratively train the intermediate prediction model by taking the saliency data set as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and perform inference on the observation interface picture data by using the trained eye movement trajectory prediction model.

[0090] Optionally, the denoising module is further configured to: determine the length of the sampling window, take the first gaze landing point data as the starting unit, frame the gaze landing point data of the corresponding length through the sampling window, and calculate the position mean value of the framed gaze landing point data; move the position of the sampling window by one bit, calculate the mean value of the gaze landing line data framed after the movement, and repeat the step until the last data enters the sampling window to calculate the position mean value, and the denoising operation ends.

[0091] Optionally, the denoising module is further configured to: calculate the data point speed of each sampling window through the position information and the collection time of the data points; process the data point speed of each sampling window by using the difference method to obtain the data point acceleration; determine whether the data point acceleration is greater than an acceleration threshold and the data point does not belong to an isolated point, and if so, determine the data point as a saccade start point; determine whether there is a target data point within a preset time interval from the saccade start point based on the saccade start point, wherein the data point acceleration of the target data point is less than a deceleration threshold; if so, determine the target data point as a saccade end point; and determine a fixation data point through the time interval between the saccade start point and the saccade end point.

[0092] Optionally, the apparatus further comprises a merging module configured to: determine whether the time interval between the saccade start point and the saccade end point is less than a preset threshold; if so, take the data points between the saccade start point and the saccade end point as saccade data, and merge the saccade start point, the saccade end point and the saccade data; and if not, determine the data points between the saccade start point and the saccade end point as fixation data points.

[0093] Optionally, the determination module is further configured to: determine whether the fixation point is the first fixation point; if so, determine the start time of the fixation point and the experimental duration as a first time factor; and if not, calculate the difference between the start time of the fixation point and the start time of the previous fixation point, and take the ratio of the difference to the experimental duration as the first time factor.

[0094] Embodiment three

[0095] Figure 3A block diagram of a computer device suitable for implementing the method for generating eye movement trajectory according to the third embodiment of the present application is shown. In this embodiment, the computer device 301 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack-mounted server (including a single server or a server cluster composed of multiple servers), etc. that executes programs. As shown in the figure, Figure 3 The computer device 301 according to this embodiment at least includes, but is not limited to, a memory 301, a processor 302, and a network interface 303 that are communicatively connected to each other through a system bus. It should be noted that, Figure 3 Only the computer device 301 with components 301-303 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0096] In this embodiment, the memory 301 includes at least one type of computer readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 301 can be an internal storage unit of the computer device 301, such as a hard disk or a memory of the computer device 301. In other embodiments, the memory 301 can also be an external storage device of the computer device 301, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 301. Of course, the memory 301 can include both the internal storage unit and the external storage device of the computer device 301. In this embodiment, the memory 301 is generally used to store an operating system and various application software installed on the computer device 301, such as program codes of the method for generating eye movement trajectory, etc.

[0097] The processor 302 in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 302 is generally used to control the overall operation of the computer device 301. For example, it performs control and processing related to data interaction or communication of the computer device 301, etc. In this embodiment, the processor 302 is used to run the program codes of the steps of the method for generating eye movement trajectory stored in the memory 301.

[0098] In the embodiment, the method for generating eye movement trajectory stored in the memory 301 can also be divided into one or more program modules and executed by one or more processors (the processor 302 in the embodiment) to complete the application.

[0099] The network interface 303 can include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 301 and other computer devices. For example, the network interface 303 is used to connect the computer device 301 with an external terminal through a network, establish a data transmission channel and a communication link between the computer device 301 and the external terminal, and the like. The network can be an intranet, the Internet, a global system of mobile communication (GSM), a wideband code division multiple access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, and the like wireless or wired network.

[0100] Embodiment four

[0101] The embodiment also provides a computer readable storage medium, including a flash memory, a hard disk, a multimedia card, a card type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, and the like, in which a computer program is stored, and the computer program is executed by a processor to implement the steps of the method for generating eye movement trajectory.

[0102] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and alternatively, each module or each step can be realized by program code executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an individual integrated circuit module or a plurality of modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the application are not limited to any particular combination of hardware and software.

[0103] It should be noted that the serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. It should be noted that the serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment.

[0105] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating eye-tracking trajectories, characterized in that, The method includes: Acquire eye-tracking data and experiment duration based on eye tracker-collected gaze point data, wherein the gaze point data includes multiple data points; The gaze point data is denoised through a sampling window, and saccade data and fixation data are determined from the denoised data points; wherein, the fixation data includes the start time, end time, duration and location of the fixation point; A first time factor is determined by the start time of the fixation point, the end time, and the duration of the experiment; a second time factor is determined by the duration and the duration of the experiment. Eye movement trajectory information is generated based on the position of the gaze point, the first time factor, and the second time factor.

2. The method according to claim 1, characterized in that, The method further includes: Prepare an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction. The observation interface image dataset consists of batch images containing gaze point data, and the observation interface image data and eye movement trajectory information are uniquely correlated. A pre-defined distance algorithm is introduced into the initial generative adversarial network to obtain an intermediate prediction model; The intermediate prediction model is iteratively trained using the observation interface image dataset as input information and eye movement trajectory information as output information until the intermediate prediction model reaches a convergent state, thus obtaining the eye movement trajectory prediction model. The trained eye-tracking prediction model is used to infer the data from the observed interface images.

3. The method according to claim 1, characterized in that, The noise reduction process for the line-of-sight data via a sampling window includes: Determine the length of the sampling window, take the first line-of-sight data point as the starting unit, select line-of-sight data points of the corresponding length through the sampling window, and calculate the position average of the selected line-of-sight data points. Move the sampling window one position to the right and calculate the average value of the line-of-sight data selected after the move. Repeat this step until the last data point enters the sampling window and the position average value is calculated. The noise reduction operation ends.

4. The method according to claim 1, characterized in that, The process of determining saccade data and fixation data from the data points obtained after denoising includes: The velocity of each sampling window is calculated using the location information and acquisition time of the data points. The acceleration of the data points in each sampling window is obtained by processing the velocity of the data points using the difference method. Determine whether the acceleration of the data point is greater than the acceleration threshold and whether the data point is not an isolated point. If so, determine the data point as the start point of eye saccade. Using the saccade start point as a reference, determine whether there is a target data point within a preset time interval from the saccade start point, wherein the data point acceleration of the target data point is less than a deceleration threshold; If so, the target data point is designated as the end point of the saccade. Fixation data points are determined by the time interval between the start and end points of the saccade.

5. The method according to claim 4, characterized in that, The method further includes: Determine whether the time interval between the start point and end point of the saccade is less than a preset threshold. If so, the data points between the start point and the end point of the saccade are taken as saccade data, and the start point, the end point, and the saccade data are merged. If not, the data points between the saccade start point and the saccade end point are determined as fixation data points.

6. The method according to claim 1, characterized in that, The determination of the first time factor based on the start time of the fixation point, the end time, and the experiment duration includes: Determine whether the fixation point is the first fixation point; If so, then the start time of the fixation point and the duration of the experiment are determined as the first time factor; If not, calculate the difference between the start time of the fixation point and the start time of the previous fixation point, and use the ratio of the difference to the duration of the experiment as the first time factor.

7. An eye-tracking trajectory generation device, characterized in that, The device includes: The acquisition module is used to acquire gaze point data and experiment duration based on eye tracker data, wherein the gaze point data includes multiple data points; The denoising module is used to denoise the gaze point data through a sampling window and determine saccade data and fixation data from the data points obtained after denoising; wherein, the fixation data includes the start time, end time, duration and position of the fixation point; The determination module is used to determine a first time factor by the start time of the fixation point, the end time, and the duration of the experiment, and to determine a second time factor by the duration and the duration of the experiment; The generation module is used to generate eye movement trajectory information based on the position of the gaze point, the first time factor, and the second time factor.

8. The apparatus according to claim 7, characterized in that, The device further includes a prediction module, specifically used for: Prepare an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction. The observation interface image dataset consists of batch images containing gaze point data, and the observation interface image data and eye movement trajectory information are uniquely correlated. A pre-defined distance algorithm is introduced into the initial generative adversarial network to obtain an intermediate prediction model; The intermediate prediction model is iteratively trained using the observation interface image dataset as input information and eye movement trajectory information as output information until the intermediate prediction model reaches a convergent state, thus obtaining the eye movement trajectory prediction model. The trained eye-tracking prediction model is used to infer the data from the observed interface images.

9. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images