Eye movement track generation method and device, equipment and readable storage medium

Abstract

The invention discloses an eye movement track generation method, apparatus and device, and a readable storage medium. The method comprises the steps of obtaining sight line drop point data and experiment duration acquired based on an eye tracker; performing denoising processing on the sight line drop point data through a sampling window, and determining eye jump data and fixation data from data points obtained after denoising processing; determining a first time factor according to the starting time, the ending time and the experiment duration of the fixation point, and determining a second time factor according to the duration and the experiment duration; eye movement trajectory information is generated based on the position of the fixation point, the first time factor, and the second time factor. According to the method, a new eye movement track representation mode is adopted, the mode not only contains information of duration and a gazing position, but also increases more specific time point information, dynamic changes of visual attention along with time can be comprehensively expressed, a new eye movement track model is constructed based on the conditional generative adversarial network, and the visual attention can be more accurately expressed. The model can accurately predict the eye movement trajectory.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and readable storage medium for generating eye movement trajectories. Background Art

[0002] Visual saliency maps are currently the main way to characterize human visual attention. In recent years, more and more models have been developed using fully convolutional network architectures to generate visual saliency maps. The generation of visual saliency maps is inseparable from the analysis of user eye movement trajectories. Traditional eye movement trajectory prediction models are generally divided into two types: PathGAN model and SaltiNet model.

[0003] Specifically, the PathGAN model is an eye movement trajectory prediction model based on the generative adversarial network (GAN) architecture. The core idea is to use an adversarial learning mechanism to allow the generator and the discriminator to compete with each other during the training process to generate realistic eye movement trajectories. The generator is responsible for simulating human eye movement behavior, outputting a series of fixations and eye saccade paths to form a complete eye movement trajectory; the discriminator is responsible for judging whether the generated trajectory is real. Through multiple rounds of training, the generator gradually learns how to generate eye movement trajectories that are closer to reality. However, due to the lack of dependent variables, the eye movement trajectories generated by the PathGAN model are too single and cannot fully simulate the diverse visual behaviors of real users in different environments and situations.

[0004] The SaltiNet model uses saliency volumes to predict eye movement trajectories on 360° images. The model uses a deep convolutional neural network (DCNN) to output the saliency volume of the image. The network is trained using stochastic gradient descent and cross entropy loss. The saliency volume adds a time dimension to the saliency map. The saliency map can be generated by adding and normalizing all time slices. In addition, a time-weighted saliency map can be obtained by weighting all time slices. At the same time, the saliency volume can obtain eye movement trajectories by sampling fixation points in the time slice. However, the model relies on a large number of time slice accumulation and normalization when generating saliency volumes, which significantly increases the computational complexity, especially when processing high-resolution 360° images, which may cause high computational costs and memory consumption. This may be difficult to meet performance requirements for devices with limited resources or real-time application scenarios. In addition, the model may show insufficient adaptability when dealing with atypical user behaviors or complex environments. Since its training is mainly based on conventional gaze patterns, the generated eye movement trajectories may lack diversity and accuracy when encountering special observation tasks or extreme situations (such as rapid switching of sight, long-term gaze, etc.).

[0005] There is currently no effective solution to the technical problem that traditional visual saliency models in the existing technology cannot fully represent individual eye movement differences. Summary of the invention

[0006] The purpose of the present invention is to provide a method, device, equipment and readable storage medium for generating eye movement trajectories, which can solve the technical problem in the prior art that traditional visual saliency models cannot fully represent individualized eye movement differences.

[0007] One aspect of the present invention provides a method for generating an eye movement trajectory, the method comprising: obtaining gaze point data and an experiment duration collected by an eye tracker, wherein the gaze point data comprises a plurality of data points; denoising the gaze point data through a sampling window, and determining saccade data and fixation data from the data points obtained after the denoising process; wherein the fixation data comprises the start time, end time, duration and position of the fixation point; determining a first time factor through the start time, end time and experiment duration of the fixation point, and determining a second time factor through the duration and experiment duration; generating eye movement trajectory information based on the position of the fixation point, the first time factor and the second time factor.

[0008] Optionally, the method also includes: preparing an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface image dataset is a batch of images consisting of gaze point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; introducing a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; iteratively training the intermediate prediction model using the observation interface image data as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and using the trained eye movement trajectory prediction model to infer the observation interface image data.

[0009] Optionally, the sight point data is denoised through a sampling window, including: determining the length of the sampling window, taking the first sight point data as the starting unit, selecting the sight point data of corresponding length through the sampling window, and calculating the position mean of the selected sight point data; moving the position of the sampling window backward by one position, and calculating the mean of the sight line data selected after the movement, repeating this step until the last bit of data enters the sampling window to calculate the position mean, and the denoising operation is completed.

[0010] Optionally, determining saccade data and fixation data from the data points obtained after denoising, includes: calculating the data point speed of each sampling window through the position information and acquisition time of the data points; processing the data point speed of each sampling window using a difference method to obtain the data point acceleration; judging whether the data point acceleration is greater than an acceleration threshold and the data point is not an isolated point, and if so, determining the data point as the saccade start point; taking the saccade start point as a reference, judging whether there is a target data point within a preset time interval from the saccade start point, wherein the data point acceleration of the target data point is less than a deceleration threshold; if so, determining the target data point as the saccade end point; and determining the fixation data point through the time interval between the saccade start point and the saccade end point.

[0011] Optionally, the method also includes: determining whether the time interval between the start point and the end point of the saccade is less than a preset threshold; if so, taking the data points between the start point and the end point of the saccade as saccade data, and merging the saccade start point, the saccade end point and the saccade data; if not, determining the data points between the start point and the end point of the saccade as fixation data points.

[0012] Optionally, the first time factor is determined by the start time, end time and experiment duration of the fixation point, including: judging whether the fixation point is the first fixation point; if so, determining the start time of the fixation point and the experiment duration as the first time factors; if not, calculating the difference between the start time of the fixation point and the start time of the previous fixation point, and taking the ratio of the difference to the experiment duration as the first time factor.

[0013] Another aspect of the present invention provides an eye movement trajectory generation device, which includes: an acquisition module, used to obtain gaze point data and experiment duration collected based on an eye tracker, wherein the gaze point data includes multiple data points; a denoising module, used to denoise the gaze point data through a sampling window, and determine eye movement data and fixation data from the data points obtained after the denoising process; wherein the fixation data includes the start time, end time, duration and position of the fixation point; a determination module, used to determine a first time factor through the start time, end time and experiment duration of the fixation point, and determine a second time factor through the duration and experiment duration; a generation module, used to generate eye movement trajectory information based on the position of the fixation point, the first time factor and the second time factor.

[0014] Optionally, the device also includes a prediction module, which is specifically used to: prepare an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface dataset is a batch of images containing gaze landing point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; introduce a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; iteratively train the intermediate prediction model using the observation interface image dataset as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and use the trained eye movement trajectory prediction model to infer the observation interface image data.

[0015] Another aspect of the present invention provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the eye movement trajectory generation method of any of the above-mentioned embodiments when executing the computer program.

[0016] Another aspect of the present invention provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating an eye movement trajectory of any of the above embodiments is implemented. Further, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of a blockchain node, etc.

[0017] The present invention can fully express the dynamic changes of visual attention over time by converting gaze point data into eye movement trajectory information containing gaze position and time information. The introduction of time point information can adapt to data sets of different experimental durations and ensure the accuracy of eye movement trajectory representation in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 An optional flow chart of the method for generating eye movement trajectories provided in the first embodiment of the present invention is shown; Figure 2 A structural block diagram of an eye movement trajectory generation device provided by Embodiment 2 of the present invention is shown; and Figure 3 A block diagram of a computer device suitable for implementing an eye movement trajectory generation method provided in a third embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0021] Embodiment 1 This embodiment provides a method for generating an eye movement trajectory. Figure 1 A flow chart of the eye movement trajectory generation method is shown in FIG. Figure 1 As shown, the eye movement trajectory generation method may include steps S101 to S104, wherein: Step S101, obtaining gaze point data and experiment duration collected by an eye tracker, wherein the gaze point data includes a plurality of data points; The gaze point data may be saliency data collected from an image by an eye tracker.

[0022] Step S102, denoising the gaze point data through a sampling window, and determining saccade data and fixation data from the data points obtained after the denoising process; wherein the fixation data includes the start time, end time, duration and position of the fixation point; De-noising can be achieved using a moving average filter.

[0023] Saccade data can include information such as saccade duration, pixel distance of saccade, visual angle distance of saccade, and dirty eye movement data rate; fixation data can include the duration of each fixation, fixation position (using the median position of the gaze point during fixation), fixation start time, fixation end time, and the variance of the gaze point during fixation. Saccade data and fixation data are saved in the form of JSON files to facilitate the effective call of subsequent eye movement trajectory generation operations.

[0024] The gaze point data is filtered by a moving average filter to eliminate the interference of invalid data on the generation of eye movement trajectories. At the same time, the saccade data and fixation data are selected from the denoised data points to improve the accuracy of eye movement trajectory generation.

[0025] Step S103, determining a first time factor according to the start time of the fixation point, the end time and the experiment duration, and determining a second time factor according to the duration and the experiment duration; The first time factor is the ratio of the fixation start time to the experimental duration, and the second time factor is the ratio of the fixation duration to the experimental duration. The multi-dimensional representation of fixation points through different types of time point information can adapt to data sets of different experimental durations and ensure the accuracy of the fixation order.

[0026] Step S104: Generate eye movement trajectory information based on the position of the gaze point, the first time factor, and the second time factor.

[0027] The eye movement trajectory is expressed in the form of [x′, y′, t1, t2], where x' and y' are the relative positions of the fixation point in the image, t1 is the ratio of the fixation start time to the total experimental time (the first time factor), and t2 is the ratio of the fixation duration (the second time factor). The position of the fixation point combined with multi-dimensional time point information provides a more comprehensive guarantee of eye movement information, and can simultaneously obtain the spatial information and time information of the landing point, thereby more accurately representing the dynamic changes of human visual attention.

[0028] Preferably, the method further comprises steps A1 to A4, wherein: Step A1, preparing an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface image dataset is a batch of images including sight point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; Step A2, introducing a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; The initial generative adversarial network can be cGAN, which is a variant of GAN. It introduces the conditional variable y to add conditions to the model, which can guide the data generation process. cGAN includes a generator and a discriminator. The generator uses VGG to extract spatial features and LSTM to process time series information to generate a complete eye movement trajectory. The discriminator is used to determine whether the generated eye movement trajectory is consistent with the real trajectory. The loss function uses the generative adversarial loss (GAN Loss) and the trajectory feature matching loss to ensure the authenticity and diversity of the generated trajectory. Use the saliency dataset for supervised training, and improve the generation effect by optimizing the adversarial process of the generator and the discriminator.

[0029] The preset distance algorithm can be the Jarodzka distance algorithm Step A3, using the observation interface image dataset as input information and the eye movement trajectory information as output information to iteratively train the intermediate prediction model until the intermediate prediction model reaches a convergence state, thereby obtaining an eye movement trajectory prediction model; Step A4: Use the trained eye movement trajectory prediction model to infer the observation interface image data.

[0030] The eye movement trajectory prediction model is explained through the following example, which does not specifically limit the eye movement trajectory prediction model.

[0031] This model is trained using RTX 3090 with 24G video memory, and the CPU uses 2.10GHz, 10-core Intel (R) Xeon (R) Gold 5218R, with 64GB memory. Use python3.8 programming environment, tensorflow2.5.0 deep learning framework, cuda11.2 parallel computing architecture, and neural network acceleration library cudnn. Opencv-python=4.5.5.60 is installed here, which can use opencv in python to read, display, process and store images. At the same time, scipy=1.8.0 is installed to read mat type data. Panda=1.3.4 is installed here to process structured data, and numpy=1.19.5 is used here for matrix operations. Pydot=1.4.2 is installed to print model diagrams.

[0032] The model hyperparameters were tuned, and the final noise z used a normal distribution with a size of 110, a mean of 0, and a variance of 1. Normally distributed noise with a mean of 0 and a variance of 4 was added to the image, and normally distributed noise with a mean of 0 and a variance of 0.1 was added to the eye movement trajectory. When the model uses the cGAN loss function, the unilateral label of the discriminator is smoothed, and the label smoothing coefficient is 0.1. Only the labels of the real eye movement trajectory are smoothed, and the labels of the generated eye movement trajectory are not processed. This can not only make the discriminator produce smooth predictions, but also weaken the incorrect behavior of the generator here. The learning rate size of the generator is 1×10-4, the learning rate size of the discriminator is 8.3×10-6, and the batchsize size is 32. The convolution of the generator uses 10 3×3 convolution kernels with a step size of 2×2, and the activation function is a linear function (linear), which is used to reduce the size of image features and reduce the complexity of the model. The output dimension of the generator's LSTM is 30, and the dropout size of the LSTM is 0.1. It is used to process the sequence and obtain the sequence output. The purpose of using noise and dropout is to improve the generalization performance of the model and prevent the model from overfitting. The output dimension of the LSTM used by the discriminator to process the eye movement track is 55. The convolution layer uses 5 3×3 convolution kernels with a step size of 2×2. The activation function is a linear function (linear). The output dimensions of the last two layers of LSTM are 10 and 1 respectively. The nonlinear activation function uses leaky_relu, and the slope size of the part less than 0 is 0.3. The model uses the RMSprop optimizer. The epoch size is 24. When the loss L^2 and cGAN loss are selected for model training, the generator only uses L^2 loss in the first epoch, which allows the generator to obtain better performance faster and prevents the generator from being too weak in the early stage of training and causing slow training. After 1 epoch, the generator uses cGAN loss and L^2 loss for joint training, where the weight α of L^2 loss is 1.

[0033] It should be noted that in addition to using the generative adversarial network (cGAN), the following technical solutions can also be used to predict eye movement trajectories: Reinforcement Learning (RL): Reinforcement learning learns complex strategies through trial and error mechanisms, which is suitable for the generation of eye movement trajectories. It treats trajectory prediction as a sequential decision problem. In this scheme, the state includes the current gaze position and image features, the action is to predict the location of the next gaze point, and the reward is based on the similarity score between the trajectory and the real data. RL can adaptively optimize trajectory generation in complex environments, and is particularly suitable for dynamic scenes or online prediction tasks.

[0034] Variational Autoencoder (VAE) + Recurrent Neural Network (RNN): Variational Autoencoder (VAE) is good at generating diverse data, and combined with Recurrent Neural Network (RNN), it can generate eye movement trajectories with time series information. This solution first uses VAE to extract and compress image features into latent vectors, and then generates the time series of eye movement trajectories through RNN (such as LSTM or GRU).

[0035] Transformer model: Transformer performs well in processing long sequences and can effectively capture long-term dependencies, so it is more suitable for generating eye movement trajectories than RNN. By inputting the feature sequence encoded by the input image into the Transformer encoder, a complete trajectory sequence can be generated, and the gaze point and duration are predicted using a self-attention mechanism. This model does not rely on sequence order and is particularly suitable for generating eye movement trajectories over long time spans, especially for predicting long-term gaze behavior.

[0036] Preferably, step S102 may include steps S1021 to S1022, wherein: Step S1021, determining the length of the sampling window, taking the first sight point data as the starting unit, selecting the sight point data of the corresponding length through the sampling window, and calculating the position mean of the selected sight point data; The length of the sampling window is set by the actual scenario requirements and is not limited here.

[0037] Step S1022, the position of the sampling window is moved backward by one position, and the average of the sight line data selected after the movement is calculated, and this step is repeated until the last data enters the sampling window to calculate the position average, and the denoising operation is completed.

[0038] Determine the length of the sampling frame, put the sampled data into the sampling frame, and calculate the average of the data in the sampling frame to obtain the filtered data. Then move the sampled data one position backward without changing the sampling frame, and calculate the average of the data in the sampling frame. Filter the entire data in sequence until the last data is filtered. When processing signals, moving average filtering is often used for denoising.

[0039] It should be noted that before moving average filtering, it is necessary to determine whether the NaN ratio of the gaze point data is less than the threshold. If so, denoising can be performed. Otherwise, it means that the group of gaze point data is unavailable and the group of gaze point data is deleted. This step avoids additional processing of invalid data and improves the efficiency of eye movement trajectory generation.

[0040] Preferably, step S102 may further include steps B1 to B6, wherein: Step B1, calculating the data point speed of each sampling window according to the position information and acquisition time of the data point; The data point speed can be obtained by dividing the data point position coordinate difference and the data point acquisition time node difference. Calculate the average value of the data point speed in each sampling window.

[0041] Step B2, processing the data point velocity of each sampling window by using a difference method to obtain the data point acceleration; The acceleration of the data point is determined by the ratio of the difference between the data point velocity and the average value and the difference between the acquisition time nodes.

[0042] Step B3, determining whether the acceleration of the data point is greater than the acceleration threshold and the data point is not an isolated point, and if so, determining the data point as the start point of the eye saccade; The acceleration of a data point greater than the acceleration threshold indicates the onset of saccades. In particular, since eye movements such as microsaccades also occur during fixation, this process is manifested as an isolated data point, and its acceleration may also be greater than the acceleration threshold. This data point does not belong to saccade data and needs to be removed.

[0043] Step B4, taking the eye saccade starting point as a reference, determining whether there is a target data point within a preset time interval from the eye saccade starting point, wherein the data point acceleration of the target data point is less than a deceleration threshold; The data point acceleration being less than the deceleration threshold indicates the end of the saccade. The preset time interval may be 25 sight point time intervals.

[0044] Step B5: if yes, determine the target data point as the end point of the eye saccade; If not, the data points in this segment that are greater than the acceleration threshold are considered to belong to fixation data.

[0045] Step B6, determining the fixation data point by the time interval between the start point of the saccade and the end point of the saccade.

[0046] Through the above operation, the data points at the beginning and the end of each saccade can be obtained, and the data points between two saccades are fixation data points.

[0047] Preferably, step B6 may further include steps B61 to B63, wherein: Step B61, determining whether the time interval between the start point of the saccade and the end point of the saccade is less than a preset threshold; Step B62, if yes, taking the data points between the eye saccade start point and the eye saccade end point as eye saccade data, and merging the eye saccade start point, the eye saccade end point and the eye saccade data; Step B63: if not, determine the data point between the saccade start point and the saccade end point as the fixation data point.

[0048] If the fixation time is less than the set number of samples (preset threshold), the fixation data is essentially saccade data, but the speed is reduced due to noise and other reasons. Therefore, it is necessary to merge the fixation data points and the saccade data points on both sides of it.

[0049] Preferably, step S103 may include steps S1031 to S1033, wherein: Step S1031, determining whether the gaze point is the first gaze point; Step S1032: If yes, determine the start time of the fixation point and the experiment duration as the first time factor; Step S1033: if not, calculate the difference between the start time of the fixation point and the start time of the previous fixation point, and use the ratio of the difference to the experiment duration as the first time factor.

[0050] For the first fixation point, the first time factor is obtained by dividing the start time of the fixation point by the total experimental time. For other fixations, the first time factor is obtained by subtracting the end time of the previous fixation point from the start time of the current fixation point and dividing it by the total experimental time. The second time factor is the fixation duration divided by the total experimental time.

[0051] This setting has the following advantages: First, the general deep learning model finally uses the sigmoid function to normalize the results to [0,1]. In this way, the value of the gaze point representation is also designed to be within [0,1], which simplifies the subsequent processing flow.

[0052] Second, since there is no unified standard for the experimental time of the observation interface image dataset, the sampling time of different datasets is often different, so the length of the eye movement trajectory generated by the same image will also be different due to the experimental time. Based on this design, the model trained with the dataset with a longer experimental time also has a larger time range for the predicted eye movement trajectory. The eye movement trajectory of the dataset with a shorter experimental time only needs to intercept the eye movement trajectory of the corresponding time period, making the generated trajectory more individual and diverse.

[0053] Third, the setting of the first time factor t1 and the second time factor t2 ensures that the end time of a certain fixation must be greater than the start time of the fixation. The start time of the fixation point can be obtained by adding the t1 and t2 of the previous fixation point and the t1 of the current fixation point. The end time of the fixation point can also be obtained by adding the t1 and t2 of the previous fixation point and the t2 of the current fixation point, thereby ensuring the accuracy of mutual reasoning of data information.

[0054] This embodiment can fully express the dynamic changes of visual attention over time by converting the gaze point data into eye movement trajectory information including gaze position and time information; the introduction of time point information can adapt to data sets of different experimental durations and ensure the accuracy of eye movement trajectory representation in different scenarios.

[0055] Embodiment 2 Embodiment 2 of the present invention further provides an eye movement trajectory generation device, which corresponds to the eye movement trajectory generation method provided in the above embodiment 1. The corresponding technical features and technical effects are not described in detail in this embodiment, and the relevant parts can refer to the above embodiment 1. Specifically, Figure 2 FIG. 4 shows a structural block diagram of the eye movement trajectory generation device. Figure 2 As shown, the eye movement trajectory generation device 200 includes an acquisition module 201, a denoising module 202, a determination module 203 and a generation module 204, wherein: An acquisition module 201 is used to acquire gaze point data and experiment duration collected by an eye tracker, wherein the gaze point data includes multiple data points; A denoising module 202 is connected to the acquisition module 201, and is used to perform denoising on the gaze point data through a sampling window, and determine the eye movement data and the fixation data from the data points obtained after the denoising; wherein the fixation data includes the start time, end time, duration and position of the fixation point; A determination module 203, connected to the denoising module 202, is used to determine a first time factor according to the start time, end time and experiment duration of the fixation point, and to determine a second time factor according to the duration and experiment duration; The generating module 204 is connected to the determining module 203 and is used to generate eye movement trajectory information based on the position of the gaze point, the first time factor and the second time factor.

[0056] Optionally, the device also includes a prediction module, which is specifically used to: prepare an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface image dataset is a batch of images consisting of gaze point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; introduce a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; iteratively train the intermediate prediction model using the saliency dataset as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state to obtain an eye movement trajectory prediction model; and use the trained eye movement trajectory prediction model to infer the observation interface image data.

[0057] Optionally, the denoising module is also used to: determine the length of the sampling window, take the first sight point data as the starting unit, select the sight point data of corresponding length through the sampling window, and calculate the position mean of the selected sight point data; move the position of the sampling window backward by one position, and calculate the mean of the sight line data selected after the movement, repeat this step until the last data enters the sampling window to calculate the position mean, and the denoising operation is completed.

[0058] Optionally, the denoising module is also used to: calculate the data point speed of each sampling window through the position information and acquisition time of the data point; process the data point speed of each sampling window using the difference method to obtain the data point acceleration; determine whether the data point acceleration is greater than the acceleration threshold and the data point is not an isolated point, and if so, determine the data point as the start point of the saccade; based on the start point of the saccade, determine whether there is a target data point within a preset time interval from the start point of the saccade, wherein the data point acceleration of the target data point is less than the deceleration threshold; if so, determine the target data point as the end point of the saccade; and determine the gaze data point through the time interval between the start point of the saccade and the end point of the saccade.

[0059] Optionally, the device also includes a merging module, which is used to: determine whether the time interval between the start point and the end point of the saccade is less than a preset threshold; if so, use the data points between the start point and the end point of the saccade as saccade data, and merge the saccade start point, the saccade end point and the saccade data; if not, determine that the data points between the start point and the end point of the saccade are fixation data points.

[0060] Optionally, the determination module is also used to: determine whether the fixation point is the first fixation point; if so, determine the start time of the fixation point and the experiment duration as the first time factor; if not, calculate the difference between the start time of the fixation point and the start time of the previous fixation point, and take the ratio of the difference to the experiment duration as the first time factor.

[0061] Embodiment 3 Figure 3 The block diagram of a computer device suitable for implementing the eye movement trajectory generation method provided by the third embodiment of the present invention is shown. In this embodiment, the computer device 301 can be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server, or a server cluster composed of multiple servers) that executes a program. Figure 3 As shown, the computer device 301 of this embodiment includes at least but is not limited to: a memory 301, a processor 302, and a network interface 303 that can be interconnected through a system bus. It should be noted that Figure 3Only a computer device 301 having components 301 - 303 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0062] In this embodiment, the memory 303 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 301 can be an internal storage unit of the computer device 301, such as a hard disk or memory of the computer device 301. In other embodiments, the memory 301 can also be an external storage device of the computer device 301, such as a plug-in hard disk equipped on the computer device 301, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 301 can also include both the internal storage unit of the computer device 301 and its external storage device. In this embodiment, the memory 301 is generally used to store the operating system and various application software installed on the computer device 301, such as the program code of the eye movement trajectory generation method, etc.

[0063] In some embodiments, the processor 302 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 302 is generally used to control the overall operation of the computer device 301. For example, it performs control and processing related to data interaction or communication with the computer device 301. In this embodiment, the processor 302 is used to run the program code of the steps of the eye movement trajectory generation method stored in the memory 301.

[0064] In this embodiment, the eye movement trajectory generation method stored in the memory 301 may also be divided into one or more program modules and executed by one or more processors (processor 302 in this embodiment) to complete the present invention.

[0065] The network interface 303 may include a wireless network interface or a wired network interface, and the network interface 303 is generally used to establish a communication link between the computer device 301 and other computer devices. For example, the network interface 303 is used to connect the computer device 301 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 301 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0066] Embodiment 4 This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating eye movement trajectories are implemented.

[0067] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0068] It should be noted that the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0069] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0070] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating eye movement trajectories, characterized in that: The method comprises: Obtaining gaze point data and experiment duration collected by an eye tracker, wherein the gaze point data includes multiple data points; De-noising the gaze point data through a sampling window, and determining the saccade data and fixation data from the data points obtained after the de-noising process; wherein the fixation data includes the start time, end time, duration and position of the fixation point; A first time factor is determined by the start time of the fixation point, the end time, and the experiment duration, and a second time factor is determined by the duration and the experiment duration; Eye movement trajectory information is generated based on the position of the gaze point, the first time factor and the second time factor.

2. The method according to claim 1, characterized in that The method further comprises: Preparing an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface image dataset is a batch of images including sight point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; Introducing a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; Iteratively training the intermediate prediction model using the observation interface image dataset as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state, thereby obtaining an eye movement trajectory prediction model; The trained eye movement trajectory prediction model is used to infer the observation interface image data.

3. The method according to claim 1, characterized in that The denoising process of the sight point data through a sampling window includes: Determine the length of the sampling window, take the first sight point data as the starting unit, select the sight point data of the corresponding length through the sampling window, and calculate the position mean of the selected sight point data; The position of the sampling window is moved backward by one position, and the average of the sight line data selected after the movement is calculated. This step is repeated until the last data enters the sampling window to calculate the position average, and the denoising operation is completed.

4. The method according to claim 1, characterized in that: The step of determining the saccade data and the fixation data from the data points obtained after the denoising process comprises: Calculate the data point speed of each sampling window by using the position information and acquisition time of the data point; Processing the data point velocity of each sampling window by using a difference method to obtain the data point acceleration; Determine whether the acceleration of the data point is greater than an acceleration threshold and the data point is not an isolated point, and if so, determine the data point as a saccade start point; Taking the eye saccade starting point as a reference, determining whether there is a target data point within a preset time interval from the eye saccade starting point, wherein the data point acceleration of the target data point is less than a deceleration threshold; If yes, the target data point is determined as the end point of the saccade; The fixation data point is determined by the time interval between the start point of the saccade and the end point of the saccade.

5. The method according to claim 4, characterized in that The method further comprises: Determine whether the time interval between the start point of the eye saccade and the end point of the eye saccade is less than a preset threshold; If yes, taking the data points between the eye saccade start point and the eye saccade end point as eye saccade data, and merging the eye saccade start point, the eye saccade end point and the eye saccade data; If not, determine the data point between the saccade start point and the saccade end point as the fixation data point.

6. The method according to claim 1, characterized in that The determining of the first time factor by the start time of the fixation point, the end time and the experiment duration comprises: Determining whether the gaze point is the first gaze point; If yes, then determine the start time of the fixation point and the duration of the experiment as the first time factor; If not, the difference between the start time of the fixation point and the start time of the previous fixation point is calculated, and the ratio of the difference to the experiment duration is used as the first time factor.

7. A device for generating eye movement trajectories, characterized in that: The device comprises: An acquisition module, used to acquire gaze point data and experiment duration collected by an eye tracker, wherein the gaze point data includes multiple data points; A denoising module, used for denoising the sight point data through a sampling window, and determining the eye saccade data and fixation data from the data points obtained after the denoising process; wherein the fixation data includes the start time, end time, duration and position of the fixation point; A determination module, configured to determine a first time factor according to the start time of the fixation point, the end time and the experiment duration, and to determine a second time factor according to the duration and the experiment duration; A generation module is used to generate eye movement trajectory information based on the position of the gaze point, the first time factor and the second time factor.

8. The device according to claim 7, characterized in that The device also includes a prediction module, which is specifically used for: Preparing an observation interface image dataset and eye movement trajectory information for eye movement trajectory prediction, wherein the observation interface image dataset is a batch of images including sight point data, and the observation interface image data uniquely corresponds to the eye movement trajectory information; Introducing a preset distance algorithm into the initial generative adversarial network to obtain an intermediate prediction model; Iteratively training the intermediate prediction model using the observation interface image dataset as input information and the eye movement trajectory information as output information until the intermediate prediction model reaches a convergence state, thereby obtaining an eye movement trajectory prediction model; The trained eye movement trajectory prediction model is used to infer the observation interface image data.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images