A method for measuring and predicting inter-observer eye movement consistency and a terminal

Abstract

The application provides a kind of dynamic observer interocular movement consistency measurement method, comprising: obtaining the eye movement data of N observers when watching video, and processing the eye movement data;Through the single frame image in the processed eye movement data, the interocular movement consistency of a single observer is obtained;Repeat until the interocular movement consistency of all observers is obtained;The interocular movement consistency of the observer is processed to obtain the value of the dynamic observer interocular movement consistency of single frame image.A kind of dynamic observer interocular movement consistency prediction method is also proposed, based on the data obtained by the dynamic observer interocular movement consistency measurement method to train model;Using the trained model to predict the dynamic observer interocular movement consistency.The application is aimed at the method for predicting the interocular movement consistency of the observer of video, and uses the technology of deep learning, combines the interframe motion feature, and obtains relatively accurate result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and terminal for measuring and predicting eye movement consistency among dynamic observers. Background Technology

[0002] In recent years, with the maturation of human eye fixation detection equipment, the analysis of the human visual system and the content of human eye attention has become a key focus in the field of computer vision and graphics processing. Human gaze behavior is influenced by various factors, such as age, education level, and the observer's physical condition. Inter-observer eye movement consistency describes the degree of dispersion in the distribution of visual attention areas when different subjects are viewing the same material.

[0003] Due to the widespread attention given to the study of visual salient regions, researchers have proposed numerous methods for predicting them. However, inter-observer eye-tracking consistency is often used as a supplementary indicator to measure the generalization ability of eye-tracking datasets and salient region prediction methods. In this context, inter-observer eye-tracking consistency is typically defined as the accuracy with which the eye-tracking data of one group of observers is used to predict the observation region of another group. Higher accuracy, i.e., higher inter-observer eye-tracking consistency, indicates a stronger representativeness of the prediction method and dataset.

[0004] Inter-observer eye-tracking consistency, as a descriptive concept of human eye movement behavior, has not been widely studied except as an auxiliary measure in visual saliency research. Currently, there are some related studies; for example, Judd proposed using the entropy of the observer's gaze to roughly represent inter-observer eye-tracking consistency, and Le Meur defined a method for calculating inter-observer eye-tracking consistency, considering features such as facial information, color harmony, depth of field, and scene complexity to predict consistency values.

[0005] However, it's worth noting that the methods mentioned above all focus on observer-to-observer eye-tracking consistency (AOS) for images. Compared to images, videos add a temporal dimension, giving AOS a temporal trend. Furthermore, scenarios where multiple people watch videos together, such as movies, are common in daily life. Both the temporal trend of AOS values ​​and the distribution of attention regions in multi-viewer scenarios are crucial. However, to our knowledge, there is currently no mature literature on dynamic AOS. Therefore, designing methods for measuring and predicting dynamic AOS is essential. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and terminal for measuring and predicting dynamic eye movement consistency among observers.

[0007] According to one aspect of the present invention, a method for measuring dynamic eye-tracking consistency among observers is provided, comprising:

[0008] S1, acquire eye movement data of N observers while watching the video, and process the eye movement data;

[0009] S2, using single-frame images from processed eye-tracking data, obtains inter-observer eye-tracking consistency for a single observer;

[0010] S3, repeat S2, until observer eye movement consistency is obtained for all observers;

[0011] S4 processes the observer eye-tracking consistency of all observers to obtain the dynamic observer eye-tracking consistency value of a single frame image.

[0012] Preferably, step S1 involves acquiring eye-tracking data from all observers while watching the video and processing the eye-tracking data, including:

[0013] S101, the eye movement point distribution of the eye movement data is extracted frame by frame into a grayscale image sequence;

[0014] S102, In the image sequence, one frame is selected every m frames as a sampling frame;

[0015] S103, Select eye-tracking data from n frames before and after the sampling frame and overlay them.

[0016] Preferably, n ≥ m, so that all fixation points are included in the calculation.

[0017] Preferably, step S2, obtaining inter-observer eye movement consistency for a single observer using single-frame images from the processed eye movement data, includes:

[0018] S201, a single frame image includes N eye-tracking data images. The i-th image is selected and convolved with a Gaussian kernel to generate a gaze density grayscale image, denoted as F. i ;

[0019] S202, the gaze points of the remaining N-1 observers are superimposed onto a single image, and convolved with a Gaussian kernel to generate a gaze density map, denoted as F. N-1 ;

[0020] S203, obtain the inter-observer eye movement consistency of the i-th observer:

[0021]

[0022] Among them, C i F is the inter-observer eye movement consistency of the i-th observer. i and FN-1 The distribution represents the gaze density map superimposed by the i-th observer and the remaining observers, F i ∩F N-1 This represents the intersection of two salient regions in an FDM image, where M represents the total number of pixels in an image, and f... i j and Let represent the pixel value of the j-th pixel on each of the two FDM images, and σ be a constant to avoid the numerator being 0.

[0023] Preferably, step S4, processing the observer-to-observer eye-tracking consistency of the observers to obtain the dynamic observer-to-observer eye-tracking consistency value of a single frame image, includes: averaging the observer-to-observer eye-tracking consistency of the observers to obtain the dynamic observer-to-observer eye-tracking consistency value of a single frame image.

[0024] According to a second aspect of the present invention, a method for predicting dynamic inter-observer eye movement consistency is provided, comprising:

[0025] S5, Construct a deep learning feature extraction and regression model that combines image information and motion information;

[0026] S6, a model is trained based on data obtained from a method for measuring dynamic eye movement consistency among observers as described in any one of claims 1-5;

[0027] S7, use the trained model to predict dynamic eye movement consistency among observers.

[0028] Preferably, in step S5, a deep learning feature extraction and regression model combining image information and motion information is constructed, including a dual-channel feature extraction subnetwork and a fully connected regression subnetwork.

[0029] Preferably, the dual-channel feature extraction subnet includes a content branch and a motion flow branch.

[0030] Preferably, the fully connected regression subnetwork includes an average pooling layer and a fully convolutional layer.

[0031] Preferably, both the content branch and the motion flow branch include one convolutional layer and four residual layers; the size of the first convolutional layer is 7×7×64; the residual layer in the content branch contains two 1×1 convolutional layers; the four residual layers in the content branch include 3, 4, 23 and 3 residual networks respectively; the residual layers in the optical flow branch include 3, 4, 6 and 3 residual networks.

[0032] Preferably, step S6, training the model using the obtained dynamic observer eye-tracking consistency data, includes:

[0033] S601. Divide the obtained dynamic observer eye-tracking consistency data and corresponding video sampling frames into training set, test set and validation set;

[0034] S602. Construct a pre-trained model for predicting image saliency and pre-train a dual-channel feature extraction subnet;

[0035] S603. Reconstruct the feature extraction and regression model using the pre-trained dual-channel feature extraction subnet;

[0036] S604. Based on the comparison between the predicted inter-observer eye movement consistency values ​​and the actual values ​​on the test set, train the model.

[0037] Preferably, the trained model predicts dynamic eye-tracking consistency among observers, including:

[0038] S701. Extract the image sequence frame by frame from the dynamic video sequence;

[0039] S702. Extract the motion information between frames of the image sequence frame by frame;

[0040] S703. Use the inter-frame motion information as input to the motion flow branch to obtain motion feature information from shallow to deep layers;

[0041] S704. Using the original image frame as input to the content branch, the motion feature information extracted from the motion flow branch is fused from shallow to deep layers.

[0042] S705. The merged information output from the last residual layer of the content branch is input into the average pooling layer and the fully convolutional layer to generate a value of visual consistency among dynamic observers.

[0043] According to a third aspect of the present invention, an electronic terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for measuring dynamic inter-observer eye-tracking consistency or to perform the method for predicting dynamic inter-observer eye-tracking consistency.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This invention provides a method for measuring dynamic eye-tracking consistency among observers. It can be effectively used to determine the concentration effect of different video frames on the human eye's focus area, and has significant implications for the development of video media.

[0046] This invention is a prediction method for dynamic observer eye movement consistency, and it utilizes deep learning technology, combined with inter-frame motion features, to obtain relatively accurate results. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart illustrating a method for measuring the consistency among dynamic observers according to an embodiment of the present invention.

[0049] Figure 2 This is a flowchart illustrating the prediction process for dynamic observer eye-tracking consistency, as described in one embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0051] One embodiment provided by the present invention, such as Figure 1 The diagram shown is a flowchart of a method for measuring consistency among dynamic observers.

[0052] S100, a general eye tracker, collects eye movement data from N viewers while watching a video. Specifically, it is important to ensure that the distance between the subject and the eye tracker is within the effective range of the eye tracker, and to select a reasonable screen size and distance from the screen to ensure that the subject's longitudinal and lateral field of view and head movement range are within the allowable range of the eye tracker.

[0053] Eye-tracking data processing includes:

[0054] The collected eye-tracking point distribution is extracted frame by frame into a grayscale image sequence;

[0055] In the image sequence, one frame is selected every m frames as the sampling frame;

[0056] The eye-tracking data from n frames before and after the sampling frame are superimposed. This operation takes into account that in videos, the same scene often lasts for several frames. For the same scene, the viewer's light may be on the same salient area, but the viewing order may be slightly different, which may cause the dynamic observer consistency value of each frame to be abnormally low. The merging of n frames by sliding window can correct the error caused by the viewing order.

[0057] As a preferred embodiment, n≥m ensures that all fixation points are included in the calculation.

[0058] Based on S100, S200 is executed to obtain the observer-to-observer eye-tracking consistency of a single frame image. Specifically,

[0059] For each frame of N eye-tracking data images, the i-th image is selected and convolved with a Gaussian kernel to generate a gaze density grayscale image, denoted as F. i ;

[0060] The gaze points of the remaining N-1 observers are superimposed onto a single image, and similarly, convolved with a Gaussian kernel to generate a gaze density map, denoted as F. N-1 ;

[0061] The inter-observer eye movement consistency of the i-th observer can be represented as:

[0062]

[0063] Among them, C i F is the inter-observer eye movement consistency of the i-th observer. i and F N-1 The distribution represents the gaze density map superimposed by the i-th observer and the remaining observers, F i ∩F N-1 This represents the intersection of the salient regions of two FDM images. M represents the total number of pixels in an image, and f i j and Let represent the pixel value of the j-th pixel on the two FDM images respectively, and σ be a very small constant to avoid the numerator being 0.

[0064] S300, performing this calculation iteratively on all observers, obtains the observer-to-observer eye-tracking consistency for all observers.

[0065] S400 calculates the inter-observer eye-tracking consistency of all observers by averaging the results. The value of the dynamic inter-observer eye-tracking consistency of the sampled frame is then obtained, and the range of the value is [0,1].

[0066] Based on S200, the same calculation is performed on all sampled frames to obtain the measured dynamic observer eye movement consistency value.

[0067] Based on the same concept as the above embodiments, an embodiment is provided, a method for predicting dynamic inter-observer eye movement consistency, comprising:

[0068] S500, constructing a deep learning feature extraction and regression model that combines image information and motion information;

[0069] S600, a model trained on data obtained from the aforementioned method for measuring dynamic inter-observer eye movement consistency;

[0070] The S700 uses a trained model to predict dynamic eye-tracking consistency among observers.

[0071] In S500, the constructed model mainly consists of a dual-channel feature extraction subnetwork and a fully connected regression subnetwork. The dual-channel feature extraction subnetwork consists of two branches: a content branch and a motion flow branch. The fully connected regression subnetwork includes an average pooling layer and a fully convolutional layer.

[0072] In a preferred embodiment, each branch includes one convolutional layer and four residual layers. The size of the first convolutional layer is [size missing]. Considering that the original video frame contains more visual semantic information than the optical flow graph, the content branch is designed as the main branch, containing a more complex residual structure. The residual layers in the content branch contain two convolutional layers and are repeated more than in the motion flow branch.

[0073] Specifically, the optical flow branch includes one convolutional layer and four residual layers. The size of the first convolutional layer is 7×7×64. The number of residual networks in the four residual layers are 3, 4, 6 and 3, respectively. The residual networks include convolutional layers of 3×3×64×2, 3×3×128×2, 3×3×256×2 and 3×3×512×2, respectively. Each residual layer extracts an output as a motion feature.

[0074] Specifically, the content branch consists of one convolutional layer and four residual layers. The first convolutional layer has a size of 7×7×64. The number of residual networks in the four residual layers are 3, 4, 6, and 3, respectively. Based on the motion branch, the residual networks add convolutional layers of 1×1×64×2, 1×1×128×2, 1×1×256×2, and 1×1×512×2, respectively. The output of each residual network is extracted as a content feature, combined with the motion features, and used as the input to the next layer.

[0075] Based on S500 in the above embodiment, S600 is executed. To better train the model, a preferred embodiment is provided, specifically including:

[0076] S601. The data obtained based on the dynamic observer eye-tracking consistency measurement method in the above embodiments and the corresponding video sampling frames are divided into training set, test set and validation set.

[0077] S602. Construct a pre-trained model for predicting image saliency and pre-train a dual-channel feature extraction subnet;

[0078] S603. Using the pre-trained dual-channel feature extraction subnet, the feature extraction and regression models are reconstructed based on the model structure.

[0079] S604. Based on the comparison between the measured inter-observer eye movement consistency value and the true value on the test set, the model is trained.

[0080] To obtain relatively suitable network weights for the feature extraction part and perform visual saliency prediction, a preferred embodiment is provided, using these weights as pre-training. Specifically, a pre-trained regression sub-network is constructed by repeating the sampling layer and performing four convolutional layers. Its output is a predicted map of salient regions. This sub-network is connected after the feature extraction part, and the saliency prediction task is performed on the pre-trained network. After pre-training, the pre-trained weights of the dual-channel feature extraction sub-network are retained, and a new training model for dynamic inter-observer consistency is constructed by combining it with the full regression sub-network. A smoothed L1 loss function is used to calculate the difference between the predicted consistency value and the measured true value, and the network weights are updated accordingly.

[0081] To better predict dynamic eye-tracking consistency among observers, a preferred embodiment is provided to execute S700, the process of which is as follows: Figure 2 As shown, it includes:

[0082] S701. Extract the image sequence frame by frame from the dynamic video sequence;

[0083] S702. Extract the motion information between frames of the image sequence frame by frame;

[0084] S703. Use inter-frame motion information as input to the motion flow branch to obtain motion feature information from shallow to deep layers;

[0085] S704. Using the original image frame as input to the content branch, the motion feature information extracted from the motion flow branch is fused from shallow to deep layers.

[0086] S705. The merged information output from the last residual layer of the content branch is input into the average pooling layer and the fully convolutional layer to generate a value for dynamic visual consistency among observers.

[0087] In this embodiment, the motion information between frames is extracted from the image sequence frame by frame. The optical flow network FlowNet 2.0 is used to generate optical flow maps in the experiment, and the optical flow maps are the motion information between frames.

[0088] Following the steps of the above embodiments, 9157 sampled frames from 12 video segments were tested using a Tobii X120 eye tracker. To verify the model's generalization ability, two dataset partitioning methods were employed. The first method involved dividing the sampled frames of each video proportionally into training, test, and validation sets (average partitioning). The second method involved partitioning the training, test, and validation sets into different videos (cross-video partitioning). The frame ratio of the training, test, and validation sets was approximately 3:1:1. The SROCC between the predicted inter-observer eye-tracking consistency value and the true value on the validation set was used to evaluate the effectiveness of the prediction method. The results showed that the support vector machine's predictions had almost no correlation with the actual collected data, with SROCC = 0.175 on the average partitioning dataset and SROCC = 0.107 on the cross-video partitioning dataset. Other methods showed statistically significant correlations (p < 0.01), but these were not strong correlations, with SROCC ranging from 0.35 to 0.65. Our method achieved good results in both dataset selection methods, with SROCC = 0.693 on the average segmentation dataset and SROCC = 0.573 on the cross-video segmentation dataset.

[0089] Based on the same technical concept described above, in another embodiment of the present invention, an electronic terminal is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it is used to perform the method for measuring dynamic inter-observer eye movement consistency in any of the above embodiments, or to perform the method for predicting dynamic inter-observer eye movement consistency in any of the above embodiments.

[0090] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The preferred features described above can be used in any combination without conflict.

Claims

1. A method for predicting eye movement consistency among dynamic observers, characterized in that, include: Construct a deep learning feature extraction and regression model that combines image and motion information; The model is trained based on data obtained from a method for measuring dynamic inter-observer eye-tracking consistency, wherein the method includes the following steps: S1, acquire eye movement data of N observers while watching the video, and process the eye movement data; S2, using single-frame images from processed eye-tracking data, obtains inter-observer eye-tracking consistency for a single observer; S3, repeat S2, until observer eye movement consistency is obtained for all observers; S4, determine the value of dynamic observer eye-tracking consistency for a single frame image based on the observer eye-tracking consistency of all observers; Use the trained model to predict dynamic inter-observer eye movement consistency; The constructed model consists of a dual-channel feature extraction subnetwork and a fully connected regression subnetwork. The dual-channel feature extraction subnetwork consists of two branches: a content branch and a motion flow branch. The fully connected regression subnetwork includes an average pooling layer and a fully convolutional layer. The optical flow branch consists of one convolutional layer and four residual layers. The size of the first convolutional layer is... The number of residual networks in the four residual layers are 3, 4, 6, and 3, respectively, and the residual networks include... , , and Each convolutional layer and each residual layer extracts an output as a motion feature; The content branch consists of one convolutional layer and four residual layers. The size of the first convolutional layer is [size missing]. The number of residual networks in the four residual layers are 3, 4, 6, and 3, respectively. Based on the motion branches, the residual networks are respectively increased by... , , and The convolutional layers are used; the output of each residual network is extracted as content features, combined with motion features, and used as input for the next layer.

2. The method for predicting dynamic eye movement consistency among observers according to claim 1, characterized in that, The data training model obtained based on a dynamic inter-observer eye movement consistency measurement method includes: S601. Divide the obtained data and corresponding video sampling frames into training set, test set and validation set; S602. Construct a pre-trained model for predicting image saliency and pre-train a dual-channel feature extraction subnet; S603. Using the pre-trained dual-channel feature extraction subnet, reconstruct the feature extraction and regression model based on the constructed model; S604. Based on the comparison between the measured inter-observer eye movement consistency value and the true value on the test set, the model is trained.

3. The method for predicting dynamic inter-observer eye movement consistency according to claim 1, characterized in that, The prediction of dynamic observer-to-observer eye movement consistency using a trained model includes: S701. Extract the image sequence frame by frame from the dynamic video sequence; S702. Extract the inter-frame motion information from the image sequence frame by frame; S703. Use the inter-frame motion information as input to the motion flow branch to obtain motion feature information from shallow to deep layers; S704. Using the original image frame as input to the content branch, the motion feature information extracted from the motion flow branch is fused from shallow to deep layers. S705. The merged information output from the last residual layer of the content branch is input into the average pooling layer and the fully convolutional layer to generate a value of visual consistency among dynamic observers.

4. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it is used to perform the prediction method for dynamic observer eye movement consistency as described in any one of claims 1-3.

Images