A Global-Local Collaborative Visual Fixation Path Prediction Method

Abstract

This invention discloses a saccade path prediction method based on global-local dynamic collaboration, relating to the fields of computer vision and human-computer interaction. The method includes: extracting hierarchical visual features using a multi-scale feature encoder; constructing a two-stream architecture, explicitly decoupling the features into a global scene structure stream for spatial localization and a local semantic detail stream for target recognition; achieving dynamic information interaction between the two streams by introducing a hierarchical two-stream interactive decoding mechanism combined with layer-by-layer bidirectional cross-attention; and using the fused features to jointly predict the gaze probability distribution and search termination signal, generating a complete scan path in an autoregressive manner. This invention effectively simulates the collaborative mechanism of the human visual system, more accurately reproducing task-driven visual search behavior, and is suitable for cognitive mechanism research and intelligent human-computer interaction.

Description

Technical Field

[0001] This invention relates to the fields of image scanning path prediction and computer vision technology, and in particular to a task-driven scanning path prediction method based on global-local dynamic collaboration. Background Technology

[0002] Sagling path prediction aims to predict the sequence of gaze points that humans will make over time when observing a visual scene. As a fundamental topic in cognitive science and computer vision, it is not only a core tool for exploring the mechanisms of human visual attention, but also has enormous application potential in cutting-edge fields such as intelligent human-computer interaction, augmented reality and virtual reality (AR / VR), and medical-assisted diagnosis.

[0003] Research in this field has undergone a significant shift from "free browsing" to "task-driven" approaches. Early free browsing methods were primarily driven by low-level visual stimuli and could not plan paths toward specific search targets. Subsequent early task-driven methods mostly used deep reinforcement learning (DRL) to model the decision-making process, but were inherently limited in handling long-range historical gaze information due to the Markov assumption. In recent years, with the development of deep learning, methods based on the Transformer architecture, with their powerful long-sequence modeling capabilities, have gradually replaced reinforcement learning as the mainstream in this field.

[0004] However, despite significant progress in capturing long-range dependencies, existing mainstream deep learning models still suffer from a fatal limitation in their design mechanisms for visual representation learning: Existing methods typically tend to directly project and fuse extracted multi-scale visual features (such as global scene context features and local detail features) into a single, homogeneous representation space. This crude "feature homogenization" fusion approach neither explicitly distinguishes the cognitive role of features at different scales nor provides a bidirectional path for the dynamic exchange of multi-scale information.

[0005] Cognitive neuroscience research shows that efficient human visual search is not a simple "feature stew," but rather highly dependent on an explicit division of labor and bidirectional dynamic coordination between peripheral vision and foveal vision: low-resolution peripheral vision perceives the macroscopic layout to guide high-resolution foveal vision for fine-grained recognition; conversely, the recognition results of foveal vision serve as feedback signals, dynamically adjusting the attention allocation of peripheral vision. Because existing models employ homogeneous feature fusion methods, this dual-channel collaborative mechanism of "global guidance of local, local calibration of global" is disrupted. This lack of explicit modeling of cognitive mechanisms makes it difficult for existing models to achieve efficient coordination of spatial localization and target recognition in complex scenes, limiting their search efficiency and the anthropomorphism of saccade paths. Summary of the Invention

[0006] Purpose of the Invention: This application provides a novel perspective for simulating human active visual perception, aiming to address the problem in existing methods where the homogeneous fusion of multi-scale features leads to an inability to effectively coordinate global localization and local recognition. Specifically, this application explicitly decouples visual representation into a global scene structure flow and a local semantic detail flow. Historical gaze context is integrated through a spatiotemporal enhancement module, and learnable task queries are used for iterative processing in a hierarchical decoder. During this process, a bidirectional cross-attention mechanism and a dynamic gating unit adaptively regulate the contribution of global macro-guidance and local detail feedback to the current saccade decision. Subsequently, the updated query features are used to jointly predict the gaze coordinates and search termination probability of the next time step. Compared with previous work, this application solves the problem of multi-scale feature entanglement, explicitly fits the human collaborative mechanism of "global guidance of local, local calibration of global," and endows the model with the ability to autonomously terminate the search, improving the accuracy of saccade path prediction.

[0007] Technical solution

[0008] A saccade path prediction method based on global-local dynamic collaboration includes the following steps:

[0009] Step S1: Extract multi-scale hierarchical visual features:

[0010] The test image is acquired and input into a multi-scale feature encoder to extract hierarchical visual features through a backbone network. An efficient multi-scale attention module is used to adaptively calibrate the channel weights of the hierarchical visual features at each stage. Subsequently, the mid-to-high-level features are input into a pixel decoder based on multi-scale deformable attention for cross-scale interaction, outputting a multi-scale feature pyramid [P1, P2, P4] containing macroscopic spatial information and microscopic detail information.

[0011] Step S2: Decouple and construct the global feature flow and the local semantic feature flow:

[0012] The multi-scale feature pyramid is explicitly decoupled into a two-stream structure:

[0013] For low-to-medium resolution features P1 and P2, they are flattened along the spatial dimension, assigned independent multi-scale sinusoidal position codes, and then concatenated to generate a global flow matrix G for encoding the macroscopic scene layout; assuming the current time step is t, based on the set of gaze coordinates from historical time steps... Historical feature sequences are obtained by sampling on the high-resolution feature map P4. and will The input spatiotemporal enhancement module integrates dual spatial embedding and learnable temporal embedding to generate a semantic flow matrix S containing dynamic context;

[0014] Step S3: Initialize task query and perform hierarchical cross-stream decoding:

[0015] Initialize the task query matrix Q, and input it along with the global stream matrix G and the semantic stream matrix S into an interactive decoder containing L decoding layers for iterative updates. Within each decoding layer, residual information is first fused between the global stream matrix G and the semantic stream matrix S through a bidirectional cross-attention module. Subsequently, the task query matrix Q is subjected to cross-stream attention calculation with the updated global stream matrix and the semantic stream matrix, respectively, and the weights are adaptively calculated through a gated fusion unit to fuse the two streams of information, outputting the updated task query matrix.

[0016] Step S4: Decode the task query to generate a gaze probability heatmap and termination probability:

[0017] The final updated task query matrix Q is input into the spatial prediction head, and mapped to a gaze embedding vector by a perceptron. The system performs dot product matching with the high-resolution feature map P4 to generate a gaze probability heatmap at the current time step t, from which the gaze coordinates at the current time step are sampled. Simultaneously, the search termination probability τ at the current time step is predicted based on the task query feature Q and the current gaze step number n. If τ < 0.5, the current gaze coordinates are fed back to step S2 for the next time step iteration; otherwise, the prediction is terminated and the complete saccade path is output.

[0018] Step S5: Train and optimize the prediction model using real human eye saccade data:

[0019] A joint loss function comprising location loss, distribution loss, and termination loss is constructed. Using the collected real human eye gaze point coordinate sequence and real termination labels, the network parameters of the multi-scale feature encoder, interactive decoder, and spatial prediction head are optimized through gradient backpropagation.

[0020] Beneficial effects:

[0021] (1) Inspired by neuroscience, this invention uses a structured decoupled dual-flow interaction framework to replace the traditional homogeneous feature fusion. This design explicitly decouples visual representation into a global flow for perceiving scene layout and a semantic flow for capturing fine-grained details, generating a more accurate and human-like visual attention saccade path.

[0022] (2) The present invention introduces bidirectional cross-attention and dynamic gating fusion unit in the decoding layer, which enables the model to adaptively arbitrate the contributions of local semantics and global spatial cues, and realizes the synergy of "global guidance of local and local calibration of global", which is in line with the real human visual cognition mechanism. Attached Figure Description

[0023] Figure 1This is a flowchart of the global-local collaborative visual gaze path prediction method provided by the present invention.

[0024] Figure 2 This is a schematic diagram of the overall architecture of the global-local cooperative gaze network provided by the present invention.

[0025] Figure 3 This is a schematic diagram of the single-layer internal structure of the interactive decoder provided by the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. For example... Figure 1 As shown, a saccade path prediction method based on global-local dynamic collaboration has the following overall network architecture: Figure 2 As shown, it includes a multi-scale feature encoder, a stream construction module, a hierarchical two-stream interaction module, and a spatial prediction head, specifically including the following steps:

[0027] Step S1: Extract multi-scale hierarchical visual features:

[0028] The image to be tested is input into the feature encoder of a global-local cooperative gaze network. The feature encoder includes a backbone network, a feature enhancement unit, and a pixel-level decoder. The specific extraction operation is as follows:

[0029] Feature extraction is performed on the image under test using the backbone network (ResNet-50 network) to obtain hierarchical features from low-level texture to high-level semantics, and the output features Resi (i=2,3,4,5) of the backbone network at different stages are retained. Through the feature enhancement unit, an efficient multi-scale attention module (EMA) is embedded after the output features Resi at each stage to adaptively recalibrate the channel weights of the hierarchical features, thereby suppressing background noise and enhancing effective feature responses. The adaptively recalibrated hierarchical features are input into the pixel-level decoder, which uses a splitting strategy to project the high-level features {Res3,Res4,Res5} into a multi-scale deformable attention module (MSD) for cross-scale interaction, resulting in enhanced semantic representations. Simultaneously, the low-level feature Res2 is directly retained through skip connections.

[0030] Finally, a top-down approach is used to fuse the enhanced semantic representation with the preserved underlying Res2 features. The decoder output contains a hierarchical feature pyramid {P1, P2, P4} with three scales. P1 and P2 have larger receptive fields to support global structure perception, while P4 integrates deep semantic guidance and underlying texture details.

[0031] Step S2: Decouple and construct the global feature flow and the local semantic feature flow:

[0032] Obtain the multi-scale hierarchical feature pyramid output from step S1, organize it into two parallel information streams: a global stream and a semantic stream, and inject explicit spatial and temporal priors during the construction phase;

[0033] The operation of constructing the global stream includes: extracting low- and medium-resolution feature maps from the feature pyramid. After being flattened along the spatial dimension, the matrix is ​​stitched together according to scale. To avoid scale semantic ambiguity caused by the mixing of features from different receptive fields, independent multi-scale sinusoidal positional encoding is introduced for each scale to obtain the global flow matrix. :

[0034] ;

[0035] in, Let P1 be the first feature vector after being flattened and superimposed with positional encoding, and ;

[0036] This represents the second feature vector after feature map P2 has been flattened and superimposed with positional encoding, and ; For flattening operation; For splicing operations; and Independent multi-scale sinusoidal position coding;

[0037] The operation of constructing the semantic stream includes: the semantic stream is used to aggregate local high-level semantics under the condition of gaze history;

[0038] In predicting the When there are multiple fixations, we are working on high-resolution feature maps. The position of historical observation Feature sampling is performed to obtain historical feature sequences. :

[0039] ;

[0040] in, Indicates the corresponding coordinates The sample is taken from the feature map.

[0041] To imbue the historical feature sequence with spatial context and temporal dynamic information, the historical feature sequence is... Input to the spatiotemporal augmentation module to reconstruct the context prior: Spatially, fixed sinusoidal position encoding is used. With learnable structure embedding Dual spatial embedding:

[0042] ;

[0043] In the time dimension, learnable temporal embeddings are introduced. The updated historical feature sequence is multiplicatively modulated to obtain the semantic flow matrix. :

[0044] ;

[0045] Here, ⊙ represents element-wise multiplication. The semantic flow matrix It integrates current local gaze information with historical context, using dynamic memory to support attention retrieval and long-range dependency integration in the subsequent decoding process.

[0046] Step S3: Initialize the task query and perform cross-stream decoding and feature fusion in the hierarchical dual-stream interactive decoder:

[0047] like Figure 3 As shown, a random, learnable task query matrix Q is initialized and used as the semantic prior input of the search intent; a structure containing =5 cascaded decoding layers of a dual-stream interactive decoder; for the first of the dual-stream decoders One decoding layer ( =1,2,3,4,5), perform the following decoding operation:

[0048] Step S3.1, Dual-stream feature residual enhancement:

[0049] A bidirectional cross-attention module (BCA) is used to establish an information channel between the global flow matrix and the semantic flow matrix, and residual fusion is performed to obtain the feature-enhanced global flow matrix. and :

[0050]

[0051] in, , They represent the first The layer's global flow matrix output and semantic flow matrix output; , They represent the first The global flow matrix output and semantic flow matrix output of the layer (when) When =1, they are the initial global flow matrices constructed in step S2. and semantic flow matrix ) is the weighted residual connectivity coefficient, with a specific value of 0.2; BCA represents bidirectional cross-attention operation.

[0052] Step S3.2, Cross-flow cross-attention interaction:

[0053] The first Layer task query matrix Each with the enhanced global flow matrix and the semantic flow matrix The input is fed into the cross-stream attention module for cross-attention interaction, resulting in global query feature matrices containing spatial cues. and local query feature matrix containing semantic evidence :

[0054]

[0055]

[0056] Among them, when When =1, The initial learnable task query matrix Q, This indicates a cross-flow attention operation.

[0057] Step S3.3, Adaptive fusion of gated features:

[0058] The contribution weights of the two components are dynamically calculated using a lightweight gated fusion unit incorporating a two-layer perceptron; specifically, the global query feature matrix... With local query feature matrix The data is spliced ​​together, passing through the first layer of network weights. Projected into a low-dimensional latent space and integrated using the ReLU activation function, then processed by the weights of the second-layer network. The weights are mapped to scalars and then processed by the Sigmoid normalization function to obtain the dynamic gating weights. :

[0059] ;

[0060] Using the dynamic gating weight For the local query feature matrix and global query feature matrix Adaptive

[0061] We obtain the fusion feature representation by weighted fusion. :

[0062] ;

[0063] Step S3.4, Iterative update of the task query matrix:

[0064] Represent the fusion features After processing by a feedforward network (FFN), and connected to the first through residual connections... Task query matrix at level -1 After addition, layer normalization (LN) is performed to obtain the first... Updated task query matrix :

[0065] ;

[0066] pass The layer iterates through steps S3.1 to S3.4, ultimately outputting the [number]th [layer]. Updated task query matrix

[0067] Step S4: Decode the final task query to generate a gaze probability heatmap and predict the termination decision signal:

[0068] Obtain the final task query set containing rich contextual information, which is output through multiple iterations of the dual-stream interactive decoder.

[0069] The final task query set is input into the spatial prediction head for task status decoding, thereby jointly realizing complete scan path prediction. The specific decoding operation is as follows:

[0070] Step S4.1: Generate a gaze probability heatmap.

[0071] Given the first Query for individual tasks First, the gaze embedding vector is obtained through a two-layer perceptron. :

[0072] ;

[0073] The gaze embedding vector Projected into a feature space with the same dimensions as the P4 channel of the high-resolution feature map, for each spatial location Calculate the dot product match between the embedding vector and the feature vector at that location, and obtain the gaze probability heatmap by applying the sigmoid function. :

[0074] ;

[0075] in, ( ) represents the two-layer perceptron operation used to generate the gaze embedding vector; Indicates the generated first The task in spatial location The probability of fixation at a location; Represents the gaze embedding vector transpose; for In position The channel feature vector, The Sigmoid function is used. This is achieved by analyzing the generated gaze probability heatmap. Perform the maximum value operation to obtain the predicted gaze coordinates at the current time step.

[0076] Step S4.2: Predict the termination decision signal:

[0077] To determine when the scan path ends, the current task is queried. Given the current gaze step number n, the termination probability of the current time step is calculated using a termination prediction perceptron. :

[0078] ;

[0079] in, ( ) represents the two-layer perceptron operation used to calculate the termination probability; This indicates the probability of the prediction terminating at the current time step; Indicates the feature concatenation operation; if the termination probability If the value is greater than 0.5, the generation of the scanning path ends; otherwise, the coordinates of the currently predicted gaze point are fed back to step S2 as the historical gaze position, and the iterative prediction of the next time step continues.

[0080] Step S5: Construct a multi-task joint loss function to train and optimize the prediction model parameters:

[0081] Optimize the model parameters of the global-local cooperative gaze network through supervised learning, and construct a model based on position loss. Distribution loss and termination loss The combined total loss function :

[0082] ;

[0083] Binary cross-entropy is used as the position loss. The true gaze probability map is obtained by Gaussian smoothing the actual gaze point information. To simulate the inherent spatial uncertainty of human gaze and constrain the model's predicted gaze point to focus on the actual gaze area:

[0084]

[0085] in, and These represent the height and width of the view, respectively. ∈[0,1] represents the model at the corresponding position. Predicted fixation probability. This represents the position in the true gaze probability map after Gaussian smoothing. The value at;

[0086] We employ KL divergence as a distribution loss to constrain the gaze probability distribution generated by the model to maintain overall consistency with the true gaze distribution at a macroscopic level.

[0087]

[0088] Among them, index Traverse all flattened spatial positions; This indicates the normalized distribution of true gaze at location. The value at;

[0089] This indicates that the model predicts the gaze distribution at location. The value at that location.

[0090] The termination loss is calculated using the binary cross-entropy method, guiding the model to automatically learn the termination condition of the scan path when generating fixation sequences.

[0091] ;

[0092] in The termination probability predicted by the model. ∈{0,1} represents the true termination label. Using the aforementioned total loss function... Perform gradient backpropagation, iteratively update and optimize model parameters until the model converges, and obtain the trained global-local cooperative gaze network model.

Claims

1. A global-local collaborative visual gaze path prediction method, comprising the following steps: Step S1: Extract multi-scale hierarchical visual features: The image to be tested is acquired and input into a multi-scale feature encoder, and hierarchical visual features are extracted through the backbone network. An efficient multi-scale attention module is used to adaptively calibrate the channel weights of visual features at each stage level; then, the mid-to-high-level features are input into a pixel decoder based on multi-scale deformable attention for cross-scale interaction, outputting a multi-scale feature pyramid [P1, P2, P4] containing macroscopic spatial information and microscopic detail information. Step S2: Decouple and construct the global feature flow and the local semantic feature flow: The multi-scale feature pyramid is explicitly decoupled into a two-stream structure: For low-to-medium resolution features P1 and P2, they are flattened along the spatial dimension, assigned independent multi-scale sinusoidal position codes, and then concatenated to generate a global flow matrix G for encoding the macroscopic scene layout; assuming the current time step is t, based on the set of gaze coordinates from historical time steps... Historical feature sequences are obtained by sampling on the high-resolution feature map P4. and will The input spatiotemporal enhancement module integrates dual spatial embedding and learnable temporal embedding to generate a semantic flow matrix S containing dynamic context; Step S3: Initialize task query and perform hierarchical cross-stream decoding: Initialize the task query matrix Q, and input it together with the global flow matrix G and the semantic flow matrix S into the interactive decoder containing L decoding layers for iterative updates; Within each decoding layer, residual information is first fused between the global stream matrix G and the semantic stream matrix S through a bidirectional cross-attention module. Then, the task query matrix Q is subjected to cross-stream attention calculation with the updated global stream matrix and the semantic stream matrix, respectively. The weights are adaptively calculated by the gated fusion unit to fuse the two streams and output the updated task query matrix. Step S4: Decode the task query to generate a gaze probability heatmap and termination probability: The final updated task query matrix Q is input into the spatial prediction head, and mapped to a gaze embedding vector by a perceptron. The system performs dot product matching with the high-resolution feature map P4 to generate a gaze probability heatmap at the current time step t, from which the gaze coordinates at the current time step are sampled. Simultaneously, the search termination probability τ at the current time step is predicted based on the task query feature Q and the current gaze step number n. If τ < 0.5, the current gaze coordinates are fed back to step S2 for the next time step iteration; otherwise, the prediction is terminated and the complete saccade path is output. Step S5: Train and optimize the prediction model using real human eye saccade data: A joint loss function comprising location loss, distribution loss, and termination loss is constructed. Using the collected real human eye gaze point coordinate sequence and real termination labels, the network parameters of the multi-scale feature encoder, interactive decoder, and spatial prediction head are optimized through gradient backpropagation.

2. The method according to claim 1, characterized in that, In step S1, the multi-scale feature encoder includes a backbone network, a feature enhancement unit, and a pixel-level decoder. The backbone network is used to extract features from the image under test, obtaining hierarchical features from low-level texture to high-level semantics, and retaining the output features Resi (i=2,3,4,5) of the backbone network at different stages. Through the feature enhancement unit, an efficient multi-scale attention module (EMA) is embedded after the output features Resi at each stage to adaptively recalibrate the channel weights of the hierarchical features, so as to suppress background noise and enhance effective feature responses.

3. The method according to claim 2, characterized in that, In step S1, the pixel-level decoder uses a splitting strategy for feature processing: the mid-to-high-level features {Res3, Res4, Res5} are projected and input into the multi-scale deformable attention module (MSD) for cross-scale interaction to obtain the enhanced semantic representation; at the same time, the low-level feature Res2 is directly retained through skip connections; the enhanced semantic representation and the retained low-level feature Res2 are fused using a top-down path to output a hierarchical feature pyramid {P1, P2, P4} containing three scales, where P1 and P2 have large receptive fields to support global structure perception, and P4 integrates deep semantic guidance and low-level texture details.

4. The method according to claim 1, characterized in that, In step S2, the operation of constructing the global flow matrix G specifically involves: flattening the low- and medium-resolution feature maps P1 and P2 in the feature pyramid along the spatial dimension and then stitching them together by scale; introducing independent multi-scale sinusoidal position encoding for each scale to obtain the global flow matrix G. ; in, Let P1 be the first feature vector after being flattened and superimposed with positional encoding, and ; This represents the second feature vector after feature map P2 has been flattened and superimposed with positional encoding, and ; For flattening operation; For splicing operations; and It is an independent multi-scale sinusoidal position coding.

5. The method according to claim 1, characterized in that, In step S2, the operation of constructing the semantic flow matrix S is specifically as follows: When predicting the t-th fixation point, we perform feature sampling on the historical fixation positions on the high-resolution feature map P4 to obtain the historical feature sequence V<t>. To endow the historical feature sequence with spatial context and temporal dynamic information, the historical feature sequence is input into the spatio-temporal enhancement module to reconstruct the context prior: Spatially, a fixed sine position encoding and a learnable structure embedding are used for dual spatial embedding: ; In the time dimension, learnable temporal embeddings are introduced. The updated historical feature sequence is multiplicatively modulated to obtain the semantic flow matrix. : ; Here, ⊙ represents element-wise multiplication.

6. The method according to claim 1, characterized in that, In step S3, the interactive decoder comprises L=5 cascaded decoding layers, and each decoding layer performs the following operations in sequence: (1) Dual-stream feature residual enhancement: By using a bidirectional cross-attention module (BCA), an information channel is established between the global stream matrix and the semantic stream matrix to perform residual fusion: in, This is the weighted residual connectivity coefficient, specifically 0.

2. (2) The first Layer task query matrix Each with the enhanced global flow matrix and the semantic flow matrix The input is fed into the cross-stream attention module for cross-attention interaction, resulting in global query feature matrices containing spatial cues. and local query feature matrix containing semantic evidence ; (3) Adaptive fusion of gating features: Dynamic gating weights are dynamically calculated through a lightweight gating fusion unit containing a two-layer perceptron. Adaptively fuse the features of the two streams to obtain a fused feature representation. ; (4) Iterative update of task query matrix: fusion feature representation After processing by a feedforward network (FFN), it is connected to the first through residual connections. After summing the query matrices of the layer tasks, layer normalization is performed to obtain the result. Updated task query matrix .

7. The method according to claim 6, characterized in that, The dynamic gating weight of the gating fusion unit Calculate the global query feature matrix as follows: With local query feature matrix The components are concatenated, projected onto a low-dimensional latent space via the first-layer network weights W1, and then integrated using the ReLU activation function. These are then mapped to scalars by the second-layer network weights W2, and finally processed using the Sigmoid normalization function to obtain the final result. ; The dynamic gating weight Γ is used to adaptively weight and fuse the two features: 。 8. The method according to claim 1, characterized in that, In step S4, the specific operation of generating the gaze probability heatmap is as follows: given the first... Task query The gaze embedding vector is obtained through a two-layer perceptron. : ; The gaze embedding vector Projected into a feature space with the same dimensions as the P4 channel of the high-resolution feature map, for each spatial location Calculate the dot product match between the embedding vector and the feature vector at that location, and obtain the gaze probability heatmap by applying the sigmoid function. : ; By generating a gaze probability heatmap Perform the maximum value operation to obtain the predicted gaze coordinates at the current time step.

9. The method according to claim 1, characterized in that, In step S4, the operation of predicting the termination decision signal specifically involves: querying the current task. Given the current gaze step number n, the termination probability of the current time step is calculated using a termination prediction perceptron. : ; in, ( ) represents the two-layer perceptron operation used to calculate the termination probability; if the termination probability If the value is greater than 0.5, the generation of the scanning path ends; otherwise, the coordinates of the currently predicted gaze point are fed back to step S2 as the historical gaze position, and the iterative prediction of the next time step continues.

10. The method according to claim 1, characterized in that, In step S5, the joint loss function L is composed of position loss. Distribution loss and termination loss composition: ; in, Using binary cross-entropy as the position loss, the true gaze probability map P* is obtained by Gaussian smoothing the true gaze point information, which constrains the model's predicted gaze points to focus on the true gaze region. Using KL divergence as the distribution loss, the model is constrained at a macro level to maintain overall consistency between the gaze probability distribution generated by the model and the true gaze distribution. The termination loss is calculated using the binary cross-entropy method, guiding the model to automatically learn the termination conditions of the scan path: ; in, The termination probability predicted by the model. ∈{0,1} is the true termination label.

Images