Saccade path prediction method, system and terminal based on active vision decision

Abstract

The application provides a saccade path prediction method, system and terminal based on active visual decision, which comprises the following steps: extracting global down-sampling features of a visual scene and local semantic features around a current fixation point through a visual perception module; combining the local semantic features with historical state information to sequentially update a visual hidden state; inputting the updated visual hidden state into a visual decision module comprising a policy network and a value network to evaluate the expected gain of the current visual state; determining whether to actively terminate observation according to the evaluation result, and if not, sampling and outputting a next fixation point position through the policy network. The application also provides a system and a terminal corresponding to the above method. The application converts passive overall perception into multi-stage active decision, truly simulates the exploration mechanism of human foveal vision from coarse to fine, effectively eliminates redundant interference, and realizes high prediction accuracy and sequence consistency in complex scenes such as panoramas.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a saccade path prediction method, system, and terminal based on active visual decision-making, and in particular, a saccade sequence generation method for complex panoramic images that combines reinforcement learning and brain-like active perception mechanisms. Background Technology

[0002] When humans dynamically explore visual scenes, eye movements mainly consist of fixation and saccades. A saccade path refers to the time sequence in which a person's eyes move from one point of interest to another while viewing a visual scene. Accurate prediction of saccade paths has significant application value in fields such as virtual reality (VR), human-computer interaction, and educational behavior analysis.

[0003] Existing saccade path prediction methods are mainly divided into traditional models based on saliency maps and data-driven models based on deep learning. However, most existing methods typically adopt a passive, holistic approach, directly extracting features and modeling the entire scene image in a single forward computation. This approach has the following shortcomings: First, it differs significantly from the actual human visual cognition mechanism. When observing complex scenes, humans often do not process all visual information at once, but rather gradually acquire local information through continuous fixation and actively adjust their exploration strategies. Second, for structurally complex or high-resolution scenes (such as 360-degree panoramic images), processing the entire image at once not only introduces a large amount of irrelevant redundant information but also incurs huge computational costs, making it difficult to achieve effective modeling for long-term temporal dependencies. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, the present invention aims to provide a scanning path prediction method, system, and terminal based on active visual decision-making. This method transforms scanning path generation from a traditional passive static regression problem into an active dynamic decision-making problem, thereby more realistically simulating the human visual system's observation mode of gradual exploration from coarse to fine.

[0005] The specific technical solution for achieving the objective of this invention is as follows:

[0006] A saccade path prediction method based on active visual decision-making includes the following steps:

[0007] The visual scene features are acquired through the visual perception module. The acquisition adopts a combination of local high-definition perception and global downsampling perception, which simulates the human foveal mechanism. The process includes extracting global downsampling features for initialization and extracting local high-level semantic features around the current gaze point at each time step.

[0008] After concatenating local high-level semantic features and historical decision-accumulated memory in the feature dimension, they are input together with the previous visual latent state representing the historical context into the recurrent neural network unit. Through nonlinear operation mechanism, the temporal dynamic fusion of multi-dimensional features is realized, thereby completing the serialization update of the visual latent state.

[0009] The updated visual hidden state is input into the visual decision-making module, which includes a policy network and a value network;

[0010] The expected information gain of continuing observation in the current visual latent state is evaluated through the value network, and the expected information gain is compared with the set objective environmental constraint threshold to determine whether to actively terminate the generation of saccade sequences.

[0011] If the expected information gain is determined to be greater than the objective environmental constraint threshold, i.e., the observation is insufficient, the next gaze point coordinates are obtained by sampling the probability distribution of the current visual latent state through the policy network, and the next time step is entered; if the expected information gain is determined to be less than or equal to the objective environmental constraint threshold, the observation is actively terminated and a complete saccade path sequence is output.

[0012] Furthermore, the acquisition of visual scene features through the visual perception module specifically includes:

[0013] The original panoramic visual image after downsampling is used as input, and semantic features representing global downsampling perception are obtained through a visual feature extractor. Used to determine the initial fixation point position of the saccade sequence. ;

[0014] At time step t, with the current gaze point Centered on, cut off the area with a size of The local fixation region, the size of the local fixation region Smaller than the resolution of the original visual environment;

[0015] The local gaze region is input into a visual feature extractor to obtain local high-level semantic features representing the current gaze state. .

[0016] Furthermore, the serialization update of the visual latent state specifically includes:

[0017] Employing a continuous visual latent state chain This indicates the internal visual state of the current time step and the history.

[0018] At time step The current visual hidden state is updated step by step according to the following state evolution equation:

[0019] ;

[0020] in, This represents the visual latent state of the previous moment. This represents the local semantic features obtained from the local gaze region at the current moment. This represents the accumulated memory matrix of historical decision-making behaviors from the initial time step to the current time step. This is a state update function based on a recurrent neural network structure.

[0021] Furthermore, the visual decision-making module adopts an actor-critic architecture, and its specific process for evaluating the expected information gain and sampling the next gaze point includes:

[0022] Value Network Receive the current visual hidden state Calculate using the current policy network In state Continue to observe the expected cumulative return, and output the state value. ;

[0023] State value With dynamically set thresholds Perform size comparison;

[0024] like < If the additional information gained from further observation is insufficient to offset the perception cost, the perception process will be terminated and a termination signal will be output.

[0025] like If this is not the case, it is determined that the current observation of the scene is insufficient, and the policy network is triggered. ;

[0026] Policy Network Based on the probability distribution of the current visual latent state output action space, random samples are taken from it to obtain the specific gaze point action coordinates for the next moment. .

[0027] Furthermore, the policy network and the value network are jointly trained using a hybrid loss function for parameter updates. Including the optimization objective loss function term and supervised learning loss function term Its expression is:

[0028]

[0029] in, Weight coefficients for supervised learning;

[0030] In the early stages of training the policy network and value network, a supervised warm-up phase is set up, and the weight coefficients are... Set to a large value, then pass Measure the error between the predicted trajectory and the actual trajectory to guide the model to quickly learn basic human eye saccade path trends;

[0031] In the later stages of model training, the weight coefficients The loss function gradually decays with each training epoch, and the model primarily relies on an optimization objective loss term that includes reinforcement learning rewards. Conduct autonomous exploration and strategy network optimization.

[0032] Furthermore, the optimization objective loss function term The gradient includes a loss term based on reinforcement learning, which aims to maximize the expected cumulative reward at time steps. Received instantaneous comprehensive reward function It consists of a weighted sum of four sub-items, and its expression is:

[0033]

[0034] in, Rewards are based on spatial probability consistency. Rewards for structured exploration, As a penalty for partial stagnation, Rewards are given for sequence structure consistency. , , , These are the corresponding weighting coefficients.

[0035] Furthermore, the comprehensive reward function The specific calculation methods for the four sub-items are as follows:

[0036] (1) Spatial probability consistency reward This is used to constrain the sampling of predicted gaze points to be located near regions with high actual gaze probability.

[0037]

[0038] in, Indicates the predicted fixation point The value of the true gaze density map constructed based on the true gaze distribution;

[0039] (2) The structured exploration reward Incorporating spatial displacement magnitude, the policy network is encouraged to explore unvisited regions:

[0040]

[0041] in, For the access history memory map to be updated with decision-making steps in action The value at that location, Indicates from action arrive The resulting spatial displacement amplitude;

[0042] (3) The local stagnation penalty Constructed based on the average displacement amplitude of the nearest K steps, it is used to suppress the collapse of the policy network during long-term dwell times within a very small spatial range:

[0043]

[0044] in, This is an indicator function; when the average displacement over K consecutive steps is less than a set small-amplitude movement threshold... At that time, the punishment A value of 1 triggers a penalty;

[0045] (4) The sequence structure consistency reward Calculate sequence alignment error using the Dynamic Time Warping (DTW) algorithm within a sliding window:

[0046]

[0047] in, For sliding windows Predicted gaze trajectory within, This corresponds to the actual eye movement fixation trajectory;

[0048] The weighting coefficient , , , A dynamic adaptive weight allocation strategy is adopted, and its value changes with the training progress. For smoothing adjustment, the expression is:

[0049]

[0050] in, , This indicates the current training progress as a percentage of the total training rounds. As the initial weights, The initial weights of the exploration reward and structured penalty are determined. Greater than the last option weight So that its corresponding weight value varies proportionally. The increase monotonically decreases; for the rewards of spatial probability consistency and sequence structure consistency, their initial weights are limited. Less than the final option weight So that its corresponding weight value varies proportionally. It increases monotonically with the increase of .

[0051] Furthermore, the features of the visual scene are derived from the panoramic image; when the visual scene is a 360-degree panoramic image, the coordinates of the next gaze point are derived from two-dimensional latitude and longitude coordinates. Convert to 3D spherical coordinates Representation and feature learning are performed to eliminate edge spatial discontinuities caused by isometric projection (ERP) of panoramic images.

[0052] A saccade path prediction system implementing the above method, the system comprising:

[0053] The visual perception module is used to acquire features of the visual scene, including extracting global downsampling features for initialization, and truncating the area around the current gaze point and extracting local semantic gaze features at each time step.

[0054] The visual state update module, connected to the visual perception module, is used to combine the acquired local semantic gaze features, historical context information and historical decision accumulated memory, and to perform serialized dynamic update of the visual latent state chain through a recurrent neural network structure.

[0055] An active visual decision-making module, connected to the visual state update module, includes a policy network and a value network. The value network receives the visual latent state, evaluates the expected information gain for continued observation, and compares the expected information gain with an environmental constraint threshold to issue a termination signal or a continuation instruction. The policy network, upon receiving a continuation instruction, outputs the probability distribution of the next gaze point behavior coordinates and samples it.

[0056] The trajectory output module is used to integrate the gaze coordinates of all historical time steps and output the final saccade path sequence when the termination signal is received.

[0057] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned saccade path prediction method based on active visual decision-making.

[0058] A panoramic image saliency prediction terminal, comprising:

[0059] At least one processor;

[0060] and a memory communicatively connected to the at least one processor;

[0061] The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the saccade path prediction method based on active visual decision-making.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] This invention transforms traditional passive learning into active perception, using reinforcement learning to enable the model to dynamically determine the next gaze position based on the currently acquired information, just like a human, and to actively stop when sufficient information has been collected.

[0064] By combining local high-definition sensing that simulates the human fovea mechanism with global downsampling sensing, the computational overhead and noise interference when processing complex panoramic images are greatly reduced.

[0065] By comprehensively utilizing hybrid reward functions for policy learning, the accuracy and naturalness of the model's predicted sequences on long-term time-series dependencies are improved, and defects such as local stagnation are overcome. Attached Figure Description

[0066] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0067] Figure 1 The flowchart shows an existing conventional panoramic image scanning path prediction method.

[0068] Figure 2 This is a flowchart of a saccade path prediction method based on active visual decision-making according to an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the overall model architecture and decision-making process of a preferred embodiment of the present invention;

[0070] Figure 4 This is a block diagram of a scanning path prediction system based on active visual decision-making in one embodiment of the present invention;

[0071] Figure 5 This is a visual comparison diagram of the method of the present invention with other methods. Detailed Implementation

[0072] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0073] Figure 1 The flowchart below illustrates a conventional panoramic image scanning path prediction method, which typically employs a "passive" overall processing architecture. The process involves first feeding the entire input scene image into a deep convolutional neural network (CNN) or Transformer encoder for one-time, full-image feature extraction. Then, a recurrent neural network (RNN / LSTM) or autoregressive structure is used to continuously output a complete gaze point sequence in a single forward computation. The drawbacks of this method are: first, it lacks cognitive initiative, as the model encodes all visual information indiscriminately, failing to simulate the dynamic decision-making process of gradual exploration from coarse to fine in humans; second, it suffers from severe feature redundancy and computational overhead, as high-resolution whole-image processing introduces a large amount of background noise unrelated to gaze points and struggles to effectively capture long-term temporal dependencies, thus limiting prediction accuracy and efficiency.

[0074] Figure 2 This is a flowchart of a saccade path prediction method based on active visual decision-making according to an embodiment of the present invention. As can be seen from the figure, the present invention changes the conventional passive whole-image encoding mode and models saccade path prediction as an active decision-making process that proceeds from coarse to fine and is sequential.

[0075] Figure 3 This is a schematic diagram illustrating the overall architecture and decision-making process of a preferred embodiment of the present invention. (Refer to...) Figure 3 As shown, in this preferred embodiment, the saccade path prediction method based on active visual decision-making can be performed according to the following steps:

[0076] S1: Visual Perception Stage. Using a visual perception module to simulate human foveal vision, the downsampled original panoramic image is first input into the perceptual neural network to extract global overview semantic features. As the initial state ; then at the time step Based on the current gaze action Cropping local gaze areas from panoramic images Local semantic features are extracted using a high-resolution encoder (such as Vision Transformer). .

[0077] S2: Visual hidden state update phase. The currently acquired local features are compared with the visual hidden state from the previous time step. And the accumulated memory of historical decision-making behavior Combined, the current visual hidden state is updated through the state update function: .

[0078] S3: Visual Decision and Policy Sampling Phase. The updated visual latent state is input into the visual decision module (Actor-Critic structure) containing the policy network and value network. The value network calculates the expected gain of the current observation. The results are compared with a resource constraint threshold. If sufficient information is available, the generation of the saccadic sequence is terminated proactively; if insufficient observation is determined, the process is handled through a policy network. The next fixation point position is obtained by sampling. .

[0079] S4: Hybrid Optimization and Reward Feedback Stage. Model training employs a hybrid loss function combining supervised learning and reinforcement learning. In the reinforcement learning stage, a spatially probabilistic consistency reward is introduced. Rewards for structured exploration based on longitude displacement Stagnation penalty to suppress local collapse And time series alignment (DTW) reward for constrained short-term trajectory structure The comprehensive reward function is constructed, and the policy is optimized by dynamically adaptive weights based on the training progress.

[0080] In some preferred embodiments, the specific hyperparameter settings of the model are as follows: the experiment is run on the PyTorch framework, the batch size is set to 64, the total number of training epochs is 300, the optimizer is AdamW, and the learning rate is set to... Gradually decrease to To facilitate smooth optimization in the early stages of training, a supervised warm-up phase was introduced for the first 20 epochs. The action distribution employed a parameterized Gaussian distribution with a discount factor. Advantage function estimation parameters The strategy update pruning factor is set to 0.2.

[0081] In one specific embodiment, the method of the present invention (including the preferred embodiments MEScan360_M and MEScan360_L) is quantitatively compared with existing baseline methods such as CLE, DeepgazeIII, ScanGAN, Pathformer3D, and ScanDTM on several publicly available panoramic image scanning path prediction evaluation sets. The results show that the method of the present invention achieves stable performance improvements over traditional passive holistic baseline methods in multiple scenarios with different visual feature distributions and complexities. Among them, the version combining long memory enhancement strategy and SCNN method (MEScan360_M) achieved the best performance in most evaluations, while the version combining long memory enhancement and robust feature extraction module (MEScan360_L) also showed excellent generalization ability; both significantly outperformed existing baselines in temporal alignment and spatial localization accuracy, and the predicted trajectory closely approximated the natural visual exploration benchmark of real humans, indicating that the simulated fovea mechanism and active visual decision-making framework introduced in the present invention achieve an excellent balance between reducing redundant information interference and long temporal sequence prediction.

[0082] In this embodiment, edit distance (LEV), scan matching degree (ScanMatch), dynamic temporal warping distance (DTW), recursion degree (REC), temporal delay embedding distance (TDE), and shape similarity (SS) are used as performance metrics. LEV, DTW, and TDE are error metrics; the smaller their values, the lower the alignment error between the predicted gaze sequence and the real eye-tracking sequence in spatiotemporal geometry. ScanMatch, REC, and SS are similarity metrics; the larger their values, the higher the consistency between the model's sampled scan path and the real human observation pattern in terms of temporal coherence and local observation distribution. The four publicly available panoramic image scan path prediction datasets include Sitzmann, Salient360!, AOI, and JUFE.

[0083] like Figure 5As shown, the method of this invention is qualitatively and visually compared with baseline methods such as DeepGazeIII, SaltiNet, ScanGAN, ScanDMM, and Pathformer3D in several typical panoramic image scenes. The results show that, when facing complex indoor and outdoor panoramic environments, the method of this invention achieves prediction results closer to the real eye movement distribution compared to traditional baseline methods. After comparing the scanning trajectory features of different methods, the gaze points generated by this invention converge more accurately in highly semantically salient regions of the image (such as crowds, core buildings, etc.), and the scanning span and trajectory direction are more natural and smooth, effectively overcoming the problems of excessive trajectory divergence, redundancy, or random jumps exhibited by baseline models such as SaltiNet. This demonstrates that the visual perception and decision-making framework described in this invention effectively improves the simulation of real human observation behavior. Furthermore, through direct comparison with real human eye movement trajectories (Ground-Truth), the scanning sequence pattern output by this invention maintains extremely high spatial distribution consistency and temporal rationality even under complex background interference, verifying the practical superiority of the method of this invention in panoramic image scanning path prediction tasks.

[0084] Table 1. Quantitative comparison of the method of this invention with other methods on four public datasets.

[0085]

[0086] Figure 4 This is a block diagram of a saccade path prediction system based on active visual decision-making according to an embodiment of the present invention. This system can be used to implement the aforementioned saccade path prediction method based on active visual decision-making. (Refer to...) Figure 4 As shown, the saccade path prediction system based on active visual decision-making in this embodiment includes: a visual perception module, a visual hidden state update module, a visual decision-making module, and a hybrid optimization training module; wherein: the visual perception module is used to extract global downsampling features and local high-level semantic features; the visual hidden state update module is used to dynamically evolve the hidden state in combination with historical context; the visual decision-making module is used to perform policy sampling and observation termination judgment; and the hybrid optimization training module is used to update the network parameters by combining supervised warm-up and multiple reward functions.

[0087] Based on the above embodiments, in another embodiment of the present invention, a panoramic image scanning path prediction terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any of the above-described scanning path prediction methods based on active visual decision-making. This method not only perfectly simulates the cognitive process of the human visual system from coarse to fine and active exploration, but also achieves extremely high accuracy and robustness in long-term complex panoramic prediction.

[0088] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0089] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0090] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0091] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0092] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding units in the device. Those skilled in the art can implement the steps of the method by referring to the technical solution of the device. That is, the embodiments in the device can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0093] Those skilled in the art will understand that, in addition to implementing the apparatus provided by the present invention in the form of purely computer-readable program code, the method steps can be logically programmed to enable the apparatus and its various units to achieve the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the apparatus provided by the present invention can be considered a hardware component, and the units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0094] The above is a description of specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above-described preferred features can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other.

Claims

1. A saccade path prediction method based on active visual decision-making, characterized in that, Includes the following steps: The visual scene features are acquired through the visual perception module. The acquisition adopts a combination of local high-definition perception and global downsampling perception, which simulates the human foveal mechanism. The process includes extracting global downsampling features for initialization and extracting local high-level semantic features around the current gaze point at each time step. After concatenating local high-level semantic features and historical decision-accumulated memory in the feature dimension, they are input together with the previous visual latent state representing the historical context into the recurrent neural network unit. Through nonlinear operation mechanism, the temporal dynamic fusion of multi-dimensional features is realized, thereby completing the serialization update of the visual latent state. The updated visual hidden state is input into the visual decision-making module, which includes a policy network and a value network; The expected information gain of continuing observation in the current visual latent state is evaluated through the value network, and the expected information gain is compared with the set objective environmental constraint threshold to determine whether to actively terminate the generation of saccade sequences. If the expected information gain is determined to be greater than the objective environmental constraint threshold, i.e., the observation is insufficient, the next gaze point coordinates are obtained by sampling the probability distribution of the current visual latent state through the policy network, and the next time step is entered; if the expected information gain is determined to be less than or equal to the objective environmental constraint threshold, the observation is actively terminated and a complete saccade path sequence is output.

2. The saccade path prediction method based on active visual decision-making according to claim 1, characterized in that, The acquisition of visual scene features through the visual perception module specifically includes: The original panoramic visual image after downsampling is used as input, and semantic features representing global downsampling perception are obtained through a visual feature extractor. Used to determine the initial fixation point position of the saccade sequence. ; At time step t, with the current gaze point Centered on, cut off the area with a size of The local fixation region, the size of the local fixation region Smaller than the resolution of the original visual environment; The local gaze region is input into a visual feature extractor to obtain local high-level semantic features representing the current gaze state. .

3. The saccade path prediction method based on active visual decision-making according to claim 1, characterized in that, The serialization update of the visual latent state specifically includes: Employing a continuous visual latent state chain This indicates the internal visual state of the current time step and the history. At time step The current visual hidden state is updated step by step according to the following state evolution equation: ； in, This represents the visual latent state of the previous moment. This represents the local semantic features obtained from the local gaze region at the current moment. This represents the accumulated memory matrix of historical decision-making behaviors from the initial time step to the current time step. This is a state update function based on a recurrent neural network structure.

4. The saccade path prediction method based on active visual decision-making according to claim 1, characterized in that, The visual decision-making module, employing an actor-critic architecture, includes the following specific processes for evaluating expected information gain and sampling the next gaze point: Value Network Receive the current visual hidden state Calculate using the current policy network In state Continue to observe the expected cumulative return, and output the state value. ; State value With dynamically set thresholds Perform size comparison; like < If the additional information gained from further observation is insufficient to offset the perception cost, the perception process will be terminated and a termination signal will be output. like If this is not the case, it is determined that the current observation of the scene is insufficient, and the policy network is triggered. ; Policy Network Based on the probability distribution of the current visual latent state output action space, random samples are taken from it to obtain the specific gaze point action coordinates at the next moment. .

5. The saccade path prediction method based on active visual decision-making according to claim 4, characterized in that, The policy network and the value network are jointly trained for parameter updates using a hybrid loss function. Includes the optimization objective loss function term and supervised learning loss function term Its expression is: ； in, Weight coefficients for supervised learning; A supervised warm-up phase is set up in the early stage of training the policy network and the value network, and the weight coefficients are... Set to a large value, then... Measure the error between the predicted trajectory and the actual trajectory to guide the model to quickly learn basic human eye saccade path trends; In the later stages of model training, the weight coefficients The model dependency gradually decreases with each training epoch, and the loss function term includes the optimization objective and reinforcement learning rewards. Conduct autonomous exploration and strategy network optimization.

6. The saccade path prediction method based on active visual decision-making according to claim 5, characterized in that, The optimization objective loss function term The gradient includes a loss term based on reinforcement learning, which aims to maximize the expected cumulative reward at time steps. Received instantaneous comprehensive reward function It consists of a weighted sum of four sub-items, and its expression is: ； in, Rewards are based on spatial probability consistency. Rewards for structured exploration, As a penalty for partial stagnation, Rewards are given for sequence structure consistency. , , , These are the corresponding weighting coefficients.

7. The saccade path prediction method based on active visual decision-making according to claim 6, characterized in that, The comprehensive reward function The specific calculation methods for the four sub-items are as follows: (1) Spatial probability consistency reward This is used to constrain the sampling of predicted gaze points to be located near regions with high actual gaze probability. ； in, Indicates the predicted fixation point The value of the true gaze density map constructed based on the true gaze distribution; (2) The structured exploration reward Incorporating spatial displacement magnitude, the policy network is encouraged to explore unvisited regions: ； in, For the access history memory map to be updated with decision-making steps in action The value at that location, Indicates from action arrive The resulting spatial displacement amplitude; (3) The local stagnation penalty Constructed based on the average displacement amplitude of the nearest K steps, it is used to suppress long-term dwell collapse of the policy network within a very small spatial range: ； in, This is an indicator function; when the average displacement over K consecutive steps is less than a set small-amplitude movement threshold... At that time, the local stagnation penalty A value of 1 triggers a penalty; (4) The sequence structure consistency reward Calculate sequence alignment error using a dynamic time warping algorithm within a sliding window: ； in, For sliding windows Predicted gaze trajectory within, This corresponds to the actual eye movement fixation trajectory; The weighting coefficient , , , A dynamic adaptive weight allocation strategy is adopted, and its value changes with the training progress. For smoothing adjustment, the expression is: ； in, , This indicates the current training progress as a percentage of the total training rounds. As the initial weights, The initial weights of the exploration reward and structured penalty are determined. Greater than the last option weight So that its corresponding weight value varies proportionally. The value of the spatial probability consistency reward and the sequence structure consistency reward decreases monotonically with the increase of the spatial probability consistency reward and the sequence structure consistency reward. Their initial weights are limited. Less than the final option weight So that its corresponding weight value varies proportionally It increases monotonically with the increase of .

8. The saccade path prediction method based on active visual decision-making according to claim 1, characterized in that, The features of the visual scene are derived from the panoramic image; when the visual scene is a 360-degree panoramic image, the coordinates of the next gaze point are derived from two-dimensional latitude and longitude coordinates. Convert to 3D spherical coordinates Representation and feature learning are performed to eliminate the edge spatial discontinuities caused by isometric projection of panoramic images.

9. A saccade path prediction system implementing the method of claim 1, characterized in that, The system includes: The visual perception module is used to acquire features of the visual scene, including extracting global downsampling features for initialization, and truncating the area around the current gaze point and extracting local semantic gaze features at each time step. The visual state update module, connected to the visual perception module, is used to combine the acquired local semantic gaze features, historical context information and historical decision accumulated memory, and to perform serialized dynamic update of the visual latent state chain through a recurrent neural network structure. An active visual decision-making module, connected to the visual state update module, includes a policy network and a value network. The value network receives the visual latent state, evaluates the expected information gain for continued observation, and compares the expected information gain with an environmental constraint threshold to issue a termination signal or a continuation instruction. The policy network, upon receiving a continuation instruction, outputs the probability distribution of the next gaze point behavior coordinates and samples it. The trajectory output module is used to integrate the gaze coordinates of all historical time steps and output the final saccade path sequence when the termination signal is received.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the saccade path prediction method based on active visual decision-making as described in any one of claims 1-8.

11. A panoramic image saliency prediction terminal, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the saccade path prediction method based on active visual decision-making as described in any one of claims 1-8.

Images