Personality-driven panoramic scene scanning path prediction method, system, terminal and storage medium
By incorporating personality traits into saccade path prediction and generating fixation points using visual encoding and adaptive attention modules, the problems of individual differences and computational overhead are addressed, thereby improving individual consistency and interpretability.
Patent Information
- Application Number
- CN202611392336.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-09
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-09-09
AI Technical Summary
Existing saccade path prediction techniques fail to effectively consider individual differences, struggle to balance long-range dependencies with local details, and incur significant computational costs, resulting in inconsistent prediction results and insufficient interpretability.
By introducing personality features as conditional signals, a visual encoder and a personality modulation parameter generation network are used to extract and modulate features from panoramic scene images. Combined with a hierarchical adaptive attention module and a spatial probability generation module, a saccade path of the gaze point is gradually generated.
It improves the individual consistency and interpretability of saccade path prediction, reduces computational burden while preserving key visual behavioral cues, and enables the simultaneous modeling of global exploration strategies and local details.
Smart Images

Figure CN122906398B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, visual attention modeling, virtual reality and artificial intelligence, and specifically to a method, system, terminal and storage medium for probabilistic prediction of individualized saccade paths in a panoramic scene by using observer personality characteristics as conditional signals. Background Technology
[0002] Sagging paths are sequences of fixations arranged in time and saccades between adjacent fixations when an observer views a visual scene. Accurate prediction of sagging paths can serve applications such as virtual reality content presentation, personalized human-computer interaction, visual quality assessment, attentional resource allocation, and auxiliary psychological and behavioral analysis. For 360-degree panoramic images, observers can freely rotate their gaze within a spherical space; the scene range is large, the spatial topology is periodic, and fixation behavior exhibits significant long-range temporal dependence.
[0003] Existing saccadic path prediction techniques typically learn average salient regions or uniform trajectory patterns based on group eye-tracking data. These techniques often use visual content as the primary input, assuming that different observers have approximately consistent attentional allocations in the same scene, thus weakening individual differences during training. In fact, stable personality traits such as openness, conscientiousness, extraversion, agreeableness, and emotional stability influence an observer's spatial exploration range, structured search tendency, and gaze shifting patterns. In the same panoramic scene, observers with different personality traits may form significantly different saccadic paths.
[0004] Furthermore, saliency map-based sampling methods struggle to adequately represent the long-range dependencies of gaze shifts; generative adversarial models may suffer from training instability and mode collapse; simple Markov models are limited by the local shift assumption; and while flat-stacked Transformer or diffusion models possess strong sequence modeling capabilities, they incur significant computational overhead on long sequences and struggle to simultaneously consider global scene semantics, local visual details, and personality conditions. Therefore, a panoramic scene saccade path prediction scheme is needed that can explicitly introduce personality priors, adaptively compress and recover sequences across multiple scale levels, and progressively generate gaze points in a spatially probabilistic manner. Summary of the Invention
[0005] To address the problems of existing technologies, such as the lack of explainable individual conditions, difficulty in balancing long-range dependence and local details, and insufficient individual consistency in predicted trajectories, the purpose of this invention is to provide a personality-driven panoramic scene scanning path prediction method, system, terminal, and storage medium.
[0006] According to a first aspect of the present invention, a personality-driven panoramic scene saccade path prediction method is provided, comprising the following steps:
[0007] S1. Acquire panoramic scene images, observer's personality characteristic information, and initial gaze point; normalize the personality characteristic information according to preset benchmark values and scale factors corresponding to different personality dimensions to obtain a personality characteristic vector corresponding to the observer; establish a historical gaze point sequence, and take the initial gaze point as the first gaze point in the historical gaze point sequence;
[0008] S2. Use a visual encoder to perform hierarchical feature extraction and dimension mapping on the panoramic scene image to obtain a visual feature sequence; use the personality feature vector as observer condition information, generate scaling parameters and offset parameters corresponding to the visual feature dimensions through a personality modulation parameter generation network, and use the scaling parameters and offset parameters to perform feature linear modulation on the visual feature sequence to obtain a personality-conditional visual feature sequence.
[0009] S3. Convert each historical gaze point in the historical gaze point sequence into a three-dimensional spatial coordinate representation and map it into a gaze embedding representation. Form a gaze embedding sequence in chronological order. Concatenate the gaze embedding sequence with the personality conditional visual feature sequence to obtain a multimodal input sequence.
[0010] S4. Input the multimodal input sequence into the hierarchical adaptive attention module, and perform hierarchical encoding, dynamic routing block division, high-level inference, deblocking and decoding processing in sequence to obtain an enhanced sequence representation that fuses the personality conditional visual feature sequence and the gaze embedding sequence.
[0011] S5. Input the enhanced sequence representation into the spatial probability generation module to obtain the three-dimensional mean parameter and the three-dimensional log-variance parameter for determining the spatial location probability distribution of the next gaze point; perform a preset range limit on the three-dimensional log-variance parameter, and perform an exponential transformation on the limited three-dimensional log-variance parameter to obtain the three-dimensional variance parameter; construct a three-dimensional spatial probability distribution based on the three-dimensional mean parameter and the three-dimensional variance parameter, and randomly sample three-dimensional candidate gaze vectors from the three-dimensional spatial probability distribution; perform unit vector normalization on the three-dimensional candidate gaze vectors to obtain the next gaze point located on a unit sphere, and add the next gaze point to the historical gaze point sequence;
[0012] S6. Determine whether the number of foves in the updated historical fove sequence has reached a preset number; if the number of foves in the historical fove sequence has not reached the preset number, repeat steps S3 to S5 based on the updated historical fove sequence, that is, repeat the formation of the fove embedding sequence and the multimodal input sequence, the acquisition of the enhanced sequence representation, and the determination and addition of the next fove, until the number of foves in the historical fove sequence reaches the preset number; when the preset number is reached, determine the historical fove sequence as a predicted saccade path.
[0013] Optionally, the personality characteristic information includes continuous scores of at least one personality dimension among extraversion, agreeableness, conscientiousness, openness, and emotional stability; for each personality dimension, the difference between the continuous score and the corresponding preset benchmark value is divided by the corresponding scaling factor, and the result is restricted to a preset numerical range to obtain the corresponding normalized personality characteristic value, and the normalized personality characteristic values are combined in the order of the preset personality dimensions to form the personality characteristic vector;
[0014] The personality modulation parameter generation network maps the personality feature vector to scaling parameters corresponding to the visual feature dimension through a multilayer perceptron. and offset parameters .
[0015] Optionally, the personality-conditional visual feature sequence satisfy: ,in, It is a visual feature sequence. This represents element-wise multiplication. For scaling parameters, This is the offset parameter.
[0016] Optionally, the visual encoder is a hierarchical visual Transformer with a shift window, and the intermediate stage output of the visual encoder that retains a preset spatial resolution is selected, and the intermediate stage output is mapped to the unified feature dimension of the hierarchical adaptive attention module through a linear projection layer.
[0017] Optionally, obtaining the multimodal input sequence specifically involves: acquiring the three-dimensional spatial coordinate representations corresponding to each historical gaze point, and obtaining the corresponding gaze embedding representations through linear mapping; arranging the gaze embedding representations according to the chronological order of each historical gaze point to form the gaze embedding sequence; using the personality-conditional visual feature sequence as a context prefix and concatenating it with the gaze embedding sequence to obtain the multimodal input sequence; wherein the gaze embedding sequence is autoregressive modeled in chronological order under causal constraints.
[0018] Optionally, the hierarchical adaptive attention module includes at least two levels of encoders, at least two levels of dynamic routing block units, a main network, deblocking units corresponding one-to-one with each of the dynamic routing block units, and at least two levels of decoders; at least one of the encoders, the main network, and the decoders is composed of one or more stacks of state space sequence layers and Transformer layers to model the long-range dependencies of the multimodal input sequence.
[0019] Each of the dynamic routing block units is used to calculate the boundary probability or retention probability corresponding to each sequence element in the current layer input sequence, and select and merge the sequence elements according to the boundary probability or retention probability to form a length-compressed block sequence; the multimodal input sequence is processed step by step by encoders and dynamic routing block units to form a length-compressed high-level input representation, and the main network performs high-level inference on the high-level input representation to obtain a high-level sequence representation; the high-level sequence representation is used to recover local temporal details step by step by deblocking units corresponding to each of the dynamic routing block units and the decoder to obtain an enhanced sequence representation aligned with the input length.
[0020] Optionally, the three-dimensional mean parameter and the three-dimensional log-variance parameter are respectively the three-dimensional mean vector and the three-dimensional log-variance vector; the spatial probability generation module outputs the three-dimensional mean vector and the three-dimensional log-variance vector for each prediction time step, and restricts the three-dimensional log-variance vector to a preset numerical range; the restricted three-dimensional log-variance vector is subjected to an exponential transformation to determine the variance corresponding to each spatial dimension, and a three-dimensional diagonal Gaussian distribution is constructed using the three-dimensional mean vector as the mean and the diagonal matrix formed by the variances corresponding to each spatial dimension as the covariance matrix;
[0021] The saccade path prediction model, which includes the visual encoder, personality modulation parameter generation network, hierarchical adaptive attention module and spatial probability generation module, is trained using the next gaze point autoregressive supervision method. During training, the sequence from the initial gaze point to the penultimate gaze point is used as the training input sequence, and the sequence from the second gaze point to the last gaze point is used as the training target sequence. The training loss is determined based on the negative log-likelihood of the target gaze point at each prediction time step under the three-dimensional diagonal Gaussian distribution.
[0022] During inference, three-dimensional candidate gaze vectors are randomly sampled from the three-dimensional diagonal Gaussian distribution. The three-dimensional candidate gaze vectors are then normalized to unit vectors to obtain the next gaze point located on a unit sphere. When mapping the next gaze point to the panoramic scene image, the three-dimensional unit sphere coordinates of the next gaze point are converted into the corresponding panoramic image coordinates.
[0023] According to a second aspect of the present invention, a personality-driven panoramic scene saccade path prediction system corresponding to the above-described method is provided, comprising:
[0024] The input and preprocessing module is used to acquire panoramic scene images, personality feature information, and initial gaze points; preprocess the personality feature information to obtain personality feature vectors; and establish a historical gaze point sequence, using the initial gaze point as the first gaze point in the historical gaze point sequence.
[0025] A personality-guided multimodal interaction module is used to extract visual features, modulate the visual features according to the personality feature vector, and concatenate the modulated visual features with the gaze embedding sequence formed by the historical gaze point sequence to obtain a multimodal input sequence.
[0026] A hierarchical adaptive attention module is used to obtain enhanced sequence representations through dynamic routing block segmentation and hierarchical sequence modeling;
[0027] The spatial probability generation module is used to determine the probability distribution parameters of the spatial location of the next gaze point based on the enhanced sequence representation, construct a three-dimensional spatial probability distribution based on the probability distribution parameters and perform random sampling to obtain a three-dimensional candidate gaze vector, and normalize the three-dimensional candidate gaze vector to a unit sphere to obtain the next gaze point;
[0028] In this process, after adding the next gaze point to the historical gaze point sequence, the personality-guided multimodal interaction module regenerates the multimodal input sequence based on the updated historical gaze point sequence. The hierarchical adaptive attention module obtains the updated enhanced sequence representation based on the updated multimodal input sequence. The spatial probability generation module determines and samples the next gaze point based on the updated enhanced sequence representation. The above process is iteratively executed until the number of gaze points in the historical gaze point sequence reaches a preset number, and the historical gaze point sequence is determined as the predicted saccadic path.
[0029] Optionally, the input and preprocessing module is further configured to divide the difference between the continuous scores of the personality dimension and the corresponding preset benchmark value by the corresponding scaling factor for different personality dimensions, and restrict the result to a preset numerical range to obtain the personality feature vector.
[0030] The personality-guided multimodal interaction module includes a hierarchical visual encoder, a personality modulation parameter generation network, a feature linear modulation unit, and a historical gaze information encoding unit.
[0031] The hierarchical adaptive attention module includes at least two encoders and at least two dynamic routing block units for hierarchical processing of multimodal input sequences, a main network for high-level sequence modeling, a deblocking unit corresponding to each of the dynamic routing block units, and at least two decoders.
[0032] According to a third aspect of the present invention, a personality-driven panoramic scene saccade path prediction terminal is provided, comprising at least one processor, a memory communicatively connected to the at least one processor, and an input / output interface; the memory stores instructions executable by the at least one processor, wherein when the instructions are executed, the at least one processor implements the method described above, and receives panoramic scene images, personality feature information, and initial fixation points through the input / output interface, and outputs a predicted saccade path.
[0033] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the personality-driven panoramic scene scanning path prediction method.
[0034] Compared with the prior art, the present invention has at least the following beneficial effects:
[0035] (1) By using personality traits with psychological meaning as explicit conditional signals, the model can predict saccadic paths with different exploration tendencies for the same scene, thereby improving the individual consistency and interpretability of the prediction results.
[0036] (2) By scaling and offsetting, the visual features are conditionally modulated, and the responses of different visual feature dimensions can be adjusted according to the observer's personality characteristics without destroying the visual space structure.
[0037] (3) Adaptive compression of sequence elements is performed by dynamic routing blocks, which reduces the burden of high-level sequence reasoning while preserving key visual behavioral clues.
[0038] (4) By hierarchically aggregating and recovering information through hierarchical structure of hierarchical encoding, high-level reasoning, unblocking and decoding, global exploration strategies, local visual details and long-range gaze dependence can be modeled simultaneously.
[0039] (5) By modeling the next gaze point through the probability distribution in three-dimensional space and normalizing the sampled three-dimensional candidate gaze vectors to the unit sphere, the problem of discontinuity in the left and right boundary positions caused by directly modeling on the equidistant cylindrical projection plane can be avoided. At the same time, the uncertainty of the next gaze position prediction is preserved through the probability distribution. Attached Figure Description
[0040] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0041] Figure 1 This is a schematic diagram of the overall process of the personality-driven panoramic scene scanning path prediction method according to an embodiment of the present invention;
[0042] Figure 2This is a schematic diagram of the multimodal interaction module structure for personality guidance according to an embodiment of the present invention;
[0043] Figure 3 This is a schematic diagram of the hierarchical adaptive attention module structure according to an embodiment of the present invention;
[0044] Figure 4 A schematic diagram of the personality-driven panoramic scene scanning path prediction system and prediction terminal provided in an embodiment of the present invention;
[0045] In the diagram: 100 - Prediction system; 110 - Input and preprocessing module; 120 - Personality-guided multimodal interaction module; 130 - Hierarchical adaptive attention module; 140 - Spatial probability generation module; 200 - Prediction terminal; 210 - Input / output interface; 220 - Processor; 230 - Memory. Detailed Implementation
[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention; equivalent substitutions and improvements made by those skilled in the art without departing from the concept of the present invention are all within the scope of protection of the present invention.
[0047] like Figure 1 As shown, this embodiment of the invention provides a personality-driven panoramic scene saccade path prediction method. Figure 1 The overall flow of the method is illustrated, including input and preprocessing, visual encoding and personality modulation, multimodal sequence construction, hierarchical adaptive attention modeling, spatial probability generation and path sampling, and the process of determining the number of fixations and predicting the saccade path output. Specifically, it includes the following steps:
[0048] S1. Input and Preprocessing: Obtain the panoramic scene image to be predicted. Information on the observer's personality traits and initial fixation point The personality feature information is normalized based on preset benchmark values and scaling factors corresponding to different personality dimensions to obtain a personality feature vector corresponding to the observer. Establish a historical gaze point sequence, and use the initial gaze point as the first gaze point in the historical gaze point sequence.
[0049] S2. Visual Encoding and Personality Modulation: A visual encoder is used to perform hierarchical feature extraction and dimension mapping on the panoramic scene image to obtain a visual feature sequence. And by the personality trait vector Generate scaling parameters and offset parameters The visual feature sequence is subjected to feature linear modulation to obtain a personality-conditional visual feature sequence. .
[0050] S3. Multimodal sequence construction: Convert each historical gaze point in the historical gaze point sequence into a three-dimensional spatial coordinate representation and map it into a gaze embedding representation, forming a gaze embedding sequence in chronological order; concatenate the personality-conditional visual feature sequence as a context prefix with the gaze embedding sequence to obtain a multimodal input sequence. .
[0051] S4. Hierarchical adaptive attention modeling; The multimodal input sequence... The input is processed by a hierarchical adaptive attention module, which sequentially performs hierarchical encoding, dynamic routing and segmentation, high-level inference, deblocking, and decoding to obtain an enhanced sequence representation that integrates personality-conditional visual features and historical gaze information. .
[0052] S5. Spatial probability generation and path sampling; based on the enhanced sequence representation. The three-dimensional spatial probability distribution parameters of the next gaze point are predicted. A three-dimensional spatial probability distribution is constructed based on the distribution parameters and random sampling is performed. The sampled three-dimensional candidate gaze vectors are normalized to a unit sphere to obtain the next gaze point. The next gaze point is added to the historical gaze point sequence.
[0053] S6. Determine the number of fixations and output the saccade path; determine whether the number of fixations in the historical fixation sequence has reached a preset number; if the preset number has not been reached, repeat steps S3 to S5 based on the updated historical fixation sequence; if the preset number has been reached, determine the historical fixation sequence as a predicted saccade path and output it.
[0054] In some preferred embodiments, such as Figure 2 As shown, Figure 2 The process of input and preprocessing, visual encoding and personality modulation, and multimodal sequence construction is illustrated, including the normalization of personality feature information, hierarchical visual Transformer encoding, visual feature dimension mapping and flattening, generation of personality modulation parameters, personality-conditional modulation of visual features, encoding of historical gaze information, and concatenation of visual features and gaze embeddings. Steps S1-S3 can be performed as follows:
[0055] S1.1 Obtain the panoramic scene image to be predicted, the observer's personality trait information, and the initial gaze point. The personality trait information includes continuous scores for at least one personality dimension: extraversion, agreeableness, conscientiousness, openness, and emotional stability.
[0056] S1.2, such as Figure 2As shown in the "normalization" process, for each personality dimension, the difference between the corresponding continuous score and the preset benchmark value corresponding to that personality dimension is divided by the corresponding scaling factor, and the result is restricted to a preset numerical range. The normalized personality feature values are combined according to the preset personality dimension order to obtain the personality feature vector.
[0057] S1.3 Establish a historical fixation point sequence, and take the initial fixation point as the first fixation point in the historical fixation point sequence.
[0058] In some preferred embodiments, such as Figure 2 As shown, visual encoding and personality modulation in S2 include hierarchical visual Transformer encoding, dimension mapping and flattening, personality modulation parameter generation, and feature linear modulation, which can be performed as follows:
[0059] S2.1, A hierarchical visual Transformer encoder with a shift window mechanism is used as the visual encoder to extract hierarchical features from the panoramic scene image. For example... Figure 2 As shown, the hierarchical feature extraction process of the hierarchical visual Transformer encoder includes three stages: Stage 1, Stage 2, and Stage 3. Stage 1, Stage 2, and Stage 3 correspond to the feature extraction processes of the first, second, and third levels, respectively. Stage 1 is used to extract first-level features from the panoramic scene image to obtain first-level visual features; Stage 2 is used to extract second-level features from the first-level visual features to obtain second-level visual features; and Stage 3 is used to extract third-level features from the second-level visual features to obtain third-level visual features. In this embodiment, the third-level visual features are selected as intermediate visual features.
[0060] S2.2. Perform dimensional mapping on the intermediate visual features. Map the intermediate visual features to the unified feature dimension corresponding to the hierarchical adaptive attention module through linear projection, and flatten them according to their spatial position. Figure 2 The visual feature sequence shown ;
[0061] S2.3, the personality feature vector The input personality modulation parameter generation network is constructed using a multilayer perceptron and outputs scaling parameters corresponding to the visual feature dimensions. With offset parameter ;
[0062] S2.4, Apply the following formula to the visual feature sequence. By applying characteristic linear modulation channel by channel, a personality-conditional visual feature sequence is obtained. :
[0063]
[0064] in, This represents the sequence of visual features output by the visual encoder. and These represent the scaling and offset parameters generated from the personality feature vector, respectively, and ⊙ indicates element-wise modulation according to the corresponding feature dimension. This represents the modulated, conditional visual characteristics of personality. The above equation allows personality information to act on the visual representation in the form of channel-level gain and bias, thereby achieving conditional modulation of the observer's personality characteristics' response to visual characteristics without changing the visual backbone structure.
[0065] In some preferred embodiments, such as Figure 2 As shown, the multimodal sequence construction in S3 includes encoding historical gaze information and concatenating the personality-conditional visual feature sequence with the gaze embedding sequence. Specifically, the encoding of historical gaze information is used to convert historical gaze information in the historical gaze point sequence into a gaze embedding sequence, which can be performed as follows:
[0066] S3.1 Obtain the three-dimensional gaze direction vector corresponding to each historical gaze point in the historical gaze point sequence, and convert the three-dimensional gaze direction vector to a preset unified three-dimensional coordinate system to obtain the three-dimensional spatial coordinate representation corresponding to each historical gaze point.
[0067] In this embodiment, the eye-tracking acquisition device outputs a three-dimensional gaze direction vector in the first coordinate system. By exchanging coordinate axes and adjusting orientation, it is converted into a 3D gaze vector in a unified 3D coordinate system. ,satisfy:
[0068] in, This represents the three-dimensional gaze direction vector output by the eye-tracking acquisition device. This represents the 3D gaze direction vector after transformation to a unified 3D coordinate system.
[0069] In another embodiment, when the historical gaze point is represented by latitude and longitude coordinates in a panoramic image, the latitude and longitude coordinates can be converted into three-dimensional unit spherical coordinates. Let the latitude be... Longitude is Then the corresponding three-dimensional unit spherical coordinates satisfy:
[0070]
[0071] S3.2, through Figure 2The historical gaze information encoding shown converts the three-dimensional spatial coordinates corresponding to each historical gaze point into a gaze embedding through linear mapping:
[0072]
[0073] in, Indicates the first The embedding of constant attention, Indicates linear projection. This represents the three-dimensional spatial coordinates obtained from S3.1. Representing a three-dimensional real number space; and arranged in chronological order according to the historical points of view. Figure 2 The gaze embedding sequence shown;
[0074] S3.3, such as Figure 2 As shown in the "splicing" process, the personality-conditional visual feature sequence is used as a visual marker sequence. The gaze embedding sequence is used as a gaze marker sequence. The visual marker sequence is used as a context prefix and concatenated with the gaze marker sequence along the sequence dimension to form a multimodal input sequence, i.e. ;
[0075] S3.4 The hierarchical adaptive attention module employs a causal sequence modeling mechanism, ensuring that the current prediction time step relies solely on previous historical gaze information; the personality-conditional visual feature sequence serves as a context prefix, providing scene visual context for each prediction time step. In this embodiment, a padding mask for identifying valid sequence elements is input to the hierarchical adaptive attention module, and causal constraints are implemented internally by the sequence modeling layer.
[0076] In some preferred embodiments, S4 can be performed as follows:
[0077] S4.1, such as Figure 3 As shown, the multimodal input sequence Input the first-level encoder E0 to perform preliminary long-range dependency modeling on visual context information and historical gaze information to obtain the first-level encoded representation;
[0078] S4.2 Input the first-level encoding representation into the dynamic routing block unit C0, calculate the boundary probability or retention probability corresponding to each sequence element in the first-level encoding representation, and select and merge the sequence elements according to the boundary probability or retention probability to form a length-compressed block sequence;
[0079] S4.3 After performing high-dimensional mapping on the block sequence, input it into the second-level encoder E1 to obtain the second-level code on the compressed sequence;
[0080] S4.4 Input the second-level code into the dynamic routing block unit C1, calculate the boundary probability or retention probability corresponding to each sequence element in the second-level code, and further select and merge the sequence elements according to the boundary probability or retention probability to obtain a further compressed high-level input representation; input the high-level input representation into the main network M, and further model long-range dependencies through the state space sequence layer and the Transformer layer to obtain a high-level sequence representation;
[0081] S4.5 The high-level sequence representation is sequentially restored through the deblocking unit DC1 corresponding to the dynamic routing block unit C1, the second-level decoder D1, the deblocking unit DC0 corresponding to the dynamic routing block unit C0, and the first-level decoder D0, and the compressed high-level information is remapped to a fine-grained representation corresponding to the length of the input sequence.
[0082] S4.6 Obtain an enhanced sequence representation that integrates conditional visual features of personality and historical gaze information. .
[0083] In other embodiments, the hierarchical adaptive attention module can set up a hierarchical structure of two or more levels according to the length of the input sequence and the complexity of the model. Each level is equipped with a corresponding encoder and dynamic routing block unit, and on the decoding side, a deblocking unit and a decoder corresponding to each of the dynamic routing block units are set up to perform sequence compression and recovery step by step.
[0084] In some preferred embodiments, S5 can be performed as follows:
[0085] S5.1, The spatial probability generation module includes a prediction head; from the enhanced sequence representation Extract the output portion corresponding to the gaze embedding sequence to obtain the gaze latent representation sequence; then, through the prediction head, output the three-dimensional mean vector and three-dimensional log-variance vector corresponding to each prediction time step based on the gaze latent representation sequence.
[0086]
[0087] in, This represents a prediction head implemented by a multilayer perceptron. Indicates the first The gaze latent representation corresponding to each prediction time step This represents the mean of the predicted next gaze point. This represents the corresponding logarithmic variance. This represents the six-dimensional real space of the prediction head output, where the mean and log-variance each occupy three components.
[0088] S5.2. Numerically truncate the logarithmic variance using the following formula:
[0089]
[0090] S5.3. Restore the variance from the truncated logarithmic variance using the following formula:
[0091]
[0092] in, This represents the logarithmic variance of the prediction; clip indicates element-wise truncation. This represents the reduced variance. The logarithmic variance truncation restricts the logarithmic variance to a certain value. Within a certain range, thus limiting the variance obtained by the exponential transformation accordingly. Within this range, its function is to: on the one hand, avoid the negative log-likelihood loss from overflowing due to excessively small variance in the early stage of training, and on the other hand, avoid the sampling range of candidate gaze vectors from being too dispersed due to excessively large variance, thereby ensuring the numerical stability of the training process.
[0093] S5.4. Using the three-dimensional mean vector as the mean and the diagonal matrix formed by the variances of each spatial dimension as the covariance matrix, construct a three-dimensional diagonal Gaussian distribution.
[0094] S5.5 In the inference phase, based on the gaze latent representation corresponding to the last historical gaze point in the current historical gaze point sequence, the three-dimensional mean vector and three-dimensional log-variance vector corresponding to the current prediction time step are determined, and a three-dimensional diagonal Gaussian distribution corresponding to the current next gaze point is constructed accordingly; three-dimensional candidate gaze vectors are randomly sampled from the three-dimensional diagonal Gaussian distribution, and the three-dimensional candidate gaze vectors are normalized to a unit vector so that they lie on a unit sphere, thereby obtaining the next gaze point; when mapping the next gaze point to the panoramic scene image, the three-dimensional unit sphere coordinates of the next gaze point are converted into the corresponding panoramic image coordinates;
[0095] S5.6 Add the next gaze point to the historical gaze point sequence.
[0096] In some preferred embodiments, S6 can be performed as follows:
[0097] S6.1 Determine whether the number of fixations in the updated historical fixation sequence has reached the preset number;
[0098] S6.2 If the preset number is not reached, the updated historical fixation point sequence is used as the current historical fixation point sequence, and steps S3 to S5 are repeated to determine and add the next fixation point.
[0099] S6.3 If the preset number is reached, the iteration stops, and the historical gaze point sequence is determined as the predicted saccade path.
[0100] In some preferred embodiments, the training process of the method can be performed as follows:
[0101] T1. Construct the training input sequence and training target sequence from the observation scan path using the following formula:
[0102]
[0103] in, This represents the training input sequence. Represents the training target sequence; This indicates the total number of fixation points included in the observation scan path; Indicates the first The coordinate vectors of each gaze point in a three-dimensional unit spherical coordinate system, and satisfying ; The time step index represents the fixation point, and its value range is... The training input sequence is the first sequence in the observation scan path. From the first fixation point to the second The training target sequence consists of _ fixations_, and the training target sequence is composed of the _th_ fixation point. From the first fixation point to the second The training target sequence is composed of several gaze points, meaning that the training target sequence is shifted backward by one time step relative to the training input sequence, so that the model learns to predict the next gaze point given historical gaze information.
[0104] T2. After ignoring constant terms that are irrelevant to the model parameters, the negative log-likelihood loss is used as the training objective according to the following formula:
[0105]
[0106] in, This represents the negative log-likelihood loss; Indicates the index of the prediction time step; Dimension index representing three-dimensional spatial coordinates , respectively, correspond to the three coordinate components of the three-dimensional unit spherical coordinate vector; Indicates the first training target sequence The first true fixation point at the 1st Coordinate values in each spatial dimension; The model represents the first time. The next gaze point 3D mean vector output at the predicted time step. The mean of each dimension; The model represents the first time. The next gaze point determined at the predicted time step is in the 1st... The variances in each spatial dimension are used to construct a diagonal covariance matrix, and a three-dimensional diagonal Gaussian distribution is constructed using the three-dimensional mean vector as the mean.
[0107] T3. Divide the model parameters into several parameter groups according to the hierarchy, and assign scaling factors to each parameter group. The parameter group is determined by the following formula in the first... Effective learning rate per step:
[0108]
[0109] in, Indicates the basic learning rate. Indicates the first Scaling factor for each parameter group Indicates the training step index. This represents a scheduling factor related to the training progress. In some preferred embodiments, the scaling factor is taken as 1.0, 1.0, and 0.5, that is, the parameter set carrying high-level semantics is updated with a smaller learning rate to avoid destroying the semantic prior implied by the pre-trained visual backbone; the scheduling factor linearly increases in the first 10% of training steps, remains constant in the next 80% of training steps, and linearly decays to zero in the last 10% of training steps.
[0110] T4. In some preferred embodiments, the model is trained using four graphics processors for 500 training rounds, with the input resolution downsampled. The pixel and visual encoder are initialized by pre-trained parameters, and the hierarchical adaptive attention module adopts a two-level hierarchical structure; the number of training rounds, batch size, basic learning rate and number of hardware can all be adjusted according to the data scale and memory capacity.
[0111] In some preferred embodiments, the data used to train the model can be constructed as follows:
[0112] D1. Select 800 panoramic images from the panoramic scene library, including five categories: urban landscape, people, large halls, small interiors and natural landscapes, with 160 images in each category, to cover viewing scenarios with different spatial scales and semantic densities.
[0113] D2. Participants wear head-mounted display devices with integrated eye-tracking modules and freely view the panoramic scene from a rotatable seat. Each image is presented for fifteen seconds, and a calibration point located at zero longitude and zero latitude is displayed before each presentation to unify the starting viewpoint for each viewing.
[0114] D3. With informed consent from the participants, scores on five personality dimensions—extroversion, agreeableness, conscientiousness, openness, and emotional stability—were obtained using a personality scale. The personality dimension scores were normalized based on preset baseline values and scaling factors for each dimension, and the results were restricted to preset numerical ranges to obtain personality feature vectors. These vectors were then associated with corresponding eye-tracking coordinates, timestamps, and image identifiers. During data processing, each participant was assigned a participant code, and their name, contact information, and other directly identifiable information were stored separately from the personality feature vectors and eye-tracking data. During model training and saccade prediction, the correspondence between the personality feature vectors and eye-tracking data was established using the participant codes, without accessing names, contact information, or other directly identifiable information.
[0115] D4. After the personality feature information is converted into a personality feature vector, it can be de-identified at the data acquisition terminal. When a remote server is used to perform model training or inference, the de-identified personality feature vector, eye-tracking data and image labels can be transmitted to the remote server.
[0116] D5. Divide the 800 panoramic images into a training set and a test set at a ratio of 80% and 20%, respectively. Each image is associated with the personalized saccade paths of multiple participants. During the training phase, input the panoramic image, personality traits, and real gaze history and predict the next real gaze point. During the testing phase, input the panoramic scene image to be predicted, the observer's personality trait information, and the initial gaze point. After obtaining the personality trait vector, establish a historical gaze point sequence, take the initial gaze point as the first gaze point in the historical gaze point sequence, and recursively determine and sample subsequent gaze points in an autoregressive manner.
[0117] In a set of comparative experiments, three classic metrics were used to evaluate the predicted saccades: Levenshtein Distance (LEV), proposed by Privitera and Stark in "Algorithms for Defining Visual Regions-of-Interest: Comparison With Eye Fixations" published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2000; Dynamic Time Warp (DTW), proposed by Berndt in "Using Dynamic Time Warping to Find Patterns in Time Series" published at the KDD workshop in 1994; and Recurrence (REC), proposed by Anderson et al. in "Recurrence Quantification Analysis of Eye Movements" published in Behavior Research Methods in 2013. Lower LEV and DTW, and higher REC, indicate better prediction performance. The experimental results are shown in Tables 1 and 2.
[0118] Table 1. Performance comparison of the present invention and different saccade path prediction methods on the constructed test set.
[0119]
[0120] As shown in Table 1, the method of the present invention achieved lower LEV and DTW and higher REC on the constructed test set, indicating that its predicted saccade path has good consistency with the actual observation behavior.
[0121] Table 2 Ablation Experiment Results of Each Key Technical Module of the Invention
[0122]
[0123] As shown in Table 2, the model performance was improved after introducing personality conditional modulation, hierarchical visual Transformer encoder, hierarchical adaptive attention module and dynamic routing mechanism, indicating that the above technical features help to improve the accuracy of personalized saccade path prediction.
[0124] The above experimental results are used to illustrate the technical effects of the present invention and do not constitute a limitation on the scope of protection of the present invention.
[0125] In some preferred embodiments, such as Figure 4 As shown, this embodiment of the invention also provides a personality-driven panoramic scene saccade path prediction system 100. The prediction system 100 includes an input and preprocessing module 110, a personality-guided multimodal interaction module 120, a hierarchical adaptive attention module 130, and a spatial probability generation module 140. The input and preprocessing module 110 is used to execute step S1; the personality-guided multimodal interaction module 120 is used to execute gaze embedding generation and multimodal sequence construction in steps S2 and S3; the hierarchical adaptive attention module 130 is used to execute hierarchical sequence modeling in step S4; the spatial probability generation module 140 is used to execute step S5; the prediction system 100 is also used to execute step S6, determining whether to continue executing steps S3 to S5 based on the number of gaze points in the historical gaze point sequence, and outputting the predicted saccade path when the number of gaze points reaches a preset number. Each module can be implemented by a software functional unit in the same processor, or distributed across a graphics processor, central processing unit, neural network accelerator, or remote server.
[0126] In some preferred embodiments, such as Figure 4 As shown, this embodiment of the invention also provides a prediction terminal 200, which includes an input / output interface 210, at least one processor 220, and a memory 230. The input / output interface 210 and the memory 230 are respectively connected to the processor 220. The memory 230 stores a computer program that can run on the processor 220. When the processor 220 executes the computer program, it implements the steps of the aforementioned personality-driven panoramic scene saccade path prediction method. The prediction system 100 can be deployed in the prediction terminal 200, and the processor 220 executes the computer program stored in the memory 230 to realize the corresponding functions of each module in the prediction system 100. The memory 230 may include random access memory, read-only memory, flash memory, solid-state memory, or a combination thereof. The input / output interface 210 can receive panoramic scene images, personality feature information, and initial fixation points provided by local files, eye-tracking devices, or network services, and output the predicted saccade path to display control, content encoding, interaction analysis, or quality evaluation modules.
[0127] In some preferred embodiments, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned personality-driven panoramic scene saccade path prediction method. The functionality can be implemented by software, or by an application-specific integrated circuit, a programmable logic device, a system-on-a-chip, or a combination of hardware and software.
[0128] Those skilled in the art should understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Without departing from the technical concept of the present invention, modifications or substitutions can be made to the personality coding method, visual encoder structure, sequence modeling layer type, dynamic routing method, probability distribution type, and coordinate representation method; any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A personality-driven panoramic scene saccade path prediction method, characterized in that, Includes the following steps: S1. Acquire panoramic scene images, observer's personality characteristic information, and initial gaze point; normalize the personality characteristic information according to preset benchmark values and scale factors corresponding to different personality dimensions to obtain a personality characteristic vector corresponding to the observer; establish a historical gaze point sequence, and take the initial gaze point as the first gaze point in the historical gaze point sequence; S2. Use a visual encoder to perform hierarchical feature extraction and dimension mapping on the panoramic scene image to obtain a visual feature sequence; use the personality feature vector as observer condition information, generate scaling parameters and offset parameters corresponding to the visual feature dimensions through a personality modulation parameter generation network, and use the scaling parameters and offset parameters to perform feature linear modulation on the visual feature sequence to obtain a personality-conditional visual feature sequence. S3. Convert each historical gaze point in the historical gaze point sequence into a three-dimensional spatial coordinate representation and map it into a gaze embedding representation. Form a gaze embedding sequence in chronological order. Concatenate the gaze embedding sequence with the personality conditional visual feature sequence to obtain a multimodal input sequence. S4. Input the multimodal input sequence into the hierarchical adaptive attention module, and perform hierarchical encoding, dynamic routing block division, high-level inference, deblocking and decoding processing in sequence to obtain an enhanced sequence representation that fuses the personality conditional visual feature sequence and the gaze embedding sequence. S5. Input the enhanced sequence representation into the spatial probability generation module to obtain the three-dimensional mean parameter and the three-dimensional log-variance parameter for determining the spatial location probability distribution of the next gaze point; perform a preset range limit on the three-dimensional log-variance parameter, and perform an exponential transformation on the limited three-dimensional log-variance parameter to obtain the three-dimensional variance parameter; construct a three-dimensional spatial probability distribution based on the three-dimensional mean parameter and the three-dimensional variance parameter, and randomly sample three-dimensional candidate gaze vectors from the three-dimensional spatial probability distribution; perform unit vector normalization on the three-dimensional candidate gaze vectors to obtain the next gaze point located on a unit sphere, and add the next gaze point to the historical gaze point sequence; S6. Determine whether the number of fixations in the updated historical fixation sequence has reached the preset number; If the number of fixations in the historical fixation sequence does not reach the preset number, steps S3 to S5 are repeated based on the updated historical fixation sequence until the number of fixations in the historical fixation sequence reaches the preset number; when the preset number is reached, the historical fixation sequence is determined to be a predicted saccade path.
2. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The personality trait information includes continuous scores of at least one personality dimension among extraversion, agreeableness, conscientiousness, openness, and emotional stability; for each personality dimension, the difference between the continuous score and the corresponding preset benchmark value is divided by the corresponding scaling factor, and the result is restricted to a preset numerical range to obtain the corresponding normalized personality trait value. The normalized personality trait values are combined in the order of the preset personality dimensions to form the personality trait vector. The personality modulation parameter generation network maps the personality feature vector to scaling parameters corresponding to the visual feature dimension through a multilayer perceptron. and offset parameters .
3. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The personality conditional visual feature sequence satisfy: ,in, It is a visual feature sequence. This represents element-wise multiplication. For scaling parameters, This is the offset parameter.
4. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The visual encoder is a hierarchical visual Transformer with a shift window. The intermediate stage output of the visual encoder with a preset spatial resolution is selected, and the intermediate stage output is mapped to the unified feature dimension of the hierarchical adaptive attention module through a linear projection layer.
5. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The process of obtaining the multimodal input sequence specifically involves: acquiring the three-dimensional spatial coordinate representations corresponding to each historical gaze point, and obtaining the corresponding gaze embedding representations through linear mapping; arranging the gaze embedding representations according to the chronological order of each historical gaze point to form the gaze embedding sequence; using the personality-conditional visual feature sequence as a context prefix and concatenating it with the gaze embedding sequence to obtain the multimodal input sequence; wherein the gaze embedding sequence is autoregressive modeled in chronological order under causal constraints.
6. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The hierarchical adaptive attention module includes at least two levels of encoders, at least two levels of dynamic routing block units, a main network, deblocking units corresponding to each of the dynamic routing block units, and at least two levels of decoders; at least one of the encoders, the main network, and the decoders is composed of one or more stacks of state space sequence layers and Transformer layers to model the long-range dependencies of the multimodal input sequence. Each of the dynamic routing block units is used to calculate the boundary probability or retention probability corresponding to each sequence element in the current layer input sequence, and select and merge the sequence elements according to the boundary probability or retention probability to form a length-compressed block sequence; the multimodal input sequence is processed step by step by encoders and dynamic routing block units to form a length-compressed high-level input representation, and the main network performs high-level inference on the high-level input representation to obtain a high-level sequence representation; the high-level sequence representation is used to recover local temporal details step by step by deblocking units corresponding to each of the dynamic routing block units and the decoder to obtain an enhanced sequence representation aligned with the input length.
7. The panoramic scene scanning path prediction method according to claim 1, characterized in that, The three-dimensional mean parameter and the three-dimensional log-variance parameter are respectively the three-dimensional mean vector and the three-dimensional log-variance vector; the spatial probability generation module outputs the three-dimensional mean vector and the three-dimensional log-variance vector for each prediction time step, and restricts the three-dimensional log-variance vector to a preset numerical range; An exponential transformation is performed on the restricted three-dimensional log-variance vector to determine the variance corresponding to each spatial dimension. A three-dimensional diagonal Gaussian distribution is constructed using the three-dimensional mean vector as the mean and the diagonal matrix formed by the variances corresponding to each spatial dimension as the covariance matrix. The saccade path prediction model, which includes the visual encoder, personality modulation parameter generation network, hierarchical adaptive attention module and spatial probability generation module, is trained using the next gaze point autoregressive supervision method. During training, the sequence from the initial gaze point to the penultimate gaze point is used as the training input sequence, and the sequence from the second gaze point to the last gaze point is used as the training target sequence. The training loss is determined based on the negative log-likelihood of the target gaze point at each prediction time step under the three-dimensional diagonal Gaussian distribution. During inference, three-dimensional candidate gaze vectors are randomly sampled from the three-dimensional diagonal Gaussian distribution. The three-dimensional candidate gaze vectors are then normalized to unit vectors to obtain the next gaze point located on a unit sphere. When mapping the next gaze point to the panoramic scene image, the three-dimensional unit sphere coordinates of the next gaze point are converted into the corresponding panoramic image coordinates.
8. A personality-driven panoramic scene saccade path prediction system implementing the method of claim 1, characterized in that, include: The input and preprocessing module is used to acquire panoramic scene images, personality feature information and initial gaze points, and preprocess the personality feature information to obtain personality feature vectors; Establish a historical fixation point sequence, and take the initial fixation point as the first fixation point in the historical fixation point sequence; A personality-guided multimodal interaction module is used to extract visual features, modulate the visual features according to the personality feature vector, and concatenate the modulated visual features with the gaze embedding sequence formed by the historical gaze point sequence to obtain a multimodal input sequence. A hierarchical adaptive attention module is used to obtain enhanced sequence representations through dynamic routing block segmentation and hierarchical sequence modeling; The spatial probability generation module is used to determine the probability distribution parameters of the spatial location of the next gaze point based on the enhanced sequence representation, construct a three-dimensional spatial probability distribution based on the probability distribution parameters and perform random sampling to obtain a three-dimensional candidate gaze vector, and normalize the three-dimensional candidate gaze vector to a unit sphere to obtain the next gaze point; Specifically, after adding the next gaze point to the historical gaze point sequence, the personality-guided multimodal interaction module regenerates the multimodal input sequence based on the updated historical gaze point sequence. The hierarchical adaptive attention module obtains the updated enhanced sequence representation based on the updated multimodal input sequence. The spatial probability generation module determines and samples the next gaze point based on the updated enhanced sequence representation. The aforementioned process is iteratively executed until the number of gaze points in the historical gaze point sequence reaches a preset number, at which point the historical gaze point sequence is determined as the predicted saccade path.
9. The panoramic scene scanning path prediction system according to claim 8, characterized in that, The input and preprocessing module is also used to divide the difference between the continuous scores of the personality dimension and the corresponding preset benchmark value by the corresponding scaling factor for different personality dimensions, and restrict the result to a preset numerical range to obtain the personality feature vector. The personality-guided multimodal interaction module includes a hierarchical visual encoder, a personality modulation parameter generation network, a feature linear modulation unit, and a historical gaze information encoding unit. The hierarchical adaptive attention module includes at least two encoders and at least two dynamic routing block units for hierarchical processing of multimodal input sequences, a main network for high-level sequence modeling, a deblocking unit corresponding to each of the dynamic routing block units, and at least two decoders.
10. A personality-driven panoramic scene scanning path prediction terminal, characterized in that, The device includes at least one processor, a memory communicatively connected to the at least one processor, and an input / output interface; the memory stores instructions executable by the at least one processor, which, when executed, cause the at least one processor to implement the method of any one of claims 1 to 7, and receive panoramic scene images, personality feature information, and initial gaze points through the input / output interface, and output a predicted saccade path.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the personality-driven panoramic scene saccade path prediction method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-visual memory unit-based glancing path prediction method
CN116563524A
Panoramic image-oriented human eye glancing track prediction method
CN118397686A
