A method for predicting human eye scanning trajectory for panoramic images

Abstract

The present invention proposes a method for predicting human eye scanning trajectory for panoramic images, namely, a three-dimensional visual scanning trajectory Transformer (3D Visual Scanpath Transformer, Pathformer3D). First, a two-dimensional panoramic image is mapped to a three-dimensional spherical coordinate system, and a three-dimensional convolutional network is used to extract its features; then, a 3D Transformer encoder is used to convert the features of each image area in the spherical coordinate system into a context-aware representation; then, the context-aware representation of the learned feature information, all historical gaze point query information, and the query information of the current gaze point are input into the 3D Transformer decoder together to calculate the hidden information of the gaze point at the current moment; finally, the hidden information is input into a three-dimensional mixed density network to learn the joint three-dimensional Gaussian distribution of the position of the gaze point at the current moment in the three-dimensional spherical coordinate system, and the final focus point position is sampled therefrom. The Pathformer3D model proposed in the present invention can generate a more realistic and reliable panoramic image scanning trajectory for 360° panoramic images.

Description

Technical Field

[0001] The invention relates to the field of genetic algorithms, and mainly to a method for predicting human eye scanning trajectories for panoramic images. Background Art

[0002] In recent years, Virtual Reality (VR) and Augmented Reality (AR) technologies have made significant progress, providing users with immersive experiences. At the same time, understanding and imitating the way humans explore 360° images in virtual environments has become increasingly important, as it can help achieve more practical and fast rendering, thereby improving user interactivity in immersive environments. Panoramic image scan trajectory prediction refers to predicting the trajectory of human gaze movement during panoramic image exploration.

[0003] Panoramic images are quite different from two-dimensional images. Panoramic images provide an immersive interactive environment where users can change their perspective by physically moving their heads, resulting in a wider gaze distribution. Panoramic images contain richer visual information, which requires more time for users to process and absorb. In addition, the data structure of panoramic images is different from that of two-dimensional images, which puts forward new requirements for data processing and analysis. Therefore, the existing two-dimensional image scanning trajectory prediction methods cannot be directly applied to the scanning trajectory prediction of panoramic images.

[0004] Early methods for predicting panorama image saccades involved sampling fixations based on saliency information to obtain the entire saccade trajectory. Subsequently, with the development of generative adversarial networks, some researchers used generative networks to generate the entire trajectory directly from the panorama image. These methods achieved preliminary results, but they ignored the modeling of temporal dependencies between fixations, which has long been shown to be a very important feature of human visual attention mechanism, and thus they often led to unstable prediction results. Recently, some research works have recognized the need to comprehensively handle the temporal dependencies in viewing behavior and model them through recurrent neural networks or Markov chains. However, these methods integrate all historical fixation information into a single hidden unit and only generate the current fixation from this hidden unit. In this case, the temporal dependencies between fixations can only be modeled indirectly, and the influence of historical fixations will be greatly weakened. Summary of the invention

[0005] Purpose of the invention: In response to the above challenges, the main purpose of this study is to propose a method for predicting human eye glance trajectories for panoramic images to solve the above problems.

[0006] Technical solution: The present invention provides a method for predicting human eye scanning trajectories for panoramic images, and the method comprises the following steps:

[0007] Step S1, converting the panoramic image into a three-dimensional spherical coordinate system to obtain a panoramic image in the three-dimensional spherical coordinate system;

[0008] Step S2: Using a multi-stage three-dimensional convolutional network to obtain feature information F of the panoramic image in a three-dimensional spherical coordinate system L ;

[0009] Step S3: Use the 3D Transformer encoder to transform the feature information F L Contextualization, obtaining feature information F L Perceptual representation of the context;

[0010] Step S4: The feature information F L The perceptual representation of the context, the historical gaze point query information, and the query information of the new gaze point at the current moment are input into the 3D Transformer decoder to predict the hidden state parameters of the new gaze point at the current moment;

[0011] Step S5: using a three-dimensional mixed density network, transforming the hidden state parameters of the new gaze point at the current moment into multiple three-dimensional Gaussian distribution parameters, establishing a joint probability density distribution map, and generating new gaze point coordinates through sampling operations;

[0012] Step S6, repeating steps S4 to S5, generating new gaze point coordinates through continuous iteration until a predicted scanning trajectory of a required length is generated.

[0013] Preferably, the implementation process of step S1 is:

[0014] S101. Convert all two-dimensional coordinate points p=[Y,X] (Y,X∈[0,1]) on the panoramic image into the longitude and latitude representation of the two-dimensional coordinate points The conversion formula is:

[0015] φ=(Y-0.5)π; λ=(X-0.5)2π

[0016] Where Y, X are the coordinate values ​​of the two-dimensional coordinate point on the Y axis and X axis in the two-dimensional plane coordinate representation, φ, λ are the latitude and longitude coordinate values ​​of the two-dimensional coordinate point in the longitude and latitude representation;

[0017] S102, convert the longitude and latitude representation of the two-dimensional coordinate point into a three-dimensional space coordinate point p=(x, y, z)(x, y, z∈[-1, 1]) in the three-dimensional spherical coordinate system, the conversion formula is:

[0018] x=cos(φ)cos(λ); y=cos(φ)sin(λ); z=sin(φ).

[0019] Where x, y, and z are the coordinate values ​​of the three-dimensional space coordinate point on the X-axis, Y-axis, and Z-axis in the three-dimensional spherical coordinate system, respectively, so as to obtain a panoramic image in the three-dimensional spherical coordinate system.

[0020] Preferably, the implementation process of step S2 is:

[0021] S201, using a multi-stage three-dimensional convolutional network SphereNet to extract three different dimensions of the panoramic image in the three-dimensional spherical coordinate system. The three different dimensions of the two-dimensional plane feature maps are upsampled to obtain feature maps F1, F2, and F3, which are expressed as:

[0022] F1,F2,F3=Upsample(SphereNet(I,θ))

[0023] Where I represents the panoramic image in the three-dimensional spherical coordinate system, and θ represents the parameters of the multi-stage three-dimensional convolutional network SphereNet;

[0024] S202, concatenate the feature maps F1, F2, and F3 in the channel dimension to obtain a global feature map F containing information of different scales, expressed as:

[0025] F=Concat(F1,F2,F3)

[0026] Where Concat represents the concatenation operation;

[0027] S203: Perform operations including average pooling and flattening on the global feature map F to convert the global feature map F into a feature vector F with a length of L. L , the expression is:

[0028] F L =Flatten(AvgPool(F))

[0029] Among them, Flatten represents the flattening operation, and AvgPool represents the average pooling.

[0030] Preferably, in step S3, the feature vector F L The following steps are used to pass the 3D Transformer encoder:

[0031] S301, feature vector F L Transform the dimensions into the internal dimensions of the 3D Transformer encoder through a linear embedding;

[0032] S302, then the eigenvector F L Add a learnable positional encoding;

[0033] S303, the feature vector F L The feature information F is obtained by sequentially passing through the four standard Transformer encoding layers in the 3D Transformer encoder. L A context-aware representation of .

[0034] Preferably, in step 4, the 3D Transformer decoder iteratively generates hidden state parameters of the new gaze point at the current moment in an autoregressive manner.

[0035] Preferably, the implementation process of step S4 is:

[0036] S401, linearly embedding the current gaze point coordinates to obtain a query vector of the current gaze point coordinates;

[0037] S402, concatenating the query vector of the current gaze point coordinates and the query vector obtained from the historical gaze point coordinates to obtain a joint query vector;

[0038] S403, adding one-dimensional absolute position encoding information of the fixation point sequence to the joint query vector;

[0039] S404: Pass the joint query vector with the one-dimensional absolute position encoding information through the 4 standard decoding layers of the 3D Transformer decoder to obtain the hidden state parameter Z of the new gaze point at the current moment. t .

[0040] Preferably, in step S5, the new fixation point p at the current time t is calculated by a three-dimensional mixed density network. t The probability of the i-th Gaussian distribution graph Covariance Mean The calculation formula is expressed as:

[0041]

[0042] Among them, f mdn represents a three-dimensional mixed density network, which includes multiple independent two-layer linear fully connected networks, Z t represents the state of the new gaze point at the current time t, θ mdn represents the parameters of the three-dimensional mixture density network, represents the mean of the i-th Gaussian distribution at the current time t, represents the covariance of the i-th Gaussian distribution at the current time t, Represents the probability of the i-th Gaussian distribution at the current time t.

[0043] Preferably, in step S5, the new fixation point p at the current moment tt The calculation formula of a single Gaussian distribution function is expressed as:

[0044]

[0045] The new fixation point p at the current time t t The calculation formula of the joint Gaussian distribution function is expressed as:

[0046]

[0047] Wherein, K represents the number of three-dimensional Gaussian distributions constituting the joint probability density distribution map, specifically, the number of independent three-dimensional Gaussian distributions constituting the joint probability density distribution map;

[0048] According to the current time t, the new gaze point p t The joint Gaussian distribution function P(p t ) draws a probability plot of the joint Gaussian probability distribution.

[0049] Preferably, in the step S5, in the step of generating new fixation point coordinates from the joint Gaussian probability distribution probability map by sampling operation, the following relationship exists:

[0050]

[0051] Among them, ~ represents the sampling operation, Indicates generating new fixation point coordinates from the joint Gaussian probability distribution map.

[0052] Preferably, the loss function used in the training process of the model is negative logarithmic loss, and the model includes a multi-stage three-dimensional convolutional network, a 3D Transformer encoder, a 3D Transformer decoder and a three-dimensional mixed density network connected in series, satisfying the formula:

[0053]

[0054] Among them, p t * represents the actual human gaze point at time t, and T represents the length of the scanning trajectory.

[0055] Beneficial effects:

[0056] The present invention extracts panoramic image information through a multi-stage three-dimensional convolutional network, taking into account image features at different levels, and can focus on both large and small objects at the same time;

[0057] The present invention solves the problem of sequence generation through the Transformer structure, which can take into account more global image information and historical gaze point information;

[0058] The present invention establishes a probability distribution map of fixation points through a mixed density network, which can better reflect the characteristic that fixation point prediction is full of uncertainty and establish a more realistic and reasonable probability distribution map of fixation points.

[0059] The present invention proposes for the first time to predict the human eye scanning trajectory in a three-dimensional spherical coordinate system and generate new fixation points by autoregression, which can better simulate the mechanism of visual working memory and generate more realistic and reliable panoramic image scanning trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of a method for predicting scanning trajectories for panoramic images proposed by the present invention;

[0061] Figure 2 The algorithm flow chart of the scanning trajectory prediction method for panoramic images proposed by the present invention; DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and drawings. In these descriptions and drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0064] See also Figure 1 and Figure 2 ,in Figure 1 is a flow chart of the present invention, Figure 2 It is the algorithm flow chart of the present invention, from Figure 2 It can be seen from the above that the present invention mainly includes two parts: texture feature representation and fixation point sequence generation. Texture feature representation is responsible for processing image information, and fixation point sequence generation gradually generates new fixations by autoregression.

[0065] The present invention provides a method for predicting human eye scanning trajectory for panoramic images. The specific principle is as follows: Figure 1 As shown, the following steps are included:

[0066] Step S1, converting the panoramic image into a three-dimensional spherical coordinate system to obtain a panoramic image in the three-dimensional spherical coordinate system;

[0067] Specifically, the panoramic image is projected into the three-dimensional spherical coordinate system through the following conversion process:

[0068] S101. Convert all two-dimensional coordinate points p=[Y,X] (Y,X∈[0,1]) on the panoramic image into the longitude and latitude representation of the coordinate points The conversion formula between them is:

[0069] φ=(Y-0.5)π; λ=(X-0.5)2π

[0070] Where Y, X are the coordinate values ​​of the two-dimensional coordinate point on the Y axis and X axis in the two-dimensional plane coordinate representation, φ, λ are the latitude and longitude coordinate values ​​of the gaze point in the longitude and latitude representation;

[0071] S102, convert the longitude and latitude representation of the two-dimensional coordinate point into a three-dimensional space coordinate point p=(x, y, z)(x, y, z∈[-1, 1]) in the three-dimensional spherical coordinate system, the conversion formula is:

[0072] x=cos(φ)cos(λ); y=cos(φ)sin(λ); z=sin(φ).

[0073] Where x, y, and z are the coordinate values ​​of the three-dimensional space coordinate point on the X-axis, Y-axis, and Z-axis in the three-dimensional spherical coordinate system, respectively, so as to obtain a panoramic image in the three-dimensional spherical coordinate system.

[0074] Step S2: Using a multi-stage three-dimensional convolutional network to obtain feature information F of the panoramic image in a three-dimensional spherical coordinate system L ;

[0075] Specifically, obtaining panoramic image feature information in a three-dimensional spherical coordinate system includes the following steps:

[0076] S201, using the multi-stage three-dimensional convolutional network SphereNet to extract the three-dimensional plane feature maps of the panoramic image in three different dimensions in the three-dimensional spherical coordinate system, and up-sample the three-dimensional plane feature maps to unify them into different feature maps F1, F2, F3 with a width of 256 and a height of 128. The expression is:

[0077] F1,F2,F3=Upsample(SphereNet(I,θ))

[0078] Where I represents the panoramic image in the three-dimensional spherical coordinate system, and θ represents the parameters of the multi-stage three-dimensional convolutional network SphereNet;

[0079] S202, concatenate the feature maps F1, F2, and F3 in the channel dimension to obtain a global feature map F containing information of different scales, expressed as:

[0080] F=Concat(F1,F2,F3)

[0081] Where Concat represents the concatenation operation;

[0082] S203, average pooling is performed on the feature map F using a convolution kernel size of 8×8, and then flattened to convert the feature map F into a feature vector F with a length of L L , the expression is:

[0083] F L =Flatten(AvgPool(F))

[0084] Among them, Flatten represents the flattening operation, and AvgPool represents the average pooling.

[0085] Step S3, contextualizing the feature information using a 3D Transformer encoder to obtain a perceptual representation of the context of the feature information;

[0086] Specifically, the 3DTransformer encoder includes the following steps:

[0087] S301, feature vector F L Through a linear embedding, the dimension is transformed into the internal dimension of the 3D Transformer encoder. Specifically, the dimension of the feature vector is 448, and the internal dimension of the Transformer is 128, and the expression is:

[0088] F' L =Embedding(F L )

[0089] S302, then the eigenvector F L Added a learnable positional encoding E pos , so that it can learn its corresponding three-dimensional space position information, the expression is:

[0090] P0=F' L +E pos

[0091] S303, the feature vector P0 obtained after adding the position information passes through M = 4 standard Transformer encoding layers to obtain the feature information F L The context-aware representation P M , the expression is:

[0092] Pi =EncoderLayer(P i-1 ) i∈[1,...,M]

[0093] Among them, P i is the output of the i-th Transformer encoding layer. Each Transformer encoding layer contains two sublayers, namely the multi-head attention mechanism layer with 8 attention heads and the feedforward network layer (two-layer fully connected network). Layer normalization and residual connection operations are added after each sublayer.

[0094] Step S4: Context-aware representation of feature information P M , the historical gaze point query information and the query information of the new gaze point at the current moment are input into the 3D Transformer decoder to predict the hidden state parameters of the new gaze point at the current moment;

[0095] Specifically, obtaining the hidden information of the new gaze point at the current time t includes the following steps:

[0096] S401, the gaze point p at the current time t-1 t-1 Linear embedding, get the fixation point p at time t t Query vector q t , the expression of this process is:

[0097] q t =Embedding(p t-1 )

[0098] The initial gaze point coordinates are set to the center of the three-dimensional space p0=(0, 0, 0).

[0099] S402, concatenate the query vector of the current gaze point coordinates and the query vector obtained from the historical gaze point coordinates to obtain a joint query vector Q'0, which is expressed as:

[0100] Q'0=[q1;q2;...;q t ]

[0101] S403: Add the one-dimensional absolute position encoding sequence information E of the gaze point sequence to the joint query vector Q'0. pos_1d , and get the new joint query vector Q0, expressed as:

[0102] Q0=Q'0+E pos_1d

[0103] S404, Q0 is passed through N=4 standard Transformer decoding layer structures, and the expression is:

[0104] Q i=DecoderLayer(Q i-1 ,P M ) j∈[1,...,N]

[0105] Among them, Q i is the output of the i-th Transformer decoding layer, P M is the output of the Transformer encoder. Each Transformer decoding layer contains three sublayers. The first layer is a multi-head attention layer with 8 attention heads and masking function, which can integrate historical gaze information; the second layer is a multi-head attention layer that integrates the encoder output and the output of the previous decoder layer; the third layer is a feedforward network layer (two-layer fully connected network). Layer normalization and residual connection operations are added after each sublayer.

[0106] Step S5: using a three-dimensional mixed density network, transforming the hidden state parameters of the new fixation point at the current moment into joint three-dimensional Gaussian distribution parameters, establishing a joint probability density distribution map, and generating new fixation point coordinates through sampling operations;

[0107] Specifically, construct the mean of the i-th Gaussian distribution map in the three-dimensional Gaussian distribution of the new fixation point at the current time t Covariance and probability The calculation expression of the calculation process is:

[0108]

[0109] where f mdn represents the mixture density network, Z t represents the hidden state of the predicted gaze point at time t, θ mdn represents the parameters of the mixture density network, represents the mean of the i-th Gaussian distribution at time t, represents the covariance of the i-th Gaussian distribution at time t, Represents the probability of the i-th Gaussian distribution at time t.

[0110] The mixed density network is implemented as multiple independent two-layer linear fully connected networks. After the first layer, the relu function is used to calculate the probability of each Gaussian distribution. When calculating the mean of the Gaussian distribution, the softmax function is used after the second layer to ensure that the sum of the probabilities of all independent Gaussian distributions is 1; When , the tanh function is used after the second layer, which can ensure that the mean of the fixation point is in the range of [-1,1]; when calculating the covariance matrix When , the calculation of the covariance matrix is ​​decomposed into the calculation of the corresponding correlation coefficient and the associated variance When calculating the correlation coefficient When , the tanh function is used after the second fully connected layer, so that the correlation coefficient range is between [-1, 1]; when calculating the correlation variance When , the exp function is used after the second fully connected layer to ensure that the variance is greater than 1.

[0111] Current time t fixation point p t The calculation formula for a single Gaussian distribution function is:

[0112]

[0113] Current time t fixation point p t The joint Gaussian distribution of the fixation points can be obtained by weighted addition to obtain the final distribution. The specific function calculation formula is expressed as:

[0114]

[0115] Here, K represents the number of three-dimensional Gaussian distributions, specifically, the number of independent three-dimensional Gaussian distributions that the joint probability density distribution map is composed of. In the specific implementation, K=5 three-dimensional Gaussian distributions are used to construct the joint Gaussian distribution map.

[0116] The present invention proposes a method for predicting glance trajectories for panoramic images. The present invention proposes for the first time to predict glance trajectories of human eyes in a three-dimensional spherical coordinate system; it uses feature maps of different scales of a multi-stage three-dimensional convolutional network to extract features, which helps to focus on image features of different scales; it uses Transformer to extract image and historical gaze point information, which can fully consider more global image information and historical gaze point coordinate information; it uses a three-dimensional mixed density network to generate new gaze points, which can more reasonably establish the distribution of gaze points. Through the reasonable combination of these modules, the present invention can generate more realistic and reliable panoramic image glance trajectories for 360° panoramic images.

[0117] The above is only a preferred embodiment of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for predicting human eye scanning trajectories for panoramic images, characterized in that: The method comprises the following steps: Step S1, converting the panoramic image into a three-dimensional spherical coordinate system to obtain a panoramic image in the three-dimensional spherical coordinate system; Step S2: Using a multi-stage three-dimensional convolutional network to obtain feature information F of the panoramic image in a three-dimensional spherical coordinate system L ; Step S3: Use the 3D Transformer encoder to transform the feature information F L Contextualization, obtaining feature information F L Perceptual representation of the context; Step S4: The feature information F L The perceptual representation of the context, the historical gaze point query information, and the query information of the new gaze point at the current moment are input into the 3D Transformer decoder to predict the hidden state parameters of the new gaze point at the current moment; Step S5: using a three-dimensional mixed density network, transforming the hidden state parameters of the new gaze point at the current moment into multiple three-dimensional Gaussian distribution parameters, establishing a joint probability density distribution map, and generating new gaze point coordinates through sampling operations; Step S6, repeating steps S4 to S5, generating new gaze point coordinates through continuous iteration until a predicted scanning trajectory of a required length is generated; The implementation process of step S2 is: S201, using a multi-stage three-dimensional convolutional network SphereNet to extract three different dimensions of the panoramic image in the three-dimensional spherical coordinate system. The three different dimensions of the two-dimensional plane feature maps are upsampled to obtain feature maps F1, F2, and F3, which are expressed as: F1,F2,F3=Upsample(SphereNet)I,θ)) Where I represents the panoramic image in the three-dimensional spherical coordinate system, and θ represents the parameters of the multi-stage three-dimensional convolutional network SphereNet; S202, concatenate the feature maps F1, F2, and F3 in the channel dimension to obtain a global feature map F containing information of different scales, expressed as: F=Concat(F1,F2,F3) Where Concat represents the concatenation operation; S203: Perform operations including average pooling and flattening on the global feature map F to convert the global feature map F into a feature vector F with a length of L. L , the expression is: F L =Flatten(AvgPool(F)) Among them, Flatten represents the flattening operation, and AvgPool represents the average pooling.

2. The method for predicting eye scan trajectories for panoramic images according to claim 1, characterized in that: The implementation process of step S1 is: S101, convert all two-dimensional coordinate points p=[Y,X], Y,X∈[0,1] on the panoramic image into the longitude and latitude representation form of the two-dimensional coordinate points p=[φ,λ], The conversion formula is: φ=(Y-0.5)π; λ=(X-0.5)2π Where Y, X are the coordinate values ​​of the two-dimensional coordinate point on the Y axis and X axis in the two-dimensional plane coordinate representation, φ, λ are the latitude and longitude coordinate values ​​of the two-dimensional coordinate point in the longitude and latitude representation; S102, convert the longitude and latitude representation of the two-dimensional coordinate point into a three-dimensional space coordinate point p=(x, y, z) in a three-dimensional spherical coordinate system, where x, y, z∈[-1, 1]. The conversion formula is: x=cos(φ)cos(λ); y=cos(φ)sin(λ); z=sin(φ); Where x, y, and z are the coordinate values ​​of the three-dimensional space coordinate point on the X-axis, Y-axis, and Z-axis in the three-dimensional spherical coordinate system, respectively, so as to obtain a panoramic image in the three-dimensional spherical coordinate system.

3. The method for predicting human eye scanning trajectories for panoramic images according to claim 1, characterized in that: In step S3, the feature vector F L The following steps are used to pass the 3D Transformer encoder: S301, feature vector F L Transform the dimensions into the internal dimensions of the 3D Transformer encoder through a linear embedding; S302, then the eigenvector F L Add a learnable positional encoding; S303, the feature vector F L The feature information F is obtained by sequentially passing through the four standard Transformer encoding layers in the 3D Transformer encoder. L A context-aware representation of .

4. The method for predicting human eye scanning trajectories for panoramic images according to claim 1, characterized in that: In step S4, the 3D Transformer decoder iteratively generates the hidden state parameters of the new gaze point at the current moment in an autoregressive manner.

5. The method for predicting eye scan trajectories for panoramic images according to claim 1, characterized in that: The implementation process of step S4 is: S401, linearly embedding the current gaze point coordinates to obtain a query vector of the current gaze point coordinates; S402, concatenating the query vector of the current gaze point coordinates and the query vector obtained from the historical gaze point coordinates to obtain a joint query vector; S403, adding one-dimensional absolute position encoding information of the fixation point sequence to the joint query vector; S404: Pass the joint query vector with the one-dimensional absolute position encoding information through the 4 standard decoding layers of the 3D Transformer decoder to obtain the hidden state parameter Z of the new gaze point at the current moment. t .

6. The method for predicting human eye scanning trajectories for panoramic images according to claim 1, characterized in that: In step S5, the new fixation point p at the current time t is calculated by a three-dimensional hybrid density network. t The probability of the i-th Gaussian distribution graph Covariance Mean The calculation formula is expressed as: Among them, f mdn represents a three-dimensional mixed density network, which includes multiple independent two-layer linear fully connected networks, Z t represents the state of the new gaze point at the current time t, θ mdn represents the parameters of the three-dimensional mixture density network, represents the mean of the i-th Gaussian distribution at the current time t, represents the covariance of the i-th Gaussian distribution at the current time t, Represents the probability of the i-th Gaussian distribution at the current time t.

7. The method for predicting eye scan trajectories for panoramic images according to claim 6, characterized in that: In step S5, the new fixation point p at the current time t t The calculation formula of a single Gaussian distribution function is expressed as: The new fixation point p at the current time t t The calculation formula of the joint Gaussian distribution function is expressed as: Where K represents the number of three-dimensional Gaussian distributions that make up the joint probability density distribution graph; According to the current time t, the new gaze point p t The joint Gaussian distribution function P(p t ) draws a probability plot of the joint Gaussian probability distribution.

8. The method for predicting eye scan trajectories for panoramic images according to claim 7, characterized in that: In step S5, in the step of generating new gaze point coordinates by sampling operation from the joint Gaussian probability distribution probability map, the following relationship exists: Among them, ~ represents the sampling operation, Indicates generating new fixation point coordinates from the joint Gaussian probability distribution map.

9. The method for predicting eye scan trajectories for panoramic images according to claim 8, characterized in that: The loss function used in the training process of the model is negative logarithmic loss. The model includes a multi-stage three-dimensional convolutional network, a 3D Transformer encoder, a 3D Transformer decoder and a three-dimensional mixed density network connected in series, satisfying the formula: in, represents the actual human gaze point at time t, and T represents the length of the scanning trajectory.

Images