Explanatable video eye movement prediction method based on visual language large model

Abstract

The invention provides an interpretable video eye movement prediction method based on a visual language large model, and belongs to the technical field of video eye movement prediction. According to the method, an attribution-guided visual decoder is added in a visual language large model architecture to form an interpretable video eye movement prediction model, and a special token: lt is added in a vocabulary of the language decoder; gt, heatmapgt; and predicting a saliency map, analyzing text description and back cognitive reasons of a salient region, and training an explainable video eye movement prediction model in a supervised fine tuning mode. According to the method, the current visual language large model which only can output texts can output pixel-level saliency prediction results at the same time, and the influence of a previous frame on an eye movement fixation point of a current frame is considered through text instruction construction and space-time residual error complementation module design in a visual decoder; the method is a universal network architecture method, can be suitable for a current mainstream visual language large model of an autoregression architecture, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of video eye movement prediction, and particularly to a multimodal interpretable video eye movement prediction model method capable of simultaneously performing saliency map prediction and text interpretation and attribution generation. Background Art

[0002] The video eye movement prediction task (Video Saliency Prediction), abbreviated as VSP, aims to predict the visual attention distribution of human viewers when watching based on the input video sequence, and generate a corresponding saliency map for each frame to mark the focus area of attention. By simulating the attention mechanism of the human eye in a dynamic scene, this task provides a quantitative basis for understanding visual perception and has important application values in fields such as video compression coding, advertising effect evaluation, and virtual reality interaction system optimization.

[0003] Early VSP models (such as STSANet [1]) used 3D convolutional networks and spatio-temporal self-attention modules to capture long-range temporal dependencies. Transformer-based models (such as VSFT [2] and TMFI-Net [3]) further advanced this development by combining self-attention mechanisms to capture short-range and long-range temporal relationships. In particular, TMFI-Net adopted a Video Swin Transformer backbone to extract multi-level spatio-temporal features. In addition, models [4][5] demonstrated that audio-video integration achieved good results in VSP. Newer methods such as MSFF-Net [6] improved the temporal aggregation and domain adaptation strategies, and adopted multi-branch and multi-scale fusion techniques to enhance saliency prediction across different datasets.

[0004] However, the shortcoming of these works is that the vast majority of existing works only focus on predicting where human attention is concentrated, while ignoring why certain regions attract attention. Only a few studies have explored the interpretable aspects of eye movement prediction, such as GazeXplain [7], which combines a simple description of gaze patterns and scan path prediction. Exploring the association between attention patterns and cognitive mechanisms and simultaneously improving the model prediction performance and interpretability contribute to promoting in-depth understanding of the core laws of the human visual attention mechanism in the field of eye movement prediction.

[0005] [1].Ziqiang Wang,Zhi Liu,Gongyang Li,Yang Wang,Tian hong Zhang,LihuaXu,and Jijun Wang.Spatio-temporal self-attention network for video saliencyprediction.IEEE Transactions on Multimedia,25:1161–1174,2021.

[0006] [2].Cheng Ma,Haowen Sun,Yongming Rao,Jie Zhou,and Jiwen Lu.Videosaliency forecasting transformer.IEEE Transactions on Circuits and SystemsforVideo Technology,32(10):6850–6862,2022.

[0007] [3].Xiaofei Zhou,Songhe Wu,Ran Shi,Bolun Zheng,ShuaiWang,HaibingYin,Jiyong Zhang,and Chenggang Yan.Transformer-based multi-scale featureintegration network for video saliency prediction.IEEE Transactions onCircuits and Systems forVideo Technology,33(12):7696–7707,2023.

[0008] [4].Qinyao Chang and Shiping Zhu.Temporal-spatial feature pyramid forvideo saliency detection,2021.

[0009] [5]. Samyak Jain, Pradeep Yarlagadda, Ramanathan Subramanian, and Vineet Gandhi. Avinet: Diving deep into audio-visual saliency prediction. ArXiv, abs / 2012.06170, 2020.

[0010] [6]. Yunzuo Zhang, Tian Zhang, Cunyu Wu, and Ran Tao. 720 Multi-scale spatiotemporal feature fusion network for video saliency prediction. IEEE Transactions on Multimedia, 26:7224183–4193, 2024.

[0011] [7]. Gazexplain: Learning to predict natural language explanations of visual scan-paths. In European Conference on Computer Vision, pages 314–333. Springer, 2024. Summary of the Invention

[0012] The objective of the present invention is to provide an interpretable video eye movement prediction method based on a vision-language large model. Based on the existing vision-language large model, an attribution-guided visual decoder is added to endow the existing vision-language large model with pixel-level saliency prediction ability, enabling the prediction of saliency maps for the representative frames of each video scene, as well as the text description of the salient regions and the analysis of the underlying cognitive reasons; in addition, through the design of text instruction construction and the spatio-temporal residual completion module in the visual decoder, the influence of the attention points of the previous frames on the current frame is considered, that is, the visual residual effect is explicitly modeled.

[0013] The technical solution of the present invention is as follows:

[0014] An interpretable video eye movement prediction method based on a vision-language large model, characterized in that an attribution-guided visual decoder is added to the vision-language large model architecture to form an interpretable video eye movement prediction model. The vision-language large model architecture includes four parts: a text tokenization and embedding layer, a visual encoder, a modality projection layer, and a language decoder. By adding special tokens to the vocabulary of the language decoder: <heatmap>, perform the prediction of the saliency map, the text description of the salient region, and the analysis of the underlying cognitive reasons; and train the interpretable video eye movement prediction model through supervised fine-tuning; the specific steps include:

[0015] A. Text result generation: Preprocess the video data to be analyzed according to the requirements of the vision-language large model for the input visual size, and obtain multiple frames of input X after processing. v , The visual encoder processes multiple frames of input X v to generate visual features f v , and through the modality projection layer, project f v into visual feature tokens: h v , at the same time, the text tokenization and embedding layer processes the text instruction input X t to generate text feature tokens: h t , concatenate the two types of tokens and input them into the language decoder to output the text result Y including the description of the key attention area and the cognitive attribution analysis. t ;

[0016] B. Saliency information extraction: Add a special token to the vocabulary of the language decoder in advance: <heatmap>, for extracting the saliency information of the text end; when the text instruction prompts the model to output a saliency prediction map, the text output generated by the language decoder contains <heatmap>, and then extract from the last hidden layer of the language decoder <heatmap>The corresponding features are obtained and the salient features h are generated by the multi-layer perceptron (MLP) of the attribution-guided visual decoder. heatmap ;

[0017] C. Saliency prediction map generation: The attribution-guided visual decoder simultaneously accepts the visual features f output by the visual encoder v And the extracted salient features h heatmap , get the significant prediction map result Y pred ;

[0018] D. Interpretable video eye movement prediction model training: the text result Y t , significance prediction map result Y pred The images were compared with the real annotations respectively, and the model was trained through supervised fine-tuning to obtain a trained interpretable video eye movement prediction model.

[0019] Furthermore, in step A, multiple frames are input Visual features Where T is the number of time frames, H×W is the video frame resolution, S is the sampling step, and C v is the hidden layer dimension of the visual encoder.

[0020] Furthermore, the significant features are generated in step B Among them C ε is the feature fusion dimension.

[0021] Furthermore, the step C of generating the significance prediction map is specifically as follows:

[0022] 1) Dimensionality reduction convolution: for the features output by the visual encoder Dimensionality reduction is achieved through 1×1 convolution Where T is the number of time frames, H×W is the video frame resolution, S is the sampling step, and C d is the first intermediate feature dimension;

[0023] 2) Spatiotemporal residual completion: The spatiotemporal residual completion module is used to fuse the features of different time frames. Specifically, the features of different times are spliced into the feature dimension, and then passed through the convolution module. Finally, the residual is added to the features of the most recent time to obtain Among them, H×W is the video frame resolution, S is the sampling step, C d' is the second intermediate feature dimension;

[0024] 3) Upsampling: gradually restore the visual feature resolution through multi-layer upsampling modules, specifically by obtaining pixel-by-pixel features of the original resolution through deconvolution operations and bilinear interpolation Among them, H×W is the video frame resolution, S is the sampling step, C ε is the feature fusion dimension;

[0025] 4) Feature fusion: The per-pixel feature f v3 and the saliency feature h heatmap perform a per-pixel dot product operation, and then pass through the final convolutional layer and sigmoid activation layer to obtain the prediction result of the saliency map where H×W is the video frame resolution.

[0026] Furthermore, the step D can explain the training of the video eye movement prediction model, specifically:

[0027] Given the predicted saliency map result Y pred , the fixation map Y generated by the truly collected fixation points fix and the saliency map Y sal , the visual saliency loss formula is based on multiple visual saliency evaluation metrics, and the visual saliency evaluation metrics include Kullback-Leibler divergence (KLD), Pearson correlation coefficient (CC), similarity (SIM), and normalized scanpath saliency (NSS). The specific formula is as follows:

[0028]

[0029] where λ1, λ2, λ3, λ4 are learnable parameters;

[0030] Given the generated text description and attribution annotation Y t , and the gold standard text description and attribution Y gt , the loss function at the text end adopts the cross-entropy loss function, and the specific formula is:

[0031]

[0032] Therefore, the final total loss is:

[0033]

[0034] where λ sal , λ t are learnable parameters.

[0035] The technical effects of the present invention are as follows:

[0036] The interpretable video eye movement prediction method based on the vision-language large model provided by the present invention, based on the existing vision-language large model, adds an attribution-guided visual decoder, and by adding special tokens to the vocabulary of the language decoder, it enables a type of vision-language large model that can currently only output text to simultaneously output pixel-level saliency prediction results. In addition, through text instruction construction and the design of the spatio-temporal residual completion module in the visual decoder, the influence of previous frames on the current frame's eye movement fixation points is considered, that is, the visual residual effect is explicitly modeled. This method is a general network architecture method and can be applied to the current mainstream vision-language large models with an autoregressive architecture. The method of the present invention achieves a saliency prediction accuracy exceeding that of traditional methods based on single frames only. Through experiments, significant improvements in multiple metrics are achieved on the SGI-ADV dataset. Compared with the latest model SUM in traditional methods, the sAUC metric is improved from 0.799 to 0.830, the SIM metric is improved from 0.605 to 0.651, and the KL divergence metric is reduced from 0.659 to 0.507. The text results of the description and cognitive attribution analysis of the key attention areas also have good effects. Scored by the GPT4o large model as a judge, the similarity between the descriptions generated by the model and the true annotations reaches an average of over 65%. Description of the Drawings

[0037] Figure 1 It is the overall framework diagram of the interpretable video eye movement prediction method based on the vision-language large model of the present invention. Detailed Embodiments

[0038] The present invention will be further clearly and completely described below with reference to the drawings and through specific embodiments.

[0039] An interpretable video eye movement prediction method based on the vision-language large model of the present invention Figure 1 is the overall framework diagram of the method of the present invention. Based on the existing vision-language large model (an autoregressive architecture composed of four parts: text tokenization and embedding layer, visual encoder, modality projection layer, and language decoder), an attribution-guided visual decoder is added, and the whole is used as an interpretable video eye movement prediction model. And by adding special tokens to the vocabulary of the language decoder: <heatmap>, endow existing vision-language large models with pixel-level saliency prediction capabilities, enabling the prediction of saliency maps for representative frames in each video, as well as the text description of salient regions and the analysis of the underlying cognitive reasons. Additionally, through the design of text instruction construction and the spatio-temporal residual completion module in the visual decoder, the influence of the attention points in previous frames on the current frame is considered, i.e., explicitly modeling the visual residual effect.

[0040] First, according to the vision-language large model, preprocess the video data to be analyzed to obtain T-frame inputs. For a video with a resolution of 1920×1080, perform shot preprocessing and then retain the key frames of each shot. Select T = 2 key frames as inputs. If the vision-language large model selects a model that specifies the size of the visual input, such as InternVL2.5, then process the input T frames with operations including padding and scaling according to the aspect ratio matching method, and then cut them into 448×448×6 blocks and input them into the model; conversely, if the vision-language large model selects a model that can support any native resolution, such as Qwen2.5-VL, then keep the original video size unchanged.

[0041] Next, based on the interpretable eye movement prediction model of the vision-language large model, predict the saliency map, describe the text of the salient region, and analyze the underlying cognitive reasons; and train the interpretable video eye movement prediction model through supervised fine-tuning.

[0042] The input of the model consists of two parts: visual input and language text instruction input. In this embodiment, the visual input is T = 2 key frames of a video, and the text instruction input is "Based on the changes between two consecutive frames and the elements in the second frame of the video, which area do you think the human attention mainly focuses on? Further, analyze the possible reasons". The output consists of two parts: text and a saliency map. The text includes a textual description of the key attention area in the video frame and an analysis of the cognitive reasons behind the attention behavior. For example, "[Attention Area]: The area that the audience focuses on highlights a hand holding a Coca-Cola bottle with a clearly visible label. [Reason Analysis]: The previous key frame showed a vibrant store scene with a girl and colorful merchandise. This scene transitions to a close-up of a man drinking water with the Coca-Cola bottle. The red label of the bottle stands out against the neutral background, and its bright color and brand recognition naturally attract the audience's attention. The drinking action is a dynamic movement that easily catches people's interest because the audience is usually attracted to movement. Additionally, the face of the girl in the first frame likely attracted the initial attention. Due to the persistence of vision effect, even after the transition, part of the attention remains in the same area, thus subtly guiding the audience's attention. The combination of color contrast, brand familiarity, dynamic movement, and persistence of vision likely guides the audience's attention to the bottle and the drinking action. [Summary]: People's attention focuses on <heatmap>Above.", the saliency map is the per-pixel saliency prediction result, and the value range of each pixel is [0, 1]. The larger the value, the greater the possibility of attention.

[0043] The processing process of the interpretable eye movement prediction model based on the vision-language large model is as follows:

[0044] (1). Visual feature extraction: The visual encoder processes T frames of input Generate visual features In a specific implementation, T is the number of time frames, which is 2, H×W is the video frame resolution, S is the sampling step, and C v is the hidden layer dimension of the visual encoder.

[0045] (2). Visual feature projection: Through the modality projection layer, project f v to visual feature tokens: h v .

[0046] (3). Text feature extraction: The text tokenization and embedding layer processes the text instruction X t , and generates text feature tokens: h t .

[0047] (4). Language decoding: Concatenate the visual feature tokens: h v and the text feature tokens: h t , then input the concatenated features into the language decoder to generate the text output Y t , including the description of the key attention area and the cognitive attribution analysis.

[0048] (5). Saliency information extraction: Add <heatmap>, when the text instruction requires the output of a saliency prediction map, the text output generated by the language decoder contains <heatmap>Extract from the last hidden layer of the language decoder <heatmap>The corresponding feature generates a saliency feature through the multi-layer perceptron (MLP) of the attribution-guided visual decoder where C ε is the feature fusion dimension.

[0049] (6) Generation of saliency prediction map: The attribution-guided visual decoder simultaneously receives the visual feature f v output by the visual encoder and the extracted saliency feature h heatmap , and obtains the saliency map through the dimensionality reduction convolution, spatio-temporal residual completion, upsampling, and feature fusion module Specifically, first perform dimensionality reduction convolution. For the feature output by the visual encoder, perform dimensionality reduction through 1×1 convolution to obtain Then, fuse the features of different time frames through the spatio-temporal residual completion module. Specifically, splice the features of different times to the feature dimension, then pass through the convolution module, and finally add it as a residual to the feature of the nearest time to obtain Then, gradually restore the visual feature resolution through the multi-layer upsampling module. Specifically, obtain the per-pixel feature of the original resolution through the transposed convolution operation and bilinear interpolation Finally, perform a per-pixel dot product operation on the per-pixel feature f v3 and the saliency feature h heatmap , and then pass through the final convolution layer and sigmoid activation layer to obtain the prediction result of the saliency map where T is the number of time frames, H×W is the video frame resolution, S is the sampling step, C v is the hidden layer dimension of the visual encoder, C ε is the feature fusion dimension, C d is the first intermediate feature dimension, C d' is the second intermediate feature dimension.

[0050] To improve the output effect of the model, model training is carried out. The training of the interpretable eye movement prediction model based on the vision-language large model includes:

[0051] Given the predicted saliency map result Y pred , the fixation map Y generated by the actually collected fixation points fix and the saliency map Y sal , their array shapes are all (1, H, W), and Y pred and Y sal after normalization satisfy Y fix The value range of is {0, 1}. The visual end saliency loss formula is based on multiple visual saliency evaluation metrics, including Kullback-Leibler divergence (KLD), Pearson correlation coefficient (CC), similarity (SIM), and normalized scanpath saliency (NSS). The specific formula is as follows:

[0052]

[0053] where λ1, λ2, λ3, λ4 are learnable parameters,

[0054]

[0055] ∈ is the regularization parameter, set to 2.2204×10 -16 ;

[0056]

[0057] where cov(·) is the covariance and σ(·) is the standard deviation;

[0058]

[0059]

[0060] where σ(·) and μ(·) represent the standard deviation and the mean respectively.

[0061] Given the generated text description and attribution annotation Y t , and the gold standard text description and attribution Y gt , the loss function at the text end adopts the cross-entropy loss function, and the specific formula is:

[0062]

[0063] Therefore, the final total loss is:

[0064]

[0065] where λ sal , λ t are learnable parameters.

[0066] Through experiments, the method of the present invention has achieved significant improvements in multiple metrics on the SGI-ADV dataset. Compared with the latest model SUM in the traditional method, the sAUC metric has increased from 0.799 to 0.830, the SIM metric has increased from 0.605 to 0.651, and the Kullback-Leibler divergence metric has decreased from 0.659 to 0.507. The text results of the description and cognitive attribution analysis of the key areas of concern are also good. Scored by the GPT4o large model as the judge, the similarity between the description generated by the model and the true annotation reaches more than 65% on average.

[0067] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention shall be defined by the scope defined in the claims.< / heatmap> < / heatmap> < / heatmap> < / heatmap> < / heatmap> < / heatmap> < / heatmap> < / heatmap> < / heatmap>

Claims

1. An interpretable video eye movement prediction method based on a vision-language large model, characterized in that, Add an attribution-guided visual decoder to the visual language large model architecture to form an interpretable video eye movement prediction model. The visual language large model architecture includes four parts: a text tokenization and embedding layer, a visual encoder, a modality projection layer, and a language decoder. By adding special tokens to the vocabulary of the language decoder: <heatmap>, perform the prediction of the saliency map, the text description of the salient region, and the analysis of the underlying cognitive reasons; < / heatmap> and train the interpretable video eye movement prediction model through supervised fine-tuning; the specific steps include: A. Text result generation: Preprocess the video data to be analyzed according to the requirements of the vision-language large model for the input vision size, and obtain multiple frames of input X after processing. v The vision encoder processes the multiple frames of input X. v Generate visual feature f. v Through the modality projection layer, project f v into visual feature tokens: h. v At the same time, the text tokenization and embedding layer processes the text instruction input X. t Generate text feature tokens: h. t After concatenating the two types of tokens, input them into the language decoder, and output the text result Y including the description of the key attention area and the cognitive attribution analysis. t ; B. Salient information extraction: Pre-add a special token to the vocabulary of the language decoder: <heatmap>, for extracting the saliency information of the text end; when the text instruction prompts the model to output a saliency prediction map, the text output generated by the language decoder contains <heatmap>, and then extract from the last hidden layer of the language decoder <heatmap>The corresponding features and generate the saliency feature h through the multi-layer perceptron (MLP) of the attribution-guided visual decoder heatmap ;< / heatmap> < / heatmap> < / heatmap> C. Generation of saliency prediction map: The attribution-guided visual decoder simultaneously receives the visual feature f output by the visual encoder v and the extracted saliency feature h heatmap , and obtains the saliency prediction map result Y pred ; D. Training of interpretable video eye movement prediction model: Compare the text result Y t and the saliency prediction map result Y pred with the true annotations respectively, and train the model through supervised fine-tuning to obtain a trained interpretable video eye movement prediction model.

2. The method according to claim 1, characterized in that, In the above step A, multiple frames are input Visual feature where T is the number of time frames, H×W is the video frame resolution, S is the sampling step, and C v is the hidden layer dimension of the visual encoder.

3. The method according to claim 1, characterized in that Generating significant features in step B where C ε is the feature fusion dimension.

4. The method according to claim 1, wherein The step C of generating the saliency prediction map is specifically as follows: 1) Dimensionality reduction convolution: For the features output by the visual encoder Dimensionality reduction is achieved through 1×1 convolution to obtain where T is the number of time frames, H×W is the video frame resolution, S is the sampling step, and C d is the first intermediate feature dimension; 2) Spatiotemporal Residual Completion: The features of different time frames are fused through the spatiotemporal residual completion module. Specifically, the features at different times are concatenated to the feature dimension, then passed through the convolutional module, and finally added to the features of the nearest time as a residual to obtain where H×W is the video frame resolution, S is the sampling step, and C d' is the second intermediate feature dimension; 3) Upsampling: Gradually restore the resolution of visual features through multiple upsampling modules. Specifically, obtain the features of each pixel at the original resolution through deconvolution operations and bilinear interpolation. where H×W is the video frame resolution, S is the sampling step, and C ε is the feature fusion dimension; 4) Feature fusion: pixel-by-pixel feature f v3 and saliency feature h heatmap perform a pixel-by-pixel multiplication operation, and then pass through the final convolutional layer and sigmoid activation layer to obtain the prediction result of the saliency map where H×W is the video frame resolution.

5. The method according to claim 1, wherein The step D of training the interpretable video eye movement prediction model is specifically as follows: Given the saliency map result Y of the prediction pred , the fixation map Y generated by the true collected fixation points fix and the saliency map Y sal , the visual saliency evaluation metrics based on which the visual end saliency loss formula includes: KL divergence KLD, Pearson correlation coefficient CC, similarity SIM, normalized scanpath saliency NSS, and the specific formulas are as follows: where λ1, λ2, λ3, λ4 are learnable parameters; Given the generated text description and attribution annotation Y t , and the gold standard text description and attribution Y gt , the loss function at the text end uses the cross-entropy loss function, and the specific formula is: The final total loss is: where λ sal , λ t are learnable parameters.

Images