Gaze spectrum prediction method based on first-view-angle high-dynamic long video

Abstract

The invention discloses a fixation spectrum estimation method based on a first-view-angle high-dynamic long video, and belongs to the field of image processing. According to the gaze spectrum prediction method, the key problems that in the prior art, time sequence information in a long-time video is lost, local and global information fusion is insufficient, and static masks are difficult to adapt to dynamic scene changes are effectively solved through three innovative modules. The enhanced long-time memory encoder effectively encodes a long-time video through multi-scale attention and a hierarchical memory mechanism, the problems of information loss and redundancy under long-time dependence are solved, and the integrity of time sequence information is ensured; the high-pass global-local information aggregation module enables a global background and local details to be effectively combined in a multi-layer network by designing a cross-layer dynamic information transmission channel, so that the gaze spectrum prediction capability in a complex dynamic scene is enhanced; and the dynamic mask fusion module adopts a self-adaptive mechanism, so that the mask weight can be adjusted in real time, the problem that static attention cannot cope with a rapid change background is solved, and the flexibility and accuracy of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular to analyzing and understanding highly dynamic and long-time first-person perspective videos in real scenes, and can effectively capture the changes in the user's gaze spectrum in complex dynamic scenes. Background Art

[0002] In recent years, with the rapid development of technologies such as augmented reality and virtual reality, gaze spectrum prediction technology based on first-person video has been increasingly used in fields such as intelligent driving and human-computer interaction. By analyzing the user's visual focus, this technology can accurately capture their cognitive process and behavioral intentions, significantly improving the interactive performance of the system. However, with the increasing complexity of video application scenarios, existing gaze spectrum prediction methods have shown many limitations when processing long-duration, highly dynamic, and complex background videos. Especially in scenes with drastic dynamic changes, the user's gaze point moves frequently, making it difficult for existing models to balance the accuracy and real-time performance of prediction. Therefore, how to achieve efficient gaze spectrum prediction in complex dynamic scenes has become a technical problem that needs to be solved urgently in this field.

[0003] The core challenge of the above problems lies in the processing of long videos and highly dynamic scenes. First, the processing of long videos brings significant challenges. Long videos often contain a large amount of temporal information, and the user's visual gaze spectrum is extremely dynamic in these temporal information. However, a large amount of irrelevant information and redundant features can easily mask the key visual clues related to the user's gaze spectrum, making it difficult for the model to effectively extract key information, thereby affecting the accuracy of the prediction. Secondly, in complex dynamic scenes, the user's hands frequently interact with objects. These local details are crucial for the prediction of the gaze spectrum, but they are highly coupled with the global background information. How to achieve effective fusion between global and local information has become a major problem for existing models. Although the current convolutional neural network and recurrent neural network models can process spatiotemporal information to a certain extent, their ability to extract local features and model the global background is insufficient when facing long time-series videos and complex scenes, which often leads to the inability to accurately characterize the dynamic changes of the gaze spectrum. In addition, the static attention mechanism is difficult to adapt to the rapidly changing scene background, which further limits the performance of existing methods in dynamic scenes. In this case, static attention not only cannot dynamically adjust the visual weight, but may cause the model to ignore key visual information, reducing the overall prediction effect. Therefore, existing gaze spectrum prediction technologies face serious performance bottlenecks when dealing with complex backgrounds and long-term dynamic scenes. Summary of the invention

[0004] The gaze spectrum prediction method of the present invention effectively solves the key problems existing in the prior art, such as the loss of temporal information in long-term videos, insufficient fusion of local and global information, and the difficulty of static masks to adapt to dynamic scene changes, through three innovative modules. The enhanced long-term memory encoder effectively encodes long-term videos through multi-scale attention and hierarchical memory mechanisms, solves the problems of information loss and redundancy under long-term dependencies, and ensures the integrity of temporal information; the Qualcomm global-local information aggregation module effectively combines global background and local details in a multi-layer network by designing a cross-layer dynamic information transmission channel, thereby enhancing the ability to predict gaze spectrum in complex dynamic scenes; the dynamic mask fusion module adopts an adaptive mechanism that can adjust the mask weight in real time, solves the problem that static attention cannot cope with rapidly changing backgrounds, and improves the flexibility and accuracy of the model.

[0005] In order to address these challenges, the present invention proposes a method for predicting gaze spectrum based on first-person perspective high-dynamic long video; the method aims to improve the accuracy and real-time performance of gaze spectrum prediction by accurately capturing the user's visual gaze spectrum in long-term dynamic scenes; the present invention achieves these goals through three key technical modules, including an enhanced long-term memory encoder, a high-pass global-local information aggregation module, and a dynamic mask fusion module. These modules deal with various challenges in long videos and complex scenes from different levels, and through the synergy between modules, more efficient gaze spectrum prediction is achieved; by designing an enhanced long-term memory encoder, the model can effectively retain the key features in long-term videos, reduce the interference of irrelevant information, and improve the efficiency of information extraction; the high-pass global-local information aggregation module ensures the efficient fusion of local hand movements and object interaction information with global scene information, thereby accurately predicting the gaze spectrum in complex dynamic scenes. In addition, the dynamic mask fusion technology can adaptively adjust the mask to respond to changes in the scene in real time, so that the model can still accurately extract key information in a rapidly changing background. Through these innovative designs, the present invention significantly improves the prediction accuracy and real-time processing capabilities in complex scenes, provides an effective technical solution for gaze spectrum prediction based on first-person perspective videos, and has broad application prospects.

[0006] The technical solution of the present invention is a gaze spectrum estimation method based on first-person perspective high dynamic long video, which includes: visual token embedding, enhanced long-term memory encoder, high-pass global-local information aggregation module, dynamic mask fusion module, decoder; Step 1: given a self-centered input video clip of a fixed long time period T, whose spatial dimension is H×W, gaze prediction is regarded as a probability distribution on a two-dimensional image plane;

[0007] The specific method of visual token embedding is: divide the input video sequence X into non-overlapping small blocks s T ×sH ×s W , the video sequence includes T time periods with a dimension of H×W; then, each small block is mapped to a D-dimensional vector space through a linear mapping function; through the above operations, local visual tokens; add learnable position embeddings to the generated local visual tokens To enhance the spatiotemporal location information of each local visual token:

[0008] L i =f(X i )+E i

[0009] Among them, L i is the i-th local visual token, f(X i ) is the feature extracted from the i-th small block, E i Represents the i-th position embedding; in order to further include global scene information in the visual token, a global visual token embedding based on convolution operation is designed; the global visual token is generated by downsampling the local visual token, and the maximum pooling is used on the local visual token to compress the local features into global features; a series of convolutional layers are used to downsample the local visual token to gradually generate the global visual token G:

[0010] G = Conv(MaxPool(L))

[0011] L represents the global view token;

[0012] Step 2: The enhanced long-term memory mechanism used combines the local and global visual tokens generated from the visual token embedding, adopts a multi-layer Transformer architecture, and introduces a memory mechanism to handle the complex temporal dependencies in long-duration high-dynamic videos; this mechanism can capture the user's gaze spectrum by effectively storing and processing historical information;

[0013] Step 3: Qualcomm's global-local information aggregation module is composed of two parts: global-local correlation attention and Highway MLP. The global-local correlation attention module is designed to capture the correlation between local visual features and global scene information in the video, especially in complex dynamic scenes, by strengthening the information interaction between local and global, and more accurately predicting the user's gaze spectrum. This module selectively enhances the part that helps predict the gaze spectrum by calculating the correlation weight between local features and global features, while suppressing background noise. By calculating local features With global features The correlation between local features and global features is selectively integrated;

[0014] Step 4: The specific method of the dynamic mask fusion module is:

[0015] The output characteristics of the enhanced long-term memory encoder are Where N represents the number of time steps and D represents the feature dimension. The dynamic mask fusion module is used to adaptively integrate the output features Z of the enhanced long-term memory encoder. mem And the processing features H(F) of the Qualcomm global-local information aggregation module are used to effectively focus on the most relevant visual information in complex dynamic scenes; the mask matrix is ​​generated by calculating the attention weights:

[0016]

[0017] Among them, W Q and W K is a learnable parameter matrix used to project the input into the same attention space; the Sigmoid activation function ensures that the mask value range is between (0, 1); the mask M performs element-by-element weighted fusion of the two feature matrices, and finally obtains the adaptive fusion feature Z fused :

[0018] Z fused =M⊙Z mem +(1-M)⊙H(F)

[0019] Among them, ⊙ represents element-by-element multiplication, and the mask M determines the output Z from the enhanced long-term memory encoder in the fused feature mem The weight of feature H(F) is processed with the high-pass global-local information aggregation module;

[0020] Step 5: Feature Z after dynamic mask fusion module fused The input is sent to the Transformer decoder to generate the final user gaze point prediction; the decoder is designed with reference to the multi-layer self-attention mechanism to ensure that the multi-scale spatiotemporal information of the fused features is fully utilized:

[0021]

[0022]

[0023] Step 6: Features of the final decoder layer output Generate the prediction result y of the gaze point through linear mapping t .

[0024] Furthermore, the specific method of step 2 is:

[0025] The specific method of the enhanced long-term memory encoder is: the local and global tokens L at each time step t t and G tWill be input into the enhanced long-term memory encoder; in order to process the timing information, the query Q, key K and value V are used to capture the dependency between the current time step and the historical time step:

[0026]

[0027] Among them, W Q , W K , W V They are the query, key, and value mapping matrices for local and global features, respectively. The superscripts L and G represent local and global.

[0028] In order to handle long-term dependencies, an enhanced long-term memory mechanism is used, which stores the key and value information of multiple previous time steps; specifically, a memory bank is introduced, which stores the local and global features of M past time steps, forming local memory and global memory respectively:

[0029]

[0030] Represents the local key information memory, Represents the local value information memory, Represents the global key information memory, Represents the global value information memory;

[0031] A multi-head self-attention mechanism is used to make the local and global query Q t The key information memory K of the previous time step mem Compare and calculate the attention weight based on the similarity, so as to dynamically focus on important historical information; the formula of the self-attention mechanism is as follows:

[0032]

[0033] V mem Represents a memory bank of value information;

[0034] Perform self-attention calculations on local and global features respectively to obtain local features and global features In order to enhance the capture of multi-level dependencies in long-term videos, a hierarchical memory mechanism is introduced in each Transformer layer of the enhanced long-term memory encoder; at each layer, the output features not only depend on the information of the current time step, but also refer to the memory state of the previous layer, ensuring the cumulative processing of temporal information.

[0035] Furthermore, the specific method of step 3 is:

[0036] The specific method of the Qualcomm global-local information aggregation module is as follows: first, and Perform linear mapping to generate query Q, key K and value V matrices:

[0037]

[0038] The inner product of the query matrix Q and the key matrix K is used to calculate the correlation weights between local and global features. In order to enhance the selectivity of the model, the suppression matrix S is subtracted to reduce the mutual influence between local features and emphasize the direct correlation between local and global features. The correlation between each local feature and the global feature, that is, the correlation weight matrix, is used to weight the global feature V to generate an enhanced correlation weight matrix:

[0039]

[0040] Each local feature will obtain corresponding information from the global feature according to its correlation with the global feature; the enhanced local feature With the original local features Perform weighted fusion to obtain the final global-local fusion feature F:

[0041]

[0042] Among them, α and β are weighting coefficients, which are used to adjust the contribution ratio of the original local features and the enhanced local features;

[0043] Highway MLP is used to control the information flow and fusion between global and local visual tokens, ensuring that local features retain the original information while achieving feature transformation; the core idea of ​​Highway MLP is to design a "gating" mechanism to allow some inputs to pass directly without nonlinear transformation, thereby retaining key information:

[0044] H(F)=T(F)·ReLU(W·F+b)+(1-T(F))·F

[0045] Among them, T(F) represents the transformation gate, which determines how much of the input needs to be nonlinearly transformed, and W and b are the parameters of the gating function. Through the above design, Highway MLP can effectively interact between global and local information to ensure the efficient fusion of the two.

[0046] Furthermore, in step 3, T(F) is a Sigmoid function.

[0047] The gaze spectrum prediction method based on first-person perspective high-dynamic long video proposed in the present invention is specifically used for user gaze prediction in long-term, high-dynamic video scenes, and can effectively cope with user visual focus prediction in complex scenes. It is suitable for various applications such as augmented reality, virtual reality and intelligent driving. The main innovation of the invention is that the transformer-based codec architecture has been carefully modified and some uniquely designed modules have been added. Through three innovative modules, the key challenges in long-term video processing are effectively addressed, including the preservation of temporal information, the efficient fusion of global and local information, and the ability to adjust the mask to adapt to dynamic scene changes. First, the enhanced long-term memory encoder encodes long-term dependent information through multi-scale attention and hierarchical memory mechanisms, reduces information redundancy and avoids temporal loss, and ensures the integrity of temporal features; secondly, the designed high-pass global-local information aggregation module introduces a cross-layer dynamic information transmission channel, so that the global background and local detail information are fully interacted in the multi-layer network, which improves the prediction effect in complex dynamic scenes; finally, the dynamic mask fusion module adopts an adaptive mechanism, which can adjust the mask weight in real time, flexibly adapt to the rapid changes of the background, and enhance the environmental adaptability and prediction accuracy of the model. The invention's F1 index on the public dataset egteagaze+ is 2.06% higher than the latest method GLC. The F1 index represents the harmonic mean of precision and recall. This method comprehensively improves the application performance of gaze spectrum prediction in long-term and high-dynamic scenes. The latest method GLC is Bolin Lai, Miao Liu, Fiona Ryan, James M. Rehg, In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0049] In order to solve the key challenges in long-term video processing in open world first-person videos, including the preservation of temporal information, the efficient fusion of global and local information, and the ability to adjust masks to adapt to dynamic scene changes, a learning framework for predicting user gaze spectra was created. As a challenging task, the invention needs to meet the requirements of temporal information preservation, efficient fusion of local and global information, and adaptive dynamic background changes in long-term, high-dynamic videos, so as to ensure the consistency and accuracy of gaze spectrum prediction. And accurate gaze spectrum prediction is performed in combination with long-term contextual information. The method extracts and fuses the significant local detail information of highly dynamic hands and various objects interacting in long-term complex background scenes and the global information of macroscopic scenes through three designs: enhanced long-term memory encoder, high-pass global-local information aggregation, and dynamic mask fusion, so that the model can extract significant feature information in videos in different degrees of dynamic scenes with long first-person perspectives, thereby generating human gaze eye movement spectra that conform to cognitive science and life sciences.

[0050] The main framework of the gaze spectrum prediction network relies on an encoder-decoder. For an egocentric input video, given a clip of a fixed length T with a spatial dimension of H×W, the input video sequence X is divided into non-overlapping patches s T ×s H ×s W Then, each small block is mapped into the D-dimensional vector space through a linear mapping function to obtain Local visual tokens. Learnable position embeddings are added to the generated local visual tokens. To enhance the spatiotemporal position information of each local visual token, the local visual token L i The global visual token is generated by downsampling the local visual token. Max pooling is used on the local visual token to compress the local features into global features. A series of convolutional layers are used to downsample the local visual token and gradually generate the global visual token G.

[0051] In order to enable the network to retain temporal information over a long time span, avoid the problem of information loss or key feature dilution under multi-frame dependencies. Especially for dynamic scenes in videos, the network needs to have the ability to maintain key information under long-term dependencies to ensure the consistency and stability of gaze spectrum prediction. An enhanced long-term memory encoder is designed to combine local and global visual tokens generated from visual token embeddings. A multi-layer Transformer architecture is adopted to handle complex temporal dependencies in long-term high-dynamic videos by introducing a memory mechanism. In order to enhance the capture of multi-level dependencies in long-term videos, a hierarchical memory mechanism is introduced in each layer of the Transformer of the enhanced long-term memory encoder. At each layer, the output features not only depend on the information of the current time step, but also refer to the memory state of the previous layer to ensure the cumulative processing of temporal information. This mechanism can capture the user's gaze spectrum by effectively storing and processing historical information. The local and global tokens L at each time step t t and G t Will be input into the enhanced long-term memory encoder. In order to process the time series information, the query Q, key K and value V are used to capture the dependency between the current time step and the historical time step:

[0052]

[0053] The memory is used to handle long-term dependencies, which stores key and value information of multiple previous time steps. Specifically, the memory stores local and global features of M past time steps, forming local memory and global memory respectively:

[0054]

[0055] A multi-head self-attention mechanism is used to make the local and global query Q t and the memory bank K of the previous time step mem Compare and calculate attention weights based on similarity, so as to dynamically focus on important historical information and obtain local features and global features Gaze spectrum prediction needs to pay attention to both local details (such as the interaction of user hands or objects) and global background (such as scene layout), and the network needs to be able to fuse these different scales of information. This fusion must not only maintain the relevance of details and the whole, but also be transmitted in multiple levels to ensure that the model can balance the impact of local and global information on prediction. Gaze spectrum prediction needs to pay attention to both local details (such as the interaction of user hands or objects) and global background (such as scene layout), and the network needs to be able to fuse these different scales of information. This fusion must not only maintain the relevance of details and the whole, but also be transmitted in multiple levels to ensure that the model can balance the impact of local and global information on prediction. Therefore, a high-pass global-local information aggregation module is designed to capture the correlation between local visual features and global scene information in the video. The module consists of two parts: global-local correlation attention and Highway MLP. By calculating the correlation weights between local features and global features, it selectively enhances the parts that contribute to gaze spectrum prediction while suppressing background noise. By calculating local features With global features The correlation between local features and global features is selectively integrated:

[0056]

[0057] The inner product of the query matrix Q and the key matrix K is used to calculate the correlation weights between local and global features, that is, the correlation weight matrix weights the global feature V to generate an enhanced correlation weight matrix:

[0058]

[0059] The enhanced local features With the original local features Perform weighted fusion to obtain the final global-local fusion feature F:

[0060]

[0061] Efficiently fuse local and global information, and use the Highway MLP gating mechanism to ensure that local details and global scene information complement each other. Highway MLP is used to control the flow and fusion of information between global and local visual tokens, thereby ensuring that local features can retain the original information while achieving feature transformation. The core idea of ​​Highway MLP is to design a "gating" mechanism that allows some inputs to pass directly without nonlinear transformation, thereby retaining key information. The addition of Highway MLP effectively enhances the capture of local and global correlations, allowing the model to incorporate global context while focusing on local features, thereby strengthening the recognition of the user's gaze area. Through Highway MLP, the output of the high-pass global-local information aggregation module will be obtained:

[0062] H(F)=T(F)·ReLU(W·F+b)+(1-T(F))·F

[0063] Dynamic mask fusion module is used to adaptively integrate the features output by the enhanced long-term memory encoder The high-pass global-local information aggregation module processes features H(F) to adaptively select areas related to the user's gaze and filter out background noise. Its attention weights generate a mask matrix:

[0064]

[0065] The mask M performs element-by-element weighted fusion of the two feature matrices, and finally obtains the adaptive fusion feature Z fused :

[0066] Z fused =M⊙Z mem +(1-M)⊙H(F)

[0067] By generating random masks, the model can dynamically adjust the region selection, highlight important visual features, and ensure the focus on key areas in different dynamic scenes. This solves the problem that traditional static attention cannot flexibly cope with rapidly changing backgrounds. The dynamic mask mechanism of this module is adaptively adjusted according to scene changes, allowing the model to more accurately capture the user's attention area in high-dynamic videos; the feature Z after mask fusion is fused Decode to generate the final gaze spectrum prediction. The decoder processes spatiotemporal information based on Transformer through a multi-head self-attention mechanism combined with residual connections and layer normalization. With reference to the multi-layer self-attention mechanism, the decoder ensures that the multi-scale spatiotemporal information of the fused features is fully utilized:

[0068]

[0069] Features output by the final decoder layer Generate the prediction result y of the gaze point through linear mapping t , the user gaze spectrum with high precision and time continuity effectively improves the responsiveness of the model at different time steps:

[0070]

Claims

1. A method for estimating gaze spectrum based on first-person perspective high dynamic long video, the method comprising: Visual token embedding, enhanced long-term memory encoder, high-pass global-local information aggregation module, dynamic mask fusion module, decoder; Step 1: The specific method of visual token embedding is: divide the input video sequence X into non-overlapping small blocks s T ×s H ×s W , the video sequence includes T time periods with a dimension of H×W; then, each small block is mapped to a D-dimensional vector space through a linear mapping function; through the above operations, local visual tokens; add learnable position embeddings to the generated local visual tokens To enhance the spatiotemporal location information of each local visual token: L i =f(X i )+E i Among them, L i is the i-th local visual token, f(X i ) is the feature extracted from the i-th small block, E i represents the i-th position embedding; The global visual token is generated by downsampling the local visual token, using maximum pooling on the local visual token to compress the local features into global features; a series of convolutional layers are used to downsample the local visual token to gradually generate the global visual token G: G = Conv(MaxPool(L)) L represents the global view token; Step 2: The enhanced long-term memory mechanism used combines the local and global visual tokens generated from the visual token embedding, adopts a multi-layer Transformer architecture, and introduces a memory mechanism to handle the complex temporal dependencies in long-duration high-dynamic videos; this mechanism can capture the user's gaze spectrum by effectively storing and processing historical information; Step 3: The Qualcomm global-local information aggregation module is designed to consist of two parts: global-local correlation attention and Highway MLP. This module selectively enhances the part that helps to predict the gaze spectrum by calculating the correlation weight between local features and global features, while suppressing background noise. With global features The correlation between local features and global features is selectively integrated; Step 4: The specific method of the dynamic mask fusion module is: The dynamic mask fusion module is used to adaptively integrate the output features Z of the enhanced long-term memory encoder. mem And the processing features H(F) of the Qualcomm global-local information aggregation module, so as to effectively focus on the most relevant visual information in complex dynamic scenes; the mask matrix is ​​generated by calculating the attention weights: Among them, W Q and W K is a learnable parameter matrix used to project the input into the same attention space; the Sigmoid activation function ensures that the value range of the mask is between (0, 1); the mask M performs element-by-element weighted fusion of the two feature matrices, and finally obtains the adaptive fusion feature Z fused : Z fused =M⊙Z mem +(1-M)⊙H(F) Among them, ⊙ represents element-by-element multiplication, and the mask M determines the output Z from the enhanced long-term memory encoder in the fused feature mem The weight of feature H(F) is processed with the high-pass global-local information aggregation module; Step 5: Feature Z after dynamic mask fusion module fused Input into the Transformer decoder to generate the final user gaze point prediction; Step 6: Features of the final decoder layer output Generate the prediction result y of the gaze point through linear mapping t .

2. The method for estimating gaze spectrum based on first-person perspective high dynamic long video according to claim 1, characterized in that: The specific method of step 2 is: The specific method of the enhanced long-term memory encoder is: the local and global tokens L at each time step t t and G t Will be input into the enhanced long-term memory encoder; in order to process the timing information, the query Q, key K and value V are used to capture the dependency between the current time step and the historical time step: Among them, W Q , W K , W V They are the query, key, and value mapping matrices for local and global features, respectively. The superscripts L and G represent local and global; In order to handle long-term dependencies, an enhanced long-term memory mechanism is used, which stores the key and value information of multiple previous time steps; specifically, a memory bank is introduced, which stores the local and global features of M past time steps, forming local memory and global memory respectively: Represents the local key information memory, Represents the local value information memory, Represents the global key information memory, Represents the global value information memory; A multi-head self-attention mechanism is used to make the local and global query Q t The key information memory K of the previous time step mem Compare and calculate the attention weight based on the similarity, so as to dynamically focus on important historical information; the formula of the self-attention mechanism is as follows: V mem Represents a memory bank of value information; Perform self-attention calculations on local and global features respectively to obtain local features and global features In order to enhance the capture of multi-level dependencies in long-term videos, a hierarchical memory mechanism is introduced in each Transformer layer of the enhanced long-term memory encoder; at each layer, the output features not only depend on the information of the current time step, but also refer to the memory state of the previous layer, ensuring the cumulative processing of temporal information.

3. The method for estimating gaze spectrum based on first-person perspective high dynamic long video according to claim 1, characterized in that: The specific method of step 3 is: The specific method of the Qualcomm global-local information aggregation module is as follows: first, and Perform linear mapping to generate query Q, key K and value V matrices: The inner product of the query matrix Q and the key matrix K is used to calculate the correlation weights between local and global features. In order to enhance the selectivity of the model, the suppression matrix S is subtracted to reduce the mutual influence between local features and emphasize the direct correlation between local and global features. The correlation between each local feature and the global feature, that is, the correlation weight matrix, is used to weight the global feature V to generate an enhanced correlation weight matrix: Each local feature will obtain corresponding information from the global feature according to its correlation with the global feature; the enhanced local feature With the original local features Perform weighted fusion to obtain the final global-local fusion feature F: Among them, α and β are weighting coefficients, which are used to adjust the contribution ratio of the original local features and the enhanced local features; Highway MLP is used to control the information flow and fusion between global and local visual tokens, ensuring that local features retain the original information while achieving feature transformation; H(F)=T(F)·ReLU(W·F+b)+(1-T(F))·F Among them, T(F) represents the transformation gate, which determines how much part of the input needs to be nonlinearly transformed, and W and b are the parameters of the gating function.

4. The method for estimating gaze spectrum based on first-person perspective high dynamic long video according to claim 3, characterized in that: In step 3, T(F) is the Sigmoid function.

Images