A gaze spectrum prediction method based on a first-view high dynamic long video

Abstract

The application discloses a gaze spectrum estimation method based on a first-view high dynamic long video and belongs to the field of image processing. The gaze spectrum prediction method of the application effectively solves the key problems existing in the prior art, such as time sequence information loss in a long video, insufficient fusion of local and global information, and difficulty of a static mask in adapting to dynamic scene changes. An enhanced long-time memory encoder effectively encodes a long video through multi-scale attention and hierarchical memory mechanism, solves the problems of information loss and redundancy under long-time dependence, and ensures the integrity of time sequence information. A high-pass global-local information aggregation module combines a global background and local details in a multi-layer network through the design of a cross-layer dynamic information transmission channel, and enhances the gaze spectrum prediction capability in a complex dynamic scene. A dynamic mask fusion module adopts an adaptive mechanism, can adjust a mask weight in real time, solves the problem that a static attention cannot cope with a rapidly changing background, and improves the flexibility and accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, in particular, to the analysis and understanding of high dynamic and long time first-person video in real scenes, which can effectively capture the gaze spectrum changes of users in complex dynamic scenes. BACKGROUND

[0002] In recent years, with the rapid development of augmented reality, virtual reality and other technologies, gaze spectrum prediction technology based on first-person video is increasingly widely used in intelligent driving and human-computer interaction fields. This technology can accurately capture the cognitive process and behavior intention of users by analyzing their visual focus, significantly improving the interaction performance of the system. However, as the complexity of video application scenarios increases, existing gaze spectrum prediction methods have many limitations in processing long time, high dynamic, and complex background videos. Especially in scenes with dramatic changes, the user's gaze point frequently moves, making it difficult for existing models to balance the accuracy and real-time performance of prediction. Therefore, in complex dynamic scenes, how to achieve efficient gaze spectrum prediction has become a technical problem to be solved in the field.

[0003] The core challenge of the above problem lies in the processing of long time video and high dynamic scene. First, long time video processing poses significant challenges. Long time video often contains a large amount of temporal information, and the user's visual gaze spectrum has strong dynamics in these temporal information. However, a large amount of irrelevant information and redundant features can easily obscure the key visual cues related to the user's gaze spectrum, making it difficult for the model to effectively extract key information and thus affecting the accuracy of the prediction. Second, in complex dynamic scenes, the user's hands and objects frequently interact, and these local details are crucial for gaze spectrum prediction, but they are highly coupled with global background information, and how to effectively integrate global and local information is a major problem for existing models. Although the current convolutional neural network and recurrent neural network model can process spatio-temporal information to some extent, its ability to extract local features and model global background is insufficient when faced with long temporal video and complex scenes, often leading to an inability to accurately depict the dynamic changes of the gaze spectrum. In addition, static attention mechanisms are difficult to adapt to rapidly changing scene backgrounds, further limiting the performance of existing methods in dynamic scenes. In this case, static attention not only cannot dynamically adjust the visual weight, but also may cause the model to ignore key visual information, reducing the overall prediction effect. Therefore, the existing gaze spectrum prediction technology faces serious performance bottlenecks when dealing with complex backgrounds and long dynamic scenes. SUMMARY

[0004] The gaze spectrum prediction method of the present application effectively solves the key problems existing in the prior art, such as the loss of time sequence information in long-time video, the insufficient fusion of local and global information, and the difficulty of static mask to adapt to dynamic scene changes. The enhanced long-time memory encoder effectively encodes the long-time video through multi-scale attention and hierarchical memory mechanism, solves the problems of information loss and redundancy under long-time dependence, and ensures the integrity of the time sequence information; the high-pass global-local information aggregation module designs a cross-layer dynamic information transmission channel, effectively combines the global background and local details in the multi-layer network, and enhances the gaze spectrum prediction ability in complex dynamic scenes; the dynamic mask fusion module adopts an adaptive mechanism to adjust the mask weight in real time, solves the problem that the static attention cannot cope with the rapidly changing background, and improves the flexibility and accuracy of the model.

[0005] To solve these challenges, the present application proposes a gaze spectrum prediction method based on first-view high-dynamic long video; this method aims to improve the accuracy and real-time performance of gaze spectrum prediction by accurately capturing the user's visual gaze spectrum in long-time dynamic scenes; the present application achieves these goals through three key technical modules, including an enhanced long-time memory encoder, a high-pass global-local information aggregation module, and a dynamic mask fusion module. These modules respectively deal with various challenges in long-time video and complex scenes from different aspects, and through the synergistic effect between modules, more efficient gaze spectrum prediction is achieved; by designing an enhanced long-time memory encoder, the model can effectively retain the key features in long-time video, reduce irrelevant information interference, and improve information extraction efficiency; the high-pass global-local information aggregation module ensures the efficient fusion of local hand movement and object interaction information with global scene information, thereby accurately predicting the gaze spectrum in complex dynamic scenes. In addition, the dynamic mask fusion technology can adaptively adjust the mask to respond to changes in the scene in real time, so that the model can still accurately extract key information in a rapidly changing background. Through these innovative designs, the present application significantly improves the prediction accuracy and real-time processing capability in complex scenes, providing an effective technical solution for gaze spectrum prediction based on first-view video, and has wide application prospects.

[0006] The technical scheme of the present application is a gaze spectrum estimation method based on first-view high-dynamic long video, which comprises: visual token embedding, enhanced long-time memory encoder, high-pass global-local information aggregation module, dynamic mask fusion module, and decoder; step 1: given a fixed long-time period T of self-centered input video clip segment with spatial dimension HxW, the gaze prediction is regarded as a probability distribution on a two-dimensional image plane;

[0007] The specific method of the visual token embedding is: dividing the input video sequence X into non-overlapping small blocks s T xsH ×s W , the video sequence includes T time periods, and the dimension is HxW; then, each small block is mapped into a D-dimensional vector space through a linear mapping function; through the above operation, a plurality of local visual tokens are generated ; in the generated local visual tokens, a learnable position embedding is added to enhance the spatiotemporal position information of each local visual token:

[0008] L i =f(X i )+E i

[0009] wherein, L i is the i-th local visual token, f(X i ) is the feature extracted from the i-th small block, and E i represents the i-th position embedding; in order to further include global scene information in the visual token, a global visual token embedding based on convolution operation is designed; the global visual token is generated by downsampling the local visual token, and the maximum pooling is used on the local visual token to compress the local feature into a global feature; a series of convolution layers are used to downsample the local visual token, and gradually generate the global visual token G:

[0010] G=Conv(MaxPool(L))

[0011] L represents the global visual token;

[0012] Step 2: The enhanced long-term memory mechanism is used to combine the local and global visual tokens generated from the visual token embedding, adopts a multi-layer Transformer architecture, and introduces a memory mechanism to process the complex temporal dependence in the long-time high-dynamic video; this mechanism can capture the user's gaze spectrum by effectively storing and processing historical information;

[0013] Step 3: The high-pass global-local information aggregation module is designed by two parts of global-local correlation attention and Highway MLP; the design of the global-local correlation attention module aims to capture the correlation between local visual features and global scene information in the video, especially in complex dynamic scenes, by strengthening the information interaction between local and global, more accurately predict the user's gaze spectrum; this module calculates the correlation weight between local features and global features, selectively enhances the part that helps to predict the gaze spectrum, and suppresses the background noise; by calculating the correlation between local features and global features , the local features are selectively fused with the global features;

[0014] Step 4: The specific method of the dynamic mask fusion module is:

[0015] The features output by the enhanced long-term memory encoder are characterized by where N represents the number of time steps, and D represents the feature dimension. The dynamic mask fusion module is used to adaptively integrate the output features Z of the enhanced long-term memory encoder mem and the processing features H(F) of the high-pass global-local information aggregation module, so as to effectively focus on the most relevant visual information in a complex dynamic scene; by calculating the attention weight, a mask matrix is generated:

[0016]

[0017] where W Q and W K are learnable parameter matrices for projecting the input into the same attention space; the sigmoid activation function ensures that the mask value ranges between (0, 1); the mask M performs element-wise weighted fusion on the two feature matrices, and finally obtains the adaptively fused feature Z fused :

[0018] Z fused = M⊙Z mem +(1-M)⊙H(F)

[0019] where ⊙ represents element-wise multiplication, and the mask M determines the weight of the output Z mem from the enhanced long-term memory encoder in the fused feature and the processing feature H(F) of the high-pass global-local information aggregation module;

[0020] Step 5: The feature Z fused after the dynamic mask fusion module is input into the Transformer decoder to generate the final user gaze point prediction; the design of the decoder refers to the multi-layer self-attention mechanism, ensuring that the multi-scale spatio-temporal information of the fused feature is fully utilized:

[0021]

[0022]

[0023] Step 6: The feature output by the final decoder layer is linearly mapped to generate the prediction result y t of the gaze point.

[0024] Further, the specific method of the second step is:

[0025] The specific method of the enhanced long-term memory encoder is: the local and global tokens L t and G twill be input into the enhanced long-term memory encoder; in order to process the time sequence information, the dependencies between the current time step and the historical time step are captured by using the way of query Q, key K and value V:

[0026]

[0027] Wherein, W Q , W K , W V are the query, key and value mapping matrices for local and global features respectively, and the superscripts L, G represent local and global;

[0028] In order to process long-time dependencies, the enhanced long-term memory mechanism stores the key and value information of multiple previous time steps; Specifically, a memory bank is introduced, which stores the local and global features of M past time steps, respectively constituting local memory and global memory:

[0029]

[0030] The local key information memory bank is represented by K The local value information memory bank is represented by V The global key information memory bank is represented by K The global value information memory bank is represented by V

[0031] The multi-head self-attention mechanism is adopted, which makes the local and global queries Q t of the current time step compare with the key information memory bank K mem of the previous time step, and calculates the attention weight according to the similarity, so as to dynamically focus on important historical information; The formula of self-attention mechanism is as follows:

[0032]

[0033] V mem represents the value information memory bank;

[0034] Self-attention calculation is performed on the local and global features respectively to obtain the local feature and the global feature In order to enhance the capture of multi-level dependencies in long videos, a hierarchical memory mechanism is introduced in each layer of the enhanced long-term memory encoder; In each layer, the output feature not only depends on the information of the current time step, but also refers to the memory state of the previous layer, ensuring the cumulative processing of time sequence information.

[0035] Further, the specific method of step 3 is:

[0036] The specific method of the high-pass global-local information aggregation module is: first, and Linear mapping is performed to generate query matrix Q, key matrix K and value matrix V:

[0037]

[0038] The relevance weight between local and global features is calculated using the inner product of query matrix Q and key matrix K; in order to enhance the selectivity of the model, the suppression matrix S is subtracted, which is used to reduce the mutual influence between local features and emphasize the direct association between local and global; using the association between each local feature and global feature, that is, the relevance weight matrix, the global feature V is weighted to generate an enhanced relevance weight matrix:

[0039]

[0040] Each local feature will obtain the corresponding information from the global feature according to its association degree with the global feature; the enhanced local feature is weighted and fused with the original local feature to obtain the final global-local fusion feature F:

[0041]

[0042] wherein, α and β are weighting coefficients, used to adjust the contribution proportion of the original local feature and the enhanced local feature;

[0043] Highway MLP is used to control the information flow and fusion between global and local visual tokens, ensuring that the local features retain the original information while realizing feature transformation; the core idea of Highway MLP is to allow part of the input to pass directly without nonlinear transformation through the design of "gate" mechanism, thereby retaining key information:

[0044] H(F) = T(F) · ReLU(W · F + b) + (1 - T(F)) · F

[0045] wherein, T(F) represents the transformation gate, which determines how much of the input needs to be nonlinearly transformed, and W and b are parameters of the gating function. Through the above design, Highway MLP can effectively interact between global and local information, ensuring efficient fusion of the two.

[0046] Further, T(F) in step 3 is a Sigmoid function.

[0047] The gaze spectrum prediction method based on the first view angle high dynamic long video is specially used for user gaze prediction in long time and high dynamic video scene, can effectively cope with user visual focus prediction in complex scene, and is suitable for various applications such as augmented reality, virtual reality and intelligent driving. The main innovation of the application lies in the careful modification of the transformer-based codec architecture, and some uniquely designed modules are added. Through three major innovative modules, the key challenges in long time video processing are effectively coped with, including the preservation of time sequence information, the efficient fusion of global and local information, and the mask adjustment ability adapting to dynamic scene changes. First, the enhanced long-time memory encoder encodes the long-time dependent information through multi-scale attention and hierarchical memory mechanism, reduces information redundancy and avoids time sequence loss, and guarantees the integrity of time sequence features; second, the designed high-pass global-local information aggregation module introduces a cross-layer dynamic information transmission channel, so that the global background and local detail information can fully interact in the multi-layer network, and the prediction effect in the complex dynamic scene is improved; finally, the dynamic mask fusion module adopts an adaptive mechanism, which can adjust the mask weight in real time, flexibly adapt to the rapid change of the background, and enhance the environmental adaptability and prediction accuracy of the model. The F1 index of the application on the public dataset egteagaze+ is improved by 2.06% compared with the latest method GLC, and the F1 index represents the harmonic mean of accuracy and recall. This method comprehensively improves the application performance of gaze spectrum prediction in long time and high dynamic scene. The latest method GLC is Bolin Lai, Miao Liu, Fiona Ryan, James M. Rehg, In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation and Beyond. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the application. DETAILED DESCRIPTION

[0049] To solve the key challenges in long-time video processing in open-world first-person videos, including the preservation of temporal information, the efficient fusion of global and local information, and the ability to adapt to dynamic scene changes, a learning framework for predicting user gaze spectra is created. As a challenging task, the invention needs to meet the needs of temporal information preservation, efficient fusion of local and global information, and adaptive dynamic background changes in long-time, high-dynamic videos, ensuring the coherence and accuracy of gaze spectrum prediction. And accurate gaze spectrum prediction is combined with long-time span context information. This method is through three designs of enhanced long-time memory encoder, high-pass global-local information aggregation and dynamic mask fusion, to extract and fuse the significant local detail information of high-dynamic hand and various different object interactions in long-time complex background scenes and the macroscopic scene global information, so that the model can extract the significant feature information in the video under different degrees of dynamic scene in the first-person long-time span to generate human gaze eye movement spectrum consistent with cognitive science and life science.

[0050] The main framework of the gaze spectrum prediction network relies on an encoder-decoder. For a self-centered input video, given a fixed long-time period T clip segment with spatial dimension HxW, the input video sequence X is divided into non-overlapping small blocks s T xs H xs W Then, each small block is mapped to a D-dimensional vector space through a linear mapping function, obtaining local visual tokens. In the generated local visual tokens, a learnable position embedding is added to enhance the spatiotemporal position information of each local visual token, obtaining local visual tokens L i Global visual tokens are generated by downsampling local visual tokens. Max pooling is used on local visual tokens to compress local features into global features; a series of convolutional layers are used to downsample local visual tokens, gradually generating global visual tokens G.

[0051] In order to make the network able to preserve the timing information in a long time span, avoid the problem of information loss or key feature dilution under multi-frame dependence. Especially for dynamic scenes in video, the network needs to have the ability to keep key information under long-time dependence to ensure the continuity and stability of gaze spectrum prediction. An enhanced long-term memory encoder is designed to combine local and global visual tokens generated from visual token embeddings. A multi-layer Transformer architecture is adopted to handle complex temporal dependencies in long-time high-dynamic videos by introducing a memory mechanism. In order to enhance the capture of multi-level dependencies in long-time videos, a hierarchical memory mechanism is introduced in each layer of the enhanced long-term memory encoder. At each layer, the output features not only depend on the information of the current time step, but also refer to the memory state of the previous layer, ensuring the cumulative processing of timing information. This mechanism can capture the user's gaze spectrum by effectively storing and processing historical information. The local and global tokens L t and G t will be input into the enhanced long-term memory encoder. In order to process the timing information, the query Q, key K and value V are used to capture the dependence between the current time step and the historical time step:

[0052]

[0053] The memory bank is used to process long-time dependence, which stores the key and value information of the previous multiple time steps. Specifically, the memory bank stores the local and global features of M past time steps, respectively constituting local memory and global memory:

[0054]

[0055] The multi-head self-attention mechanism is adopted to make the local and global queries Q t of the current time step compare with the memory bank K mem of the previous time step, and calculate the attention weight according to the similarity, so as to dynamically focus on important historical information, and obtain the local feature and the global feature The gaze spectrum prediction needs to pay attention to both local details (such as the interaction of the user's hands or objects) and global background (such as the layout of the scene), and the network needs to have the ability to fuse these different scale information. This fusion not only needs to maintain the relevance of details and the whole, but also needs to be transmitted in multiple levels, to ensure that the model can balance the influence of local and global information on prediction. The gaze spectrum prediction needs to pay attention to both local details (such as the interaction of the user's hands or objects) and global background (such as the layout of the scene), and the network needs to have the ability to fuse these different scale information. This fusion not only needs to maintain the relevance of details and the whole, but also needs to be transmitted in multiple levels, to ensure that the model can balance the influence of local and global information on prediction. Therefore, a high-pass global-local information aggregation module is designed to capture the correlation between local visual features and global scene information in the video. The module is designed by two parts of global-local correlation attention and Highway MLP, by calculating the correlation weight between local features and global features, selectively enhancing the part that helps gaze spectrum prediction, while suppressing background noise. By calculating the correlation between local features and global features , the local features are selectively fused with the global features:

[0056]

[0057] The inner product of the query matrix Q and the key matrix K is used to calculate the correlation weight between the local and global features, that is, the correlation weight matrix is weighted to the global features V, to generate the enhanced correlation weight matrix:

[0058]

[0059] The enhanced local features are weighted and fused with the original local features to obtain the final global-local fusion features F:

[0060]

[0061] The local and global information is efficiently fused, and the Highway MLP gating mechanism is used to ensure that the local details and global scene information complement each other. The Highway MLP is used to control the information flow and fusion between global and local visual tokens, so as to ensure that the local features can retain the original information while realizing feature transformation. The core idea of Highway MLP is to allow part of the input to pass directly without nonlinear transformation through the design of the "gate" mechanism, so as to retain key information. The addition of Highway MLP effectively enhances the capture of local and global relevance, so that the model can integrate global context when focusing on local features, and strengthens the recognition of the user's gaze area. Through the Highway MLP, the output of the high-pass global-local information aggregation module is obtained:

[0062] H(F) = T(F) · ReLU(W · F + b) + (1 - T(F)) · F

[0063] The dynamic mask fusion module is used to adaptively integrate the features output by the enhanced long-term memory encoder The processing feature H(F) of the high-pass global-local information aggregation module is used to adaptively select the area related to the user's gaze and filter out background noise. The attention weight generates a mask matrix:

[0064]

[0065] The mask M performs element-wise weighted fusion on the two feature matrices, and finally obtains the adaptively fused feature Z fused :

[0066] Z fused = M O Z mem + (1 - M) O H(F)

[0067] By generating a random mask, the model can dynamically adjust the region selection, highlight important visual features, and ensure the focus on key areas in different dynamic scenes. Thus, the traditional static attention cannot flexibly cope with the rapid changes in the background. The dynamic mask mechanism of this module adaptively adjusts according to the scene changes, so that the model can more accurately capture the user's attention area in high dynamic video; the feature Z fused after mask fusion is decoded to generate the final gaze spectrum prediction. The decoder is based on Transformer and uses multi-head self-attention mechanism combined with residual connection and layer normalization to process spatio-temporal information. Referring to the multi-layer self-attention mechanism, the decoder ensures that the multi-scale spatio-temporal information of the fused features is fully utilized:

[0068]

[0069] The feature output by the final decoder layer A prediction result y of the gaze point is generated through a linear mapping t The user gaze spectrum with high precision and time continuity effectively improves the response ability of the model at different time steps:

[0070]

Claims

1. A method for estimating gaze spectrum based on first-person perspective high dynamic range long video, the method comprising: Visual token embedding, enhanced long-term memory encoder, high-pass global-local information aggregation module, dynamic mask fusion module, decoder; Step 1: The specific method of visual token embedding is: divide the input video sequence X into non-overlapping small blocks s T ×s H ×s W , the video sequence consists of T time periods with a dimension of H×W; then, each small block is mapped to a D-dimensional vector space through a linear mapping function; through the above operations, local visual tokens; add learnable position embeddings to the generated local visual tokens To enhance the spatiotemporal location information of each local visual token: L i =f(X i )+E i Among them, L i is the i-th local visual token, f(X i ) is the feature extracted from the i-th small block, E i represents the i-th position embedding; The global visual token is generated by downsampling the local visual token. The maximum pooling is used on the local visual token to compress the local features into global features. The local visual token is downsampled using a series of convolutional layers to gradually generate the global visual token G: G = Conv(MaxPool(L)) L represents the global view token; Step 2: The enhanced long-term memory mechanism used combines local and global visual tokens generated from visual token embeddings, adopts a multi-layer Transformer architecture, and introduces a memory mechanism to handle the complex temporal dependencies in long, high-dynamic-range videos. This mechanism can capture the user's gaze spectrum by effectively storing and processing historical information. Step 3: The high-pass global-local information aggregation module is designed to consist of two parts: global-local correlation attention and Highway MLP. This module selectively enhances the part that helps to predict the gaze spectrum by calculating the correlation weight between local features and global features, while suppressing background noise. With global features Correlation between local features and global features; Step 4: The specific method of the dynamic mask fusion module is: The dynamic mask fusion module is used to adaptively integrate the output features Z of the enhanced long-term memory encoder. mem And the processing features H(F) of the Qualcomm global-local information aggregation module, so as to effectively focus on the most relevant visual information in complex dynamic scenes; the mask matrix is ​​generated by calculating the attention weights: Among them, W Q and W K is a learnable parameter matrix used to project the input into the same attention space; the Sigmoid activation function ensures that the mask value range is between (0, 1); the mask M performs element-by-element weighted fusion of the two feature matrices, and finally obtains the adaptive fusion feature Z fused : Z fused =M⊙Z mem +(1-M)⊙H(F) Among them, ⊙ represents element-by-element multiplication, and the mask M determines the output Z from the enhanced long-term memory encoder in the fusion feature mem The weight of feature H(F) is processed with the high-pass global-local information aggregation module; Step 5: Feature Z after dynamic mask fusion module fused Input to the Transformer decoder to generate the final user gaze point prediction; Step 6: Features output by the final decoder layer Generate the predicted result y of the gaze point through linear mapping t .

2. The method for estimating gaze spectrum based on first-person perspective high dynamic range long video according to claim 1, characterized in that: The specific method of the second step is: The specific method of the enhanced long-term memory encoder is: the local and global tokens L at each time step t t and G t Will be input into the enhanced long-term memory encoder; in order to process the temporal information, the query Q, key K and value V are used to capture the dependency between the current time step and the historical time step: Among them, W Q 、W K 、W V are the query, key, and value mapping matrices for local and global features, respectively. The superscripts L and G represent local and global. To handle long-term dependencies, an enhanced long-term memory mechanism stores key and value information from multiple time steps in the past. Specifically, a memory bank is introduced, which stores local and global features of M past time steps, forming local memory and global memory respectively: Represents the local key information memory, Represents a local value information memory, Represents the global key information memory, Represents the global value information memory; A multi-head self-attention mechanism is used to make the local and global query Q t and the key information memory K of the previous time step mem Compare and calculate the attention weight based on the similarity, so as to dynamically focus on important historical information; the formula of the self-attention mechanism is as follows: V mem Represents a memory bank of value information; Perform self-attention calculation on local and global features respectively to obtain local features and global features To enhance the capture of multi-level dependencies in long videos, a hierarchical memory mechanism is introduced in each Transformer layer of the enhanced long-term memory encoder; at each layer, the output features not only depend on the information of the current time step, but also refer to the memory state of the previous layer, ensuring the cumulative processing of temporal information.

3. The method for estimating gaze spectrum based on first-person perspective high dynamic range long video according to claim 1, characterized in that: The specific method of step 3 is: The specific method of the Qualcomm global-local information aggregation module is as follows: first, and Perform linear mapping to generate query Q, key K and value V matrices: The inner product of the query matrix Q and the key matrix K is used to calculate the correlation weight between local and global features. In order to enhance the selectivity of the model, the suppression matrix S is subtracted to reduce the mutual influence between local features and emphasize the direct correlation between local and global features. The correlation between each local feature and the global feature, that is, the correlation weight matrix, is used to weight the global feature V to generate the enhanced correlation weight matrix: Each local feature will obtain corresponding information from the global feature according to its correlation with the global feature; the enhanced local feature With the original local features Perform weighted fusion to obtain the final global-local fusion feature F: Among them, α and β are weighting coefficients, which are used to adjust the contribution ratio of original local features and enhanced local features; Highway MLP is used to control the information flow and fusion between global and local visual tokens, ensuring that local features retain the original information while achieving feature transformation; H(F)=T(F)·ReLU(W·F+b)+(1-T(F))·F Among them, T(F) represents the transformation gate, which determines how much of the input needs to be nonlinearly transformed, and W and b are the parameters of the gating function.

4. The method for estimating gaze spectrum based on first-person perspective high dynamic range long video according to claim 3, characterized in that: In step 3, T(F) is the Sigmoid function.

Images