Long-time viewport prediction method based on time-aware adaptive attention mechanism

Abstract

The invention discloses a long-time viewport prediction method based on a time perception adaptive attention mechanism, and belongs to the technical field of 360-degree video viewport prediction. The method comprises the following steps: firstly, taking a historical head movement track when a user watches a 360-degree video as input, and extracting video inter-frame space-time characteristics through a convolutional long-short-term memory network; thirdly, a time perception adaptive attention module is introduced, attention weights are dynamically distributed according to the importance of time steps and a time decay rule, and long-time dependent adaptive feature fusion is achieved; and finally, the original multi-head attention is adaptively fused and attention output is enhanced through a dynamic gating mechanism, and the stability of the model under long-time prediction is improved. Experimental results show that the average prediction accuracy in the prediction range from 1s to 5s exceeds 97%, the defect that an existing model is difficult to model a time decay rule and global time sequence dependence in long time sequence prediction is effectively overcome, and the prediction accuracy and the model stability are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of 360-degree video stream viewport prediction technology, specifically involving a long-term viewport prediction method based on a time-aware adaptive attention mechanism. Technical Background

[0002] With the rapid development of virtual reality and panoramic video technologies, 360-degree video has become a core carrier of immersive content. During 360-degree video playback, users control their viewing angle through head movements. The system needs to predict the user's future viewing direction in real time to preload the corresponding area, thereby improving bandwidth utilization, reducing latency, and enhancing video smoothness.

[0003] Most existing viewport prediction methods are based on recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or convolutional long short-term memory networks (ConvLSTMs) to model the user's historical head movements. While these methods can learn certain temporal dependencies, they still have the following problems: traditional LSTM structures have fixed weight distributions in the time dimension, making it impossible to dynamically adjust the importance of different time steps; the model lacks a time decay mechanism, with long-term features excessively influencing the prediction results, leading to error accumulation; the attention mechanism does not fully consider temporal information, making it difficult to model local and global dependencies simultaneously; and the prediction results are highly volatile, resulting in insufficient model stability.

[0004] To address the aforementioned issues, this invention proposes a long-term viewport prediction framework that integrates a time-aware adaptive attention mechanism with a dynamic gating fusion structure. This framework captures long-term temporal dependencies while simultaneously considering global feature modeling and dynamic feature weight adjustment, thereby improving prediction accuracy and system robustness. Summary of the Invention

[0005] This invention proposes a long-term viewport prediction method based on a time-aware adaptive attention mechanism. By introducing time decay modeling, a time weight generation network, and a dynamic gating fusion mechanism, it achieves adaptive weighting of time dependence, self-adjusting selection of attention paths, and overall end-to-end collaborative optimization, thereby improving the stability of viewport prediction in long-term prediction tasks.

[0006] Step 1: Collect head motion trajectory data of users while watching 360-degree videos, smooth, interpolate and normalize the data, and input the preprocessed time series into ConvLSTM to extract spatiotemporal features.

[0007] Step 2: Based on the spatiotemporal features introduced in Step 1, a learnable time decay factor and time position encoding are introduced. An adaptive importance weight is assigned to each time step through a time weight generator to obtain a time-aware weighted feature sequence.

[0008] Step 3: Input the time-weighted features from Step 2 into the multi-head attention module, dynamically allocate attention weights based on temporal importance and temporal dependence, and generate stable enhanced features through normalization and residual connections.

[0009] Step 4: Design a gating network based on global statistical features to generate gating coefficients, and adaptively fuse the original attention output with the time-enhanced attention output to balance short-term and long-term dependencies.

[0010] Step 5: Based on the fused features from Step 4, input the convolutional output network, recover the spatial features through dilated convolution and channel weighting modules, and output the viewport probability distribution for future time periods.

[0011] Step 1 specifically includes the following:

[0012] ConvLSTM is used to jointly model the time series input to simultaneously capture spatial structure and temporal dependency features. At time step... The gating update process can be represented as follows:

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] in, This represents the convolution operation. Represents element-wise product. This represents the Sigmoid function. Indicates input features, In hidden state, This structure serves as a memory state. While preserving local spatial features, it models temporal progressive dependencies, providing a dynamic spatiotemporal representation for attention computation.

[0019] Step 2 specifically includes the following:

[0020] A time-aware adaptive attention mechanism is introduced to adaptively allocate the importance of different time steps. This mechanism consists of three parts: time position encoding, time decay modulation, and time weight generation.

[0021] First, let the output of the convolutional long short-term memory network be... Through learnable position encoding vectors Timing modulation is performed, and a time decay factor is introduced. The time modulation characteristics are obtained:

[0022]

[0023] in, These are learnable parameters used to simulate the decreasing trend of temporal distance's contribution to features.

[0024] Subsequently, the importance weights for each time step are calculated using a time weight generator. This generator consists of two layers of linear transformation and nonlinear activation, which globally aggregates the input to generate scalar weights. :

[0025]

[0026] in, This indicates a global pooling operation. For learnable parameters, Indicates the first Adaptive weights for time steps.

[0027] Finally, temporal features are fused using a time-weighted and multi-head attention mechanism to obtain a time-aware weighted result. Each attention head calculates the relevance distribution among the query, key, and value vectors, achieving global dependency modeling between time steps. The output, after residual connections and layer normalization, yields a stable time-enhanced feature representation.

[0028] Step 4 specifically includes the following:

[0029] A dynamic gating fusion module is introduced to balance the contribution ratio between the original attention output and the time-enhanced attention output, thereby achieving adaptive integration of multi-path features.

[0030] This module first enhances the output time. Perform global statistical aggregation to obtain descriptive features. The fusion coefficients were then calculated using a gating network. :

[0031]

[0032] in A gating factor is used to control the relative proportions of the two attention paths. The final output is:

[0033]

[0034] in This represents the original multi-head attention result. Through a gating adjustment mechanism, the model can adaptively select the information flow path based on the input complexity and the strength of time dependence, thereby balancing short-term response and long-term stability.

[0035] Step 5 specifically includes the following:

[0036] The convolutional output network is used to map the fused global features to a spatial viewport probability distribution over a future time period. This network consists of dilated convolutional layers, deconvolutional layers, and channel-weighted modules.

[0037] In the channel weighting stage, the feature map is first processed. Global average pooling is used to obtain the channel description vector. Subsequently, channel weight vectors are generated through compression and excitation operations. :

[0038]

[0039] And weighting is achieved through channel-by-channel product:

[0040]

[0041] in, This is the weight matrix. This represents the saliency weight of each channel. The weighted features are deconvolved to restore the spatial dimension, generating a viewport probability distribution map for future time steps. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the description are only some embodiments of the present invention.

[0043] Figure 1 This is a step diagram of the present invention. Detailed Implementation

[0044] To clarify the technical problems, technical solutions, implementation processes, and performance demonstrations, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Various exemplary embodiments, features, and aspects of this disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0045] This embodiment proposes a long-term viewport prediction method based on a time-aware adaptive attention mechanism, including the following steps:

[0046] Step 1: By collecting and preprocessing the head motion trajectory of users watching 360-degree videos, a time-series input convolutional long short-term memory network is obtained. Spatial features are extracted and the time dependencies between frames are modeled using recurrent memory units, thereby generating a comprehensive spatiotemporal feature representation.

[0047] Step 2: Based on the spatiotemporal characteristics of Step 1, introduce time position encoding, learnable time decay parameters, and time weight generation modules to achieve adaptive weighting for different time steps. This enables the model to highlight key information and suppress redundant historical features in long-term series, thereby enhancing the model's ability to perceive and model time dependencies.

[0048] Step 3: Input the time-weighted features from Step 2 into the multi-head attention module. The multi-head structure enables parallel modeling of features in different subspaces. Combined with layer normalization and residual connections, a stable enhanced feature representation is obtained, improving the model's ability to express complex spatiotemporal features.

[0049] Step 4: By learning to generate gating coefficients, the original attention output and the time-enhanced attention result are adaptively weighted and fused. The fusion ratio is automatically adjusted according to the complexity of the input features and the strength of temporal dependencies, so as to achieve dynamic balance and adaptive optimization between short-term and long-term dependencies.

[0050] Step 5: Input the spatiotemporal features fused in Step 4 into the decoding convolutional network, combine dilated convolution, deconvolution and channel weighting modules to restore the spatial feature distribution, and output the viewport probability distribution map for future time moments.

[0051] Step 1 specifically includes the following:

[0052] ConvLSTM can capture long-term dependencies in the temporal dimension and preserve spatial distribution features through convolutional structures, achieving joint modeling of temporal and spatial features. The state update at time step k can be represented as:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] in, This represents the convolution operation. Represents element-wise product. This represents the Sigmoid function. By stacking multiple layers of ConvLSTM, a deeper spatiotemporal dependency modeling capability can be obtained, and the final output hidden state sequence serves as the input for subsequent attention.

[0059] Step 2 specifically includes the following:

[0060] Based on the spatiotemporal features output by ConvLSTM, a time-aware adaptive attention mechanism is introduced to describe the dynamic decay features of long-term dependencies. This module consists of three parts: time position encoding, time weight generator, and learnable decay factor.

[0061] First, a learnable location vector is assigned to each time step. And the time modulation features are obtained by superimposing them with the features:

[0062]

[0063] in, is the time decay coefficient, used to describe the feature contribution that decreases as the time interval increases. Then, the time weight generator generates dynamic importance weights based on the multilayer perceptron structure:

[0064]

[0065] in This indicates a global average pooling operation.

[0066] By inputting time-weighted features into the multi-head attention module, the model can enhance key time-slice features and suppress redundant historical information in long-term prediction tasks, thereby significantly improving its ability to model time-series dependencies.

[0067] Step 3 specifically includes the following:

[0068] The time-weighted feature sequence is input into a multi-head self-attention module for global dependency modeling. This module enhances the richness of feature representation by capturing the correlations of different subspaces through multiple attention heads in parallel, including:

[0069] For the Each attention head is used to calculate the query, key, and value vectors respectively:

[0070]

[0071] Then, the attention output is obtained through a scaled dot product attention mechanism:

[0072]

[0073] in, The dimension is the key vector. The results from multiple attention heads are concatenated and then linearly mapped to obtain the final output:

[0074]

[0075] To balance sensitivity to local features, this step also adds a convolutional branch module after multi-head attention, enhancing the spatial neighborhood response through local convolutional operations. Finally, combining layer normalization and residual connections, a stable enhanced feature representation is generated:

[0076]

[0077] This design ensures complete modeling of global dependencies while maintaining the stability and convergence of spatiotemporal feature representations.

[0078] Step 4 specifically includes the following:

[0079] To achieve dynamic information balance among different attention paths, a gated adaptive fusion mechanism is proposed. This mechanism generates gating coefficients based on global statistical features to adaptively control the fusion ratio of the original attention output and temporal enhancement features. This includes:

[0080] First, global average pooling is performed on the time-weighted features to obtain the description vector:

[0081]

[0082] The fusion weights are obtained by passing through a linear layer and a Sigmoid activation function:

[0083]

[0084] The final fusion feature is represented as follows:

[0085]

[0086] in, This is the original multi-head attention output. Through learnable gating weights, the model can dynamically adjust the contributions of the two attention branches based on temporal complexity and feature relevance, achieving synergistic optimization of short-term and long-term dependencies.

[0087] Step 5 includes the following:

[0088] The SE module obtains the channel description vectors through global average pooling. Channel weights are generated through two layers of fully connected transformation and nonlinear mapping. :

[0089]

[0090] Finally, perform channel recalibration:

[0091]

[0092] After upsampling and convolution transpose operations, the predicted heatmap is output. This represents the probability distribution of the user's gaze area within a future time window. This output can be used for bitrate allocation and bandwidth optimization of subsequent 360-degree video tiles, thereby achieving efficient transmission of the video stream and seamless playback.

[0093] In summary, this invention proposes a long-term viewport prediction method based on a time-aware adaptive attention mechanism. This method achieves an adaptive balance between long-term dependencies and short-term changes by introducing time decay modeling and a gating dynamic fusion strategy. Experimental results show that, under a 3-second prediction scenario training, the model achieves accuracies of 97.55%, 97.35%, 97.0%, 97.2%, and 97.7% in 1-second, 2-second, 3-second, 4-second, and 5-second prediction tasks, respectively, with corresponding F1 scores of 0.841, 0.829, 0.804, 0.819, and 0.851. This demonstrates that the proposed time-aware adaptive attention mechanism effectively improves the model's prediction accuracy and generalization ability over long time scales, exhibiting good robustness and practical application value.

[0094] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still covered by the invention.

Claims

1. A long-term viewport prediction method based on a time-aware adaptive attention mechanism, characterized in that, The specific steps include the following: Step 1: Obtain the input spatiotemporal feature sequence and extract the spatial feature representation of each time step through a convolutional long short-term memory network.

2. Step 2: Introduce a learnable time decay factor and time position encoding into the extracted time-series features, and assign adaptive importance weights to each time step through a time weight generator, thereby forming time-aware weighted features.

3. Step 3: Input the time-aware weighted features into the time-aware adaptive attention mechanism, dynamically calculate the multi-head attention weights based on the importance of time steps and temporal dependencies, and obtain the time-sensitive attention output. After layer normalization and residual connection, a stable enhanced feature representation is formed.

4. Step 4: Through a dynamic gating fusion mechanism, gating coefficients are generated based on global statistical features to achieve adaptive weighted fusion between the original attention output and the enhanced temporal attention output, so as to balance short-term and long-term dependency features.

5. Step 5: Input the fused features into the prediction network, and after convolution transformation and channel weighting, obtain the final spatiotemporal prediction result.

6. The long-term viewport prediction method based on a time-aware adaptive attention mechanism as described in claim 1, characterized in that, The time-aware adaptive attention mechanism includes a time encoding module, a time weight generation module, and a multi-head attention calculation module. The time encoding module generates a learnable position vector for each time step, the time weight generation module dynamically generates time importance weights based on input features, and the multi-head attention calculation module calculates the attention distribution based on time-weighted features to achieve time-dependent modeling.

7. The long-term viewport prediction method based on a time-aware adaptive attention mechanism as described in claim 1, characterized in that, Linear layers, layer normalization, nonlinear activation layers, and the Sigmoid function are used to generate time weight coefficients in the range (0, 1) for each time step. These coefficients work together with the time decay weights to achieve two-layer adaptive modeling of time dependence.

8. The long-term viewport prediction method based on a time-aware adaptive attention mechanism as described in claim 1, characterized in that, The dynamic gating fusion mechanism includes a linear layer and local feature mean generation gating coefficients. Attention output fusion is achieved through the following formula:

9. Among them, For time-aware attention output, For the original attention output, This represents the attention representation after fusion.

10. The long-term viewport prediction method based on a time-aware adaptive attention mechanism as described in claim 1, characterized in that, The time-aware adaptive attention module and the dynamic gating fusion mechanism are jointly trained in an end-to-end manner. During the training process, the parameters of the time decay factor, the time weight generator, and the gating network are optimized simultaneously, thereby improving the model's ability to dynamically learn the importance and dependencies of temporal features.

Images