Long-time viewport prediction method based on spatial deformation attention network

Abstract

The invention provides a long-time viewport prediction method based on a space deformation attention network, and belongs to the technical field of 360-degree panoramic video viewport prediction. The method comprises the following steps: firstly, modeling a historical head movement track of a user by using a convolutional long-short-term memory network, and extracting basic spatio-temporal characteristics; then introducing a spatial deformation attention network, dynamically adjusting a convolution sampling position through a biased field prediction module, and adaptively compensating spatial distortion generated by isometric columnar projection through deformable sampling; meanwhile, in combination with a space and channel dual attention gating mechanism, performing multi-dimensional significance enhancement on the deformed feature map; and finally, generating a probability distribution diagram of a future viewport through feature fusion and edge enhancement output network. Experimental results show that the average prediction accuracy of the method exceeds 97% within the range of 1s to 5s, the limitation that a traditional fixed convolution kernel is difficult to adapt to the non-uniform distortion of the panoramic image is effectively broken through, and the perception ability of the model to the change of the space structure and the stability of long-term prediction are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of 360-degree video stream viewport prediction technology, specifically involving a long-term viewport prediction method based on spatial deformation attention network, which is used to optimize bandwidth allocation and viewport prediction accuracy in 360-degree video transmission. Technical Background

[0002] With the deep integration of virtual reality technology and 5G networks, 360-degree panoramic video has become the mainstream form of immersive media consumption. To provide a high-quality viewing experience with limited bandwidth, viewport-adaptive streaming technology is widely adopted. The core of this technology lies in the system's ability to accurately predict the user's head movement trajectory over a future period, allowing for high-bitrate loading of video tiles for the predicted area in advance, while simultaneously loading or omitting the background area at a low bitrate.

[0003] Existing viewport prediction methods primarily rely on deep learning models, such as convolutional neural networks (CNNs) or convolutional long short-term memory networks (ConvLSTMs), to extract spatiotemporal features of the user's historical trajectory. However, these methods face two main challenges when processing 360-degree video:

[0004] First, there's the issue of geometric distortion. 360-degree videos are typically stored in an equidistant cylindrical projection format. When this projection method unfolds a sphere into a plane, it produces severe nonlinear stretching distortion in high-latitude regions (near the poles). Traditional standard convolutional kernels use fixed geometric structures for sampling and cannot adapt to this spatial deformation, leading to distorted feature extraction.

[0005] Second, dynamic uncertainty in long-term prediction. In long-term prediction tasks, the movement of the user's viewport is often accompanied by complex non-rigid changes. Convolutional operations with a fixed receptive field struggle to accurately capture these rapidly moving or shape-changing target features, causing the prediction results to diverge rapidly over time, resulting in a significant drop in accuracy.

[0006] To address the aforementioned issues, this invention proposes a long-term viewport prediction method based on a spatially deformable attention network. By introducing offset field prediction and deformable sampling mechanisms, the fixed grid limitation is broken. Combined with dual attention gating and edge enhancement techniques, the model's adaptability to distortion features and prediction stability are significantly improved. Summary of the Invention

[0007] This invention proposes a long-term viewport prediction method based on a spatial deformation attention network. By constructing a network architecture that includes offset field regression, geometric deformation sampling, spatial-channel joint filtering, and edge sharpening, it solves the feature alignment problem caused by projection distortion in panoramic video and achieves high-precision long-term viewport prediction.

[0008] The method of the present invention includes the following five specific steps:

[0009] Step 1: Collect the user's head motion sequence and input it into ConvLSTM. Extract the cross-frame spatiotemporal dependencies through its convolutional recursive structure to form the basic representation for subsequent geometric deformation modeling.

[0010] Step 2: Input the initial features extracted in Step 1 into the offset prediction module, and use local texture and structural information to regress the sampling displacement vector to generate a dynamic offset field that reflects the panoramic projection distortion pattern.

[0011] Step 3: Based on the dynamic offset field generated in Step 2, adjust the convolution sampling position, construct geometrically adaptive features through deformable convolution and bilinear resampling, and use spatial saliency weighting to highlight key viewport-related regions.

[0012] Step 4: Apply channel attention to the deformed features to enhance the semantically sensitive dimension, and perform residual fusion with the initial features to achieve a more stable and robust spatial representation.

[0013] Step 5: Input the fused features into the decoding network, generate a high-resolution viewport heatmap through stepwise upsampling and boundary enhancement, and finally obtain the viewport probability distribution for future time periods.

[0014] Step 1 specifically includes the following:

[0015] Collect user's head motion sequence The input is then fed into a ConvLSTM to extract local spatiotemporal patterns across frames. At each time step t, the input tensor is... Hidden state is The memory state is The gating update process can be represented as:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] in, This represents the convolution operation. Represents element-wise product. This represents the Sigmoid function. This hidden state... Maintained The two-dimensional spatial structure provides a basic feature map for subsequent spatial deformation operations.

[0023] Step 2 specifically includes the following:

[0024] The extracted initial features The input offset prediction module regresses the two-dimensional displacement vector at each sampling location through a local convolutional structure. The offset prediction network can be represented as:

[0025]

[0026] in It is a non-linear activation. This represents the planar offset at each convolution kernel position. This represents the number of sampling points for the convolution kernel. These are the learnable convolution kernel parameters.

[0027] Offset field It is used to describe the distortion rules of the mapping from a sphere to a plane in panoramic projection, so that the convolution operation can follow the dynamic changes of the geometric structure.

[0028] Step 3 specifically includes the following:

[0029] According to the offset field Adjust the convolution sampling coordinates; the new sampling position is:

[0030]

[0031] in These are the sampling points for the standard convolution kernel. The feature map is then resampled using bilinear interpolation.

[0032]

[0033] in These are bilinear weights. The deformable convolution output is:

[0034]

[0035] This operation enables the model to adapt to the uneven spatial stretching and bending characteristics of panoramic images.

[0036] After geometric deformation, for features Apply spatial significance weights Its generation method is as follows:

[0037]

[0038] in If it is Sigmoid, the weight values ​​will be in the range [0,1].

[0039] The final significance enhancement feature is:

[0040]

[0041] This step emphasizes key spatial locations relevant to the viewport area and suppresses background interference.

[0042] Step 4 specifically includes the following:

[0043] In deformation features A channel attention mechanism is applied to enhance semantic sensitivity. First, global pooling is performed on each channel:

[0044]

[0045] And a weight vector is generated through a two-layer perceptron:

[0046]

[0047] in It is ReLU.

[0048] The result after channel weighting is:

[0049]

[0050] Finally, residual fusion is used to integrate the original features. With deformation enhancement features :

[0051]

[0052] This strategy enhances geometric sensitivity while maintaining the stability of the underlying representation, thereby improving the overall robustness of the features.

[0053] Step 5 specifically includes the following:

[0054] Fusion features The input decoding network generates a high-resolution viewport heatmap through a series of upsampling and boundary refinement modules. Let the upsampling operation of the decoding network be:

[0055]

[0056] Gradient-enhanced convolutions are added to the boundary regions:

[0057]

[0058] in This represents the edge enhancement operator. The final predicted viewport probability distribution is:

[0059]

[0060] in This indicates that the user's gaze will fall on a pixel in the future. The probability of. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the description are only some embodiments of the present invention.

[0062] Figure 1 This is a step diagram of the present invention. Detailed Implementation

[0063] To clarify the technical problems, technical solutions, implementation processes, and performance demonstrations, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Various exemplary embodiments, features, and aspects of this disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0064] This embodiment proposes a long-term viewport prediction method based on a spatial deformation attention network to address the problems of spatial distortion and uneven distribution of viewport-related features caused by panoramic projection. The overall method includes the following steps:

[0065] Step 1 specifically includes the following:

[0066] First, the viewport center trajectory of a user watching a 360-degree video is converted into a sequence of viewport saliency maps, which serves as the input to the model. The dimension of the input tensor is... ,in For batch size, For time step, For the number of channels, For height and width.

[0067] The sequence is then input into a ConvLSTM network. Each ConvLSTM unit contains an input gate. Forgotten Gate Output gate and memory unit Its state update depends on the current input. and the hidden state of the previous moment .

[0068] Input gate The forgetting gate determines how much new information is retained in memory at any given moment. How many memory units from the previous moment are forgotten, when you go out... Controls the output of the currently hidden state.

[0069] The hidden state of the final output It not only contains historical time information, but also preserves... The model extracts a two-dimensional spatial structure. By stacking multiple layers of ConvLSTM, the model can extract a high-level abstract spatiotemporal feature map, which contains trend information of user head movement and spatial structure information of video content, providing rich information for subsequent processing.

[0070] Step 2 specifically includes the following:

[0071] The basic features are input into the offset prediction module, and local texture gradients and projection structure information are extracted through a lightweight convolutional network. The resulting sampled displacement vector field for each spatial location is obtained through regression.

[0072]

[0073] in It is a learnable offset prediction network.

[0074] The offset field explicitly describes the spatial non-uniform distortion mode introduced by the projection from the sphere to the ERP, enabling dynamic adjustment of the geometry under rotating viewpoints.

[0075] Step 3 specifically includes the following:

[0076] Using the offset field The original convolution sampling positions are deformed so that the convolution kernel can adaptively align geometrically with the deformed positions. The sampling process of deformable convolution is represented as follows:

[0077]

[0078] in, As the central location, Preset sampling points for the convolution kernel. This is the corresponding dynamic offset vector.

[0079] To avoid spatial imbalances caused by local deformation, a spatial saliency weighting map is introduced. It is generated by channel compression and spatial softmax:

[0080]

[0081] The final significance enhancement feature is:

[0082]

[0083] This module can highlight spatial areas that are highly correlated with future viewport changes, improving the discernibility of geometrically sensitive features.

[0084] Step 4 specifically includes the following:

[0085] To further improve the semantic selectivity of deformation features, in Channel attention is applied to enhance the response to key semantic dimensions. Channel weights are constructed using a Squeeze-Excitation structure.

[0086]

[0087]

[0088] in This is the ReLU activation function.

[0089] The output after channel recalibration is:

[0090]

[0091] To maintain the stability of the low-level geometric structure and the high-level semantic response, it is combined with the basic features. Fusion using residuals:

[0092]

[0093] This design significantly enhances the network's robustness to changes in spatial structure and reduces the spread of geometric errors.

[0094] Step 5 specifically includes the following:

[0095] Fusion features By inputting into a multi-layer decoding network, spatial resolution is restored through stepwise deconvolution, bilinear upsampling, and boundary enhancement convolution branches, thus obtaining the viewport probability distribution at future time steps.

[0096] The final output heatmap is as follows:

[0097]

[0098] in For predicting time intervals, Decoding the network.

[0099] Final prediction results The calculation formula is: .

[0100] This heatmap can be used to allocate 360-degree video tile bitrates and viewport priority areas to achieve high-efficiency streaming media transmission.

[0101] In summary, this invention proposes a long-term viewport prediction method based on a spatially deformable attention network. This method achieves adaptive modeling of geometric distortion and long-term motion dependence in panoramic videos by introducing an offset field prediction and gating dynamic fusion strategy. Experimental results show that, under a 3-second training scenario, the model achieves accuracies of 97.8%, 97.4%, 97.1%, 97.1%, and 97.0% in prediction tasks at 1s, 2s, 3s, 4s, and 5s, respectively, significantly outperforming traditional benchmark models and demonstrating good robustness and practical application value.

[0102] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still covered by the invention.

Claims

1. A long-term viewport prediction method based on a spatially deformable attention network, characterized in that, The specific steps include the following: Step 1: First, the input historical frame sequence is encoded using a convolutional long short-term memory structure to obtain a preliminary spatiotemporal feature representation across time frames, providing a stable input for subsequent spatial deformation modeling; Step 2: Introduce an offset prediction network onto the initial feature map. By learning the geometric change trends of local regions, a two-dimensional offset field is generated, enabling the model to adaptively adjust to the spatial non-uniformity caused by spherical projection. Step 3: Based on the generated offset field, the convolution sampling position is repositioned, and fine-grained modeling of the resampled features is achieved through deformable convolution, thereby effectively alleviating spatial distortion and viewpoint-dependent structural offset in panoramic images. Step 4: Add a dual attention mechanism to the reconstructed representation. Spatial attention highlights salient regions related to location, while channel attention emphasizes feature dimensions with high semantic contribution. The two are combined under the effect of learnable fusion coefficients to further enhance effective features and suppress noise components. Step 5: Input the enhanced representation into the lightweight prediction head, and obtain the final prediction result through convolutional mapping and channel reweighting to obtain the final future spatiotemporal view probability distribution.

2. The long-term viewport prediction method based on spatially deformable attention networks as described in claim 1, characterized in that, The spatial deformable attention network includes an offset field prediction module, a deformable sampling module, and a spatial attention calculation module. The offset field prediction module learns the sampling displacement at each spatial location, the deformable sampling module adjusts the convolution sampling points based on the predicted offset, and the spatial attention calculation module generates a spatial correlation map based on the deformed features to highlight areas with significant geometric distortion.

3. The long-term viewport prediction method based on spatially deformable attention networks as described in claim 1, characterized in that, The spatial attention module is used to generate a single-channel spatial saliency map. It maps features to the (0,1) interval using the Sigmoid function and multiplies them element-wise with the deformed feature map to suppress background noise introduced by spatial deformation and focus on the core viewport region.

4. The long-term viewport prediction method based on spatially deformable attention networks as described in claim 1, characterized in that, The channel attention module adopts a squeeze-and-excitation structure, including a global average pooling layer, a squeezed convolutional layer, an activation layer, and a recovery convolutional layer, which is used to capture the dependencies between feature channels and assign importance weights to different semantic features.

5. The long-term viewport prediction method based on spatially deformable attention networks as described in claim 1, characterized in that, The prediction decoding network includes an edge enhancement module, which extracts edge gradient information of the prediction map through convolution to generate edge weights and superimposes them into the preliminary prediction results to sharpen the boundaries of the heatmap predicted by the viewport.

Images