A method for predicting user field of view based on deep learning

Abstract

The invention discloses a method for predicting a user's field of view based on deep learning. The steps are: (1) Map the panoramic video from the spherical surface to the 6 surfaces of the spherical inscribed cube to obtain the videos corresponding to the 6 surfaces, generate the dynamic features and saliency sequence diagrams of the videos respectively, and perform block and numbering ; (2) Judging the severity w of video content viewpoint switching according to the dynamic features; (3) Recording the user's head turning with the helmet and processing it; (4) Selecting the prediction network through the value of w, and using the network prediction to obtain the user's The field of view of the next n frames of video frames can be processed to obtain the number of video blocks that overlap with the field of view; (5) render and transmit the predicted video blocks, and repeat the steps until the last n frames are predicted. The method of the present invention reduces the influence of panorama distortion on the input video features, and at the same time adds the pre-judgment and classification of video information, and can predict the field of view when the user watches the video in the VR HMD with high accuracy.

Description

technical field [0001] The invention relates to the fields of computer vision and deep learning, in particular to a method for predicting a user's field of view based on deep learning. Background technique [0002] At present, many innovative applications have appeared in the VR industry, and VR is gradually entering mobile terminals such as mobile phones. However, the problems of high bandwidth and low latency required for smooth VR playback have not been resolved. Human perception requires smooth and accurate movement of vision, so unsmooth playback and high delay may cause VR users to experience symptoms such as nausea and dizziness, seriously affecting the user's immersive experience. Adding field of view prediction during VR video rendering and transmission can reduce the amount of transmitted data, thereby reducing the rendering and transmission time and effectively reducing transmission delay. [0003] LSTM (Long Short Term Memory) network is a special type of recurr...

AI Technical Summary

Problems solved by technology

[0002] At present, many innovative applications have appeared in the VR industry, and VR is also gradually entering mobile terminals such as mobile phones, but the problems of high bandwidth and low latency required for smooth VR playback have not been resolved.

In addition, isometric mapping etc. make the distortion of objects in the panorama very obvious, thus also affecting the accuracy of the obtained video features, which is an embarrassing problem for field of view prediction

Images