Method for predicting field of view of user based on deep learning

A deep learning and user-friendly technology, applied in the field of computer vision and deep learning, can solve problems affecting the accuracy of video features, embarrassing field of view prediction, high bandwidth and low latency that have not been resolved, and achieve improved VR experience and low bandwidth costs , Reduce the effect of transmission delay

Active Publication Date: 2018-09-04
NANJING UNIV
View PDF5 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] At present, many innovative applications have appeared in the VR industry, and VR is also gradually entering mobile terminals such as mobile phones, but the problems of high bandwidth and low latency required for smooth VR playback hav

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for predicting field of view of user based on deep learning
  • Method for predicting field of view of user based on deep learning
  • Method for predicting field of view of user based on deep learning

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the implementation method of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0021] A method of predicting the user's field of view based on deep learning in this embodiment, the steps are as follows:

[0022] (1) Map the panoramic video from the spherical surface to the 6 faces of the cube inscribed on the sphere, and obtain the video corresponding to the 6 faces of the cube from the 2D panoramic video. Number the faces of the cube from 1 to 6, and expand them in sequence from 1 to 6 (see attached image 3 ).

[0023] (2) Use the optical flow algorithm to generate the dynamic feature sequence diagram of the six faces of the cube corresponding to the video, and then use the coordinate transformation relationship from the cube to the 2D plane and its numbering sequence to synthesize the panoramic dynamic featu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for predicting the field of view of a user based on deep learning, which comprises the steps of (1) mapping a panoramic video from a spherical surface to six surfacesof a sphere inscribed cube to obtain videos corresponding to the six surfaces, respectively generating a dynamic feature sequence diagram and a saliency sequence diagram of the videos, and performingblocking and numbering; (2) judging the video content viewpoint switching intensity w according to dynamic features; (3) recording the head turning of the user by a helmet, and processing the recordedhead turning; (4) selecting a prediction network according to the size of the w value, obtaining the field of view of the user in the last n video frames by network prediction, and processing to obtain the number of a video block overlapped with the field of view; and (5) rendering the transmitting the video block obtained by prediction, and repeating the steps until the final n frames are predicted. The method disclosed by the invention reduces influences imposed on input video features by the panorama distortion, and pre-judgment grading of the video information is added at the same time, so that the field of view of the user when watching video in a VR HMD (Head Mount Display) can be predicted with high accuracy.

Description

technical field [0001] The invention relates to the fields of computer vision and deep learning, in particular to a method for predicting a user's field of view based on deep learning. Background technique [0002] At present, many innovative applications have appeared in the VR industry, and VR is gradually entering mobile terminals such as mobile phones. However, the problems of high bandwidth and low latency required for smooth VR playback have not been resolved. Human perception requires smooth and accurate movement of vision, so unsmooth playback and high delay may cause VR users to experience symptoms such as nausea and dizziness, seriously affecting the user's immersive experience. Adding field of view prediction during VR video rendering and transmission can reduce the amount of transmitted data, thereby reducing the rendering and transmission time and effectively reducing transmission delay. [0003] LSTM (Long Short Term Memory) network is a special type of recurr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06T7/246G06K9/00G06F3/01
CPCG06F3/012G06T7/246G06T2207/10016G06V20/40
Inventor 蒲志远沈秋郭佩瑶马展
Owner NANJING UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap