Method of converting 2d video to 3D video using machine learning

Abstract

Machine learning method that learns to convert 2D video to 3D video from a set of training examples. Uses machine learning to perform any or all of the 2D to 3D conversion steps of identifying and locating objects, masking objects, modeling object depth, generating stereoscopic image pairs, and filling gaps created by pixel displacement for depth effects. Training examples comprise inputs and outputs for the conversion steps. The machine learning system generates transformation functions that generate the outputs from the inputs; these functions may then be used on new 2D videos to automate or semi-automate the conversion process. Operator input may be used to augment the results of the machine learning system. Illustrative representations for conversion data in the training examples include object tags to identify objects and locate their features, Bézier curves to mask object regions, and point clouds or geometric shapes to model object depth.

Description

[0001]This application is a continuation in part of U.S. Utility patent application Ser. No. 14 / 857,704, filed 17 Sep. 2015, the specification of which is hereby incorporated herein by reference.BACKGROUND OF THE INVENTION[0002]Field of the Invention[0003]One or more embodiments of the invention are related to the field of image processing. More particularly, but not by way of limitation, one or more embodiments of the invention enable a method of converting 2D video to 3D video using machine learning. Embodiments of the invention train a machine learning system to perform one or more 2D to 3D conversion steps. The machine learning system is trained on a training set that includes 2D to 3D conversion examples; it derives generalized 2D to 3D transformation functions from this training set. Embodiments of the invention may also obtain 3D models of objects, such as characters, and process the frames of a video to locate and orient these models in the frames. Depth maps and stereoscopi...

AI Technical Summary

Benefits of technology

[0009]One or more embodiments described in the specification are related to a method of converting of 2D video to 3D video using 3D object models that provides increased artistic and technical flexibility and rapid conversion of movies for stereoscopic viewing. Embodiments of the invention convert a large set of highly granular depth information inherent in

Problems solved by technology

Two-dimensional images however may include shading and lighting that provide the observer a sense of depth for portions of the image, however, this is not considered a three-dimensional view of an image.

Current solutions for conversion of two-dimensional images to stereoscopic images generally require large amounts of manual labor for highly accurate results.

The 2D to 3D conver

Images