Method of converting 2d video to 3D video using machine learning

Abstract

Machine learning method that learns to convert 2D video to 3D video from a set of training examples. Uses machine learning to perform any or all of the 2D to 3D conversion steps of identifying and locating objects, masking objects, modeling object depth, generating stereoscopic image pairs, and filling gaps created by pixel displacement for depth effects. Training examples comprise inputs and outputs for the conversion steps. The machine learning system generates transformation functions that generate the outputs from the inputs; these functions may then be used on new 2D videos to automate or semi-automate the conversion process. Operator input may be used to augment the results of the machine learning system. Illustrative representations for conversion data in the training examples include object tags to identify objects and locate their features, Bézier curves to mask object regions, and point clouds or geometric shapes to model object depth.

Description

[0001]This application is a continuation in part of U.S. Utility patent application Ser. No. 14 / 857,704, filed 17 Sep. 2015, the specification of which is hereby incorporated herein by reference.BACKGROUND OF THE INVENTION[0002]Field of the Invention[0003]One or more embodiments of the invention are related to the field of image processing. More particularly, but not by way of limitation, one or more embodiments of the invention enable a method of converting 2D video to 3D video using machine learning. Embodiments of the invention train a machine learning system to perform one or more 2D to 3D conversion steps. The machine learning system is trained on a training set that includes 2D to 3D conversion examples; it derives generalized 2D to 3D transformation functions from this training set. Embodiments of the invention may also obtain 3D models of objects, such as characters, and process the frames of a video to locate and orient these models in the frames. Depth maps and stereoscopi...

AI Technical Summary

Benefits of technology

The invention is a method for converting 2D video to 3D video using 3D object models that provides increased artistic and technical flexibility and rapid conversion of movies for stereoscopic viewing. The method converts a large set of highly granular depth information in a depth map associated with a two-dimensional image to a smaller set of rotated planes associated with masked areas in the image. This eliminates many problems associated with importing external depth maps. The method also allows for the grouping of planes to provide a manageable number of parts and enables the movement of planes relative to one another to self align with respect to the other planes. The conversion of 3D data to a 3D object model may include retopologizing the 3D data, fitting one or more planes to the 3D data, and generating a 3D object model from these planes. The 3D object model can be located and oriented in a frame using one or more model features. The technical effects of the invention include increased flexibility and efficiency in the conversion process and improved quality of the resulting 3D video.

Problems solved by technology

Two-dimensional images however may include shading and lighting that provide the observer a sense of depth for portions of the image, however, this is not considered a three-dimensional view of an image.

Current solutions for conversion of two-dimensional images to stereoscopic images generally require large amounts of manual labor for highly accurate results.

The 2D to 3D conversion processes described above require large amount of manual labor.

There are no known systems that automate the conversion process.

Machine learning techniques are known in the art, but they have not been applied to 2D to 3D conversion.

There are no known systems that apply machine learning techniques to develop 2D to 3D conversion methods using a database of conversion examples.

Images