Light field video synthesis method based on monocular RGB camera and three-dimensional body representation

An RGB camera and video synthesis technology, applied in computer components, image data processing, 3D modeling, etc., can solve problems such as model drift, initial value sensitivity, and easy accumulation of errors, so as to improve convenience and time domain consistency sexual effect

Pending Publication Date: 2022-04-12
JILIN UNIV
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, compared with multi-eye reconstruction, the accuracy and robustness of the monocular reconstruction model have not been substantially resolved, which seriously affects the quality of the reconstruction model and restricts its practical application in the field of autostereoscopic display content generation.
One of the main reasons is that the current monocular reconstruction algorithm cannot effectively use the time domain information, so each frame of the RGB image stream needs to be processed independently, resulting in a lack of constraint relationship between the reconstruction models of the two frames before and after, and the reconstruction model flickers and shakes, etc. Instability
Currently, solutions based on temporal tracking are prone to accumulating errors and are sensitive to initial values, which can lead to drift of the reconstructed model
While recent approaches have demonstrated that drift can be mitigated using anchor frames, it still requires a particularly large engineering effort to handle common but challenging application scenarios.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Light field video synthesis method based on monocular RGB camera and three-dimensional body representation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] The implementation process of the present invention will be further described below in conjunction with the accompanying drawings. The light field video synthesis method based on monocular RGB camera and three-dimensional volume representation, its overall framework is as follows figure 1 As shown, the gray box in the figure indicates the input of this method, and the dotted line box indicates the output of this method, which specifically includes the following steps:

[0035] 1. Extracting features for the t-th frame RGB image, including the following steps:

[0036] 1.1 Use the image segmentation algorithm to extract the t-th frame RGB image I t The foreground object in , get the foreground image F of the tth frame t ;

[0037] 1.2 Use the feature extraction encoder to initially extract the foreground image F of the t-th frame t The features in the t-th frame get the initial feature J t , the feature extraction encoder consists of 4 convolutional layers;

[0038]...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A light field video synthesis method based on a monocular RGB camera and three-dimensional body representation belongs to the technical field of free three-dimensional display, and comprises the following steps: 1, extracting features for a current frame RGB image by using a feature extraction and feature optimization encoder; 2, calculating an estimated value of a light field image for the foreground image of the current frame by using a pixel alignment implicit function and an efficient ray casting mechanism; 3, calculating the time domain consistency loss of the estimated value of the current frame light field image by using the forward and backward light stream information; 4, performing weighted summation on global loss, local loss and time domain consistency loss to construct a target function; 5, obtaining the optimal estimation of the current frame of light field image through minimizing the objective function; by adopting the monocular RGB camera and the three-dimensional body representation, the time domain consistency can be improved on the basis of ensuring the consistency of the spatial domain and the angle domain of the light field image, so that the convenience and the robustness of the collection of the free three-dimensional display content and the stability of the dynamic human body picture are improved.

Description

technical field [0001] The invention belongs to the technical field of autostereoscopic display, and relates to a content generation method for autostereoscopic display, in particular to a light field video synthesis method based on a monocular RGB camera and three-dimensional volume representation. Background technique [0002] Autostereoscopic display technology is the cutting-edge technology in today's international display field, and it will gradually replace the existing flat display and visual aid three-dimensional display media. Autostereoscopic display has been actively explored at home and abroad, especially in the improvement of resolution, viewing angle, depth of field and other aspects have made significant progress. However, the bottleneck of the relative lack of high-quality content has not been substantively resolved, which seriously restricts its popularization in many fields and the practical application of hardware devices such as 3D mobile phones, 3D TVs, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06T17/00G06T7/194G06T7/586G06T7/90G06V10/40
Inventor李志烽韦健郭校源贾勇田锴顾宇杰王学军王世刚陈立何韧冬
OwnerJILIN UNIV