Static three-dimensional light field content generation method and system

Abstract

The invention discloses a static three-dimensional light field content generation method and system, and the method comprises the following steps: S1, collecting a video of static scene content, and preferentially extracting a discrete parallax image sequence from the video through an automatic frame obtaining algorithm; s2, calculating a relative pose parameter of the discrete parallax image; s3, calculating a synthetic coding parameter mapping function according to the relative pose parameters and a sub-pixel arrangement rule of the light field display; and S4, based on the synthetic coding parameter mapping function, performing synthetic coding on the discrete parallax image sequence by using CUDA and OpenGL rendering engines, and rendering the discrete parallax image sequence to a light field display for display. According to the invention, high-quality static light field content in a real scene can be rapidly collected and generated, and an actual light field display effect can be rapidly adjusted according to actual demands.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional light field display, and in particular relates to a method and system for generating static three-dimensional light field content. Background Art

[0002] Three-dimensional light field display technology produces stereoscopic vision through physiological or psychological factors. Common three-dimensional display methods, such as grating 3D display, integrated imaging 3D display, volumetric three-dimensional display and other 3D display technologies use physiological and psychological factors such as binocular parallax to display parallax images on 2D display screens, so that the audience can feel the depth information of objects on the screen and present a three-dimensional sense. However, the acquisition of high-quality light field content has always been a difficult point in the industry. The common acquisition method uses sparse camera array acquisition, reconstructs the 3D scene through three-dimensional reconstruction, and then renders dense viewpoint images, and finally synthesizes light field images. The three-dimensional reconstruction method used is difficult to achieve high-quality reconstruction effects, and the rendering algorithm is complex and requires a lot of computing resources.

[0003] Therefore, it is of great significance to develop a simple and effective light field acquisition and high-quality light field content generation solution. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, a static three-dimensional light field content generation method and system provided by the present invention solves the problem of complex rendering algorithm when obtaining high-quality light field content in the prior art.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a method for generating static three-dimensional light field content, comprising the following steps:

[0006] S1, collect videos of static scene contents, and extract discrete parallax image sequences from the videos through an automatic frame-taking algorithm;

[0007] S2, calculating the relative pose parameters of the discrete disparity image;

[0008] S3, calculating a synthetic coding parameter mapping function according to the relative posture parameters and the sub-pixel arrangement rule of the light field display;

[0009] S4. Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines, and rendered to a light field display for display.

[0010] Further: in said S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence;

[0011] The method for extracting the non-still scene image sequence is specifically as follows:

[0012] Extract key point information of human actions from the video, delete frames with action amplitudes greater than a threshold through an action detection algorithm, and obtain a non-still scene image sequence;

[0013] The specific method for extracting still scene image sequences is as follows:

[0014] Extract any stable feature points in the scene, fit the relative position relationship curve of each frame of the image according to the pixel coordinates of the extracted feature points, and generate a still scene image sequence through the frame-taking algorithm.

[0015] Further: S2 is specifically:

[0016] The relative position parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated through the motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

[0017] Further: in S3, the expression of the synthetic coding parameter mapping function F is specifically:

[0018]

[0019] Where θ is the tilt angle of the display, L x is the number of lines of the display, N is the number of disparity maps, h is the vertical resolution of the display, and h 11 is the first parameter of the transformation matrix H(z), h 12 is the second parameter of the transformation matrix H(z), h 13 is the third parameter of the transformation matrix H(z), h 21 is the fourth parameter of the transformation matrix H(z), h 22 is the fifth parameter of the transformation matrix H(z), h 23 is the sixth parameter of the transformation matrix H(z), h 31 is the seventh parameter of the transformation matrix H(z), h 32 is the eighth parameter of the transformation matrix H(z), h 33 is the ninth parameter of the transformation matrix H(z), x is the horizontal coordinate of the pixel corresponding to the target viewpoint, y is the vertical coordinate of the pixel corresponding to the target viewpoint, f is the pixel correspondence between the synthetic coding image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the remainder, and k is the sub-pixel index.

[0020] Further: S4 is specifically:

[0021] Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines and rendered to the light field display for display.

[0022] A static three-dimensional light field content generation system, the system comprising:

[0023] A video acquisition device placed on the guide rail is used to acquire video of static scene content;

[0024] The motion structure recovery algorithm module is used to calculate the relative posture parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix;

[0025] A synthetic coding parameter mapping function calculation module is used to calculate a synthetic coding parameter mapping function according to the number of lines and the inclination angle of the light field display and the resolution of the display;

[0026] The synthesis encoding module is used to rasterize and render the synthesized image onto the light field display according to the CUDA and OpenGL rendering engines.

[0027] The beneficial effects of the present invention are as follows: the present invention provides a method and system for generating static three-dimensional light field content, using a video of static scene content of a video acquisition device placed on a horizontal rail, extracting a discrete parallax image sequence from the video by an automatic frame-taking algorithm, calculating the relative posture parameters of the discrete parallax image, calculating a zero-plane adjustable transformation matrix according to the relative posture parameters, calculating a synthetic coding parameter mapping function according to the sub-pixel arrangement rule of the light field display and the above transformation matrix, using CUDA and OpenGL rendering engines, synthesizing and encoding the above parallax image sequence in real time and rendering it to the light field display for display, and adjusting the screen depth in real time according to the display effect. The present invention can quickly collect and generate high-quality static light field content in real scenes, can quickly adjust the actual light field display effect according to actual needs, and solves the problem of complex rendering algorithms when obtaining high-quality light field content in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a method for generating static three-dimensional light field content according to the present invention.

[0029] Figure 2 It is a schematic diagram of the structure of a static three-dimensional light field content generation system of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0031] like Figure 1 As shown, in one embodiment of the present invention, a method for generating static three-dimensional light field content includes the following steps:

[0032] S1, collect videos of static scene contents, and extract discrete parallax image sequences from the videos through an automatic frame-taking algorithm;

[0033] S2, calculating the relative pose parameters of the discrete disparity image;

[0034] S3, calculating a synthetic coding parameter mapping function according to the relative posture parameters and the sub-pixel arrangement rule of the light field display;

[0035] S4. Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines, and rendered to a light field display for display.

[0036] In this embodiment, the present invention can quickly collect real scene content and directly generate three-dimensional light field content, and can adjust the actual stereoscopic display effect by adjusting the zero plane position.

[0037] In S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence;

[0038] The method for extracting the non-still scene image sequence is specifically as follows:

[0039] Extract key point information of human actions from the video, delete frames with action amplitudes greater than a threshold through an action detection algorithm, and obtain a non-still scene image sequence;

[0040] The specific method for extracting still scene image sequences is as follows:

[0041] Extract any stable feature points in the scene, fit the relative position relationship curve of each frame of the image according to the pixel coordinates of the extracted feature points, and generate a still scene image sequence through the frame-taking algorithm.

[0042] In this embodiment, under the premise of ensuring that all viewing angles are covered as much as possible, a uniform sampling algorithm is preferentially used to select frame images from a video of static scene content as an original discrete parallax image sequence.

[0043] The S2 is specifically:

[0044] The relative position parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated through the motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

[0045] In this embodiment, the motion structure recovery algorithm uses the matching points between multiple images to calculate the relative pose parameters of each image, specifically:

[0046] S21. Extract key points from each image using a feature detection algorithm, wherein the feature detection algorithm includes SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF);

[0047] S22, finding corresponding feature point pairs between the images by using a feature matching algorithm, the feature matching algorithm including BFMatcher (Brute-Force Matcher) or FLANN (Fast Library for Approximate Nearest Neighbors);

[0048] S23. Using the matched feature points, by calculating the essential matrix or the fundamental matrix, the relative pose parameters of the camera corresponding to each disparity image are estimated, including the relative rotation matrix and the relative translation matrix.

[0049] In S3, the expression of the synthetic coding parameter mapping function F is specifically:

[0050]

[0051] Where θ is the tilt angle of the display, L x is the number of lines of the display, N is the number of disparity maps, h is the vertical resolution of the display, and h 11 is the first parameter of the transformation matrix H(z), h 12 is the second parameter of the transformation matrix H(z), h 13 is the third parameter of the transformation matrix H(z), h 21 is the fourth parameter of the transformation matrix H(z), h 22 is the fifth parameter of the transformation matrix H(z), h 23 is the sixth parameter of the transformation matrix H(z), h 31 is the seventh parameter of the transformation matrix H(z), h 32 is the eighth parameter of the transformation matrix H(z), h 33 is the ninth parameter of the transformation matrix H(z), x is the horizontal coordinate of the pixel corresponding to the target viewpoint, y is the vertical coordinate of the pixel corresponding to the target viewpoint, f is the pixel correspondence between the synthetic coding image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the remainder, and k is the sub-pixel index.

[0052] By adjusting the size of z, since the projection coordinate points on each frame of the image are related to the zero plane depth, by adjusting the size of z, the final transformation matrix is ​​related to the zero plane depth z. By adjusting, different transformation matrices can be obtained, thereby realizing the adjustment of the synthetic coding mapping function, thereby adjusting the entry and exit screen depth of the actual three-dimensional light field display.

[0053] The principle of calculating the synthetic coding parameter mapping function is as follows:

[0054] First, the display inclination and line number measured according to the display light control element parameters and actual assembly parameters are used to calculate the mapping relationship f between the synthetic coding image (x, y, k) and the discrete parallax image sequence. The specific expression is:

[0055] V index (x,y,k)=F(x,y,k)

[0056] Where V index (·) is the disparity map index, F(·) is the synthetic coding mapping function, and finally a synthetic coding mapping function can be obtained, in which the depth of the three-dimensional light field display content entering and exiting the screen can be freely adjusted.

[0057] A transformation matrix H(z) related to the set zero plane position is calculated according to the relative posture parameters, wherein the zero plane is the confocal plane between dense viewpoints of the three-dimensional display and is also the critical plane between the out-screen and the input-screen of the three-dimensional display. After the discrete disparity image sequence is transformed by the transformation matrix, an approximate off-axis transformed disparity sequence image can be obtained; according to the light control element parameters and the actual assembly parameters, the sub-pixel arrangement rule of the light field display is determined, the pixel correspondence f between the synthetic coding image and the discrete disparity image sequence is calculated, and the synthetic coding parameter mapping function is obtained.

[0058] The principle of calculating the transformation matrix is ​​as follows:

[0059] Set an initial zero plane with a size of H×W and a distance of Z from the acquisition device. Take the camera of the middle frame as the origin of the coordinate system. The coordinates of the four vertices of the zero plane are P1(-W / 2,H / 2,Z), P2(W / 2,H / 2,Z), P3(-W / 2,-H / 2,Z), and P4(W / 2,-H / 2,Z). Based on the pose parameters of each frame calculated by the SFM algorithm, the four vertices of the zero plane can be projected onto each frame to obtain the corresponding pixel coordinates p i,j (u,v), and the transformation matrix is ​​calculated to be the transformation matrix H(z) which is expressed as:

[0060]

[0061] Through the transformation matrix H(z), the above-mentioned ordinary acquisition method can be transformed into an approximately staggered view volume setting, achieving both the out-of-screen effect and the in-screen effect, without the distortion caused by the convergence setting. And by adjusting the position of the initial zero plane and the transformation matrix, the zero plane adjustment of the light field display content can be achieved, presenting different stereoscopic display effects.

[0062] The S4 is specifically:

[0063] Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines and rendered to the light field display for display.

[0064] In this embodiment, the present invention utilizes CUDA and OpenGL rendering engines to synthesize and encode the above-mentioned disparity image sequence in real time and render it to the light field display for display, and adjusts the screen depth in real time according to the display effect. In the synthesis encoding process, the calculation of sub-pixel off-axis transformation and light field encoding are performed simultaneously, that is, a sub-pixel in the required discrete disparity image is located according to the mapping relationship, and only the off-axis transformation is applied to the sub-pixel, which can greatly reduce the calculation redundancy, and the whole process is processed in parallel at high speed in the GPU, and the synthesized encoded image is displayed using OpenGL real-time rasterization rendering, and the zero plane depth is changed in real time through interactive design to display different stereoscopic effects.

[0065] like Figure 2 As shown, a static three-dimensional light field content generation system includes:

[0066] A video acquisition device placed on the guide rail is used to acquire video of static scene content;

[0067] The guide rail, which can be a straight or curved guide rail, can carry the acquisition device to acquire scene video at an approximately uniform and stable speed according to instructions. The acquisition range needs to be able to meet the display viewing angle range of the display, and the acquisition device is placed at a distance from the scene that is approximately the best viewing position of the display.

[0068] The motion structure recovery algorithm module is used to calculate the relative posture parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix;

[0069] A synthetic coding parameter mapping function calculation module is used to calculate a synthetic coding parameter mapping function according to the number of lines and the inclination angle of the light field display and the resolution of the display;

[0070] The synthesis encoding module is used to rasterize and render the synthesized image onto the light field display according to the CUDA and OpenGL rendering engines.

[0071] In this embodiment, the video acquisition device is placed at a distance from the scene that is approximately the optimal viewing position of the display, the acquisition range can meet the display viewing angle range of the display, and the slider on the guide rail can carry the acquisition device at an approximately uniform speed and stably to acquire scene video according to instructions.

[0072] The beneficial effects of the present invention are as follows: the present invention provides a method and system for generating static three-dimensional light field content, using a video of static scene content of a video acquisition device placed on a horizontal rail, extracting a discrete parallax image sequence from the video by an automatic frame-taking algorithm, calculating the relative posture parameters of the discrete parallax image, calculating a zero-plane adjustable transformation matrix according to the relative posture parameters, calculating a synthetic coding parameter mapping function according to the sub-pixel arrangement rule of the light field display and the above transformation matrix, using CUDA and OpenGL rendering engines, synthesizing and encoding the above parallax image sequence in real time and rendering it to the light field display for display, and adjusting the screen depth in real time according to the display effect. The present invention can quickly collect and generate high-quality static light field content in real scenes, can quickly adjust the actual light field display effect according to actual needs, and solves the problem of complex rendering algorithms when obtaining high-quality light field content in the prior art.

[0073] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims

1. A method for generating static three-dimensional light field content, characterized in that: The following steps are involved: S1, collect videos of static scene contents, and extract discrete parallax image sequences from the videos through an automatic frame-taking algorithm; S2, calculating the relative pose parameters of the discrete disparity image; S3, calculating a synthetic coding parameter mapping function according to the relative posture parameters and the sub-pixel arrangement rule of the light field display; S4. Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines, and rendered to a light field display for display.

2. The method for generating static 3D light field content according to claim 1, characterized in that: In S1, the discrete parallax image sequence includes a still scene image sequence and a non-still scene image sequence; The method for extracting the non-still scene image sequence is specifically as follows: Extract key point information of human actions from the video, delete frames with action amplitudes greater than a threshold through an action detection algorithm, and obtain a non-still scene image sequence; The specific method for extracting still scene image sequences is as follows: Extract any stable feature points in the scene, fit the relative position relationship curve of each frame of the image according to the pixel coordinates of the extracted feature points, and generate a still scene image sequence through the frame-taking algorithm.

3. The method for generating static 3D light field content according to claim 2, characterized in that: The S2 is specifically: The relative position parameters of the camera corresponding to each image in the discrete parallax image sequence are calculated through the motion structure recovery algorithm, including the relative rotation matrix and the relative translation matrix.

4. The method for generating static 3D light field content according to claim 1, characterized in that: In S3, the expression of the synthetic coding parameter mapping function F is specifically: Where θ is the tilt angle of the display, L x is the number of lines of the display, N is the number of disparity maps, h is the vertical resolution of the display, and h 11 is the first parameter of the transformation matrix H(z), h 12 is the second parameter of the transformation matrix H(z), h 13 is the third parameter of the transformation matrix H(z), h 21 is the fourth parameter of the transformation matrix H(z), h 22 is the fifth parameter of the transformation matrix H(z), h 23 is the sixth parameter of the transformation matrix H(z), h 31 is the seventh parameter of the transformation matrix H(z), h 32 is the eighth parameter of the transformation matrix H(z), h 33 is the ninth parameter of the transformation matrix H(z), x is the horizontal coordinate of the pixel corresponding to the target viewpoint, y is the vertical coordinate of the pixel corresponding to the target viewpoint, f is the pixel correspondence between the synthetic coding image and the discrete disparity image sequence, z is the transformation matrix and the zero plane depth, % is the remainder, and k is the sub-pixel index.

5. The method for generating static 3D light field content according to claim 4, characterized in that: The S4 is specifically: Based on the synthetic coding parameter mapping function, the discrete parallax image sequence is synthetically encoded using CUDA and OpenGL rendering engines and rendered to the light field display for display.

6. A static three-dimensional light field content generation system, applied to the static three-dimensional light field content generation method according to any one of claims 1 to 5, characterized in that: The system includes: A video acquisition device placed on the guide rail is used to acquire video of static scene content; The motion structure recovery algorithm module is used to calculate the relative posture parameters of discrete parallax images, including the relative rotation matrix and the relative translation matrix; A synthetic coding parameter mapping function calculation module is used to calculate a synthetic coding parameter mapping function according to the number of lines and the inclination angle of the light field display and the resolution of the display; The synthesis encoding module is used to rasterize and render the synthesized image onto the light field display according to the CUDA and OpenGL rendering engines.

Images