Rapid compression and decomposition method and system for high-dimensional light field video

Abstract

The invention discloses a rapid compression and decomposition method and system for a high-dimensional light field video, and belongs to the technical field of light field display. According to the rapid compression decomposition method for the high-dimensional light field video, a light field acquisition model, a first frame extraction model, a first optimization simulation model, a second frame extraction model, a second optimization simulation model and an object generation model are constructed; according to the method, the first optimization value of the video key frame nearest to the video non-key frame is used as the second iteration initial value of the compressed light field, so that the iteration initial value of the non-key frame can be quickly obtained by utilizing the correlation between the video key frame and the non-key frame, the iteration times of the light field video can be greatly reduced, and the efficiency of the light field video is improved. The compression and decomposition speed of the dynamic compression light field video information is improved, and compression and decomposition of a higher-resolution and more complex object can be realized. Furthermore, the method overcomes the limitations that a traditional iterative optimization method is easy to converge to a local optimal solution and the signal-to-noise ratio of a display result is not high.

Description

Technical Field

[0001] The invention relates to a method and system for fast compression and decomposition of high-dimensional light field video, and belongs to the technical field of light field display. Background Art

[0002] Compressed light field display technology is an important naked eye 3D display technology, which uses multi-layer flat display to construct high-dimensional light field information to achieve optical reconstruction of 3D images. The compressed light field display system can provide binocular parallax information and continuous motion parallax information, and has the advantages of high resolution, simple structure and good compatibility, so it has attracted much attention.

[0003] In compressed light field display technology, how to compress and decompose high-dimensional light field information into multi-layer images is a very challenging task. The compression decomposition algorithm of high-dimensional light field needs to record a large amount of light information, and solve the intersection position and intersection pixel value between the light and each display layer according to the brightness and direction of the light. Since the number of input light equations is much larger than the number of pixels to be solved on each layer of the screen, traditional compressed light field technology usually expresses the problem in the form of solving an overdetermined linear system of equations. Such equations usually need to be solved using optimization algorithms.

[0004] The current optimization algorithm methods include four categories: least square method, non-negative tensor decomposition method, tomography algorithm and machine learning. However, these optimization algorithms usually need to calculate all the collected light rays at the same time during the light field decomposition process. Even a scene with a low resolution rate will involve hundreds of millions of light rays. Calculating these light rays at the same time will inevitably occupy a large amount of memory and consume a long time, making it difficult to meet the requirements of fast light field display of high-resolution complex scenes. The information disclosed in this background technology is only used to understand the background of the concept of the present invention, so it may include information that does not constitute prior art. Summary of the invention

[0005] In view of the above problem or one of the above problems, an object of the present invention is to provide a method and system for fast compression and decomposition of high-dimensional light field video, which uses the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field, so that the correlation between the video key frame and the non-key frame can be used to quickly obtain the iteration initial value of the non-key frame, thereby greatly reducing the number of iterations of the light field video, improving the compression and decomposition speed of the dynamic compressed light field video information, and further greatly improving the compression and decomposition efficiency of the compressed light field video information, and achieving higher resolution and compression and decomposition of more complex objects.

[0006] In response to the above problem or one of the above problems, the second object of the present invention is to provide a method and system for fast compression decomposition of high-dimensional light field videos. By constructing a high-dimensional light field pre-decomposition unit and / or a high-dimensional light field reverse transmission unit to calculate the iteration initial value, the limitations of traditional iterative optimization methods that they easily converge to a local optimal solution and the signal-to-noise ratio of the displayed results are not high are overcome, thereby achieving high-quality compression decomposition of the high-dimensional light field.

[0007] In response to the above problem or one of the above problems, the third object of the present invention is to provide a method and system for rapid compression and decomposition of high-dimensional light field videos, which further solves the problems of large number of iterations and long optimization time in traditional iterative optimization methods by constructing a high-dimensional light field pre-decomposition unit and / or a high-dimensional light field reverse transmission unit to calculate the iteration initial value, thereby achieving rapid compression and decomposition of high-dimensional light fields.

[0008] To achieve one of the above purposes, the first technical solution of the present invention is:

[0009] A fast compression and decomposition method for high-dimensional light field video comprises the following steps:

[0010] Step 1: Using a pre-built light field acquisition model, high-dimensional light field information of the three-dimensional object is temporally and spatially sampled to obtain a high-dimensional light field information video of the three-dimensional object;

[0011] Step 2: Using a pre-built first frame extraction model, the high-dimensional light field information video is processed to extract the video key frame and obtain the first iteration initial value of the compressed light field of the three-dimensional object at the corresponding moment;

[0012] Step 3, using a pre-built first optimization simulation model and a first iteration initial value to perform iterative optimization to obtain a first optimization value of the compressed light field at a time corresponding to a key frame of the video;

[0013] Step 4, using the pre-built second frame extraction model to extract the video non-key frame, and taking the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field;

[0014] Step 5, using the pre-built second optimization simulation model, using the second iteration initial value to perform iterative optimization, and obtaining the second optimization value of the compressed light field at the corresponding moment of the non-key frame of the video;

[0015] Step six, using the pre-built object generation model, the first optimization value and the second optimization value are combined into a compressed light field video of the three-dimensional object in a time series, thereby completing the rapid compression decomposition of the high-dimensional light field video.

[0016] After continuous exploration and experiments, the present invention constructs a light field acquisition model, a first frame extraction model, a first optimization simulation model, a second frame extraction model, a second optimization simulation model and an object generation model, and uses the first optimization value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field, so that the correlation between the video key frame and the non-key frame can be used to quickly obtain the iteration initial value of the non-key frame, thereby greatly reducing the number of iterations of the light field video, improving the compression decomposition speed of the dynamic compressed light field video information, and further greatly improving the compression decomposition efficiency of the compressed light field video information, and can achieve higher resolution and compression decomposition of more complex objects.

[0017] As the preferred technical measures:

[0018] Step 1: Using a pre-built light field acquisition model, high-dimensional light field information of a three-dimensional object is sampled in time and space to obtain a high-dimensional light field information video of the three-dimensional object as follows:

[0019] Get the dimension information (u, v) of the spatial light information variable;

[0020] According to the dimension information (u, v), set the position coordinates of each camera unit;

[0021] Establishing a camera array according to the position coordinates of each camera unit;

[0022] The dynamic three-dimensional object is photographed by a camera array, and time-space sampling is performed to obtain color image information of the three-dimensional object;

[0023] Based on the color image information of the three-dimensional object, depth information of the three-dimensional object is calculated;

[0024] According to the depth information, a high-dimensional light field information video is obtained.

[0025] As the preferred technical measures:

[0026] Spatiotemporal sampling is the sampling of the 5-dimensional light field function that describes a 3D object;

[0027] The 5-dimensional light field function is L(u, v, s, t, T);

[0028] Among them, u, v, s, t are variables for spatial light information, and T is the time information variable;

[0029] Or / and, the camera array includes an entity physical camera array or a virtual digital camera array.

[0030] As the preferred technical measures:

[0031] Step 2: Using the pre-built first frame extraction model, the high-dimensional light field information video is processed to extract the video key frame, and the method for obtaining the first iteration initial value of the compressed light field of the three-dimensional object at that moment is as follows:

[0032] By using the mapping law between the decomposition result of the compressed light field and the three-dimensional object, a high-dimensional light field pre-decomposition unit is constructed to pre-decompose the high-dimensional light field information video to obtain the first iteration initial value of the compressed light field;

[0033] Or, using the mapping law between the spatial light of the high-dimensional light field and the three-dimensional object, a high-dimensional light field reverse transmission unit is constructed to refocus the image on the compressed light field plane to obtain the first iteration initial value of the compressed light field;

[0034] Alternatively, a large-aperture camera is used to record video key frames, and images captured by focusing the camera on different depth planes are used as initial values ​​for the first iteration, wherein the plane focused by the camera corresponds to the information display plane of the compressed light field.

[0035] As the preferred technical measures:

[0036] The method of constructing a high-dimensional light field pre-decomposition unit is as follows:

[0037] Extract a key frame and obtain the RGBD information recorded by the camera array at the corresponding moment, which includes the RGB information and D information of the object point;

[0038] Obtaining coordinate information of array units in the camera array, obtaining position information and resolution information of multiple layers of the compressed light field, and establishing a world coordinate system;

[0039] Based on the world coordinate system, the RGB information of the object point is decomposed into the layer of the compressed light field adjacent to the object point according to the D information of the object point;

[0040] When there are two adjacent layers of compressed light fields around an object point, the decomposition ratio between the two adjacent layers of compressed light fields is inversely proportional to the nth power of the ratio of the distance between the object point and the layer to the distance between the two layers, where n is greater than or equal to 1;

[0041] When there is only one layer of compressed light field adjacent to an object point, the RGB information of the object point is decomposed only on this layer, and the decomposition ratio is inversely proportional to the ratio of the distance between the object point and the layer to the nth power of the distance between the two layers, where n is greater than or equal to 1;

[0042] The RGB values ​​decomposed into each layer of the compressed light field are accumulated according to the weights and the pre-decomposition results are output;

[0043] The pre-decomposition result is used as the iterative initial value of the compressed light field of the three-dimensional object at that moment.

[0044] As the preferred technical measures:

[0045] The method of constructing a high-dimensional light field reverse transmission unit is as follows:

[0046] Obtain a key frame, and based on the key frame, obtain the light field information L(u,v,s,t) recorded by the camera array at the corresponding moment;

[0047] Obtain the position and resolution information of multiple layers of the compressed light field, and construct the compressed light field L'(u',v',s',t');

[0048] There is a certain distance between the light field information L and the compressed light field L', and a world coordinate system XYZ is established, where the X coordinate axis is parallel to the u coordinate axis, and the Y coordinate axis is parallel to the v coordinate axis;

[0049] Reversely transmit the light in the light field information L to each plane of the compressed light field L';

[0050] The pixel value on each plane of the compressed light field is equal to the weighted average of the brightness of all reverse transmission rays intersecting each plane;

[0051] The weighted average value is used as the initial value of the first iteration of the compressed light field of the three-dimensional object at that moment.

[0052] As the preferred technical measures:

[0053] Step 3: Using the pre-built first optimization simulation model and the first iteration initial value to perform iterative optimization, the method for obtaining the first optimization value of the compressed light field at the time corresponding to the video key frame is as follows:

[0054] According to the characteristics of the initial value of the first iteration, select an optimization algorithm from the gradient descent algorithm, the stochastic gradient descent algorithm, the mini-batch stochastic gradient descent method, and the momentum method;

[0055] Based on the optimization algorithm, an optimization iteration target is set, an iterative solver is established, and an iterative solver based on non-negative matrix decomposition or an iterative solver based on high-dimensional linear least squares method is formed to iteratively optimize the initial value of the first iteration;

[0056] The optimization iteration objectives include the signal-to-noise ratio of the image and the number of iterations;

[0057] Using an iterative solver, the optimized values ​​of multiple layers of information of the compressed light field at the time corresponding to the key frame of the video are obtained;

[0058] The optimized value is the optimized value of the transmittance, reflectivity, brightness or polarization parameter of the layer;

[0059] According to the optimized values ​​of transmittance, reflectivity, brightness or polarization parameters of the layer, a first optimized value of the compressed light field is established.

[0060] As the preferred technical measures:

[0061] Step 5, using the pre-built second optimization simulation model, using the second iteration initial value to perform iterative optimization, and obtaining the second optimization value of the compressed light field at the corresponding moment of the non-key frame of the video;

[0062] According to the characteristics of the initial value of the second iteration, an optimization algorithm is selected from the gradient descent algorithm, the stochastic gradient descent algorithm, the mini-batch stochastic gradient descent method, and the momentum method;

[0063] Based on the optimization algorithm and setting the optimization iteration target, an iterative solver is established to form an iterative solver based on non-negative matrix decomposition or an iterative solver based on high-dimensional linear least squares method, which is used to iteratively optimize the second iteration initial value;

[0064] The optimization iteration objectives include the signal-to-noise ratio of the image and the number of iterations;

[0065] Using an iterative solver, the optimized values ​​of multiple layers of information of the compressed light field at the corresponding time of the non-key frame of the video are obtained;

[0066] The optimized value is the optimized value of the transmittance, reflectivity, brightness or polarization parameter of the layer;

[0067] According to the optimized values ​​of transmittance, reflectivity, brightness or polarization parameters of the layer, a second optimized value of the compressed light field is established.

[0068] To achieve one of the above purposes, the second technical solution of the present invention is:

[0069] A fast compression and decomposition method for high-dimensional light field video comprises the following steps:

[0070] Step 1: Perform spatiotemporal sampling on the high-dimensional light field information of the dynamic three-dimensional object to obtain a high-dimensional light field information video of the three-dimensional object;

[0071] Step 2: extract the video key frame to obtain the iterative initial value of the compressed light field of the three-dimensional object at the corresponding moment;

[0072] Step 3: Input the initial value of the iteration into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the key frame;

[0073] Step 4: extract a non-key frame of the video, use the optimized value of the compressed light field of the high-dimensional light field video frame at the nearest time to a non-key frame as the iterative initial value of the compressed light field of the non-key frame, input the initial value into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the non-key frame;

[0074] Step 5: Combining the optimized values ​​of the compressed light field in step 3 and step 4 into a compressed light field video of a three-dimensional object in a time series for output and display.

[0075] After continuous exploration and experiments, the present invention uses the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field, so that the correlation between the video key frame and the non-key frame can be used to quickly obtain the iteration initial value of the non-key frame, thereby greatly reducing the number of iterations of the light field video, improving the compression decomposition speed of the dynamic compressed light field video information, and further greatly improving the compression decomposition efficiency of the compressed light field video information, and achieving higher resolution and compression decomposition of more complex objects.

[0076] To achieve one of the above purposes, the third technical solution of the present invention is:

[0077] A fast compression and decomposition system for high-dimensional light field video, comprising a light field acquisition module, a first frame extraction module, a first optimization simulation module, a second frame extraction module, a second optimization simulation module and an object generation module;

[0078] The light field acquisition module performs spatiotemporal sampling on the high-dimensional light field information of the three-dimensional object to obtain the high-dimensional light field information video of the three-dimensional object;

[0079] A first frame extraction module processes the high-dimensional light field information video, extracts the video key frame, and obtains the first iteration initial value of the compressed light field of the three-dimensional object;

[0080] A first optimization simulation module performs iterative optimization using a first iteration initial value to obtain a first optimization value of the compressed light field at a time corresponding to a key frame of the video;

[0081] A second frame extraction module extracts a non-key frame of the video, and uses a first optimized value of a key frame of the video that is the nearest neighbor to the non-key frame of the video as a second iteration initial value of the compressed light field;

[0082] A second optimization simulation module performs iterative optimization using a second iteration initial value to obtain a second optimization value of the compressed light field of a non-key frame of the video;

[0083] The object generation module combines the first optimization value and the second optimization value into a compressed light field video of a three-dimensional object according to a time series, thereby completing the rapid compression and decomposition of the high-dimensional light field video.

[0084] After continuous exploration and experiments, the present invention sets a light field acquisition module, a first frame extraction module, a first optimization simulation module, a second frame extraction module, a second optimization simulation module and an object generation module, and uses the first optimization value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field, so that the correlation between the video key frame and the non-key frame can be used to quickly obtain the iteration initial value of the non-key frame, thereby greatly reducing the number of iterations of the light field video, improving the compression decomposition speed of the dynamic compressed light field video information, and further greatly improving the compression decomposition efficiency of the compressed light field video information, and can achieve higher resolution and compression decomposition of more complex objects.

[0085] To achieve one of the above purposes, the fourth technical solution of the present invention is:

[0086] An electronic device comprising:

[0087] one or more processors;

[0088] A storage device for storing one or more programs;

[0089] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for fast compression and decomposition of high-dimensional light field video.

[0090] To achieve one of the above purposes, the fifth technical solution of the present invention is:

[0091] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for rapid compression and decomposition of a high-dimensional light field video.

[0092] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0093] After continuous exploration and experiments, the present invention uses the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field, so that the correlation between the video key frame and the non-key frame can be used to quickly obtain the iteration initial value of the non-key frame, thereby greatly reducing the number of iterations of the light field video, improving the compression decomposition speed of the dynamic compressed light field video information, and further greatly improving the compression decomposition efficiency of the compressed light field video information, and achieving higher resolution and compression decomposition of more complex objects.

[0094] Furthermore, the present invention overcomes the limitations of traditional iterative optimization methods that they easily converge to local optimal solutions and have low signal-to-noise ratios of displayed results by constructing a high-dimensional light field pre-decomposition unit and / or a high-dimensional light field reverse transmission unit to calculate the iteration initial value, thereby achieving high-quality compressed decomposition of high-dimensional light fields.

[0095] The present invention further solves the problems of large number of iterations and long optimization time in traditional iterative optimization methods by constructing a high-dimensional light field pre-decomposition unit and / or a high-dimensional light field reverse transmission unit to calculate the iteration initial value, and realizes the rapid compression decomposition of the high-dimensional light field. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A schematic diagram of a process of performing rapid compression decomposition using a light field pre-decomposition method according to the present invention;

[0097] Figure 2 A schematic diagram of the light field pre-decomposition method of the present invention;

[0098] Figure 3 A schematic diagram of a process of performing rapid compression and decomposition by a reverse transmission method according to the present invention;

[0099] Figure 4 A schematic diagram of the reverse transmission method of the present invention;

[0100] Figure 5 A schematic diagram of a process of performing rapid compression and decomposition using a camera multi-plane focusing method according to the present invention;

[0101] Figure 6 A schematic diagram of the multi-plane focusing method of the camera of the present invention. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0103] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0104] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0105] The first specific embodiment of the method for rapid compression and decomposition of high-dimensional light field video of the present invention:

[0106] A fast compression and decomposition method for high-dimensional light field video comprises the following steps:

[0107] Step 1: Using a pre-built light field acquisition model, high-dimensional light field information of the three-dimensional object is temporally and spatially sampled to obtain a high-dimensional light field information video of the three-dimensional object;

[0108] Step 2: Using a pre-built first frame extraction model, the high-dimensional light field information video is processed to extract the video key frame and obtain the first iteration initial value of the compressed light field of the three-dimensional object at the corresponding moment;

[0109] Step 3, using a pre-built first optimization simulation model and a first iteration initial value to perform iterative optimization to obtain a first optimization value of the compressed light field at a time corresponding to a key frame of the video;

[0110] Step 4, using the pre-built second frame extraction model to extract the video non-key frame, and taking the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field;

[0111] Step 5, using the pre-built second optimization simulation model, using the second iteration initial value to perform iterative optimization, and obtaining the second optimization value of the compressed light field at the corresponding moment of the non-key frame of the video;

[0112] Step six, using the pre-built object generation model, the first optimization value and the second optimization value are combined into a compressed light field video of the three-dimensional object in a time series, thereby completing the rapid compression decomposition of the high-dimensional light field video.

[0113] A second specific embodiment of the method for rapid compression and decomposition of high-dimensional light field video of the present invention:

[0114] A fast compression and decomposition method for high-dimensional light field video comprises the following steps:

[0115] Step 1: Perform spatiotemporal sampling on the high-dimensional light field information of the dynamic three-dimensional object to obtain a high-dimensional light field information video of the three-dimensional object;

[0116] Step 2: extract the video key frame to obtain the iterative initial value of the compressed light field of the three-dimensional object at that moment;

[0117] Step 3: Input the initial value of the iteration into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the key frame.

[0118] Step 4: extract non-key frames from the video, use the optimized value of the compressed light field of the high-dimensional light field video frame at the nearest time to a non-key frame as the iterative initial value of the compressed light field of the frame, input the initial value into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the non-key frame.

[0119] Step 5: Combining the optimized values ​​of the compressed light field in step 3 and step 4 into a compressed light field video of a three-dimensional object in a time series for output and display.

[0120] In this embodiment, the spatiotemporal sampling described in step 1 is the sampling of a 5-dimensional light field function describing a dynamic 3D object. The 5-dimensional light field function is described as L(u, v, s, t, T), which includes the sampling of spatial light information variables (u, v, s, t) and the sampling of time information variables T.

[0121] The spatiotemporal sampling process can be obtained by using a camera array to shoot a video of a dynamic three-dimensional object. The position coordinates of each camera unit in the camera array correspond to the (u, v) dimension of the spatial light information variable. The pixel coordinates of the video shot by the camera array correspond to the (s, t) dimension. The shooting time of all cameras in the camera array is synchronized, corresponding to the variable T.

[0122] The camera array includes a physical camera array and a virtual digital camera array. The cameras in the array preferably record the color image information and depth information of the three-dimensional object, that is, the information RGBD. The cameras in the array record the color image information of the three-dimensional object and obtain the depth information of the three-dimensional object through calculation.

[0123] In this embodiment, the method for obtaining the iterative initial value of the compressed light field of the three-dimensional object at this moment described in step 2 includes: (1) a pre-decomposition method of a high-dimensional light field, (2) a reverse transmission method of a high-dimensional light field, and (3) a camera multi-plane focusing method.

[0124] (1) The high-dimensional light field pre-decomposition method includes the following contents:

[0125] The high-dimensional light field is pre-decomposed by using the mapping law between the decomposition result of the compressed light field and the three-dimensional object, including the following process:

[0126] Input the RGBD information recorded by the camera array at the time corresponding to a key frame.

[0127] Input the coordinate information of the array element in the camera array.

[0128] Input position and resolution information for multiple layers of a compressed light field.

[0129] A world coordinate system is established, and the RGB information of the object point recorded by each camera is decomposed into the compressed light field layer adjacent to the object point according to the D information of the object point. When there are two adjacent compressed light field layers around the object point, the decomposition ratio between the two adjacent compressed light field layers is inversely proportional to the nth power of the ratio of the distance between the object point and the layer to the spacing between the two layers, where n is greater than or equal to 1. When there is only one adjacent layer around the object point, the RGB information of the object point is only decomposed on this one layer, and its decomposition ratio is inversely proportional to the nth power of the ratio of the distance between the object point and the layer to the spacing between the two layers, where n is greater than or equal to 1.

[0130] The RGB values ​​decomposed into each compressed light field layer are added up according to the weights and then the pre-decomposition result is output. The pre-decomposition result is the iterative initial value of the compressed light field of the three-dimensional object at that moment. (2) The reverse transmission method of the high-dimensional light field includes the following contents:

[0131] The mapping law between the spatial light of the high-dimensional light field and the three-dimensional object is used to refocus the image on the compressed light field plane. The following process is included:

[0132] Input the light field information L(u,v,s,t) recorded by the light field camera at the time corresponding to a key frame.

[0133] The position and resolution information of multiple layers of the compressed light field are input to construct the compressed light field L'(u',v',s',t').

[0134] Set the distance between light fields L and L', and establish a world coordinate system (XYZ), where the X coordinate axis is parallel to the u coordinate axis, and the Y coordinate axis is parallel to the v coordinate axis. Reversely transmit the light in the light field L to each plane of the compressed light field L'. The pixel value on each plane of the compressed light field is equal to the weighted average of the brightness of all reversely transmitted light rays intersecting each plane. The weighted average is the iterative initial value of the compressed light field of the three-dimensional object at that moment.

[0135] (3) The camera multi-plane focusing method includes the following contents:

[0136] An image captured by a large-aperture camera focused on different depth planes is used as an iteration initial value, and the plane focused by the camera corresponds to the information display plane of the compressed light field. In this embodiment, the iterative solver described in step 3 includes an iterative solver based on non-negative matrix decomposition and an iterative solver based on high-dimensional linear least squares. The solver can use an optimization algorithm, and the optimization algorithm includes a gradient descent algorithm, a stochastic gradient descent algorithm, a small batch stochastic gradient descent method, a momentum method, etc.

[0137] In this embodiment, the optimized values ​​of the compressed light field described in step 3 and step 4 include optimized values ​​of multiple layer information of the compressed light field. The physical meaning of the optimized values ​​of the layer information may be the optimized values ​​of the transmittance, reflectivity, brightness or polarization parameters of the layer. The layer of the compressed light field is greater than or equal to 2 layers.

[0138] In this embodiment, the optimization value of the compressed light field described in step 3 and step 4 is determined by a set optimization iteration target, and the optimization iteration target includes the signal-to-noise ratio of the image and the number of iterations.

[0139] In this embodiment, the compressed light field video described in step 5 includes a video generated in real time through the process of steps 1 to 4 and a video file generated offline through the process of steps 1 to 4. The present invention uses pre-decomposition technology and / or reverse transmission technology to calculate the initial value of iteration, overcomes the limitations of traditional iterative optimization methods that are easy to converge to a local optimal solution and the signal-to-noise ratio of the displayed results is not high, and realizes high-quality compression decomposition of high-dimensional light fields.

[0140] The present invention adopts pre-decomposition technology and / or reverse transmission technology to calculate the initial value of iteration, solves the problems of large number of iterations and long optimization time in traditional iterative optimization methods, and realizes rapid compression and decomposition of high-dimensional light fields.

[0141] The present invention optimizes the initial value by utilizing the correlation of the compressed light field between non-key frames, greatly reduces the number of iterations of the light field video, and improves the compression decomposition speed of the dynamic compressed light field video information.

[0142] The present invention greatly improves the compression decomposition efficiency of compressed light field video information, and can achieve compression decomposition of higher resolution and more complex objects.

[0143] See also Figure 1 , Figure 2 , the third specific embodiment of the method for rapid compression and decomposition of high-dimensional light field video of the present invention:

[0144] A fast compression and decomposition method for high-dimensional light field video, comprising the following contents:

[0145] First, the camera array is used to collect the 5-dimensional light field function L(u,v,s,t,T) of the dynamic 3D object to complete the spatiotemporal sampling of the high-dimensional light field. Then, the key frames of the high-dimensional light field video are set, and the key frames and non-key frames are processed separately. For the key frames, the light field pre-decomposition method is used to obtain the iterative initial value of the decompressed light field, and then the iterative initial value and the iterative solver based on non-negative matrix decomposition are used for iterative optimization. The optimization method uses the stochastic gradient descent algorithm to obtain the optimized value of the compressed light field of the key frame. For non-key frames, the optimized value of the compressed light field of its nearest neighbor key frame is used as the initial value. Then, the initial value and the iterative solver based on non-negative matrix decomposition are used for iterative optimization. The optimization method uses the momentum method to obtain the optimized value of the compressed light field of the non-key frame. Finally, the optimized values ​​of the compressed light field of the key frames and non-key frames are combined into a compressed light field video of the 3D object according to the time series for output and display.

[0146] like Figure 2 The light field pre-decomposition method in this embodiment is shown as a schematic diagram. First, a world coordinate system XYZ is established, in which the dotted hexagon represents the three-dimensional scene to be imaged. The light field camera is located in the Z=0 plane, and each camera unit x on the plane i , i∈(1,……,n) records the RGBD information of a certain perspective of the three-dimensional scene at that moment. The compressed light field is composed of two display layers, where the first display layer is located in the Z=l plane, and the coordinates on this plane are represented by (u',v'), and the second display layer is located in the Z=l+h plane, and the coordinates on this plane are represented by (s',t'). Taking the two-dimensional light field coordinates as an example, for the pre-resolved values ​​f(u' i ) and f(s' i ) are:

[0147]

[0148] Among them I xi (u i ') indicates the xth i The camera records the first display layer u' i The brightness value of the object point corresponding to the coordinate, z xi (u i ') represents the depth value of the object point. xi (s i ') indicates the xth i The camera records the second display layer s' i The brightness value of the object point corresponding to the coordinate, z xi (s i ') represents the depth value of the object point.

[0149] Taking the two-dimensional light field coordinates as an example, in another implementation of this embodiment, the pre-resolved values ​​f(u' i ) and f(s' i ) are defined as:

[0150]

[0151] See also Figure 3 , Figure 4 , the fourth specific embodiment of the method for rapid compression and decomposition of high-dimensional light field video of the present invention:

[0152] A fast compression and decomposition method for high-dimensional light field video, comprising the following contents:

[0153] First, a light field camera is used to collect the 5-dimensional light field function L(u,v,s,t,T) of a dynamic 3D object to complete the spatiotemporal sampling of the high-dimensional light field. Then, the key frames of the high-dimensional light field video are set, and the key frames and non-key frames are processed separately. For the key frames, the reverse transmission method of the high-dimensional light field is used to obtain the iterative initial value of the decompressed light field, and then the iterative initial value and the iterative solver based on the high-dimensional linear least squares method are used for iterative optimization. The optimization method uses the gradient descent algorithm to obtain the optimized value of the compressed light field of the key frame. For non-key frames, the optimized value of the compressed light field of its nearest neighboring key frame is used as the initial value. Then, the initial value and the iterative solver based on the high-dimensional linear least squares method are used for iterative optimization. The optimization method uses the small batch stochastic gradient descent method to obtain the optimized value of the compressed light field of the non-key frame. Finally, the optimized values ​​of the compressed light field of the key frames and non-key frames are combined into a compressed light field video of the 3D object according to the time series for output and display.

[0154] like Figure 4 The figure shows a schematic diagram of the reverse transmission method of the high-dimensional light field in this embodiment. First, a world coordinate system XYZ is established. The light field information recorded by the light field camera corresponding to the key frame at a certain moment is L(u, v, s, t). The compressed light field to be constructed is L'(u', v', s', t'). The distance Z between the light fields L and L' is set 1 and Z 2 , the light in the light field L is transmitted back to each plane of the compressed light field L' through the light transmission matrix. The pixel value on each plane of the compressed light field is equal to the weighted average of the brightness of all the reversely transmitted light rays intersecting each plane. The weighted average is the iterative initial value of the compressed light field of the three-dimensional object at that moment:

[0155]

[0156] Where L i (s,t) and L i(u,v) is the brightness of the light recorded by the i-th camera that is transmitted to the same pixel on the (s,t) or (u,v) plane.

[0157] See also Figure 5 , Figure 6 , the fifth specific embodiment of the method for rapid compression and decomposition of high-dimensional light field video of the present invention:

[0158] A fast compression and decomposition method for high-dimensional light field video, comprising the following contents:

[0159] First, the camera array is used to collect the 5-dimensional light field function L(u,v,s,t,T) of the dynamic 3D object to complete the spatiotemporal sampling of the high-dimensional light field. Then, the key frames of the high-dimensional light field video are set, and the key frames and non-key frames are processed separately. For the key frames, the camera multi-plane focusing method is used to obtain the iterative initial value of the decompressed light field, and then the iterative initial value and the iterative solver based on non-negative matrix decomposition are used for iterative optimization. The optimization method uses the gradient descent algorithm to obtain the optimized value of the compressed light field of the key frame. For non-key frames, the optimized value of the compressed light field of its nearest neighbor key frame is used as the initial value. Then, the initial value and the iterative solver based on non-negative matrix decomposition are used for iterative optimization. The optimization method uses the small batch stochastic gradient descent method to obtain the optimized value of the compressed light field of the non-key frame. Finally, the optimized values ​​of the compressed light field of the key frames and non-key frames are combined into a compressed light field video of the 3D object according to the time series for output display.

[0160] like Figure 6 FIG. 1 is a schematic diagram of the camera multi-plane focusing method in this embodiment. Figure 6 As shown, the camera takes a picture of a three-dimensional image. Focus the camera on the information plane (s', t') of the compressed light field and take a picture, which is used as the initial value on the (s', t') information plane. Focus the camera on the information plane (u', v') of the compressed light field and take another picture, which is used as the initial value on the (u', v') information plane.

[0161] An embodiment of a device applying the method of the present invention:

[0162] An electronic device comprising:

[0163] one or more processors;

[0164] A storage device for storing one or more programs;

[0165] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for fast compression and decomposition of high-dimensional light field video.

[0166] A computer medium embodiment using the method of the present invention:

[0167] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for rapid compression and decomposition of a high-dimensional light field video.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, and computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

[0169] The present application is described by flowcharts or / and block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process or / and block in the flowchart or / and block diagram and the combination of the processes or / and blocks in the flowchart or / and block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart or / and block diagram. Figure 1 Process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] The model in this application is an object that objectively describes the morphological structure with the help of physical or virtual representation. The object is not equal to the object and is not limited to physical and virtual. It can be a data processing function, software program, processing mode, usage method, operation method, workflow, application process, electronic hardware, circuit module, processing system, system imitation or simulation object.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field can still modify or replace the specific implementation methods of the present invention with equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A fast compression and decomposition method for high-dimensional light field video, characterized in that: The following steps are involved: Step 1: Using a pre-built light field acquisition model, high-dimensional light field information of the three-dimensional object is temporally and spatially sampled to obtain a high-dimensional light field information video of the three-dimensional object; Step 2: Using a pre-built first frame extraction model, the high-dimensional light field information video is processed to extract the video key frame and obtain the first iteration initial value of the compressed light field of the three-dimensional object at the corresponding moment; Step 3, using a pre-built first optimization simulation model and a first iteration initial value to perform iterative optimization to obtain a first optimization value of the compressed light field at a time corresponding to a key frame of the video; Step 4, using the pre-built second frame extraction model to extract the video non-key frame, and taking the first optimized value of the video key frame that is the nearest neighbor to the video non-key frame as the second iteration initial value of the compressed light field; Step 5, using the pre-built second optimization simulation model, using the second iteration initial value to perform iterative optimization, and obtaining the second optimization value of the compressed light field at the corresponding moment of the non-key frame of the video; Step six, using the pre-built object generation model, the first optimization value and the second optimization value are combined into a compressed light field video of the three-dimensional object in a time series, thereby completing the rapid compression decomposition of the high-dimensional light field video.

2. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 1, characterized in that: Step 1: Using a pre-built light field acquisition model, high-dimensional light field information of a three-dimensional object is sampled in time and space to obtain a high-dimensional light field information video of the three-dimensional object as follows: Get the dimension information (u, v) of the spatial light information variable; According to the dimension information (u, v), set the position coordinates of each camera unit; Establishing a camera array according to the position coordinates of each camera unit; The dynamic three-dimensional object is photographed by a camera array, and time-space sampling is performed to obtain color image information of the three-dimensional object; Based on the color image information of the three-dimensional object, depth information of the three-dimensional object is calculated; According to the depth information, a high-dimensional light field information video is obtained.

3. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 2, characterized in that: Spatiotemporal sampling is the sampling of the 5-dimensional light field function that describes a 3D object; The 5-dimensional light field function is L(u, v, s, t, T); Among them, u, v, s, t are variables for spatial light information, and T is the time information variable; Or / and, the camera array includes an entity physical camera array or a virtual digital camera array.

4. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 1, characterized in that: Step 2: Using the pre-built first frame extraction model, the high-dimensional light field information video is processed to extract the video key frame, and the method for obtaining the first iteration initial value of the compressed light field of the three-dimensional object at that moment is as follows: By using the mapping law between the decomposition result of the compressed light field and the three-dimensional object, a high-dimensional light field pre-decomposition unit is constructed to pre-decompose the high-dimensional light field information video to obtain the first iteration initial value of the compressed light field; Or, using the mapping law between the spatial light of the high-dimensional light field and the three-dimensional object, a high-dimensional light field reverse transmission unit is constructed to refocus the image on the compressed light field plane to obtain the first iteration initial value of the compressed light field; Alternatively, a large-aperture camera is used to record video key frames, and images captured by focusing the camera on different depth planes are used as initial values ​​for the first iteration, wherein the plane focused by the camera corresponds to the information display plane of the compressed light field.

5. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 4, characterized in that: The method of constructing a high-dimensional light field pre-decomposition unit is as follows: Extract a key frame and obtain the RGBD information recorded by the camera array at the corresponding moment, which includes the RGB information and D information of the object point; Obtaining coordinate information of array units in the camera array, obtaining position information and resolution information of multiple layers of the compressed light field, and establishing a world coordinate system; Based on the world coordinate system, the RGB information of the object point is decomposed into the layer of the compressed light field adjacent to the object point according to the D information of the object point; When there are two adjacent layers of compressed light fields around an object point, the decomposition ratio between the two adjacent layers of compressed light fields is inversely proportional to the nth power of the ratio of the distance between the object point and the layer to the distance between the two layers, where n is greater than or equal to 1; When there is only one layer of compressed light field adjacent to an object point, the RGB information of the object point is decomposed only on this layer, and the decomposition ratio is inversely proportional to the ratio of the distance between the object point and the layer to the nth power of the distance between the two layers, where n is greater than or equal to 1; The RGB values ​​decomposed into each layer of the compressed light field are accumulated according to the weights and the pre-decomposition results are output; The pre-decomposition result is used as the iterative initial value of the compressed light field of the three-dimensional object at that moment.

6. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 4, characterized in that: The method of constructing a high-dimensional light field reverse transmission unit is as follows: Obtain a key frame, and based on the key frame, obtain the light field information L(u,v,s,t) recorded by the camera array at the corresponding moment; Obtain the position and resolution information of multiple layers of the compressed light field, and construct the compressed light field L'(u',v',s',t'); There is a certain distance between the light field information L and the compressed light field L', and the world coordinate system XYZ is established. The X coordinate axis is parallel to the u coordinate axis, and the Y coordinate axis is parallel to the v coordinate axis; Reversely transmit the light in the light field information L to each plane of the compressed light field L'; The pixel value on each plane of the compressed light field is equal to the weighted average of the brightness of all reverse transmission rays intersecting each plane; The weighted average value is used as the initial value of the first iteration of the compressed light field of the three-dimensional object at that moment.

7. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 1, characterized in that: Step 3: Using the pre-built first optimization simulation model and the first iteration initial value to perform iterative optimization, the method for obtaining the first optimization value of the compressed light field at the time corresponding to the video key frame is as follows: According to the characteristics of the initial value of the first iteration, select an optimization algorithm from the gradient descent algorithm, the stochastic gradient descent algorithm, the mini-batch stochastic gradient descent method, and the momentum method; Based on the optimization algorithm, an optimization iteration target is set, an iterative solver is established, and an iterative solver based on non-negative matrix decomposition or an iterative solver based on high-dimensional linear least squares method is formed to iteratively optimize the initial value of the first iteration; The optimization iteration objectives include the signal-to-noise ratio of the image and the number of iterations; Using an iterative solver, the optimized values ​​of multiple layers of information of the compressed light field at the time corresponding to the key frame of the video are obtained; The optimized value is the optimized value of the transmittance, reflectivity, brightness or polarization parameter of the layer; According to the optimized values ​​of transmittance, reflectivity, brightness or polarization parameters of the layer, a first optimized value of the compressed light field is established.

8. The method for rapid compression and decomposition of a high-dimensional light field video according to claim 1, characterized in that: Step 5, using the pre-built second optimization simulation model, using the second iteration initial value to perform iterative optimization, and obtaining the second optimization value of the compressed light field at the corresponding moment of the non-key frame of the video; According to the characteristics of the initial value of the second iteration, an optimization algorithm is selected from the gradient descent algorithm, the stochastic gradient descent algorithm, the mini-batch stochastic gradient descent method, and the momentum method; Based on the optimization algorithm and setting the optimization iteration target, an iterative solver is established to form an iterative solver based on non-negative matrix decomposition or an iterative solver based on high-dimensional linear least squares method, which is used to iteratively optimize the second iteration initial value; The optimization iteration objectives include the signal-to-noise ratio of the image and the number of iterations; Using an iterative solver, the optimized values ​​of multiple layers of information of the compressed light field at the corresponding time of the non-key frame of the video are obtained; The optimized value is the optimized value of the transmittance, reflectivity, brightness or polarization parameter of the layer; According to the optimized values ​​of transmittance, reflectivity, brightness or polarization parameters of the layer, a second optimized value of the compressed light field is established.

9. A method for rapid compression and decomposition of high-dimensional light field video, characterized in that: The steps include: Step 1: Perform spatiotemporal sampling on the high-dimensional light field information of the dynamic three-dimensional object to obtain a high-dimensional light field information video of the three-dimensional object; Step 2: extract the video key frame to obtain the iterative initial value of the compressed light field of the three-dimensional object at the corresponding moment; Step 3: Input the initial value of the iteration into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the key frame; Step 4: extract a non-key frame of the video, use the optimized value of the compressed light field of the high-dimensional light field video frame at the nearest time to a non-key frame as the iterative initial value of the compressed light field of the non-key frame, input the initial value into the iterative solver for iterative optimization, and obtain the optimized value of the compressed light field of the three-dimensional object at the time corresponding to the non-key frame; Step 5: Combining the optimized values ​​of the compressed light field in step 3 and step 4 into a compressed light field video of a three-dimensional object in a time series for output and display.

10. A fast compression and decomposition system for high-dimensional light field video, characterized in that: It includes a light field acquisition module, a first frame extraction module, a first optimization simulation module, a second frame extraction module, a second optimization simulation module and an object generation module; The light field acquisition module performs spatiotemporal sampling on the high-dimensional light field information of the three-dimensional object to obtain the high-dimensional light field information video of the three-dimensional object; A first frame extraction module processes the high-dimensional light field information video, extracts the video key frame, and obtains the first iteration initial value of the compressed light field of the three-dimensional object; A first optimization simulation module performs iterative optimization using a first iteration initial value to obtain a first optimization value of the compressed light field at a time corresponding to a key frame of the video; A second frame extraction module extracts a non-key frame of the video, and uses a first optimized value of a key frame of the video that is the nearest neighbor to the non-key frame of the video as a second iteration initial value of the compressed light field; A second optimization simulation module performs iterative optimization using a second iteration initial value to obtain a second optimization value of the compressed light field of a non-key frame of the video; The object generation module combines the first optimization value and the second optimization value into a compressed light field video of a three-dimensional object according to a time series, thereby completing the rapid compression and decomposition of the high-dimensional light field video.

Images