A light field stitching method and terminal based on grid deformation and low-order approximation

Abstract

The present invention provides a light field stitching method based on grid deformation and low-order approximation, including: obtaining a depth map of the light field to be stitched; dividing the central sub-aperture image into foreground and background according to the depth map; matching the background of the left and right light fields after extracting feature points; stitching the central sub-aperture image based on the traditional two-dimensional image; using the control point guided network deformation method to calibrate two different light fields to a panoramic image; using the low-rank features of the light field, without using four-dimensional graph cuts, finding the stitching line of the calibrated light field and fusing it. The present invention uses control point guided grid deformation technology to calibrate two different light fields to a panoramic image, and applies low-rank approximation to directly use traditional two-dimensional graph cuts instead of four-dimensional graph cuts to reduce the time complexity of the method. Compared with other light field stitching algorithms, it has better stitching quality, angle domain consistency and lower time complexity.

Description

Technical Field

[0001] The present invention relates to the technical field of light field splicing, and in particular to a light field splicing method and a terminal based on grid deformation and low-order approximation. Background Art

[0002] Light field imaging is a technology that has developed rapidly in recent years. Compared with traditional imaging, light field imaging can obtain richer scene information. Unlike traditional photography, traditional cameras only capture the two-dimensional projection of light in space, while light field imaging can collect the radiation of light from all directions and obtain the angle information lost in traditional photography. On the one hand, this higher-dimensional data manifests itself as a powerful ability to analyze scenes, greatly improving the performance of traditional computer vision problems; on the other hand, the high-dimensional characteristics of light fields also bring new challenges to data capture, data compression, content editing, and light field stitching.

[0003] At present, light field imaging is limited by the field of view. For example, the field of view of the commercial Lytro Illum is only 43.5° horizontally and 62° vertically. One of the potential applications of light field is to produce videos related to virtual reality (VR). As we all know, VR videos require 360° panoramic images. If we want to apply light field to VR-related fields, it is very important to expand the field of view of the light field camera. One way to expand the field of view is to improve the hardware, but this method is too expensive and difficult to promote. Therefore, we often increase the field of view of the light field by stitching multiple light fields with different perspectives.

[0004] Existing light field stitching methods can be divided into two categories: 2D stitching methods derived from traditional images and 4D methods specifically for light field stitching. Traditional stitching methods usually estimate the best global transformation of the entire image, which can obtain visually acceptable stitching results when the camera translation is negligible or the scene is close to a plane. However, traditional stitching methods may introduce ghosting effects, misalignment and local distortion to images with large parallax. And 4D stitching methods specifically for light fields often increase the amount of computation due to the use of 4D graph cuts in the graph cut process. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a light field splicing method and terminal based on grid deformation and low-order approximation.

[0006] According to one aspect of the present invention, there is provided a light field stitching method based on grid deformation and low-order approximation, comprising: obtaining a depth map of a light field to be stitched;

[0007] Dividing the central sub-aperture image into a foreground and a background according to the depth map;

[0008] After extracting the feature points, the backgrounds of the left and right light fields are matched;

[0009] Based on the traditional two-dimensional image, stitching the central sub-aperture image;

[0010] Determine the control points of the stitched central sub-aperture image;

[0011] Based on the control points, calibrate two different light fields onto the panorama using a control point guided network deformation method;

[0012] Using the low-rank features of the light field, we find the seams of the calibrated light fields and fuse them without using four-dimensional graph cuts.

[0013] Preferably, the background of the left and right light fields is matched using feature points of a scale-invariant feature transformation based on the depth map, and the feature points are matched only in points whose depth is less than a threshold value T in the central sub-aperture image.

[0014] Preferably, natural image stitching (NISwGSP) is used to stitch the central sub-aperture images.

[0015] Preferably, the determining of the control points comprises: re-dividing the panoramic image after stitching the central sub-aperture images into equidistant grids, and using the nodes of the grids as control points.

[0016] Preferably, the determining of the control point comprises:

[0017] The panoramic image composed of two central sub-aperture images is divided into equal intervals to obtain grid nodes, which are expressed as {c i，j};

[0018] Each {c i，j} are on the deformed grids of the two central subaperture images;

[0019] The deformed mesh is labeled as {g i，j}, where i represents the index number of the deformed grid and j is used to distinguish the two light fields;

[0020] Each deformed grid can calculate a homography matrix H i，j ;

[0021] The control points of the left light SAI and the right light field SAI {p u，v，i，j} is the mesh vertex {c i，j}Calculated;

[0022]

[0023] where u c and v care the angular domain coordinates of the central sub-aperture respectively.

[0024] Preferably, the method of calibrating two different light fields onto the panoramic image using a control point guided grid deformation method comprises:

[0025] The remaining SAIs to be spliced ​​are divided into uniform grids;

[0026] Then, by corresponding to the control point {p u，v，i，j}Guiding the uniform grid to deform.

[0027] Preferably, the energy function includes a disparity term; the disparity term enables each SAI grid to maintain the disparity on the central sub-aperture after deformation and conform to correct depth information.

[0028] Preferably, the energy function comprises a similarity term; the similarity term propagates the disparity constraint from the overlapping region to the non-overlapping region.

[0029] Preferably, the method of using the low-rank features of the light field to find and fuse the stitching lines of the calibrated light field without using four-dimensional graph cuts includes:

[0030] Calibrate all SAIs in each light field to the position of the central viewpoint;

[0031] The calibrated light field is combined into a matrix in the form of column vectors by using the low-rank property of the light field;

[0032] Decompose the matrix using singular value decomposition (SVD) or orthogonal triangular decomposition (QR) to obtain

[0033] I=U·W·V=B·C

[0034] Due to the low-rank property of the light field, each column of B is considered as a basis image of the central subaperture image;

[0035] Use the stitching lines previously calculated when stitching the central sub-aperture image to stitch each column of the B matrix of the left and right light fields. The stitched B matrix is ​​denoted as B s ,

[0036] I s =B s C1

[0037] Among them I s is the spliced ​​light field, C1 is the C matrix of the left light field;

[0038] Take out I s Each column of gets the sub-aperture image of the complete light field after stitching.

[0039] According to a second aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above method when executing the program.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The light field stitching method based on grid deformation and low-order approximation provided by the embodiment of the present invention achieves good image quality by guiding the grid deformation through control points, and the root mean square error correlation coefficient (CRMSE) of the image is better than other existing technologies;

[0042] The light field stitching method based on grid deformation and low-order approximation provided by the embodiment of the present invention can effectively maintain the angular domain continuity of the light field, which can be verified by observing the epipolar plane image of the stitched light field.

[0043] The light field stitching method based on grid deformation and low-order approximation and the method of stitching lines of central sub-apertures based on low-rank multiplexing of light fields provided in the embodiments of the present invention only need to use the graph cut method to calculate the stitching lines of the central sub-apertures, which significantly reduces the amount of calculation and does not involve complex and time-consuming four-dimensional graph cut calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0045] Figure 1 A flowchart of a light field stitching method based on grid deformation and low-order approximation according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of obtaining control points according to a preferred embodiment of the present invention;

[0047] Figure 3 A schematic diagram of grid calibration according to a preferred embodiment of the present invention;

[0048] Figure 4 The original image of the data set used in an application example of the present invention;

[0049] Figure 5 It is a subjective effect diagram of stitching quality and an EPI image in an application example of the present invention;

[0050] Figure 6 This is a diagram of the refocusing effect in an application example of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0052] S1, obtain the depth map of the light field to be spliced;

[0053] S2, dividing the central sub-aperture image into foreground and background according to the depth map, extracting feature points, and matching the background of the left and right light fields;

[0054] S3, using the extracted feature points and based on a traditional two-dimensional image stitching method, stitching the central sub-aperture image;

[0055] S4, determining control points of the stitched central sub-aperture image; based on the control points, using a control point guided network deformation method to calibrate two different light fields onto the panoramic image (the stitched central sub-aperture image);

[0056] S5, uses the low-rank features of the light field to find and fuse the seams of the calibrated light field without using four-dimensional graph cuts.

[0057] As a preferred embodiment, the currently more mature light field depth method EPINET is used to obtain the depth map of the light field to be spliced;

[0058] As a preferred embodiment, the scale-invariant feature transform (SIFT) method is used to extract feature points, and then the backgrounds of the left and right light fields are matched. Further, the feature points are matched among the points whose depth is less than a threshold value T in the central sub-aperture image.

[0059] As a preferred embodiment, NISwGSP is used to stitch the central sub-aperture image (SAI), and the control point of the central viewpoint can be obtained through the stitched central SAI, and the central control point can be pushed to each other viewpoint through the disparity map. This method adopts a local distortion model and uses a grid to guide the deformation of each image. In addition to good alignment and minimal local distortion, it also increases the global similarity of the energy function. The energy function constrains the distortion of each image well. In addition, since this embodiment also adopts grid-based deformation, NISwGSP is very suitable for application in the preprocessing stage.

[0060] The present invention provides a preferred embodiment to perform S14.

[0061] First, a certain number of control points are obtained. Generally speaking, the panorama after stitching the central sub-aperture images is re-divided into equidistant grids, and the nodes of the grids are used as control points.

[0062] Specifically, Figure 2 As shown, it is the result of stitching two light field central sub-aperture images (SAI) using NISwGSP in S13. Since NISwGSP is also a panorama produced based on mesh deformation, the blue and red dots in the figure represent the mesh vertices obtained after NIwGSP calibrates the two central sub-aperture images. Figure 2 A typical panorama calibrated by NISwGSP grid deformation is shown. The panorama is re-divided into equally spaced grids. The green dots represent the grid nodes that divide the panorama equally, which are represented as {c i，j}.

[0063] It can be seen that the green mesh nodes must be on a red or blue deformed mesh. The red or blue deformed mesh is marked as {g i，j}, where i represents the index number of the deformed grid, and j is used to distinguish the two light fields. Each deformed grid can calculate a homography matrix, and the corresponding homography matrix is ​​expressed as H i，j For each green point, we can find the blue or red grid where it is located and use the homography matrix H corresponding to that grid i，j Reverse warp it to the original central viewpoint SAI and find its coordinates in the original light field.

[0064] like Figure 3 As shown, the green points in (a) are control points in the SAI of the left light field, and the green points in (b) are control points in the SAI of the right light field with the same angular coordinates as (a). The control points in (a) and (b) are the mesh vertices {c i，j} calculated, which is Figure 2 Indicated by green dots, using {p u，v，i，j} represents the control points on each sub-aperture image. The disparity map can be written as:

[0065]

[0066] where u c and v c are the angular domain coordinates of the central sub-aperture respectively.

[0067] Secondly, the light field is calibrated using the control point guided grid deformation method. Specifically, the remaining SAI to be spliced ​​is divided into a uniform grid with the same size as the grid divided in the preprocessing stage NIWsGP. Then, with the help of the control points {p u，v，i，j}And use the grid deformation method proposed in the present invention that combines disparity terms and similarity terms to guide the deformation of these grids.

[0068] Finally, this type of problem is transformed into a least squares problem and the energy minimization problem is solved by a sparse linear solver.

[0069] In order to maintain the angle consistency of the panorama without damaging the image quality, the present invention provides a preferred embodiment. The energy function used is composed of a similarity term and a parallax term.

[0070] E=E d +λ·E s

[0071] The default value of λ is set to 1, E d is the disparity term of the energy function, E s is the similarity term of the energy function.

[0072] Specifically, in order to maintain the parallax on the central SAI as much as possible after the mesh of each SAI is deformed, so that the scene in the spliced ​​light field can conform to the correct depth information, thereby improving the angle domain consistency of the calibrated light field, the parallax energy term is defined in this embodiment:

[0073]

[0074]

[0075] Find the grid cell where the control point is located, and obtain the weight value by performing inverse bilinear interpolation on the vertices of this grid, which is represented as w i，j，k . is the disparity vector at the control point position, the disparity vector is obtained by the preprocessed disparity map, and and are the disparities of adjacent SAIs in the x and y directions on the disparity map, u and v are the angular domain coordinates of the sub-aperture image, u c and v c are the angular domain coordinates of the central subaperture image.

[0076] The similarity term is used for regularization and propagates the alignment constraints from the overlapping area to the non-overlapping area. This embodiment chooses to use the Laplace mesh deformation method to make the deformed mesh shape as consistent as possible. Laplace coordinates contain local feature information of the surface. This form of coordinates has important applications in mesh deformation, mesh smoothing and mesh denoising. This embodiment defines the similarity energy term:

[0077]

[0078] in:

[0079]

[0080] in is set to the degree of the grid node, δ i，j Represents the Laplace coordinates of the SAI mesh vertices to be stitched. Finally, combining the above energy terms, we can get the following linear least squares problem.

[0081] WV=d

[0082] in:

[0083]

[0084]

[0085] This equation can be solved using a direct linear transformation. Figure 3 As shown in (c), the SAI grid vertices of the left and right light fields can be calculated after solving the equations, represented by the red and yellow dots in the figure respectively. The blue dots are the grid vertices of the central sub-aperture image panorama. The red and yellow points can be calibrated to a new panorama. In this new panorama, after the grid nodes are calculated, the original light field is warped to the position of the grid to obtain the calibrated light field.

[0086] The present invention provides a preferred embodiment to perform S15. The calibrated sub-aperture images on the left and right sides are stacked in the form of column vectors to form a large matrix I. Each SAI can be represented as I i ∈R X，Y , where X and Y represent the length and width of each subaperture image. Therefore, the complete light field can be represented by the matrix I∈R m，n Indicates that m is the number of pixels in a single sub-aperture image, and n is the number of sub-aperture images. I can be written as follows:

[0087] I=[vec(I1)|vec(I2)|…vec(I n )]

[0088] Where vec(I i ) represents the i-th sub-aperture image. Due to the low-rank characteristics of the light field, if all sub-aperture images in the light field are calibrated to the position of the central viewpoint, each column of B can be considered as the basis image of the central sub-aperture image, and the calibrated sub-aperture image is almost the same as the central sub-aperture image. Therefore, the dimension of the light field angle domain can be reduced by low-rank approximation of the matrix I to reduce the amount of computation. In order to minimize the rank of the matrix (ideally the rank is 1), the light field dimensionality reduction problem is often expressed as:

[0089]

[0090] where A is a rank k matrix, τ(·) is the backward warping operator, and the flow vector u∈Rm，n , v∈R m，n , these two vectors are used to shift each pixel of each sub-aperture image to the center view position. After calibrating all sub-aperture images, the present invention uses singular value decomposition (SVD) to reduce the dimension of the matrix to obtain the matrix A. Note τ u，v (I) is I L

[0091] I L =U·W·V=B·C

[0092] Keep the k columns with the largest singular values ​​in W and set the rest to zero. B is a matrix of rank k, and since the B matrix is ​​actually the basis image of the central subaperture, each column of the matrix B is very similar to the central subaperture image. L After singular value decomposition, we get B1, C1 and B2, C2. The stitching line previously calculated in the stitching center SAI can be used to stitch the images represented by the first k columns of B1 and B2, and the remaining nk columns are set to zero, so that we can get an enhanced matrix B with a larger field of view. s . Remember I s =B s ·C, where C can be arbitrarily selected as C1 or C2. Here, C1 is selected, and I s Decode by column to get the spliced ​​light field LF s .

[0093] In order to verify the beneficial effects of the embodiments provided by the present invention, the method in the embodiments is evaluated on a multi-view light field of two scenes.

[0094] like Figure 4 As shown in the figure, the light field is captured on the rail and the camera is focused at infinity during the capture process. Each set of light fields contains two sufficiently large overlapping areas. Due to the severe halo effect, an 8×8 angular domain resolution is selected for each light field and the boundary images are discarded.

[0095] like Figure 5 As shown, the stitching quality and angle domain consistency of the method will be subjectively evaluated. The angle domain consistency can be evaluated by observing the continuity of EPI. Figure 5 The leftmost SAI and the rightmost SAI in the sixth row of the light field sub-aperture images after stitching by various methods and the EPI in the sixth row of the light field sub-aperture images after stitching are shown. Figure 5 The first row is the subjective experimental results of the method in the present invention, such as Figure 5As shown in (c), due to the low-rank characteristics of the light field, the experimental results show that it is reasonable to apply the graph cut results of the central sub-aperture image to the entire light field. This method can not only maintain a good stitching quality, but also achieve good angular domain consistency when observing EPI. The second row is the subjective experimental results of the improved baseline method for light field stitching (MBFLFS). MBFLFS is a light field stitching method based on the calibration of the four-dimensional homography matrix and the improved two-dimensional graph cut. It can be seen that Figure 5 In the red box of (f), the EPI of MBFLFS has obvious discontinuities. The third row compares the results of this method with those of the traditional baseline method. Figure 5 As shown in (i), it can be seen that the EPI image in the experimental results of the traditional baseline has many disordered points, which proves that the traditional method cannot be used directly to stitch light field images. The fourth row shows the experimental results of the Ray Space Motion Matrix Method (RSMM). This method uses a 5×6 matrix to calibrate the light field and uses a four-dimensional graph cut. By observing the EPI image, it can be seen that there are discontinuities in some parts, that is, the straight lines in the red box. This shows the advantage of the subjective effect of the present invention.

[0096] Figure 6 The light field refocusing performance of the present invention is demonstrated. Figure 6 (a) refocuses the light field to the back field and (b) focuses the light field to the front field. It can be seen that the leaves in the green frame in (a) are blurred while the guitar in the yellow frame is clear, while the leaves in the green frame in (b) are clear while the guitar in the yellow frame is blurred, proving that the experimental results have good refocusing performance.

[0097] The present invention uses C RMSE To evaluate the image quality of the stitched light field, C RMSE The smaller the value, the better the quality.

[0098]

[0099]

[0100] It can be seen that the performance of the present invention is better than the above method in both scenarios.

[0101] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A light field stitching method based on grid deformation and low-order approximation, characterized in that: include: Obtain a depth map of the light field to be stitched; Dividing the central sub-aperture image into a foreground and a background according to the depth map; After extracting the feature points of the left and right light fields, the backgrounds of the left and right light fields are matched; Using the feature points, stitching the central sub-aperture image of the light field based on a traditional two-dimensional image stitching method; Determine the control points of the stitched central sub-aperture image; Based on the control points, two different light fields are calibrated using a control point guided mesh deformation method; Using the low-rank features of the light field, we can find the seams of the calibrated light fields and fuse them without using 4D graph cuts. The determining of the control point comprises: The panoramic image composed of two central sub-aperture images is divided into equal intervals to obtain grid nodes, which are expressed as ; Each All on the deformed grid of the two central subaperture images; The deformed mesh is labeled ,in Indicates the index number of the deformed mesh, Used to distinguish two light fields; For each deformed mesh, a homography matrix can be calculated ; Control points of the left and right light field SAI are the mesh vertices in the central panorama that was previously stitched together Calculated; ， in and are the angular domain coordinates of the central sub-aperture respectively; The method of using control point guided mesh deformation to calibrate two different light fields includes: The remaining SAIs to be spliced ​​are divided into uniform grids, and the size of the grid is consistent with the size of the grid divided in the preprocessing stage NIWsGP; Control points on SAI Using a grid deformation method combining a disparity term and a similarity term to guide the uniform grid to deform; The method uses the low-rank features of the light field to find and fuse the stitching lines of the calibrated light field without using four-dimensional graph cuts, including: Calibrate all SAIs in each light field to the position of the central viewpoint; The calibrated light field is combined into a matrix in the form of column vectors by using the low-rank property of the light field; Decompose the matrix using singular value decomposition SVD or orthogonal triangular decomposition QR to obtain ， in , , , , is the number of subaperture image pixels, is the number of subaperture images; , and C are both unitary matrices, is a matrix consisting of singular values; Due to the low-rank property of the light field, each column of B is considered as a basis image of the central subaperture image; The left and right light fields are stitched together using the stitching lines computed using 2D graph cuts in NISwGSP when stitching the central subaperture images. Each column of the matrix, record the concatenated The matrix is , ， in is the spliced ​​light field, The left light field matrix; take out Each column of gets the sub-aperture image of the complete light field after stitching.

2. The light field stitching method based on grid deformation and low-order approximation according to claim 1, characterized in that: The background of the left and right light fields is matched using feature points of a scale-invariant feature transformation based on the depth map, and the feature points are matched only in points whose depth is less than a threshold value T in the central sub-aperture image.

3. The light field stitching method based on grid deformation and low-order approximation according to claim 1, characterized in that: The natural image stitching (NISwGSP) method is used to stitch the central sub-aperture image.

4. The light field stitching method based on grid deformation and low-order approximation according to claim 1, characterized in that: The determining of the control points includes: re-dividing the panoramic image after stitching the central sub-aperture images into equidistant grids, and using the nodes of the grids as control points.

5. The light field stitching method based on grid deformation and low-order approximation according to claim 1, characterized in that: The energy function of the grid deformation method combining the disparity term and the similarity term includes a disparity term; the disparity term enables each SAI grid to maintain the disparity on the central sub-aperture after deformation and conform to the correct depth information.

6. The light field stitching method based on grid deformation and low-order approximation according to claim 1, characterized in that: The energy function of the mesh deformation method combining the disparity term and the similarity term includes a similarity term; the similarity term propagates the disparity constraint from the overlapping area to the non-overlapping area.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 6.

Images