A light field stitching method based on a four-dimensional homography matrix

Abstract

This invention provides a light field stitching method based on a four-dimensional homography matrix. First, the light field data is decoded and preprocessed to obtain sub-aperture images. Then, feature point detection is performed on the sub-aperture images, intra-frame matching is conducted, and each feature point to be matched is calculated. r The value is then converted into a four-dimensional coordinate pattern with depth information for each feature point to be matched; inter-frame matching is then performed to obtain a set of matching feature points for different light fields; four feature point pairs are randomly selected iteratively from the feature point set, and the coefficient matrix is ​​obtained using the Kronecker product. A ,right A Singular value decomposition is performed to solve for the homography transformation matrix of the light field. The optimal global homography transformation matrix is ​​then iteratively obtained using a random sample consensus algorithm. Finally, the light fields to be stitched are fused to obtain the stitched panoramic light field. This invention incorporates the depth information inherent in the light field into the homography matrix estimation, ensuring that the stitched light field maintains spatial and angular consistency, thus achieving a precise and efficient light field stitching method.

Description

Technical Field

[0001] This invention relates to the field of computer vision and digital image processing technology, specifically to a light field stitching method based on a four-dimensional homography matrix. Background Technology

[0002] Light, as the most important medium for human perception of the world, carries rich information about the three-dimensional environment. Traditional images record the two-dimensional projection of light by integrating the angles of light rays at each pixel, thus discarding multiple dimensions of visual information. We can only observe three-dimensional scenes through two-dimensional imaging effects. Unlike traditional images, light fields record the distribution of light rays in free space. Therefore, light fields not only record the spatial information of pixels but also retain angular information. This makes light fields have a wider range of applications than two-dimensional images, such as view composition, deblurring, and depth estimation.

[0003] The recent developments in virtual reality and augmented reality have led to a surge in research using light fields to represent virtual scenes. However, the limitations of using light field data to represent scenes are apparent: it cannot simultaneously achieve high spatial and angular resolution. To broaden the application of light fields in practice, we need to expand the field of view of light field images, investigate how perspective shifts change in each light field image, and explore how to combine multiple light fields into a large panoramic light field image.

[0004] Existing light field stitching methods focus on focal stacks, transforming the light field into a focal stack. Theoretically, the stitched light field obtained using this method is better than that obtained by direct stitching methods. However, the drawback is that the focal stack transformation increases computational cost, and the accuracy of the result is dependent on the focal stack transformation algorithm. Some works transform the light field into ray space and estimate its motion matrix within it. This method yields better results and is independent of other algorithms, but it requires accurate camera rotation data and a large overlap between light fields. Other works directly apply 2D image stitching algorithms to light field stitching, resulting in a panoramic light field image that loses the properties of the light field. In general, most light field stitching methods cannot simultaneously maintain the spatial and angular continuity of the stitched light field, or suffer from excessive computational cost. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a light field stitching method based on a four-dimensional homography matrix that can simultaneously maintain the spatial and angular continuity of the stitched light field and has a relatively small computational load.

[0006] To solve the above-mentioned technical problems, the technical solution proposed by this invention is: a light field stitching method based on a four-dimensional homography matrix, comprising the following steps: A1: Acquire the light field data to be stitched, decode and preprocess it to obtain all sub-aperture images of the light field, extract feature points from all decoded light field sub-aperture images, and perform intra-frame matching; calculate the depth information of each feature point to be matched. r value; A2: After obtaining the set of feature points to be matched for each light field, the light field points are converted into a four-dimensional coordinate format containing depth information, and then inter-frame matching is performed to obtain the set of matching feature points between different light fields. A3: Iteratively select 4 pairs of feature points from the feature point set, and use the Kronecker product to obtain the coefficient matrix in each iteration. A And then A The singular value decomposition is used to solve the homography transformation matrix of the optical field, and then the optimal global homography transformation matrix is ​​obtained iteratively based on the random sample consensus algorithm. A4: Based on the global light field homography transformation matrix obtained in A3, the light field is stitched together to obtain the panoramic light field.

[0007] The above-mentioned light field stitching method based on a four-dimensional homography matrix, preferably, wherein... r The value is obtained by calculating using the following formula:

[0008] in, d ij Representing the i Individual aperture camera and the first j Parallax between individual aperture cameras b ij This represents the baseline length between the two sub-aperture cameras mentioned above.

[0009] In the above-described light field stitching method based on a four-dimensional homography matrix, preferably, in step A2, the light rays in the light field... l i Relative target point P The conversion relationship is as follows:

[0010] in, Z c This represents the distance from the target object to the camera. f Represents the camera's focal length, ( c x ,c y () represents the principal point coordinates of the central sub-aperture image. R i and t i Represents the displacement matrix and vector from the world coordinate system to the light field coordinate system.

[0011] In the above-mentioned light field stitching method based on a four-dimensional homography matrix, preferably, in step A3, the transformation relationship between different light fields is as follows:

[0012] in, l r and l w These represent feature points in the light field to be stitched together. and This represents the depth of that point. The transformation matrix represents two light fields, containing rotation matrices and translation vectors. F The intrinsic parameter matrix representing the light field.

[0013] In the above-mentioned light field stitching method based on a four-dimensional homography matrix, preferably, in step A3, the Kronecker product takes the form of:

[0014] in, It is a skew-symmetric matrix, consisting of 6 fundamental matrices. vec ( H )∈ R 16 Let be the homography transformation matrix of the optical field.

[0015] The above-mentioned light field stitching method based on a four-dimensional homography matrix preferably uses a coefficient matrix. Perform singular value decomposition on the coefficient matrix A:

[0016] in, n This represents the number of feature point pairs. B 1- B 6 represents 6 basic matrices, and each pair of points represents a transformation relationship.

[0017] Compared with existing technologies, the advantages of this invention are as follows: This invention combines traditional panoramic image stitching algorithms with the homography of light fields. By analyzing the data structure of the four-dimensional light field, the scene depth information provided by the light field is integrated into the representation of light field feature points. Then, equations are listed through the correspondence of feature points, and the three-dimensional direct linear transformation algorithm is extended to four dimensions to solve for the homography transformation matrix of the light field. Compared with existing methods, the method provided by this invention has a relatively small computational load, while ensuring the spatial and angular consistency of the stitched light field and effectively expanding the viewing angle of the light field. Attached Figure Description

[0018] Figure 1 This is a flowchart of the light field splicing method of the present invention. Implementation

[0019] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0020] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0021] refer to Figure 1 This invention provides a light field stitching method based on a four-dimensional homography matrix, which can obtain a stitched light field with spatial and angular consistency through four steps A1-A4. The following embodiment will use the stitching of two light fields as an example to specifically illustrate the scheme of the light field stitching method of this invention.

[0022] The first step, A1, involves acquiring the light field data to be stitched, decoding and preprocessing it to obtain all sub-aperture images of the light field. Feature points are extracted from all decoded sub-aperture images, and intra-frame matching is performed. This involves extracting, matching, and filtering feature points within all sub-aperture images of the light field, setting a threshold. A feature point is included in the matching set only if the number of sub-aperture images containing that feature point exceeds the threshold. Then, depth information is calculated for each feature point to be matched. r value.

[0023] Specifically, in this embodiment, after decoding and preprocessing the two light field data, a reference light field can be obtained. L r ( x,y,u,v ) and the light field to be spliced L w ( x,y,u,v Then, SIFT feature point detection and matching are performed on the sub-aperture images of the two light fields mentioned above. The Continuous Consistent Sampling Detection (RANSAC) method is used to filter feature point pairs in the corresponding sub-aperture images, where:

[0024] Next, a threshold T is set such that the selected feature point must meet the following condition: the feature point must appear in at least T sub-aperture images to enhance the reliability of feature point selection.

[0025] In this embodiment, depth information is included. r The value is obtained by calculating using the following formula:

[0026] in,d ij Representing the i Individual aperture camera and the first j Parallax between individual aperture cameras b ij This represents the baseline length between the two sub-aperture cameras mentioned above.

[0027] Then proceed to step A2: After obtaining the set of feature points to be matched for each light field, convert the light field points into a four-dimensional coordinate format containing depth information, and then perform inter-frame matching, that is, feature point matching between adjacent light fields, to obtain the set of matching feature points between different light fields.

[0028] Specifically, in this embodiment, the light rays in the light field l i Relative target point P The conversion relationship is as follows:

[0029] in, Z c This represents the distance from the target object to the camera. f Represents the camera's focal length, ( c x ,c y () represents the principal point coordinates of the central sub-aperture image. R i and t i Represents the displacement matrix and vector from the world coordinate system to the light field coordinate system.

[0030] The ray coordinates in the light field can be represented by the above transformation relationship as follows:

[0031] in, l i For light field i Feature points, ( x c ,y c () represents the coordinates of the feature point in the central sub-aperture. r This is the value with depth information obtained in step A1.

[0032] Then inter-frame matching is performed, that is, for and( x,y,u,v ) Matching is performed to obtain matching feature point pairs between the two light fields.

[0033] Next is step A3: Iteratively select 4 pairs of feature points from the feature point set, and use the Kronecker product to obtain the coefficient matrix in each iteration. A And then A The singular value decomposition is used to solve the homography transformation matrix of the optical field, and then the optimal global homography transformation matrix is ​​obtained iteratively based on the Random Sample Consensus (RANSAC) algorithm.

[0034] Specifically, A3 includes the following steps: A31: Identify the transformation relationships between light fields; in this embodiment, the transformation relationships between different light fields are as follows:

[0035] in, l r and l w These represent feature points in the light field to be stitched together. and This represents the depth of that point. The transformation matrix represents two light fields, containing rotation matrices and translation vectors. F The intrinsic parameter matrix representing the light field.

[0036] In this embodiment, the predicted global homography transformation matrix of the optical field OR The calculation formula is:

[0037] in, These are the matching feature point pairs obtained through step A2.

[0038] The four-dimensional homography matrix can be obtained through calculation. H The form is:

[0039] A32: Solving for the homography transformation matrix H In this embodiment, The form was changed to Then, the four-dimensional matrix is ​​obtained through the Kronecker product. H The solution.

[0040] Specifically, the Kronecker product takes the form of:

[0041] in, It is a skew-symmetric matrix, consisting of 6 fundamental matrices. vec ( H )∈ R 16 Let be the homography transformation matrix of the optical field.

[0042]

[0043] Because in this embodiment S The matrix consists of 6 basic matrices. S Since the rank of the matrix is ​​3, we only need to randomly select 3 basic matrices to substitute into the calculation.

[0044] In this embodiment, the coefficient matrix Perform singular value decomposition on the coefficient matrix A:

[0045] in, n This represents the number of feature point pairs. B 1- B The six matrices represent six fundamental matrices, and each pair of points represents a transformation relationship. A The rank of the matrix is ​​3. H There are 12 unknown parameters, and at least 4 feature point pairs are needed in each round to calculate them. H All parameters of the matrix.

[0046] In this embodiment, we iteratively and randomly select 4 feature point pairs and 3 fundamental matrices from the feature point pair set. B i To calculate the matrix A And obtain the matrix H After each calculation H The matrix is ​​then substituted with the remaining data to calculate the error. Through multiple iterations, the matrix with the smallest total error is selected. H The matrix is ​​the desired homography transformation matrix for the optical field.

[0047] Finally, step A4 is used to stitch the light field together based on the global light field homography transformation matrix obtained in A3 to obtain the panoramic light field.

[0048] In this embodiment, the optical field homography transformation matrix H is a global transformation matrix. The central sub-aperture image of the stitched two optical fields is used. and For example, the stitching steps for the remaining sub-aperture images are similar.

[0049] First, the depth information of each pixel in the image is obtained; real data is obtained from Lytro Desktop, and virtual data is obtained from Blend. The sub-aperture image coordinates are then converted to the required format for calculation, and the depth information of each pixel is calculated. r value:

[0050] in, Z This is the depth value at that point. f This refers to the focal length of a sub-aperture camera.

[0051]

[0052] in, H This is the homography light field transformation matrix obtained earlier. It is the input light field sub-aperture image. It is the mapped sub-aperture image.

[0053] In this embodiment, a pixel-weighted average image fusion method is used to stitch the light field. Based on the grayscale information of the sub-aperture images to be stitched, a weight is assigned to the grayscale value of each corresponding pixel in the fusion region. The grayscale of the fused image is the weighted sum of the grayscale values ​​of the two images.

[0054] in, w 1 and w 2 represents the calculated weight. F This is the intersection region of the two light fields.

[0055] This invention combines traditional panoramic image stitching algorithms with the homography of light fields. By analyzing the data structure of four-dimensional light fields, the scene depth information provided by the light fields is integrated into the representation of light field feature points. Then, equations are listed based on the correspondence of feature points. The three-dimensional direct linear transformation algorithm is extended to four dimensions to solve for the homography transformation matrix of the light fields.

[0056] Compared with existing methods, the method provided in this invention employs a four-dimensional light field coordinate representation method with built-in depth data, transforming multiple light fields into the same coordinate system based on the homography transformation relationship of the light field. Furthermore, it improves the accuracy of the estimated homography matrix through intra-frame matching and inter-frame matching. By solving the four-dimensional homography matrix of the light field based on the Kronecker product, it theoretically requires only a minimum of four pairs of feature points to stitch the light fields together. Therefore, the method of this invention can achieve spatial and angular consistency in the stitched light fields, effectively expanding the viewing angle of the light field, while having a relatively small computational load, making it highly practical.

Claims

1. A light field stitching method based on a four-dimensional homography matrix, characterized in that, Includes the following steps: A1: Collect the light field data to be stitched, decode and preprocess to obtain all sub-aperture images of the light field, extract feature points from all decoded light field sub-aperture images and perform intra-frame matching; calculate the ρ value with depth information for each feature point to be matched. A2: After obtaining the set of feature points to be matched for each light field, the light field points are converted into a four-dimensional coordinate format containing depth information, and then inter-frame matching is performed to obtain the set of matching feature points between different light fields. A3: Iteratively select 4 pairs of feature points in the feature point set, use the Kronecker product to obtain the coefficient matrix A in each iteration, then perform singular value decomposition on A to solve the light field homography transformation matrix, and then iteratively obtain the optimal global light field homography transformation matrix based on the random sample consensus algorithm. A4: Based on the global light field homography transformation matrix obtained in A3, the light field is stitched together to obtain the panoramic light field; The value of ρ is calculated using the following formula: ； Where, d ij b represents the parallax between the i-th sub-aperture camera and the j-th sub-aperture camera. ij This represents the baseline length between the two sub-aperture cameras mentioned above; In step A2, the light ray l in the light field i The transformation relationship with the real target point P is as follows: ； Among them, Z c f represents the distance from the target object to the camera, and f represents the focal length of the camera. x ,c y R represents the principal point coordinates of the central sub-aperture image. i and t i Represents the displacement matrix and vector from the world coordinate system to the light field coordinate system; In step A3, the transformation relationship between different light fields is as follows: ； Among them, l r and l w These represent feature points in the light field to be stitched together. and Represents the depth of that point, T∈ The transformation matrix represents two light fields, including the rotation matrix and translation vector, and F represents the intrinsic parameter matrix of the light field; In step A3, the Kronecker product takes the form of: ； Where, S∈ It is a skew-symmetric matrix consisting of 6 fundamental matrices, vec(H)∈R. 16 Let be the desired homography transformation matrix of the optical field; The coefficient matrix Perform singular value decomposition on the coefficient matrix A: ； Where n represents the number of feature point pairs, and B1-B6 are six basic matrices, each point pair representing a transformation relationship.

Images