A light field image calibration method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a light field image calibration method and device, electronic equipment and storage medium, obtain a coordinate transfer matrix of a lens group after calibration, when the absolute value of an element in the coordinate transfer matrix is less than a preset tolerance, set the element to zero to obtain a pruned coordinate transfer matrix, generate a black RGB canvas image of the same size as a light field image captured by the lens group, determine the coordinates of the light field image corresponding to the coordinates of the black RGB canvas image through the pruned coordinate transfer matrix, obtain the RGB average of the surrounding area of the coordinates of the light field image, assign the RGB average to the coordinates of the black RGB canvas image, and obtain the calibrated light field image. In the embodiments of the present application, the coordinate transfer matrix is pruned, the elements with an absolute value less than the preset tolerance in the coordinate transfer matrix are set to zero, and then in the process of coordinate transfer matrix calculation, the zero operation is skipped to realize the acceleration of image calibration by sparse matrix under the condition of controllable precision loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a light field image calibration method, a light field image calibration device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The light field image is shot by a light field camera array, which records the position information and direction information of light in space through a lens array, calculates the three-dimensional space coordinates by light field imaging technology, and makes up for the deficiency of traditional cameras that can only record two-dimensional images.

[0003] Because the data transmitted by the light field camera array is tens of times that of a traditional camera, and the calibration process is a necessary pre-step for each light field image, there are a large number of sparse matrix calculations in the operation process of the existing light field image calibration technology, and the processing efficiency is low. SUMMARY

[0004] Embodiments of the present application provide a light field image calibration method, device, electronic device and computer readable storage medium to solve the problem of a large number of sparse matrix calculations in the operation process of the existing light field image calibration technology and low processing efficiency.

[0005] Embodiments of the present application disclose a light field image calibration method applied to a light field camera array, the light field camera array comprising a reference lens and a plurality of sub-lenses, each of the plurality of sub-lenses forming a lens group with the reference lens, the method comprising:

[0006] obtaining a coordinate transfer matrix of the calibrated lens group;

[0007] when the absolute value of an element in the coordinate transfer matrix is less than a preset tolerance, setting the element to zero to obtain a pruned coordinate transfer matrix;

[0008] obtaining a light field image captured by the lens group, and generating a black RGB canvas image of the same size as the light field image based on the light field image;

[0009] determining the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transfer matrix;

[0010] obtaining the RGB mean value of the surrounding area of the coordinates of the light field image;

[0011] assigning the RGB mean value to the coordinates of the black RGB canvas image to obtain a calibrated light field image.

[0012] Optionally, the method for obtaining the coordinate transfer matrix of the calibrated lens group comprises:

[0013] obtaining a calibration image of a calibration site photographed by the lens group;

[0014] calculating distortion coefficients, a rotation coefficient matrix, a translation coefficient matrix, a focal length and a lens principal point of the lens group based on the calibration image;

[0015] determining a coordinate translation matrix of the lens group after calibration according to the distortion coefficients, the rotation coefficient matrix, the translation coefficient matrix and the lens principal point.

[0016] Optionally, the calibration site comprises a black-and-white chessboard or other feature indicators.

[0017] Optionally, determining the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate translation matrix comprises:

[0018] traversing the coordinates in the black RGB canvas image and calculating the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate translation matrix.

[0019] Optionally, calculating the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate translation matrix further comprises:

[0020] skipping the multiplication operation of the zero-value elements in the pruned coordinate translation matrix.

[0021] Optionally, before obtaining the RGB mean value of the surrounding area of the coordinates of the light field image, further comprising:

[0022] scaling up the light field image by a predetermined multiple using linear interpolation.

[0023] Optionally, obtaining the RGB mean value of the surrounding area of the coordinates of the light field image comprises:

[0024] obtaining the RGB values of a predetermined number of data points in the surrounding area of the coordinates of the light field image;

[0025] calculating the average of the RGB values of the predetermined number of data points to obtain the RGB mean value of the surrounding area of the coordinates of the light field image.

[0026] The embodiment of the application also discloses a light field image calibration device applied to a light field camera array, the light field camera array comprising a reference lens and a plurality of sub-lenses, each of the plurality of sub-lenses forming a lens group with the reference lens, and comprising:

[0027] a matrix obtaining module configured to obtain a coordinate translation matrix of the lens group after calibration.

[0028] a matrix pruning module, configured to set to zero an element in the coordinate transfer matrix when an absolute value of the element is less than a preset tolerance, to obtain a pruned coordinate transfer matrix;

[0029] an image acquisition module, configured to acquire a light field image captured by the lens group, and generate a black RGB canvas image with a same size as the light field image based on the light field image;

[0030] a coordinate determination module, configured to determine, through the pruned coordinate transfer matrix, a coordinate of the light field image corresponding to a coordinate in the black RGB canvas image;

[0031] an RGB mean value acquisition module, configured to acquire an RGB mean value of a surrounding area of the coordinate of the light field image;

[0032] an RGB mean value assignment module, configured to assign the RGB mean value to the coordinate of the black RGB canvas image, to obtain a calibrated light field image.

[0033] Optionally, the matrix acquisition module comprises:

[0034] an image acquisition sub-module, configured to acquire a calibration image of a calibration site captured by the lens group;

[0035] a parameter calculation sub-module, configured to calculate, based on the calibration image, a distortion coefficient, a rotation coefficient matrix, a transfer coefficient matrix, a focal length and a lens principal focal point of the lens group;

[0036] a matrix determination sub-module, configured to determine, according to the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix and the lens principal focal point, a calibrated coordinate transfer matrix of the lens group.

[0037] Optionally, the calibration site comprises a black-and-white chessboard or other feature indicators.

[0038] Optionally, the coordinate determination module comprises:

[0039] a coordinate calculation sub-module, configured to traverse a coordinate in the black RGB canvas image, and calculate, through the pruned coordinate transfer matrix, a coordinate of the light field image corresponding to the coordinate in the black RGB canvas image.

[0040] Optionally, the method further comprises:

[0041] a zero operation skipping module, configured to skip a multiplication operation of a zero value element in the pruned coordinate transfer matrix.

[0042] Optionally, the method further comprises:

[0043] An image magnification sub-module is configured to use a linear interpolation method to magnify the light field image by a preset multiple.

[0044] Optionally, the RGB mean value acquisition module comprises:

[0045] An RGB value acquisition sub-module is configured to acquire RGB values of a preset number of data points in a surrounding area of the coordinate of the light field image.

[0046] An RGB mean value calculation sub-module is configured to calculate mean values of the RGB values of the preset number of data points to obtain the RGB mean value of the surrounding area of the coordinate of the light field image.

[0047] The embodiment of the application further discloses an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0048] The memory is used to store a computer program.

[0049] The processor is used to execute the program stored on the memory to realize the method as described in the embodiment of the application.

[0050] The embodiment of the application further discloses a computer readable storage medium, which stores instructions, and when executed by one or more processors, causes the processor to execute the method as described in the embodiment of the application.

[0051] The embodiment of the application has the following advantages:

[0052] The embodiment of the application is used to prune the coordinate transformation matrix, set the elements with absolute values less than a preset tolerance to zero, then skip the zero operation in the process of calculation based on the coordinate transformation matrix to realize the acceleration of the image calibration of the sparse matrix under the controllable precision loss, and the pruning technology is fully compatible, so that the controllable secondary acceleration is realized, and the problem of low processing efficiency caused by a large number of sparse matrix calculations in the operation process of the existing light field image calibration technology is solved. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a step flow chart of a light field image calibration method provided in the embodiment of the application;

[0054] Figure 2 is a structural block diagram of a light field image calibration device provided in the embodiment of the application;

[0055] Figure 3 is a hardware structural block diagram of an electronic device provided in each embodiment of the application. DETAILED DESCRIPTION

[0056] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0057] The light field image is taken by a light field camera array, which records the position information and direction information of light in space through a lens array, calculates the three-dimensional space coordinates by light field imaging technology, and makes up for the deficiency of traditional cameras that can only record two-dimensional images. Light field images have wide applications, such as digital refocusing, depth estimation, three-dimensional imaging, and have great prospects in the fields of automatic driving of cars, industrial detection, security monitoring, medical imaging, etc. Among them, light field image calibration is the premise of light field imaging application, which guarantees the accuracy of three-dimensional space information calculation, and also affects the complexity and compression efficiency of light field video coding. Since the data transmitted by the light field camera array is tens of times that of a traditional camera, and the calibration process is a necessary pre-step for each light field image, a fast and efficient calibration method is urgently needed to solve the problem of light field imaging.

[0058] However, the existing method calculates pixel distortion and parallax distortion by light field information of the camera array, and uses absolute coordinates in three-dimensional space to displace image pixel points associated with the lens to the correct position. Because the feature matrix and the parameter matrix are mostly sparse matrices, there is a lot of calculation redundancy (0 vector multiplication occupies computing resources) in the operation process of the calibration algorithm. And in actual application, the distortion degree of each lens is different, but the calibration always needs to traverse all the pixel points, which leads to the fact that the calculation amount of the lens group with slight distortion is actually the same as that of the lens group with serious distortion. Finally, the lens groups of the camera array do not affect each other, and the serial workflow has a lot of room for improvement in efficiency. Therefore, the current calibration method has problems such as calculation redundancy and inability to dynamically adjust the calculation amount at the actual application level.

[0059] Based on this, the present application embodiment discloses a light field image calibration method, device, electronic equipment and storage medium to solve the above problems.

[0060] Reference Figure 1 A step flowchart of a light field image calibration method provided by the present application embodiment is shown, the light field image calibration method is applied to a light field camera array, the light field camera array includes a reference lens and a plurality of sub-lenses, each of the plurality of sub-lenses forms a lens group with the reference lens, and can specifically include the following steps:

[0061] Step 101, obtaining the coordinate transfer matrix of the calibrated lens group.

[0062] The light field camera array is composed of a reference lens and a plurality of sub-lenses, and there is a repeated view angle between the reference lens and the sub-lenses, each sub-lens and the reference lens form a group, and there are N-1 groups (N is the total number of camera array lenses).

[0063] Specifically, before calibrating the light field image, the lens group in the light field camera array needs to be calibrated, and the coordinate transfer matrix of the calibrated lens group is obtained. The coordinate transfer matrix is the transfer matrix of the original light field image coordinates corresponding to the calibrated coordinates.

[0064] It should be noted that the calibration of the lens group can be realized by using the conventional technical means in the art, such as calibrating the light field camera by using the corner points of the checkerboard or the line features of the calibration board. The specific setting can be made according to the needs, and the present embodiment does not limit this.

[0065] Step 102, when the absolute value of an element in the coordinate transfer matrix is less than a preset tolerance, the element is set to zero, and the pruned coordinate transfer matrix is obtained.

[0066] Specifically, after obtaining the coordinate transfer matrix of the calibrated lens group, the approximately zero elements in the coordinate transfer matrix need to be pruned (the approximately zero elements are set to 0), and the position of the 0 vector is recorded. The pruning process of the coordinate transfer matrix is as follows:

[0067]

[0068] In the formula: K nx is an element in the coordinate transfer matrix, is a tolerance, if the absolute value of the matrix coefficient is less than then it is set to zero. The size of the tolerance can be adjusted according to the actual precision. If there is no precision loss.

[0069] In the above embodiment, the approximately zero values in the matrix are conditionally pruned, and the sparse matrix in the transfer matrix is marked with a zero vector. This is a pre-step for matrix calculation acceleration in the operation process, and also sets a variable for controlling the result precision and secondary acceleration.

[0070] Step 103, obtaining the light field image shot by the lens group, and generating a black RGB canvas image with the same size as the light field image based on the light field image.

[0071] Specifically, the light field image shot by the lens group is obtained, and a black RGB (RGB color mode, red, green, and blue) canvas image with the same size as the original light field image is generated according to the light field image data in the light field image.

[0072] In the above embodiment, the light field image to be calibrated is detected, a pure black canvas image of the same size is generated as a calibration mask, and the input of the calibration algorithm is the pixel coordinates, which are sequentially traversed from the top left corner of the canvas to ensure the input and output environment of the calibration algorithm.

[0073] In step 104, the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image are determined through the pruned coordinate transformation matrix.

[0074] Specifically, the coordinates (u', v') of the light field image corresponding to each coordinate (u, v) in the black RGB canvas image are calculated through the pruned coordinate transformation matrix. In the calculation process through the pruned coordinate transformation matrix, the calculation needs to be accelerated, and the specific process is as follows: the positions of the non-zero values recorded in the coordinate transformation matrix are searched, and the coordinate transformation matrix is stored in a sparse manner. In the calculation process of the coordinate transformation matrix, the current element to be multiplied is checked, and if a 0 value is encountered, the position of the next element to be multiplied is shifted until a non-zero value is encountered. That is, in the calculation process of the coordinate transformation matrix, the zero sub-element is directly skipped, thereby reducing the calculation of the coordinate transformation matrix.

[0075] In step 105, the RGB mean value of the surrounding area of the coordinates of the light field image is obtained.

[0076] Specifically, after calculating the coordinates (u', v') of the light field image corresponding to each coordinate (u, v) in the black RGB canvas image through the pruned coordinate transformation matrix, the RGB mean value of the surrounding area of the coordinates (u', v') of the light field image is obtained.

[0077] In step 106, the RGB mean value is assigned to the coordinates of the black RGB canvas image to obtain a calibrated light field image.

[0078] Specifically, after obtaining the RGB mean value of the surrounding area of the coordinates (u', v') of the light field image, the RGB value is assigned to the (u, v) position of the black RGB canvas image, and the coordinates on the black RGB canvas image are assigned until the coordinates on the black RGB canvas image are assigned, thereby obtaining a calibrated light field image.

[0079] In the embodiment of the application, by pruning the coordinate transformation matrix, the elements with an absolute value less than a predetermined tolerance in the coordinate transformation matrix are set to zero, and then in the calculation process through the coordinate transformation matrix, the zero operation is skipped to accelerate the image calibration process of the sparse matrix under the condition of controllable precision loss, and it is fully compatible with the pruning technology, thereby achieving controllable secondary acceleration, and solving the problem of low processing efficiency caused by a large number of sparse matrix calculations in the existing light field image calibration technology.

[0080] On the basis of the above-mentioned embodiments, a variant embodiment of the above-mentioned embodiments is proposed, and it should be noted that, in order to make the description brief, only the differences from the above-mentioned embodiments are described in the variant embodiment.

[0081] In an embodiment of the present application, the obtaining of the coordinate transfer matrix of the lens group after calibration comprises: obtaining a calibration image photographed by the lens group to a calibration site; calculating a distortion coefficient, a rotation coefficient matrix, a transfer coefficient matrix, a focal length and a lens principal focal length point of the lens group based on the calibration image; and determining the coordinate transfer matrix of the lens group after calibration according to the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix and the lens principal focal length point.

[0082] The calibration site contains a black-and-white chessboard or other characteristic indicators.

[0083] Specifically, the calibration image photographed by the lens group is taken as input. The calibration image is generally a photograph taken by a light field camera on a scene containing a black-and-white chessboard or other characteristic indicators. The intersection of the black-and-white chessboard is used to calculate the horizontal plane in the background, so as to obtain the distortion coefficient and the calibration parameter in the picture. Each lens group can be independently bound to a CPU for parallel processing.

[0084] The calculation of the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix, the focal length and the lens principal focal length point by the calibration image specifically comprises:

[0085] The distortion coefficient and its transformation are as follows:

[0086]

[0087] is expressed as a normalized two-dimensional coordinate in a plane from a three-dimensional space coordinate, which is also called a world coordinate (x, y, z). The three-dimensional space coordinate is calculated from a clear intersection point (generally the intersection point of a black-and-white chessboard) on a calibration image to obtain a three-dimensional coordinate relative to a two-dimensional image. It serves as an anchor for the lens group. Its normalized expression is:

[0088]

[0089] The rotation coefficient matrix R and the transfer coefficient matrix T jointly construct an external parameter matrix Q, P, which is expressed as follows:

[0090] Q=R -1 -R -1 T∈R 3×4

[0091] P=[RT∈R 3×4

[0092] The external parameter matrix Q, P, the original image coordinate group X ωAnd the corresponding coordinate group X is expressed as follows:

[0093]

[0094]

[0095] Wherein, focal length f, lens main focal point (c u ,c v ) jointly construct internal parameter matrix K c :

[0096]

[0097] Wherein, γ is the bending coefficient, and δ is the aspect ratio, and the matrix K is the internal parameter matrix generated by comparison of the reference lens and the lens to be calibrated, and if the reference lens and the lens to be calibrated are compared, a secondary internal matrix is generated As follows:

[0098]

[0099] Wherein γ = 0, and δ = 1.

[0100] According to the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix and the lens main focal point, the coordinate transfer matrix of the lens group after calibration is determined, and the following is obtained:

[0101]

[0102]

[0103] Wherein, The pixel coordinates on the black RGB canvas image are The corresponding pixel coordinates on the light field image to be processed are.

[0104] In the above embodiment, the distortion parameters, the internal parameters, the external parameters, the selection of various parameters and the setting of special values ensure the accuracy of the calibration algorithm, and the relative coordinates in three-dimensional space are used as the reference to generate the transfer matrix, which also ensures the accuracy of the calibration algorithm.

[0105] In an embodiment of the present application, the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image are determined by the pruned coordinate transfer matrix, including: traversing the coordinates in the black RGB canvas image, and calculating the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image by the pruned coordinate transfer matrix.

[0106] Specifically, the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image are calculated by the pruned coordinate transfer matrix, and the calculation is realized by traversing the coordinates in the black RGB canvas image, for example, the RGB canvas image is traversed from the top left corner (0, 0) in the pixel coordinate (u, v), and the corresponding coordinate point (u', v') of the original image is obtained by the pruned coordinate transfer matrix and matrix acceleration calculation.

[0107] In an embodiment of the present application, the calculation of the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image by the pruned coordinate transfer matrix further comprises: skipping the multiplication operation of the zero value element in the pruned coordinate transfer matrix.

[0108] Specifically, in the process of calculating the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image by the pruned coordinate transfer matrix, the elements to be multiplied in the coordinate transfer matrix are detected, and when a zero value element is encountered, it is directly ignored and skipped, that is, when a zero value element is encountered, the multiplication operation is not performed, and the position is shifted to the next element to be multiplied until a non-zero value element is encountered, and the calculation is performed.

[0109] In the above embodiment, the coordinate transfer matrix is used to generate the calibrated pixel coordinates, and in the calibration operation process, the elements to be multiplied in the matrix vector position are checked, and if it is zero, it is shifted to the next number until a non-zero value is encountered, so that the redundant calculation is logically cancelled, and the matrix calculation is accelerated.

[0110] In an embodiment of the present application, before the RGB mean value of the surrounding area of the coordinates of the light field image is obtained, the light field image is enlarged by a preset multiple by using the linear interpolation method.

[0111] Specifically, the new coordinates (u', v') obtained by the pruned coordinate transfer matrix operation belong to the real number range and must be processed by integerization, which will cause a certain degree of information loss. In order to maintain the accuracy after integerization, the original light field image is enlarged by the same proportion as the new coordinate point by using the Lanczos linear interpolation method, and the specific preset multiple can be set according to actual needs.

[0112] In an embodiment of the present application, the RGB mean value of the surrounding area of the coordinates of the light field image is obtained, comprising: obtaining the RGB values of a preset number of data points in the surrounding area of the coordinates of the light field image; calculating the average value of the RGB values of the preset number of data points to obtain the RGB mean value of the surrounding area of the coordinates of the light field image.

[0113] Specifically, after calculating the coordinates (u', v') of the light field image corresponding to each coordinate (u, v) of the black RGB canvas image, the RGB values of a preset number of data points in the surrounding area of the coordinates (u', v') of the light field image are obtained, for example, the RGB values of nine points around the coordinate point (u', v') are obtained, and then the average value of the RGB values of the nine data points is calculated, specifically as follows:

[0114]

[0115] Thus, the RGB average value of the surrounding area of the coordinates of the light field image is obtained (u′,v′) .

[0116] In the above embodiment, the original image is proportionally enlarged by using the interpolation method, and the average value of the surrounding nine pixel points is taken for the integerized coordinate value, thereby saving the image information lost by integerization and ensuring the accuracy of the calibration operation.

[0117] It should be noted that the preset number can be set as needed, and the embodiments of the present application do not limit this.

[0118] In order to better understand the embodiments in the present application, the following will be described through an example.

[0119] (1) Group the light field camera array, each lens (sub-lens) and the reference lens form a group, a total of N-1 groups (N is the total number of camera array lenses), and the calibration image shot by the lens group is taken as input. The calibration image is generally a photograph of a light field camera containing a black and white chessboard or other feature indicator in the scene, and the intersection of the black and white chessboard is used to calculate the horizontal plane in the background, so as to obtain the distortion coefficient and the calibration parameter in the picture. Each lens group can be independently bound with a CPU for parallel processing.

[0120] (2) Calculate the distortion coefficient, rotation coefficient matrix, transfer coefficient matrix, focal length, and lens main focal point through the calibration image, specifically as follows:

[0121] The distortion coefficient and its transformation are:

[0122]

[0123] Where (a1, a2) is the distortion coefficient,

[0124] is expressed as a two-dimensional plane coordinate normalized from a three-dimensional space coordinate, and its three-dimensional space coordinate, also called world coordinate (x, y, z), is a three-dimensional coordinate relative to the two-dimensional image obtained by calculating the depth of field from the explicit intersection point (generally the intersection point of the black and white chessboard) on the calibration image, which serves as the anchor reference of the lens group. Its normalized expression is:

[0125]

[0126] Rotation coefficient matrix R and transfer coefficient matrix T jointly construct external parameter matrix Q, P, which are expressed as follows:

[0127] Q = [R -1 -R -1 T] ∈ R 3×4

[0128] P = [RT] ∈ R 3×4

[0129] Using external parameter matrix Q, P, original image coordinate group X ω and corresponding coordinate group X are expressed as follows:

[0130]

[0131]

[0132] Wherein, focal length f, lens main focal point (c u ,c v ) jointly construct internal parameter matrix K c :

[0133]

[0134] Wherein, γ is the bending coefficient, and δ is the aspect ratio. This matrix K is the internal parameter matrix generated by comparing the lens to be calibrated with the reference lens. If the reference lens is compared with the lens to be calibrated, the secondary internal matrix is expressed as follows:

[0135]

[0136] Wherein Y = 0, 6 = 1.

[0137] (3) The coefficients calculated in step (2) are used to generate a transfer matrix (coordinate transfer matrix) through a formula, which is as follows:

[0138]

[0139]

[0140] Wherein, is the pixel coordinate on the canvas, is the corresponding pixel coordinate on the light field image to be processed.

[0141] (4) The coefficients approximately equal to zero in the transfer matrix are pruned, and the positions of the 0 vectors are recorded. The pruning process of the matrix is as follows:

[0142] Let the tolerance be If the absolute value of the matrix coefficient (element) is less than Then it is set to zero.

[0143]

[0144] Where the size of the tolerance can be adjusted according to the actual precision, if There is no loss of precision.

[0145] (5) Obtain the light field image data to be processed of the lens group, generate a black RGB canvas image of the same size as the original image, and traverse all pixel coordinates from the top left corner coordinate (u, v) of the image (0, 0).

[0146] (6) The zero vector position output by step (4) is decomposed for matrix calculation, and the corresponding original image coordinates (u', v') are obtained by inputting the canvas coordinates (u, v). The matrix calculation acceleration process is as follows:

[0147] Search matrix, record the position of non-zero value, and store the matrix sparsely. In the matrix calculation process, check the current multiplier, and if a 0 value is encountered, move to the next multiplier position until a non-zero value is encountered.

[0148] Specific step (6) traverses the RGB canvas image generated in step (5) from the top left corner (0, 0) using pixel coordinates (u, v), and obtains the corresponding coordinate points (u', v') of the original image through the transfer matrix and matrix acceleration calculation.

[0149] (7) Output the original image coordinates (u', v') corresponding to the current coordinates through step (6). To ensure accuracy, use linear interpolation to scale up the original image and the original image coordinates proportionally, and calculate the average of the nine data points near the coordinates (u', v'), and assign the RGB average value to the original coordinates (u, v). Repeat this step until the RGB canvas image is traversed. The traversed canvas image is the calibrated light field image.

[0150] The new coordinates obtained by step (6) are real number range, which must be processed by integerization. Integerization processing will cause a certain degree of information loss. In order to maintain the accuracy after integerization processing, the original image is scaled up by the same proportion as the new coordinate point using Lanczos linear interpolation method, and the RGB average of the nine points around the enlarged coordinate point is taken, which is represented as follows:

[0151]

[0152] This RGB value will be assigned to the canvas pixel (u, v) position, and the loop processing will be repeated until the canvas traversal is completed.

[0153] The processes of steps (1) to step (7) are only effective for a single lens group, and each lens group works independently, which means that the calibration process between lens groups is completely parallel. According to the number of CPUs and the number of lens groups, multiple lens groups can perform calibration work at the same time, thereby improving the efficiency of the work flow.

[0154] The above embodiments have the following advantages:

[0155] 1) Efficiency improvement. The existing technology has a large number of operations involving sparse matrices, and the zero operation can accelerate the processing of sparse matrices with controllable precision loss, and it is fully compatible with pruning technology, thereby achieving controllable secondary acceleration;

[0156] 2) Controllable accuracy. The pruning threshold in the operation process can be customized, and the balance between calculation speed and calculation accuracy can be adjusted as needed;

[0157] 3) Controllable cost. Calibration only requires CPU support, and does not require GPU for training and calculation. Parallel work flow can fully utilize computing resources.

[0158] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0159] Referring to Figure 2 , a structural block diagram of a light field image calibration device provided in an embodiment of the present application is shown, the light field image calibration device is applied to a light field camera array, the light field camera array includes a reference lens and a plurality of sub-lenses, each of the plurality of sub-lenses forms a lens group with the reference lens, and specifically can include the following modules:

[0160] The matrix acquisition module 201 is configured to acquire a coordinate transfer matrix of the lens group after calibration.

[0161] The matrix pruning module 202 is configured to set an element in the coordinate transfer matrix to zero when an absolute value of the element is less than a preset tolerance, and obtain a pruned coordinate transfer matrix.

[0162] The image acquisition module 203 is configured to acquire a light field image captured by the lens group, and generate a black RGB canvas image with the same size as the light field image based on the light field image.

[0163] The coordinate determination module 204 is configured to determine the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transfer matrix.

[0164] The RGB mean value acquisition module 205 is configured to acquire the RGB mean value of the surrounding area of the coordinates of the light field image.

[0165] The RGB mean value assignment module 206 is configured to assign the RGB mean value to the coordinates of the black RGB canvas image to obtain the calibrated light field image.

[0166] Optionally, the matrix acquisition module comprises:

[0167] The image acquisition sub-module is configured to acquire the calibration image of the calibration site shot by the lens group.

[0168] The parameter calculation sub-module is configured to calculate the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix, the focal length and the lens principal focal point of the lens group based on the calibration image.

[0169] The matrix determination sub-module is configured to determine the calibrated coordinate transfer matrix of the lens group according to the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix and the lens principal focal point.

[0170] Optionally, the calibration site comprises a black-and-white chessboard or other feature indicators.

[0171] Optionally, the coordinate determination module comprises:

[0172] The coordinate calculation sub-module is configured to traverse the coordinates in the black RGB canvas image and calculate the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transfer matrix.

[0173] Optionally, the method further comprises:

[0174] The skip zero operation module is configured to skip the multiplication operation of the zero value element in the pruned coordinate transfer matrix.

[0175] Optionally, the method further comprises:

[0176] The image magnification sub-module is configured to use the linear interpolation method to magnify the light field image by a preset multiple.

[0177] Optionally, the RGB mean value acquisition module comprises:

[0178] The RGB value acquisition sub-module is configured to acquire the RGB values of a preset number of data points in the surrounding area of the coordinates of the light field image.

[0179] The RGB mean value calculation submodule is configured to calculate the average value of the RGB values of the preset number of data points to obtain the RGB mean value of the surrounding area of the coordinate of the light field image.

[0180] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the part of the method embodiment.

[0181] In addition, the embodiment of the present application also provides an electronic device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the processes of the above-mentioned model generation method embodiment are realized, and the same technical effects are achieved. To avoid repetition, this will not be repeated here.

[0182] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by the processor, the processes of the above-mentioned model generation method embodiment are realized, and the same technical effects are achieved. To avoid repetition, this will not be repeated here. The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0183] Figure 3 A hardware structure diagram of an electronic device for implementing the embodiments of the present application.

[0184] The electronic device 300 includes, but is not limited to, a radio frequency unit 301, a network module 302, an audio output unit 303, an input unit 304, a sensor 305, a display unit 306, a user input unit 307, an interface unit 308, a memory 309, a processor 310, and a power supply 311, etc. Those skilled in the art can understand that the electronic device 300 can include more or less components, or combine certain components, or different component arrangements. Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or less components than the figure, or combine certain components, or different component arrangements. In the embodiment of the present application, the electronic device includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle terminal, a wearable device, and a pedometer, etc.

[0185] It should be appreciated that in the embodiments of the present application, the radio frequency unit 301 can be used for receiving and sending signals in the process of transmitting information or talking. Specifically, after receiving the downlink data from the base station, the radio frequency unit 301 processes the data for the processor 310. In addition, the radio frequency unit 301 sends the uplink data to the base station. Generally, the radio frequency unit 301 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 301 can also communicate with the network and other devices through a wireless communication system.

[0186] The electronic device provides the user with wireless broadband Internet access through the network module 302, such as helping the user to send and receive emails, browse web pages, and access streaming media, etc.

[0187] The audio output unit 303 can convert audio data received by the radio frequency unit 301 or the network module 302 or stored in the memory 309 into an audio signal and output as sound. Moreover, the audio output unit 303 can also provide audio output related to a specific function performed by the electronic device 300 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 303 includes a speaker, a buzzer, a receiver, etc.

[0188] The input unit 304 is used to receive audio or video signals. The input unit 304 can include a graphics processor (GPU) 3041 and a microphone 3042. The graphics processor 3041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 306. The image frame processed by the graphics processor 3041 can be stored in the memory 309 (or other storage medium) or transmitted via the radio frequency unit 301 or the network module 302. The microphone 3042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 301 in the case of a telephone call mode.

[0189] The electronic device 300 further includes at least one sensor 305, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 3061 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 3061 and / or the backlight when the electronic device 300 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, can detect the magnitude and direction of gravity, and can be used to identify the electronic device posture (such as screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, knocking), and the like. The sensor 305 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which will not be described here.

[0190] The display unit 306 is configured to display information input by a user or information provided to the user. The display unit 306 can include a display panel 3061, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0191] The user input unit 307 can be configured to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 307 includes a touch panel 3071 and other input devices 3072. The touch panel 3071, also known as a touch screen, can collect user touch operations (such as user operations on or near the touch panel 3071 using a finger, a stylus, or any suitable object or accessory) on or near it. The touch panel 3071 can include two parts, a touch detection device and a touch controller. The touch detection device detects the user's touch position and detects the signals generated by the touch operation, and transmits the signals to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 310, receives commands from the processor 310 and executes them. In addition, the touch panel 3071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 3071, the user input unit 307 can also include other input devices 3072. Specifically, the other input devices 3072 can include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, on-off buttons, etc.), trackballs, mice, joysticks, and the like, which will not be described here.

[0192] Further, the touch panel 3071 can be overlaid on the display panel 3031, and when the touch panel 3071 detects a touch operation thereon or adjacent thereto, it transmits to the processor 310 to determine the type of touch event, and then the processor 310 provides corresponding visual output on the display panel 3061 according to the type of touch event. Although in the above embodiment, the touch panel 3071 and the display panel 3031 are implemented as two independent components to realize the input and output functions of the electronic device, in some embodiments, the touch panel 3071 and the display panel 3061 can be integrated to realize the input and output functions of the electronic device, which is not limited here. Figure 3

[0193] The interface unit 308 is an interface for connecting external devices to the electronic device 300. For example, the external devices can include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, an earphone port, and the like. The interface unit 308 can be used to receive input (e.g., data information, power, and the like) from external devices and transmit the received input to one or more elements within the electronic device 300 or can be used to transmit data between the electronic device 300 and external devices.

[0194] The memory 309 can be used to store software programs and various data. The memory 309 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs (such as a sound play function, an image play function, and the like) required by at least one function, and the like; the data storage area can store data (such as audio data, a phone book, and the like) created according to the use of the mobile phone, and the like. In addition, the memory 309 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0195] The processor 310 is the control center of the electronic device, connects all parts of the electronic device through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 309 and calling data stored in the memory 309, and thus monitors the whole electronic device. The processor 310 can include one or more processing units; preferably, the processor 310 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 310.

[0196] ​The electronic device 300 can further include a power supply 311 (such as a battery) for powering the various components of the electronic device 300. In preferred embodiments, the power supply 311 is preferably coupled to the processor 310, and when operating the processor 310 with power management system, such that power required for the various components of the electronic device 300 is supplied and controlled by the power management system.

[0197] In addition, the electronic device 300 includes some function modules which are not shown here and will not be described here.

[0198] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0199] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0200] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, which are only illustrative and not limiting. Those skilled in the art can make many forms without departing from the purpose of the present application and the scope of the claims, which are all within the protection of the present application.

[0201] Those skilled in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0203] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0204] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0205] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0206] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various program codes that can be stored.

[0207] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for calibrating a light field image, characterized in that, The method, applied to a light field camera array, wherein the light field camera array includes a reference lens and a plurality of sub-lenses, each plurality of sub-lenses forming a lens group with the reference lens, comprises: Obtain the coordinate transition matrix of the calibrated lens group; When the absolute value of an element in the coordinate transition matrix is ​​less than a preset tolerance, the element is set to zero to obtain the pruned coordinate transition matrix. Acquire the light field image captured by the lens group, and generate a black RGB canvas image of the same size as the light field image based on the light field image; The coordinates of the light field image corresponding to the coordinates in the black RGB canvas image are determined by the pruned coordinate transfer matrix. In the process of determining the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image by the pruned coordinate transfer matrix, zero-value elements are skipped. Obtain the RGB mean value of the area surrounding the coordinates of the light field image; The RGB mean value is assigned to the coordinates of the black RGB canvas image to obtain the calibrated light field image.

2. The method according to claim 1, characterized in that, The process of obtaining the coordinate transformation matrix after the lens group calibration includes: Acquire calibration images of the calibration site captured by the lens group; Based on the calibration image, calculate the distortion coefficients, rotation coefficient matrix, transfer coefficient matrix, focal length, and principal focal length of the lens group; The coordinate transfer matrix of the calibrated lens group is determined based on the distortion coefficient, the rotation coefficient matrix, the transfer coefficient matrix, and the lens principal focal point.

3. The method according to claim 2, characterized in that, The designated area includes a black and white checkerboard or other distinctive markers.

4. The method according to claim 1, characterized in that, Determining the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image using the pruned coordinate transformation matrix includes: Traverse the coordinates in the black RGB canvas image, and calculate the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image using the pruned coordinate transition matrix.

5. The method according to claim 4, characterized in that, The step of calculating the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image using the pruned coordinate transition matrix further includes: Skip the multiplication operations of zero-value elements in the pruned coordinate transition matrix.

6. The method according to claim 1, characterized in that, Before obtaining the RGB mean value of the area surrounding the coordinates of the light field image, the method further includes: The light field image is magnified proportionally by a preset factor using linear interpolation.

7. The method according to claim 1, characterized in that, The step of obtaining the RGB mean value of the surrounding area of ​​the coordinates of the light field image includes: Obtain the RGB values ​​of a preset number of data points within the area surrounding the coordinates of the light field image; Calculate the average RGB values ​​of the preset number of data points to obtain the average RGB values ​​of the area surrounding the coordinates of the light field image.

8. A light field image calibration device, characterized in that, Applied to a light field camera array, the light field camera array includes a reference lens and several sub-lenses, each of the several sub-lenses forming a lens group with the reference lens, including: The matrix acquisition module is used to acquire the coordinate transformation matrix of the calibrated lens group; The matrix pruning module is used to set the elements to zero when the absolute value of an element in the coordinate transition matrix is ​​less than a preset tolerance, so as to obtain the pruned coordinate transition matrix. The image acquisition module is used to acquire the light field image captured by the lens group, and generate a black RGB canvas image of the same size as the light field image based on the light field image; The coordinate determination module is used to determine the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transition matrix, and skips zero-value elements in the process of determining the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transition matrix. An RGB mean value acquisition module is used to acquire the RGB mean value of the area surrounding the coordinates of the light field image; The RGB mean assignment module is used to assign the RGB mean to the coordinates of the black RGB canvas image to obtain a calibrated light field image.

9. The apparatus according to claim 8, characterized in that, The matrix acquisition module includes: The image acquisition submodule is used to acquire calibration images of the calibration site captured by the lens group; The parameter calculation submodule is used to calculate the distortion coefficients, rotation coefficient matrix, transfer coefficient matrix, focal length, and principal focal length of the lens group based on the calibration image. The matrix determination submodule is used to determine the coordinate transfer matrix of the lens group after calibration based on the distortion coefficients, the rotation coefficient matrix, the transfer coefficient matrix, and the lens principal focal length point.

10. The apparatus according to claim 9, characterized in that, The designated area includes a black and white checkerboard or other distinctive markers.

11. The apparatus according to claim 8, characterized in that, The coordinate determination module includes: The coordinate calculation submodule is used to traverse the coordinates in the black RGB canvas image and calculate the coordinates of the light field image corresponding to the coordinates in the black RGB canvas image through the pruned coordinate transition matrix.

12. The apparatus according to claim 11, characterized in that, Also includes: The zero-operation skip module is used to skip the multiplication operations of zero-value elements in the pruned coordinate transition matrix.

13. The apparatus according to claim 8, characterized in that, Also includes: The image magnification submodule is used to magnify the light field image proportionally by a preset factor using linear interpolation.

14. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.

15. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images