An adaptive sub-pixel precision light field data calibration method

Abstract

The present invention discloses an adaptive sub-pixel precision light field data calibration method, which relates to the technical field of sensor equipment, and includes the following steps: collecting original two-dimensional light field data, obtaining the horizontal and vertical deviation distances of the center point of the original two-dimensional light field data through row projection, column projection, and energy distribution rearrangement; determining the pixel center point of the original two-dimensional light field data according to the horizontal and vertical deviation distances of the center point, cutting and rearranging the original two-dimensional light field data according to the pixel center point, and obtaining four-dimensional phase space light field data; calculating the sub-pixel center of the original two-dimensional light field data according to the pixel center and the four-dimensional phase space light field data; scaling the original two-dimensional light field data using different scaling sizes according to the sub-pixel center, and after rearrangement in the four-dimensional phase space, filtering out the calibrated sub-pixel precision light field data according to the corresponding energy value. The adaptive sub-pixel precision light field data calibration of the present invention improves the resolution and quality of light field reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor equipment, and more particularly to an adaptive sub-pixel precision light field data calibration method. Background Art

[0002] Conventional sensors can only record the two-dimensional intensity information of light and lose the viewing angle information. Therefore, the two-dimensional imaging result based on the conventional lens and sensor system is the projection of the three-dimensional object onto the two-dimensional plane, and the depth information of the sample is lost. The four-dimensional phase space light field imaging technology based on microlenses can simultaneously record the spatial information and angle information of light, and can more completely describe the light in three-dimensional space. By modeling the point spread function of the system and using the point spread function to deconvolute the four-dimensional phase space data, the three-dimensional tomography results of the sample can be obtained.

[0003] The four-dimensional phase space data is generated by preprocessing the two-dimensional original light field data through pixel calibration, cropping and rearrangement, wherein the pixel calibration mainly includes the alignment of the light field perspective and the alignment of the microlens magnification. The misalignment of the light field perspective will cause the light field data to not match the point spread function used for reconstruction, which will lead to a decrease in the lateral and axial resolution of the reconstructed tomography results; the misalignment of the microlens magnification will cause aliasing and vignetting of light field data at different perspectives, which will lead to a decrease in imaging resolution and quality. Therefore, accurate four-dimensional phase space pixel calibration is of great significance for high-quality reconstruction. In the prior art, the original two-dimensional light field data is rearranged into a regular grid by manually calibrating the central perspective of the four-dimensional phase space, thereby achieving pixel calibration. However, the pixel calibration achieved by manual calibration has poor adaptability to different systems and different data, and can only achieve pixel-precision data calibration through regular sampling reconstruction.

[0004] Therefore, how to achieve sub-pixel precision light field data calibration and improve the adaptability of pixel calibration is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention provides an adaptive sub-pixel precision light field data calibration method, which implements adaptive sub-pixel precision light field data calibration for different systems and different sample data, thereby improving the resolution and quality of light field reconstruction.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An adaptive sub-pixel precision light field data calibration method comprises the following steps:

[0008] Step 1: Collect original two-dimensional light field data, and obtain the horizontal deviation distance and the vertical deviation distance of the center point of the original two-dimensional light field data through row projection, column projection, and energy distribution rearrangement;

[0009] Step 2: Determine the pixel center point of the original two-dimensional light field data according to the horizontal deviation distance of the center point and the vertical deviation distance of the center point, and cut and rearrange the original two-dimensional light field data according to the pixel center point to obtain four-dimensional phase space light field data;

[0010] Step 3: Calculate the sub-pixel center of the original two-dimensional light field data based on the pixel center and the four-dimensional phase space light field data;

[0011] Step 4: Scale the original two-dimensional light field data using different scaling sizes according to the sub-pixel center, construct several groups of scaled four-dimensional phase space light field data through four-dimensional phase space rearrangement, and filter out calibrated sub-pixel precision light field data according to the energy values ​​corresponding to the scaled four-dimensional phase space light field data.

[0012] Preferably, the specific steps of step 1 are:

[0013] Step 11: Collecting original two-dimensional light field data; the original two-dimensional light field data is a two-dimensional light field image, whose lateral and longitudinal dimensions are u*h and v*w respectively, where h and w represent the number of microlenses in the lateral and longitudinal directions of the microlens array light field imaging system, i.e., spatial coordinates; u and v represent the lateral and longitudinal field of view pixels of a single microlens, i.e., angular coordinates;

[0014] Performing line projection on the original two-dimensional light field data to obtain the lateral energy distribution of the original light field data;

[0015] The lateral energy distribution of the original light field data is rearranged into the lateral energy distribution of the microlens, and the lateral energy distribution of the microlens is line-projected to obtain a number (v) of lateral viewing angle energy distributions; the size of the lateral energy distribution of the microlens is v*w;

[0016] The distance from the center of the maximum lateral viewing angle energy distribution in the lateral viewing angle energy distribution is taken as the lateral deviation distance of the center point of the original two-dimensional light field data. ;

[0017] Step 12: Perform column projection on the original two-dimensional light field data to obtain the longitudinal energy distribution of the original light field data; rearrange the longitudinal energy distribution of the original light field data into the longitudinal energy distribution of the microlens, and perform column projection on the longitudinal energy distribution of the microlens to obtain a number (u) of longitudinal viewing angle energy distributions; take the distance of the maximum longitudinal viewing angle energy distribution from the center in the longitudinal viewing angle energy distribution as the longitudinal deviation distance of the center point of the original two-dimensional light field data ; The size of the longitudinal energy distribution of the microlens is u*h.

[0018] Preferably, the specific steps of step 2 include:

[0019] Step 21: Horizontal deviation distance according to the center point Longitudinal deviation distance from the center point Re-determine the pixel center point of the original 2D light field data ( , ), expressed as:

[0020] ;

[0021] in, and represents the horizontal coordinate and the vertical coordinate of the pixel center point respectively; h and w represent the number of horizontal microlenses and the number of vertical microlenses in the microlens array light field imaging system respectively, as spatial coordinates; u and v represent the horizontal field of view pixel and the vertical field of view pixel of a single microlens respectively, as angular coordinates;

[0022] Step 22: According to the pixel center point ( , ) The original two-dimensional light field data is cropped to obtain a center-aligned two-dimensional light field image; the center-aligned two-dimensional light field image is rearranged to obtain four-dimensional phase space light field data.

[0023] Preferably, the specific steps of rearranging the center-aligned two-dimensional light field image include:

[0024] Step 221: fix the angle coordinates, obtain the pixel values ​​of each group of angle coordinates in the center-aligned two-dimensional light field image at different spatial coordinates, and form a spatial coordinate sub-image of the angle coordinate with a size of h*w;

[0025] Step 222: Arrange the spatial coordinate sub-images corresponding to all angle coordinates in order to construct four-dimensional phase space light field data with a size of u*v*h*w.

[0026] Preferably, the specific steps of step 3 are:

[0027] Step 31: sum the pixel values ​​of the spatial coordinate sub-image under each set of angle coordinates in the four-dimensional phase space light field data to obtain the energy distribution of all viewing angles, the size of which is u*v;

[0028] Step 32: Fit the energy distribution of all viewing angles with a two-dimensional Gaussian function to obtain the two-dimensional coordinates of the maximum value of the energy distribution ( , );

[0029] Step 33: According to the pixel center point ( , ) and the two-dimensional coordinates of the maximum energy distribution ( , ), determine the sub-pixel center of the original two-dimensional light field data ( ), expressed as:

[0030] .

[0031] Preferably, the specific steps of step 4 include:

[0032] Step 41: Sub-pixel center ( ) as the origin, within a preset size range, traverse different scaling sizes at preset intervals to interpolate the original two-dimensional light field data to obtain scaled two-dimensional light field data corresponding to different scaling sizes;

[0033] Step 42: Rearrange the scaled two-dimensional light field data corresponding to different scaling sizes in four-dimensional phase space to obtain scaled four-dimensional phase space light field data; the rearrangement step is the same as the above steps 221 to 222;

[0034] Step 43: sum up the intensities of all pixel values ​​of the spatial coordinate sub-image corresponding to the preset angle coordinates of each group of scaled four-dimensional phase space light field data to obtain the total energy of the preset angle coordinates, and select the scaled four-dimensional phase space light field data corresponding to the maximum total energy as the calibrated sub-pixel precision light field data. The scaled size corresponding to the maximum total energy is the optimal scaled size.

[0035] Preferably, the preset size range is [u*h-10, u*h+10]; the preset interval is 1; and the preset angle coordinates are [2:u-1, 2:v-1].

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an adaptive sub-pixel precision light field data calibration method, which performs light field data calibration on the original light field data of different systems and different samples through an automatic calculation method. Different from the traditional solution, the present invention performs a coarse calibration by calculating and adjusting the deviation distance, preliminarily determines the pixel center of the light field data, and then performs a fine calibration based on the adjusted square data pixel center to determine the sub-pixel center of the light field data and the correct sampling interval, and finally obtains the calibrated sub-pixel precision light field data. The present invention ultimately improves the resolution and quality of light field reconstruction through a sub-pixel precision pixel calibration scheme. Compared with manual calibration, the present invention realizes adaptive pixel calibration of light field data of different systems and different samples; compared with regular sampling, the present invention realizes sub-pixel precision calibration of light field data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of an adaptive sub-pixel precision light field data calibration method in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The embodiment of the present invention discloses an adaptive sub-pixel precision light field data calibration method, such as Figure 1 As shown, the following steps are included:

[0041] S1: Collect original two-dimensional light field data, and obtain the horizontal deviation distance and the vertical deviation distance of the center point of the original two-dimensional light field data through row projection, column projection, and energy distribution rearrangement;

[0042] S2: determining the pixel center point of the original two-dimensional light field data according to the horizontal deviation distance of the center point and the vertical deviation distance of the center point, and cutting and rearranging the original two-dimensional light field data according to the pixel center point to obtain four-dimensional phase space light field data;

[0043] S3: Calculate the sub-pixel center of the original two-dimensional light field data according to the pixel center and the four-dimensional phase space light field data;

[0044] S4: scaling the original two-dimensional light field data using different scaling sizes according to the sub-pixel center, constructing scaled four-dimensional phase space light field data through four-dimensional phase space rearrangement, and filtering out calibrated sub-pixel precision light field data according to energy values ​​corresponding to the scaled four-dimensional phase space light field data.

[0045] On the other hand, in a specific embodiment, an adaptive sub-pixel precision light field data calibration method specifically includes:

[0046] S1, coarse calibration;

[0047] S11, collecting original two-dimensional light field data, the horizontal size and vertical size of the original two-dimensional light field image are u*h and v*w respectively, where h and w represent the number of microlenses in the horizontal and vertical directions in the microlens array light field imaging system, i.e., spatial coordinates; u and v represent the horizontal and vertical field of view pixels of a single microlens, i.e., angular coordinates;

[0048] S12, perform line projection on the original two-dimensional light field data to obtain the lateral energy distribution of the original light field data; rearrange the lateral energy distribution of the original two-dimensional light field data into a microlens lateral energy distribution with a size of v*w; continue to perform line projection on the microlens lateral energy distribution to eliminate the influence of the shooting sample on the energy distribution of the viewing angle, and obtain the energy distribution of v lateral viewing angles; find the distance from the center of the maximum value in the energy distribution of v lateral viewing angles as the lateral deviation distance of the center point of the original light field data ; Perform column projection on the original two-dimensional light field data to obtain the longitudinal energy distribution of the original light field data; rearrange the longitudinal energy distribution of the original two-dimensional light field data into the longitudinal energy distribution of the microlens with a size of u*h; continue to perform column projection on the longitudinal energy distribution of the microlens to eliminate the influence of the shooting sample on the energy distribution of the viewing angle, and obtain the energy distribution of the longitudinal u viewing angles; find the distance from the center of the maximum value in the energy distribution of the longitudinal u viewing angles as the longitudinal deviation distance of the center point of the original light field data ;

[0049] S13. According to and Re-determine the pixel center point of the original 2D light field data and , expressed as:

[0050] ;

[0051] After cropping and rearranging, the four-dimensional phase space light field data of the light field is obtained: according to the pixel center point ( , ) The original two-dimensional light field data is cropped to obtain a center-aligned two-dimensional light field image; the center-aligned two-dimensional light field image is rearranged to obtain four-dimensional phase space light field data; the rearrangement process is: fix the angle coordinate, obtain the pixel value of the angle coordinate in the center-aligned two-dimensional light field image at different spatial coordinates, and form a spatial coordinate sub-image of the angle coordinate with a size of h*w; the spatial coordinate sub-images of all angle coordinates are arranged in order to form four-dimensional phase space light field data with a size of u*v*h*w; the above operation rearranges the pixels of the original white image two-dimensional light field image into four-dimensional phase space light field data;

[0052] S2, fine calibration;

[0053] S21, summing the pixel values ​​of the spatial coordinate sub-image at each angle coordinate of the four-dimensional phase space data, that is, summing the last two dimensions of the four-dimensional phase space data, to obtain the energy distribution of all viewing angles, the size of which is u*v;

[0054] S22, the visual energy distribution is similar to Gaussian distribution, and the form is as follows:

[0055] ;

[0056] in and Represents the center point of the Gaussian function; the energy distribution of all the above perspectives is fitted with a two-dimensional Gaussian function to obtain the two-dimensional coordinates of the maximum value of the energy distribution and ;

[0057] S23, pixel accuracy center obtained by row and column projection in S13 ( , ), and the two-dimensional coordinates of the maximum value of the energy distribution of the Gaussian fit ( , ) (i.e., sub-pixel precision offset), determine the sub-pixel center of the light field data ( ):

[0058] ;

[0059] S24, the sub-pixel center of the original two-dimensional light field data ( ) is the origin, from u*h-10 to u*h+10, with 1 as the interval, traverse different scaling sizes to interpolate the two-dimensional light field data to obtain the scaled image; the scaled two-dimensional light field data is rearranged in four-dimensional phase space; the sum of all pixel value intensities of the spatial coordinate sub-image corresponding to the angle coordinate [2:u-1,2:v-1] of the four-dimensional phase space data is calculated to obtain the total energy of the angle coordinate. When the energy is the largest, it is the optimal scaling size; the four-dimensional phase space rearrangement process is the same as the above S13 process;

[0060] S25, selecting the four-dimensional phase space data obtained by rearrangement of the optimal scaling size as the corrected sub-pixel precision light field data.

[0061] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0062] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An adaptive sub-pixel precision light field data calibration method, characterized in that: The following steps are involved: Step 1: Collect original two-dimensional light field data, and obtain the horizontal deviation distance and the vertical deviation distance of the center point of the original two-dimensional light field data through row projection, column projection, and energy distribution rearrangement; Step 2: determining the pixel center point of the original two-dimensional light field data according to the horizontal deviation distance of the center point and the vertical deviation distance of the center point, and cutting and rearranging the original two-dimensional light field data according to the pixel center point to obtain four-dimensional phase space light field data; Step 3: Calculate the sub-pixel center of the original two-dimensional light field data according to the pixel center point and the four-dimensional phase space light field data; Step 4: scaling the original two-dimensional light field data using different scaling sizes according to the sub-pixel center, constructing several groups of scaled four-dimensional phase space light field data through four-dimensional phase space rearrangement, and screening out calibrated sub-pixel precision light field data according to the corresponding energy values ​​of the scaled four-dimensional phase space light field data; The specific steps of step 1 are: Step 11: Collect the original two-dimensional light field data, perform line projection on the original two-dimensional light field data, and obtain the lateral energy distribution of the original light field data; rearrange the lateral energy distribution of the original light field data into a microlens lateral energy distribution, and perform line projection on the microlens lateral energy distribution to obtain a plurality of lateral viewing angle energy distributions; take the distance of the maximum lateral viewing angle energy distribution in the lateral viewing angle energy distribution from the center as the lateral deviation distance X of the center point of the original two-dimensional light field data shift ; Step 12: Perform column projection on the original two-dimensional light field data to obtain the longitudinal energy distribution of the original light field data; rearrange the longitudinal energy distribution of the original light field data into the longitudinal energy distribution of the microlens, and perform column projection on the longitudinal energy distribution of the microlens to obtain a plurality of longitudinal viewing angle energy distributions; take the distance of the maximum longitudinal viewing angle energy distribution from the center in the longitudinal viewing angle energy distribution as the longitudinal deviation distance Y of the center point of the original two-dimensional light field data shift ; The specific steps of step 2 include: Step 21: Determine the horizontal deviation distance X according to the center point shift The longitudinal deviation distance Y from the center point shift Re-determine the pixel center point (X) of the original two-dimensional light field data coarse , Y coarse ), expressed as: Among them, X coarse and Y coarse represents the horizontal coordinate and the vertical coordinate of the pixel center point respectively; h and w represent the number of horizontal microlenses and the number of vertical microlenses in the microlens array light field imaging system respectively, as spatial coordinates; u and v represent the horizontal field of view pixel and the vertical field of view pixel of a single microlens respectively, as angular coordinates; Step 22: According to the pixel center point (X coarse , Y coarse ) cropping the original two-dimensional light field data to obtain a center-aligned two-dimensional light field image; rearranging the center-aligned two-dimensional light field image to obtain the four-dimensional phase space light field data; The specific steps of rearranging the center-aligned two-dimensional light field map include: Step 221: fix the angle coordinates, obtain the pixel values ​​of each group of the angle coordinates in different spatial coordinates in the center-aligned two-dimensional light field image, and form a spatial coordinate sub-image of the angle coordinates; Step 222: Arrange the spatial coordinate sub-images corresponding to all the angle coordinates in order to construct the four-dimensional phase space light field data; The specific steps of step 3 are: Step 31: summing the pixel values ​​of the spatial coordinate sub-image under each group of the angle coordinates in the four-dimensional phase space light field data to obtain energy distribution of all viewing angles; Step 32: Fit the energy distribution of all viewing angles with a two-dimensional Gaussian function to obtain the two-dimensional coordinates of the maximum value of the energy distribution (subX shift ,subY shift ); Step 33: According to the pixel center point (X coarse , Y coarse ) and the two-dimensional coordinates of the maximum value of the energy distribution (subX shift , subX shift ), determine the sub-pixel center (X fine ,Y fine ), expressed as: X fine Y fine =X coarse +subX shift ,Y coarse +subY shift ; The specific steps of step 4 include: Step 41: Taking the sub-pixel center (X fine , Y fine ) as the origin, within a preset size range, traverse different scaling sizes at preset intervals to interpolate the original two-dimensional light field data to obtain scaled two-dimensional light field data corresponding to different scaling sizes; Step 42: Rearrange the scaled two-dimensional light field data corresponding to different scaling sizes in four-dimensional phase space to obtain the scaled four-dimensional phase space light field data; Step 43: Sum the intensities of all pixel values ​​of the spatial coordinate sub-image corresponding to the preset angle coordinates of each group of the scaled four-dimensional phase space light field data to obtain the total energy of the preset angle coordinates, and select the scaled four-dimensional phase space light field data corresponding to the maximum total energy as the calibrated sub-pixel precision light field data.

2. The adaptive sub-pixel precision light field data calibration method according to claim 1, characterized in that: The preset size range is [u*h-10, u*h+10]; the preset interval is 1; the preset angle coordinates are [2:u-1,2:v-1]; h represents the number of lateral microlenses in the microlens array light field imaging system, and u and v represent the lateral field of view pixels and the longitudinal field of view pixels of a single microlens, respectively.

Images