A light field camera calibration method to eliminate primary lens distortion

By calibrating and resampling the light field camera light equation, using the LCD display as the target, the main lens distortion of the light field camera is eliminated, and the problem of low three-dimensional reconstruction accuracy of the light field camera is solved, and high-precision three-dimensional reconstruction is achieved Effect.

CN114782545BActive Publication Date: 2024-11-19SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210272045.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-11-19
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

The prior art cannot effectively eliminate the distortion of the main lens of the light field camera, resulting in the impact on three-dimensional reconstruction, and the light field camera parameters cannot be accurately calibrated.

Method used

The light field camera is regarded as a black box system that realizes pixel-to-light mapping. By calibrating and resampling the light equation, the main lens distortion is eliminated, the LCD display is used as a target to estimate the target plane attitude, fit the light equation, and The electrode plan is extracted in a distortion-free light field and uniform resampling is performed to eliminate linear structural deformation.

Benefits of technology

It effectively eliminates the influence of the main lens distortion of the light field camera on three-dimensional reconstruction, improves the accuracy of the three-dimensional reconstruction, is a concise and clear calibration method, and uses a simple and easy-to-get LCD to obtain high-precision feature point sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782545B_ABST
    Figure CN114782545B_ABST
Patent Text Reader

Abstract

A light field camera calibration method for eliminating the distortion of the main lens, which regards the light field camera as a black box system for realizing pixel and light mapping, and makes the central viewpoint imaging model of the light field camera equivalent to the classical camera model, and estimates the attitude parameters of the target plane; uses the camera extrinsic matrix to obtain the coordinates of the intersection of the light emitted from each pixel of the light field camera and the target plane, transforms the relative position of the camera and the target multiple times to generate multiple intersections, fits the parameter equation of the light corresponding to the pixel in three-dimensional space, performs uniform resampling processing on the light, eliminates the calibration error, and obtains the distortion-free light field distribution; based on the three-dimensional reconstruction of the light field of the polar plane diagram, the polar plane diagram extracted in the distortion-free light field is used to calibrate and resample the light equation of the light field camera, and the linear structure deformation phenomenon caused by the distortion of the main lens is eliminated. The present invention solves the problem of linear structure deformation of the polar plane diagram caused by the distortion of the main lens of the light field camera, and eliminates the distortion of the main lens of the light field camera.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for calibrating a light field camera, and belongs to the field of light field three-dimensional imaging and measurement. Background Art

[0002] The classic imaging system records a two-dimensional image, which is equivalent to the two-dimensional projection of spatial three-dimensional information on the image plane from a mathematical point of view. The depth information of the target and the scene is lost during the projection process. If you want to restore the target more realistically, or conduct quantitative analysis of the target's geometric shape, position, internal structure, etc., you need to restore the lost depth information and three-dimensional structure from the two-dimensional image. The relevant technologies are collectively referred to as three-dimensional imaging technology. With the continuous development of science and technology and the gradual improvement of production requirements, the research on three-dimensional measurement technology has made great progress to meet the growing and increasingly stringent measurement needs. Driven by the needs of social production, three-dimensional measurement technology is increasingly widely used in various fields such as surveying and mapping navigation, machine vision, reverse engineering, industrial manufacturing, quality control, cultural relics protection, biomedicine, film and television entertainment, etc.

[0003] Light field imaging technology has emerged in recent years. Compared with classical imaging, it records the direction of light in addition to the intensity of light, breaking through some limitations of classical imaging. This feature also makes light field imaging widely used and has more possibilities in the fields of three-dimensional imaging and three-dimensional measurement. On this basis, the Stanford Computer Graphics Laboratory designed a microlens array light field camera and a light field microscope. The laboratory analyzed the integration process of the plenoptic camera, transformed the digital focusing process into the Fourier domain, applied the Fourier slice theorem to achieve rapid changes in the depth of focus of the image, and successively launched two consumer-level light field cameras.

[0004] Compared with traditional cameras, light field cameras add a microlens array between the sensor and the main lens, located at the focal plane of the main lens. The sensor is located at the focal plane of the microlens. In addition to recording light intensity information, it also records light direction information, enabling it to extract multi-viewpoint images under a single exposure. The number of viewpoints is equal to the number of sensor pixels behind a single microlens, and the resolution of a single viewpoint image is equal to the number of microlenses. Since the edge viewpoint distortion model of the light field camera is quite different from that of the classic camera, it is impossible to accurately calibrate the parameters of the light field camera.

[0005] Because the distortion of the main lens of a light field camera can easily cause deformation of the linear structure of the polar plane graph, the existing technology has not yet achieved successful modeling of the distortion of a light field camera, and cannot solve the problem of eliminating the distortion of the main lens of a light field camera, and cannot eliminate the impact of the distortion of the main lens of a light field camera on three-dimensional reconstruction. Summary of the invention

[0006] The present invention aims at the problem of main lens distortion in existing light field cameras, and provides a light field camera calibration method that can eliminate main lens distortion. The light field camera is regarded as a black box system that realizes pixel and light mapping, and the main lens distortion of the light field camera is eliminated by calibrating and resampling the light equation. The method effectively eliminates the influence of the main lens distortion of the light field camera on three-dimensional reconstruction when the light field camera distortion has not been successfully modeled.

[0007] The light field camera calibration method for eliminating main lens distortion of the present invention adopts the following technical solutions:

[0008] The light field camera is regarded as a black box system that realizes the mapping between pixels and light. The central viewpoint imaging model of the light field camera is equivalent to the classical camera model (pinhole model), and the attitude parameters of the target plane are estimated. The camera extrinsic matrix (rotation and translation matrix) is used to obtain the coordinates of the intersection of the light emitted from each pixel of the light field camera and the target plane. The relative position of the camera and the target is transformed multiple times to produce multiple intersections, and then the parametric equation of the light in three-dimensional space corresponding to the pixel is fitted (at this time, it is believed that the calibrated light equation is not affected by the distortion of the main lens of the light field camera), and the light is uniformly resampled to eliminate the calibration error, so as to obtain a distortion-free light field distribution. The polar plane image is extracted in the distortion-free light field, and the deformation of the linear structure of the inclined straight line in the polar plane image caused by the distortion of the main lens is eliminated.

[0009] The target is a display (flat liquid crystal display). As a flat target, a pattern containing phase information is displayed on the display and captured by a light field camera. Since the light field camera has the function of obtaining multi-viewpoint images with a single exposure, the distortion model of the central viewpoint is approximately regarded as a classic camera distortion model, and then the target plane posture is estimated. A flat liquid crystal display is used as the flat target required for camera calibration. The required devices are simple and easy to obtain; and different patterns can be conveniently displayed. Using the pixels in the display as feature points, a dense and high-precision feature point set can be flexibly and conveniently obtained with the help of phase information.

[0010] The light field reconstruction process of the polar plane image is:

[0011] A four-dimensional light field is used, that is, all light rays inside the light field are characterized by four-dimensional functions of the two parallel planes through which they pass; two vertical direction parameters are fixed, and horizontal two-dimensional slices of the four-dimensional light field, namely the epipolar plane, are extracted. The vertical two-dimensional slices are similar. The epipolar plane has a clear linear structure, in which each inclined straight line corresponds to a scene target point, and the depth of the point has a mapping relationship with the slope of the straight line, so the final scene depth reconstruction depends on solving the slope of the inclined straight line of the epipolar plane.

[0012] The intersection points of the parametric equation of the light in three-dimensional space (the light equation obtained by calibration) are distributed at a certain depth plane, and a uniform grid is established. The distortion of the main lens is eliminated by resampling the four-dimensional light field. Compared with the polar plane map before resampling, the linear structure deformation phenomenon is eliminated, which improves the accuracy of three-dimensional reconstruction.

[0013] The present invention adopts the three-dimensional reconstruction of the light field based on the polar plane graph, utilizes the calibration and resampling of the light equation of the light field camera, solves the problem of linear structure deformation of the polar plane graph caused by the distortion of the main lens of the light field camera, and successfully eliminates the distortion of the main lens of the light field camera.

[0014] The present invention has the following characteristics:

[0015] (1) When the light field camera parameters cannot be accurately calibrated, the present invention abandons the structured model description and regards the light field camera as a black box system that realizes pixel and light mapping, and realizes the light field camera calibration by calibrating the light equation.

[0016] (2) The present invention approximates the central viewpoint model of the light field camera as a classical camera, estimates the target plane posture information, and obtains the intersection coordinates through the rotation and translation matrix. The light equation is fitted from the intersection coordinates, and then the light is uniformly resampled to eliminate the calibration error, so that the distortion-free light field distribution can be obtained. The polar plane diagram is extracted from the distortion-free light field, and the linear structure deformation caused by the main lens distortion is eliminated, thereby improving the three-dimensional reconstruction accuracy of the light field camera.

[0017] (3) The present invention uses a liquid crystal display as a camera calibration target, and the required devices are simple and easy to obtain; the pixels in the display are used as feature points, so that a dense, high-precision feature point set can be obtained flexibly and conveniently;

[0018] (4) Based on geometric optics, the present invention designs a high-precision calibration method that is concise, clear in thought, and easy to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram of the calibration result of the light equation in the present invention (views at two different angles).

[0020] Figure 2 Schematic diagram of uniform light resampling in the present invention, (a) is a schematic diagram of the distribution of light intersections containing errors before resampling, and (b) is a schematic diagram of the uniform distribution of light intersections after resampling.

[0021] Figure 3 The present invention is a flow chart of a light field camera calibration method for eliminating main lens distortion. DETAILED DESCRIPTION

[0022] Since the edge viewpoint distortion model of the light field camera is quite different from that of the classical camera, it is impossible to accurately calibrate the light field camera parameters. Therefore, in the absence of a reasonable light field camera distortion model, the present invention abandons the structured model description and regards the light field camera as a black box system that realizes pixel and light mapping, and realizes the calibration of the light field camera by calibrating the light equation. The central viewpoint imaging model of the light field camera is approximately equivalent to the classical camera model, the target plane posture is estimated, and the intersection of the light and the target surface is calculated using the conversion relationship of the target plane posture parameters (i.e., the rotation and translation matrix), and then the light equation is fitted. A depth plane is selected outside the main lens for light resampling to obtain a distortion-free light field distribution. The original light field of the scene to be measured is captured by the light field camera, and the light field distribution that is not affected by distortion can be obtained by two-dimensional linear interpolation. At this time, the polar plane image is re-extracted, and the linear structure deformation in the polar plane image can be eliminated.

[0023] The present invention only regards the central viewpoint model of the light field camera as a classic camera model (also known as the pinhole model) in order to determine the target posture. The pinhole model describes the transformation relationship of four coordinate systems in the camera imaging process, including: (1) two-dimensional image coordinate system; (2) two-dimensional imaging plane coordinate system; (3) three-dimensional camera coordinate system; (4) three-dimensional world coordinate system. The purpose of camera calibration is to find the correspondence between three-dimensional space and two-dimensional image. This correspondence can be achieved through two transformation relationships: first, the world coordinate system is converted to the camera coordinate system, and the parameters in the solution process are the external parameters of the camera; second, the camera coordinate system is converted to the image coordinate system. In this part of the solution process, the parameters are the internal parameters of the camera (the following formula is represented by K). The final transformation relationship between the imaging plane coordinate system and the world coordinate system is expressed as:

[0024] Since the edge viewpoint distortion model of the light field camera is quite different from that of the classical camera, it is impossible to accurately complete the light field camera parameter calibration through this process. To obtain the light equation, the spatial coordinates of multiple points on the straight line need to be known, so a plane target containing dense target points needs to be transformed multiple times relative to the camera. The present invention uses a liquid crystal display as the plane target required for camera calibration. The experimental system consists of a liquid crystal display and a light field camera, wherein the single viewpoint image resolution of the light field camera is 625×434, and the viewpoint resolution is 15×15.

[0025] The vertical and horizontal stripes containing phase information are captured by the light field camera after being displayed on a flat liquid crystal display. The present invention uses phase auxiliary information and high-density display pixels as the feature points required for calibration. After the external parameters of the target plane under different postures relative to the camera are calibrated, the coordinates of the intersection of the light and the target surface can be solved: in is the phase value in the horizontal direction; l H is the horizontal length of a single pixel on the display; T is the stripe period; The vertical phase expansion value l V Z is the vertical length of a single pixel on the display; w = 0. Finally, multiple intersection coordinates are used to fit the light equation of the pixel: Among them, x0, y0, m, and n are all parameters of the fitted straight line equation. At this time, it is believed that the calibrated light equation is not affected by the distortion of the main lens of the light field camera. Figure 1 The calibration results of the ray equations for two views at different angles are given.

[0026] Based on the above four-dimensional light field dual-plane model, any depth value plane can be selected outside the main lens during the resampling process. Different depths of the resampling plane will only lead to different slopes and depth mapping coefficients. Substituting the plane selected depth value into the light equation can obtain the intersection distribution of all light rays at this depth, observe the intersection distribution, establish a suitable equidistant grid, and uniformly resample the light on this plane, eliminating the error in the calibration of the light equation. The uniform sampling diagram is shown in the figure. Figure 2 At this point, the light equation calibration is complete.

[0027] By calibrating the light equation, we can obtain the light field distribution without distortion on this plane, and then use it as a sample to interpolate the light field distribution in this grid that is not affected by distortion under different viewpoints using two-dimensional linear interpolation. At this point, the undistorted light field is re-extracted, and the straight lines in the re-extracted polar plane image no longer have linear structural deformation. Then the subsequent mapping coefficient calibration is completed.

[0028] like Figure 3 The process shown in the figure uses one-dimensional linear interpolation to find the precise sub-pixel coordinates on the straight line in the polar plane, and then fits the slope of the straight line. The polynomial fitting method is selected to estimate the relationship between the slope and the depth mapping. Taking the third-order polynomial as an example, the scene depth Z = p1×s 3 +p2×s 2 ×p3×s+p4, where s is the slope of the line in the polar plane. Next, the slope of the polar plane extracted after depth resampling is calibrated with the depth mapping coefficients p1, p2, p3, and p4 under fixed focus.

[0029] Under the central viewpoint, the light emitted by each pixel has corresponding intersection points with the target surface of different postures. The depth values ​​of these multiple intersection points and the slope value corresponding to the pixel can be used to fit the slope depth mapping coefficient of the pixel. When reconstructing any object later, its depth coefficient will not change under fixed focus.

[0030] Finally, when performing three-dimensional reconstruction of any object, the undistorted light field can be resampled through the calibrated intersection distribution and grid coordinates to solve the depth coordinates.

Claims

1. A light field camera calibration method for eliminating primary lens distortion, characterized by: The light field camera is regarded as a black box system that realizes the mapping between pixels and light. The central viewpoint imaging model of the light field camera is equivalent to the classical camera model, and the attitude parameters of the target plane are estimated. The camera extrinsic matrix, that is, the attitude parameters of the target plane, is used to obtain the coordinates of the intersection of the light emitted from each pixel of the light field camera and the target plane. The relative position of the camera and the target is transformed multiple times to generate multiple intersections, and then the parametric equation of the light in three-dimensional space corresponding to the pixels is fitted. Based on the four-dimensional light field two-plane model, an arbitrary depth value plane is selected outside the main lens. The different depths selected by the resampling plane will only lead to different slopes and depth mapping coefficients. The depth value selected by the plane is substituted into the light equation to obtain the intersection distribution of all light rays at this depth. An equidistant grid is established, and the light is uniformly resampled on this plane to eliminate the error in the calibration of the light equation; through the calibration of the light equation, the light field distribution without distortion on this plane is obtained, and then it is used as a sample, and two-dimensional linear interpolation is used to interpolate the light field distribution that is not affected by distortion in this grid under different viewpoints. At this time, the undistorted light field is re-extracted, and the straight line in the re-extracted polar plane image no longer has linear structural deformation; one-dimensional linear interpolation is used to find the precise sub-pixel coordinates of the straight line on the polar plane image, and then the slope of the straight line is fitted, and the polynomial fitting method is selected to estimate the relationship between the slope and the depth mapping; under the fixed focus condition, the slope and depth mapping coefficient of the polar plane image extracted after the depth resampling are calibrated.

2. The light field camera calibration method for eliminating primary lens distortion according to claim 1, wherein: The target is a flat liquid crystal display. As a planar target, a pattern containing phase information is displayed on the display and captured by a light field camera. Since the light field camera has the function of obtaining multi-viewpoint images with a single exposure, the distortion model of the central viewpoint is approximately regarded as the classic camera distortion model, and the target planar posture information is then estimated.