A calibration method for a polarization camera array applicable to a multi-layer medium environment

Abstract

The present invention discloses a calibration method for a polarization camera array applicable to a multi-layer medium environment. First, a refraction imaging model of the camera in the multi-layer medium environment is established, and a corresponding calibration algorithm is designed based on this refraction imaging model to calibrate the internal parameters of the camera and related refraction parameters. At the same time, the external parameters of the calibration cameras of the polarization camera array are calibrated. Two sets of three-dimensional point cloud data are obtained respectively from the intensity image and the polarization image obtained by the camera array, and the two sets of point cloud data are fused, and the external parameters of the camera array are calibrated using the fused three-dimensional point cloud data. The present invention can achieve higher-precision calibration of the external parameters of the camera and is more applicable to scenarios with a single texture and unable to extract effective feature points.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a calibration method for a polarization camera array. Background Art

[0002] One of the basic tasks of computer vision is to calculate the geometric information of an object in three-dimensional space based on the image information obtained by a camera, and thereby reconstruct and identify the object. The mutual relationship between the three-dimensional geometric position of a certain point on the surface of a spatial object and the corresponding point in the image is determined by the geometric model of camera imaging, and these geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very crucial link, and the accuracy of its calibration results and the stability of the algorithm directly affect the accuracy of the results generated by the camera's work. Therefore, doing a good job in camera calibration is a prerequisite for doing subsequent work, and improving the calibration accuracy is the focus of scientific research work.

[0003] The calibration and vision measurement technology of cameras in the air has been very mature. Different from taking pictures in the air, refraction phenomena will occur in a multi-layer medium environment, and the refraction phenomena will affect the traditional perspective imaging model in the air, resulting in the inability to use existing camera calibration algorithms. At the same time, under relatively harsh environmental conditions, the images captured by the camera may be of poor quality, which will also have a great impact on the calibration accuracy of the camera.

[0004] In summary, when calibrating a camera in a multi-layer medium environment, the traditional calibration method in the air cannot be used. It is necessary to establish a camera imaging model in the multi-layer medium environment and design a corresponding calibration algorithm according to this imaging model. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a calibration method for a polarization camera array applicable to a multi-layer medium environment. First, a refraction imaging model of the camera in the multi-layer medium environment is established, and a corresponding calibration algorithm is designed based on this refraction imaging model to calibrate the internal parameters of the camera and related refraction parameters; at the same time, the external parameters of the polarization camera array are calibrated. Two sets of three-dimensional point cloud data are obtained from the intensity image and the polarization image obtained by the camera array respectively, and the two sets of point cloud data are fused, and the external parameters of the camera array are calibrated using the fused three-dimensional point cloud data. The present invention can achieve a higher-precision calibration of the camera's external parameters and is more applicable to scenarios with a single texture and no effective feature points that can be extracted.

[0006] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0007] Step 1: First, analyze the influence of refraction on camera imaging in the presence of single-layer and double-layer media, derive the corresponding camera imaging model, and finally summarize the camera imaging model in a multi-layer media environment as follows:

[0008] Step 1-1: The camera imaging model in air is approximated as a simple perspective imaging model; the three-dimensional points in space correspond one-to-one with the two-dimensional image pixel points formed on the camera plane, which is expressed by Equation (1):

[0009]

[0010] where the P matrix represents the imaging process, that is, the correspondence between the three-dimensional coordinates in space and the two-dimensional coordinates of the pixel points on the imaging plane; [u v 1] T represents the two-dimensional image pixel point, [x w y w z w T represents the three-dimensional point coordinates in the world coordinate system;

[0011] Specifically, it is expressed as follows:

[0012]

[0013] where k represents the proportionality coefficient, represents the internal parameters of the camera, f represents the camera focal length, represents the transformation matrix, [x w y w z w T represents the three-dimensional point coordinates in the world coordinate system;

[0014] Step 1-2: Equation (2) is the perspective imaging model in air. Considering the influence of refraction, a camera imaging model in a multi-layer media environment is established; first, a single-layer media refraction imaging model is established;

[0015] Let d be the perpendicular distance from the refraction plane to the camera center, (α l1 β l1 γ l1 ) T and (α a β a γ a ) T respectively represent the direction vectors of the light rays in different media layers, (x r y r z r ) T is the focus of the incident light ray and the refraction plane, θ l1 and θ a ​​are the incident angle and the refraction angle of the light; the relationship between the direction vectors of the light before and after refraction is expressed as:

[0016]

[0017] Meanwhile, the refraction angle and the incident angle satisfy Snell's law: n l1 sinθ l1 = n a sinθ a , then we get:

[0018]

[0019]

[0020]

[0021]

[0022] where, n a respectively represent the refractive index of the air medium, and n l1 represents the refractive index of the medium layer 1;

[0023] The relationship between the direction vectors of the light before and after refraction is obtained as:

[0024]

[0025] Assume that (x u , y u ) is the two-dimensional physical coordinate of the imaging point, and the direction vector of the incident light after refraction is obtained as:

[0026]

[0027] Let Calculate the relationship between the direction vector of the incident light before refraction and the two-dimensional physical coordinate of the imaging point as:

[0028]

[0029] Assume that the coordinate of the object in the camera coordinate system is represented in the following form:

[0030]

[0031] where, [x c y c z c T represents the coordinate of the object in the camera coordinate system, and [x r y r z r T ​​Indicates the intersection position of the light ray and the refraction surface:

[0032]

[0033] where (x u , y u ) T are the two-dimensional physical coordinates of the imaging point;

[0034] Obtain:

[0035]

[0036] The relationship between the coordinates of the object point in the camera coordinate system and the two-dimensional physical coordinates of the object imaging point is:

[0037]

[0038] The physical coordinates of the object pixel point are obtained as:

[0039]

[0040] Assume is the external parameter matrix of the camera, is the internal parameter matrix of the camera, then the single-layer medium refraction imaging model is finally obtained as:

[0041]

[0042] where n0 = n a / n l1 ,

[0043] Step 1-3: When there are two layers of media, according to Snell's law: n l1 sinθ l1 = n l2 sinθ l2 = n a sinθ a , at the intersection position (x r y r z r ) T , after adding an additional refraction layer, the position is shifted to (x' r y' r z' r ), and according to the geometric relationship, equation (12) is corrected to: T where θ

[0044]

[0045] where θ l2 represents the incident angle of the light ray in medium layer 2, tl2 represents the thickness of the dielectric layer 2, n l2 represents the refractive index of the dielectric layer 2;

[0046] Derive the imaging model of the double-layer dielectric:

[0047]

[0048] where n0 = n a / n l1 ;

[0049] Step 1-4: When there are multiple layers of dielectrics, due to multiple refractions, the intersection point (x r y r z r ) T changes in position:

[0050]

[0051] From this, the refraction imaging model under multiple layers of dielectrics is derived as:

[0052]

[0053] Step 2: Improve on the basis of the calibration algorithm in air, and use the multi-layer dielectric imaging model established in Step 1 to design a suitable calibration method to calibrate the camera internal parameters and refraction parameters;

[0054] First, use the Zhang-Zhengyou calibration method to calibrate the camera internal parameters in air, keep the position of the checkerboard calibration board fixed, and obtain the camera internal parameters and the external parameters of the camera relative to the checkerboard; then take pictures of the calibration board under the condition of multiple layers of dielectrics to obtain the correspondence between the three-dimensional points in space and the two-dimensional pixel points in the image; finally, solve the model parameters by solving the multi-layer dielectric refraction imaging model;

[0055] Step 3: Perform camera external parameter calibration, use a polarization camera array to take a set of calibration pictures, and then process the captured data to obtain a visible light intensity image and a polarization phase angle image;

[0056] Step 4: Calculate the corresponding two sets of three-dimensional point cloud data using the two sets of images obtained in Step 3, and then fuse the two sets of three-dimensional point clouds;

[0057] Step 5: Use the fused three-dimensional point cloud data obtained in Step 4 and use the EPnP algorithm to solve for the poses of each camera.

[0058] The beneficial effects of the present invention are as follows:

[0059] Traditional camera calibration methods in air are no longer applicable to camera calibration under multi-layer media conditions. Therefore, it is necessary to analyze the influence of refraction on imaging under multi-layer media conditions, and based on this, establish a refraction imaging model for multi-layer media. On the basis of this model, a new calibration algorithm is designed to calibrate the camera. This method takes into account the influence brought by the refraction generated by multi-layer media and has higher accuracy than directly using the calibration method in air.

[0060] For the external calibration of the camera: A method for external calibration of a camera array based on the fusion of intensity and polarization three-dimensional point cloud information is proposed. Different from traditional camera external calibration methods, this method obtains two sets of three-dimensional point cloud data according to the intensity image and polarization image obtained by the camera array respectively, fuses the two sets of point cloud data, and then calculates the pose of each camera according to the obtained fused three-dimensional point cloud, which can achieve higher-precision external calibration of the camera and is more suitable for scenarios with single texture and unable to extract effective feature points. Brief Description of the Drawings

[0061] Figure 1 is the polarization camera array of the embodiment of the present invention.

[0062] Figure 2 is the refraction imaging model in the case of a single-layer medium layer of the present invention.

[0063] Figure 3 is the refraction imaging model in the case of a double-layer medium layer of the present invention.

[0064] Figure 4 is the refraction imaging model in the case of a multi-layer medium layer of the present invention.

[0065] Figure 5 is the camera imaging model in air of the present invention.

[0066] Figure 6(a) is the reprojection error after calibration using the imaging model in air in the embodiment of the present invention.

[0067] Figure 6(b) is the reprojection error after calibration using the multi-layer medium imaging model in the embodiment of the present invention.

[0068] Figure 7(a) is the three-dimensional reconstruction point cloud of the visible light image in the embodiment of the present invention.

[0069] Figure 7(b) is the three-dimensional reconstruction point cloud of the polarization image in the embodiment of the present invention.

[0070] Figure 7(c) is the fused three-dimensional point cloud in the embodiment of the present invention.

[0071] Figure 8 is the pose of the camera array obtained by solving in the embodiment of the present invention. Detailed Embodiments

[0072] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0073] The object of the present invention is to provide a calibration method for a polarization camera array applicable to a multi-layer dielectric environment to solve the problem of internal and external parameter calibration of the polarization camera array under multi-layer dielectrics.

[0074] A calibration method for a polarization camera array applicable to a multi-layer dielectric environment includes the following steps:

[0075] Step 1: First, analyze the influence of refraction phenomena on camera imaging in the presence of single-layer and double-layer dielectrics, derive the corresponding camera imaging models, and finally summarize the camera imaging models in a multi-layer dielectric environment, which are specifically as follows:

[0076] Step 1-1: The camera imaging model in air is approximately a simple perspective imaging model; the three-dimensional points in space correspond one-to-one with the two-dimensional image pixel points formed on the camera plane, and are represented by Equation (1):

[0077]

[0078] Among them, the P matrix represents the imaging process, that is, the correspondence between the three-dimensional coordinates in space and the two-dimensional coordinates of the pixel points on the imaging plane;

[0079] Specifically, it is expressed as follows:

[0080]

[0081] Step 1-2: Equation (2) is the perspective imaging model in air. Considering the influence of refraction, a camera imaging model in a multi-layer dielectric environment is established; first, a single-layer dielectric refraction imaging model is established;

[0082] Let d be the vertical distance from the refraction plane to the camera center, (α l1 β l1 γ l1 ) T and (α a β a γ a ) T respectively represent the direction vectors of the light rays in different dielectric layers, (x r y r z r ) T is the focus of the incident light ray and the refraction plane, θ m1 and θ a are the incident angle and refraction angle of the light ray; the relationship between the direction vectors of the light ray before and after refraction is expressed as:

[0083]

[0084] Meanwhile, the refraction angle and the incident angle satisfy Snell's law: n l1 sinθ l1 =n a sinθ a Then, we get:

[0085]

[0086]

[0087]

[0088]

[0089] The relationship between the direction vectors of the light rays before and after refraction is obtained as:

[0090]

[0091] Assume that (x u , y u ) are the two-dimensional physical coordinates of the imaging point. The direction vector of the incident light ray after refraction is obtained as:

[0092]

[0093] Let Calculate the relationship between the direction vector of the incident light ray before refraction and the two-dimensional physical coordinates of the imaging point as:

[0094]

[0095] Assume that the coordinate representation of the object in the camera coordinate system is in the following form:

[0096]

[0097]

[0098] We get:

[0099]

[0100] The relationship between the coordinates of the object point in the camera coordinate system and the two-dimensional physical coordinates of the object imaging point is obtained as:

[0101]

[0102] The physical coordinates of the object pixel point are obtained as:

[0103]

[0104] Assume is the external parameter matrix of the camera, is the internal parameter matrix of the camera, and finally the single-layer medium refraction imaging model is obtained as:

[0105]

[0106] where n0 = n a / n l1 ,

[0107] Step 1-3: When there are two layers of media, according to Snell's law: n l1 sinθ l1 = n l2 sinθ l2 = n a sinθ a , at the intersection position (x r y r z r ) T , after adding an additional refraction layer, the position is shifted to (x' r y' r z' r ). According to the geometric relationship, formula (12) is corrected to: T , and the imaging model of the double-layer medium is derived:

[0108]

[0109] Step 1-4: When there are multiple layers of media, due to multiple refractions, the intersection point (x

[0110]

[0111] y r z r r T ) T changes:

[0112]

[0113] Therefore, the refraction imaging model under multiple layers of media is derived as:

[0114]

[0115] Step 2: Improve on the calibration algorithm in air, and use the multi-layer medium imaging model established in Step 1 to design a suitable calibration method to calibrate the camera internal parameters and refraction parameters;

[0116] First, the Zhang-Zhengyou calibration method is used to calibrate the internal parameters of the camera in the air. Keeping the position of the checkerboard calibration board fixed, the internal parameters of the camera and the external parameters of the camera relative to the checkerboard are obtained. Then, pictures of the calibration board are taken under the condition of multi-layer media to obtain the corresponding relationship between three-dimensional spatial points and two-dimensional image pixel points. Finally, by solving the refraction imaging model of multi-layer media, the model parameters are solved.

[0117] Step 3: Perform external parameter calibration of the camera. Use a polarized camera array to take a set of calibration pictures, and then process the captured data to obtain a visible light intensity image and a polarization phase angle image.

[0118] Step 4: Calculate the corresponding two sets of three-dimensional point cloud data respectively using the two sets of images obtained in Step 3, and then fuse the two sets of three-dimensional point clouds.

[0119] Step 5: Use the fused three-dimensional point cloud data obtained in Step 4 and solve the poses of each camera using the EPnP algorithm. Specific embodiments:

[0121] The present invention provides a method for calibrating internal and external parameters of a camera array applicable to multi-layer media. As Figure 1 shown, when calibrating the internal parameters of the camera, refraction occurs when there are other media between the camera and the object, and the imaging model of the camera changes. Using the traditional imaging model will be inaccurate. Therefore, it is necessary to analyze the influence of refraction on imaging, establish a corresponding refraction imaging model, and at the same time consider the influence brought by multi-layer media as the number of media layers increases.

[0122] The external parameter calibration of the camera mainly obtains the relative poses of each camera in the camera array. Use a camera array built with multiple polarized cameras to take calibration pictures, and obtain two sets of three-dimensional point cloud data respectively according to the intensity image and the polarization image obtained by the camera array. The two sets of point cloud data are fused, and the external parameters of the camera array are calculated through the EPnP algorithm for the fused three-dimensional point cloud.

[0123] The method is specifically implemented according to the following steps:

[0124] Camera internal parameter calibration

[0125] 1. As Figure 2 shown, analyze the propagation path of the light rays emitted from the object. Since the refractive indices of different media are different, the light rays are refracted at the interface of the media. According to Snell's law and geometric relationships, the relationship between the direction vectors of the light rays before and after refraction can be obtained, and from this, the mapping relationship between the three-dimensional spatial coordinate points and the two-dimensional image points under single-layer media is deduced, which is the refraction imaging model of the camera under single-layer media.

[0126] 2. On the basis of the single-layer media refraction imaging model in Step 1, further deduce the multi-layer media refraction imaging model. AsFigure 3 As shown, when the incident light is under a double-layer medium, refraction occurs at the interface between the two layers. However, the refraction does not change the angular comparison relationship of the light in the first medium and the last medium. Therefore, a general mathematical model applicable to multi-layer media can be analyzed and summarized.

[0127] 3. According to the multi-layer medium imaging model of the camera established in step 2, design the corresponding calibration algorithm to calibrate the internal parameters of the camera and the relevant parameters of the medium layer.

[0128] The parameter calibration uses the one-to-one correspondence between the three-dimensional points in space and the two-dimensional points in the image to obtain the internal parameters and external parameters of the camera. First is the internal parameter of the camera. Since the internal parameter is the internal parameter of the camera and does not change with the environment, it can be calibrated in the air. To calibrate the external parameters such as multi-layer media, use the calibrated internal parameters of the camera and the corresponding points, and calculate according to the established multi-layer medium imaging model.

[0129] Camera external parameter calibration

[0130] 1. Use multiple polarization cameras to form a polarization camera array. The distance between the cameras is determined according to the actual situation so that all cameras can obtain the target image. Then, synchronously collect a group of images of the same scene through this camera array as calibration data.

[0131] 2. Process the experimental data collected in step 1 into visible light images and polarization images, respectively extract and match the feature points, solve the initial pose of the camera, and generate their respective three-dimensional point clouds.

[0132] 3. Fuse the two sets of point cloud data to obtain a dense three-dimensional point cloud, and use the EPnP algorithm to calculate a more accurate camera pose.

[0133] In the technical solution of the present invention, camera calibration mainly consists of camera internal parameter calibration under multi-layer media and external parameter calibration of the camera array. The principle of camera internal parameter calibration is as Figure 5 , when there is no other medium between the camera and the object, the light travels in a straight line into the camera and forms an image on the imaging plane. The camera imaging model in the air can be approximated as a simple perspective imaging model. The three-dimensional points in space correspond one-to-one with the two-dimensional image pixel points formed on the camera plane, and this relationship can be expressed by the following formula:

[0134]

[0135] Among them, the P matrix represents the imaging process, that is, the correspondence between the three-dimensional coordinates in space and the two-dimensional coordinates of the pixel points on the imaging plane. The specific expression can be shown as follows:

[0136]

[0137] This is the perspective imaging model in air. However, under the condition of multi-layer media, light will refract at the junction of media layers, and the imaging model in air is no longer applicable. Therefore, it is necessary to consider the influence of refraction to establish a camera imaging model under multi-layer media. First, start with a single-layer medium to analyze the influence of refraction on the imaging model. The camera imaging model of a single-layer medium is as Figure 2 shown.

[0138] f is the camera focal length, d is the vertical distance from the refraction plane to the camera center, (α l1 β l1 γ l1 ) T and (α a β a γ a ) T represent the direction vectors of light in different media layers, (x r y r z r ) T is the focus of the incident light and the refraction plane, θ M1 and θ a are the incident angle and refraction angle of light. Express the relationship between the direction vectors of light before and after refraction as:

[0139]

[0140] At the same time, the refraction angle and incident angle satisfy Snell's law: n m1 sinθ m1 =n a sinθ a , then we can get

[0141]

[0142]

[0143]

[0144]

[0145] The relationship between the direction vectors of light before and after refraction can be obtained as:

[0146]

[0147] Assume (x u ,y u ) is the two-dimensional physical coordinate of the imaging point. The direction vector of the incident light after refraction can be obtained as:

[0148]

[0149] Let The relationship between the direction vector of the incident light before refraction and the two-dimensional physical coordinates of the imaging point can be calculated as follows:

[0150]

[0151] Assume that the coordinates of the object in the camera coordinate system can be expressed in the following form:

[0152]

[0153] where (x u , y u ) T represents the coordinates of the intersection point of the incident light and the refraction plane in the camera coordinate system:

[0154]

[0155] It can be obtained that:

[0156]

[0157] The relationship between the coordinates of the object point in the camera coordinate system and the two-dimensional physical coordinates of the object imaging point is obtained as:

[0158]

[0159] In this way, the physical coordinates of the object pixel points can be obtained as:

[0160]

[0161] Assume is the external parameter matrix of the camera, is the internal parameter matrix of the camera, then finally the single-layer medium refraction imaging model can be obtained as:

[0162]

[0163] where n0 = n a / n l1 ,

[0164] When there are two layers of media (as shown in Figure 3 ), according to Snell's law: n m1 sinθ m1 = n m2 sinθ m2 = n a sinθ a, it can be seen that although the light undergoes two refractions through the refraction layer, the refraction does not change the angular relationship of the light in the first layer and the last layer. The only influence is the intersection position (x r y r z r ) T on the refraction surface of the first layer. After adding an extra refraction layer, this position shifts to (x' r y' r z' r ) T . According to the geometric relationship, the formula is corrected to:

[0165]

[0166] If the derivation for a single layer still holds later, the imaging model for a double-layer medium can be deduced:

[0167]

[0168] When there are multiple layers of media (as shown in Figure 4 ), due to multiple refractions, the intersection point (x r y r z r ) T changes its position:

[0169]

[0170] Thus, the refraction imaging model for multiple layers of media can be deduced as:

[0171]

[0172] Further, the traditional calibration algorithm uses the camera imaging model in air and is also for camera calibration in an air environment, and cannot be directly used for camera calibration under multiple layers of media. Therefore, it is necessary to improve the existing calibration algorithm according to the established camera imaging model under multiple layers of media so that it can more accurately calibrate the internal parameters of the camera and the refraction influence parameters.

[0173] To solve the corresponding parameters in the model according to the imaging model, first, the three-dimensional spatial point coordinates and the corresponding two-dimensional point coordinates need to be obtained, and then a system of equations can be established to solve the parameters. By taking pictures of a planar calibration board in air, first calibrate the internal parameters of the camera, then keep the position of the calibration board unchanged, and use the imaging model in air to obtain the external parameters corresponding to the calibration board. At the same time, the corresponding points of the three-dimensional spatial points and the two-dimensional points are also obtained. Finally, solve the system of equations of the multiple-layer medium model to solve the model parameters.

[0174]

[0175] For the external parameter calibration of the camera, the obtained intensity and polarization three-dimensional point cloud data are first fused, and then the poses of the camera array are calculated based on the EPnP algorithm. In a single and physically sparse environment, it is difficult to extract effective feature points from traditional visible light images and easy to form false matches, which greatly affects the accuracy of the calculation results. The polarization information provided by polarization images can well make up for this. Therefore, in the external parameter calibration of the camera array, using visible light images combined with polarization images can extract more feature points, thus making the calibration results more accurate.

[0176] In the specific operation process of the present invention, the internal parameter calibration of the camera is first carried out, that is, the influence of refraction on the imaging process under multiple layers of media is first analyzed, and a camera imaging model for multiple layers of media is established. Based on this imaging model, the existing calibration algorithm is improved: first, a calibration board image is taken in the air to obtain the internal parameters of the camera and the corresponding points in space, and then a system of equations is established according to the imaging model, and the corresponding parameters are obtained by solving the system of equations. For the external parameter calibration of the camera array, a set of images are taken using a polarization camera array, processed into polarization images and visible light images, two sets of three-dimensional point cloud data are obtained respectively according to the intensity images and polarization images obtained by the camera array, the two sets of point cloud data are fused, and finally the accurate external parameters of the camera array are obtained using the EPnP algorithm.

[0177] The camera used in this embodiment is a polarization camera. Multiple polarization cameras are fixed on a tripod to form a polarization camera array, and an external trigger ensures synchronous acquisition of all cameras.

[0178] The built polarization camera array is as Figure 1 shown. First, the Zhang's calibration method is used to calibrate the internal parameters of the camera. By collecting calibration board images at different angles in the air, the internal parameters of the camera are calculated. Multiple layers of media are simulated using glass, water, and air. Calibration board images in the presence of different media layers are taken, and the calibration results are calculated. Compared with the traditional method that does not consider the influence of refraction on the imaging model, the reprojection error is shown in Figure 6. It can be seen from the experimental results that the reprojection error of the camera internal parameter calibration method considering the influence of refraction is smaller, and the multi-layer medium imaging model can well eliminate the influence brought by refraction.

[0179] For the external parameter calibration of the polarization camera array, a set of images are synchronously collected using the polarization camera array, which are respectively processed into visible light images and polarization images, and intensity and polarization three-dimensional point cloud data are respectively generated as Figure 7(a) 、 7(b) shown. The two sets of point clouds are fused to obtain a dense point cloud, as shown in Figure 7(c). Finally, the external parameters of the camera are calculated using the EPnP algorithm, as Figure 8 shown.

Claims

1. A calibration method for a polarization camera array applicable to a multi-layer medium environment, characterized in that It includes the following steps: Step 1: First, analyze the influence of refraction phenomena on camera imaging in the presence of single-layer and double-layer media, derive the corresponding camera imaging model, and finally summarize the camera imaging model in a multi-layer media environment, as follows: Step 1-1: The camera imaging model in air is approximately a simple perspective imaging model; the three-dimensional points in space correspond one-to-one with the two-dimensional image pixel points formed on the camera plane, which is represented by Equation (1): Among them, the P matrix represents the imaging process, that is, the correspondence between the three-dimensional spatial coordinates and the two-dimensional coordinates of the pixel points on the imaging plane; [uv 1] T represents the two-dimensional image pixel point, [x w y w z w T represents the three-dimensional point coordinates in the world coordinate system;​ Specifically, it is expressed as follows: where k represents the proportionality coefficient, represents the internal parameters of the camera, f represents the camera focal length, represents the transformation matrix, [x w y w z w T represents the three-dimensional spatial point coordinates in the world coordinate system;​ Step 1-2: Equation (2) is the perspective imaging model in air. Considering the influence of refraction, a camera imaging model in a multi-layer media environment is established; first, a single-layer media refraction imaging model is established; Let d be the perpendicular distance from the refraction plane to the camera center, (α l1 β l1 γ l1 ) T and (α a β a γ a ) T represent the direction vectors of the light rays in different medium layers respectively, (x r y r z r ) T is the focus of the incident light ray and the refraction plane, θ l1 and θ a are the incident angle and the refraction angle of the light ray; the relationship between the direction vectors before and after the refraction of the light ray is expressed as: Meanwhile, the refraction angle and the incident angle satisfy Snell's law: n l1 sinθ l1 = n a sinθ a , then we get: where n a respectively represents the refractive index of the air medium, and n l1 represents the refractive index of the medium layer 1; The relationship between the direction vectors of the light rays before and after refraction is obtained as: Assume that (x u , y u ) is the two-dimensional physical coordinate of the imaging point, and the direction vector after the incident light is refracted is obtained as follows: Let The relationship between the direction vector of the incident light before refraction and the two-dimensional physical coordinates of the imaging point is calculated as follows: Assume that the coordinates of the object in the camera coordinate system are represented in the following form: Among them, [x c y c z c T represents the coordinates of the object in the camera coordinate system, [x r y r z r T represents the intersection position of the light ray and the refracting surface:​​ where (x u , y u ) T are the two-dimensional physical coordinates of the imaging point; It is obtained that: The relationship between the coordinates of the object point in the camera coordinate system and the two-dimensional physical coordinates of the object imaging point is obtained as: The physical coordinates of the object pixel point are obtained as: Hypothesis is the external parameter matrix of the camera, is the internal parameter matrix of the camera, then the finally obtained single-layer medium refraction imaging model is as follows: where n0 = n a / n l1 , Step 1-3: When there are two layers of media, according to Snell's law, we have: n l1 sinθ l1 = n l2 sinθ l2 = n a sinθ a , at the intersection position (x r y r z r ) T of the first-layer refraction surface, adding an additional refraction layer causes this position to shift to (x' r y' r z' r ). According to the geometric relationship, Equation (12) is corrected to: T ​ Among them, θ l2 represents the incident angle of light on the dielectric layer 2, and t l2 represents the thickness of the dielectric layer 2, and n l2 represents the refractive index of the dielectric layer 2; The imaging model of the double-layer media is deduced: where n0 = n a / n l1 ; Steps 1-4: When there are multiple layers of media, due to multiple refractions, the intersection point (x r y r z r ) T changes in position: From this, the refraction imaging model in a multi-layer media is deduced as: Step 2: Improve on the calibration algorithm in air, and use the multi-layer media imaging model established in Step 1 to design a suitable calibration method to calibrate the camera internal parameters and refraction parameters; First, use the Zhang Zhengyou calibration method to calibrate the camera internal parameters in air, keep the position of the checkerboard calibration board unchanged, and obtain the camera internal parameters and the external parameters of the camera relative to the checkerboard; then take pictures of the calibration board in the case of multi-layer media to obtain the correspondence between the three-dimensional points in space and the two-dimensional pixel points in the image; finally, solve the model parameters by solving the multi-layer media refraction imaging model; Step 3: Perform camera external parameter calibration. Use a polarization camera array to take a set of calibration pictures, and then process the captured data to obtain a visible light intensity image and a polarization phase angle image; Step 4: Use the two groups of images obtained in Step 3 to calculate the corresponding two groups of three-dimensional point cloud data respectively, and then fuse the two groups of three-dimensional point clouds; Step 5: Use the fused three-dimensional point cloud data obtained in Step 4, and use the EPnP algorithm to solve for the poses of each camera.

Images