Underwater camera calibration method and device based on forward projection refraction model

Abstract

The application provides a kind of underwater camera calibration method and device based on forward projection refraction model, comprising: initializing the pose of camera and calibration board;After fixing the pose of camera, change the pose of calibration board multiple times, and carry out multi-pose calibration underwater;Based on Zhang Zhengyou calibration principle, identify and solve the initial value of camera internal and external parameters;Based on the multi-medium refraction model considering the influence of medium thickness and the third-order radial and tangential distortion model constructed by forward projection refraction model, the initial value of the internal and external parameters of the camera is iteratively optimized, and the final value of the internal and external parameters of the camera is identified and solved to complete camera calibration.The application does not need to obtain the same aerial and underwater target image in the same position and attitude, nor does it need to calibrate in advance in the air to obtain the inherent parameters and distortion coefficients of the camera, which can be directly used for underwater camera calibration, with short calibration time, high calibration accuracy, multiple calibration parameters and wide application range.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of underwater calibration, in particular to an underwater camera calibration method and device based on a forward projection refraction model. BACKGROUND

[0002] Underwater camera calibration technology is of great importance in underwater archaeology, underwater manipulator grasping, underwater environment monitoring, underwater vehicle navigation, underwater shape reconstruction, underwater deformation measurement and other underwater application scenarios. When imaging underwater, refraction occurs when light passes through the interface between different media, resulting in distortion that is difficult to eliminate. Therefore, the traditional calibration model in the air is no longer applicable to underwater imaging, and if it is continued to be applied, it will result in considerable system error.

[0003] Due to the complexity of the underwater environment, many scholars have conducted relevant research on how to weaken or eliminate the influence of the underwater environment on the calibration accuracy, which can be mainly divided into three categories: pinhole imaging method, equivalent compensation method and theoretical compensation method. The pinhole imaging method is to directly apply the pinhole model to camera calibration and corresponding reconstruction in underwater imaging without considering the particularity of underwater imaging. The equivalent compensation method is to equivalently compensate the influence of underwater refraction and the change of the radial distortion of the lens or the focal length of the camera. The theoretical compensation method is to construct a forward projection equation (from the world coordinate system to the pixel coordinate system) or a reverse projection equation (from the pixel coordinate system to the world coordinate system) according to the actual trajectory of light in different media. Generally speaking, the forward projection method needs continuous iterative solutions to optimize and calculate the optimal image points, which is time-consuming but has high accuracy. In contrast, the reverse projection method does not need complex and large iterative optimization to solve the spatial point, which is time-efficient but has low accuracy.

[0004] Current research shows that some models need to be calibrated in the air in advance to calculate the intrinsic parameters and distortion coefficients of the camera, and then subsequent underwater calibration is performed. Worse still, some models must obtain the same air and underwater target images at the same position and attitude. In addition, several key problems need to be well solved, such as short calculation time, high measurement accuracy, and simple and unified multi-layer refraction model. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide an underwater camera calibration method and device based on a forward projection refraction model.

[0006] According to a first aspect of the present application, an underwater camera calibration method based on a forward projection refraction model is provided, comprising:

[0007] initializing the pose of the camera and the calibration board;

[0008] After fixing the camera pose, the pose of the calibration board is changed multiple times, and multi-pose calibration is performed underwater.

[0009] Based on the Zhang Zhengyou calibration principle, the initial values of the internal and external parameters of the camera are identified and solved.

[0010] Based on the forward projection refraction model and the multi-medium refraction model considering the influence of medium thickness, the initial values of the internal and external parameters of the camera are iteratively optimized, the final values of the internal and external parameters of the camera are identified and solved, and the camera calibration is completed.

[0011] Preferably, the multiple changes of the pose of the calibration board and the multi-pose calibration performed underwater comprise:

[0012] Within the effective working distance range of the camera, the calibration board is moved to the jth pose in turn, and the camera captures the image of the calibration board at this pose;

[0013] The pose of the calibration board is repeatedly moved until the number of poses of the calibration board meets the number requirement of the multi-pose calibration.

[0014] Preferably, the initial values of the internal and external parameters of the camera are identified and solved based on the Zhang Zhengyou calibration principle, comprising:

[0015] For each underwater calibration board image captured, the corresponding coordinate values of the multiple marker points of the calibration board in the world coordinate system and the image coordinate system are obtained respectively;

[0016] Based on the corresponding coordinate values of the multiple marker points in the world coordinate system and the image coordinate system, the initial values of the internal and external parameters of the camera are identified and solved according to the homography matrix.

[0017] Preferably, the multi-medium refraction model uses the third-order radial and tangential distortion model to iteratively identify and solve the final values of the internal and external parameters of the camera using the initial values of the internal and external parameters.

[0018] Preferably, the third-order radial and tangential distortion model is:

[0019]

[0020] In the formula, (x, y) is the image coordinate value before distortion, r 2 = x 2 +y 2 , (x d , y d ) is the image coordinate value after distortion, P1 is the first-order tangential distortion coefficient of the camera, P2 is the second-order tangential distortion coefficient of the camera, k1 is the first-order radial distortion coefficient of the camera, k2 is the second-order radial distortion coefficient of the camera, and k3 is the third-order radial distortion coefficient of the camera.

[0021] Preferably, the mathematical expression of the multi-medium refraction model considering the influence of medium thickness based on the forward projection refraction model is:

[0022]

[0023] wherein (x w ,y w ,z w ) is the three-dimensional point coordinate value in the world coordinate system, R is a rotation transformation matrix of 3 rows and 3 columns, T is a translation transformation vector of 3 rows and 1 column, (u0,v0) is the origin coordinate value of the image coordinate system in the pixel coordinate system, (u,v) is the corresponding two-dimensional coordinate value in the pixel coordinate system, f is the camera focal length, (f x ,f y ) is the number of pixels per unit distance in the image coordinate system, (c x ,c y ) is the corresponding ratio, i.e., c x =f x / f, c y =f y / f.

[0024] Preferably, the mathematical expression of the coefficient factor Δ is:

[0025]

[0026] wherein d c is the thickness of medium I (the distance from the camera optical center to medium II), d m is the thickness of medium II, z c is the three-dimensional point Z-axis coordinate value in the corresponding camera coordinate system, n1 is the refractive index of medium I, n2 is the refractive index of medium II, n3 is the refractive index of medium III, n A is the ratio of the refractive indices of medium I and medium II, i.e., n A =n1 / n2, n B is the ratio of the refractive indices of medium I and medium III, i.e., n B =n1 / n3.

[0027] Preferably, the mathematical expression of the multi-medium refraction model, from right to left, represents rigid body transformation from the world coordinate system to the camera coordinate system, perspective transformation from the camera coordinate system to the image coordinate system, and secondary transformation from the image coordinate system to the pixel coordinate system.

[0028] The rigid body transformation of the multi-medium refraction model has the same form and external parameter coefficients as the pinhole model in air; the perspective transformation of the multi-medium refraction model has the same form but different coefficients as the pinhole model in air; and the quadratic transformation of the multi-medium refraction model has the same form and internal parameter coefficients as the pinhole model in air.

[0029] Preferably, the camera and the calibration board are always located underwater during the calibration process.

[0030] According to a second aspect of the present application, there is provided an underwater camera calibration device based on a forward projection refraction model, comprising a planar calibration board, an industrial camera and a host computer, wherein the underwater calibration device is calibrated by using any of the underwater camera calibration methods based on the forward projection refraction model.

[0031] Compared with the prior art, the embodiments of the present application have at least one of the following beneficial effects:

[0032] The underwater camera calibration method and device based on the forward projection refraction model in the embodiments of the present application do not need to obtain the same aerial and underwater target images in the same position and attitude, nor do they need to be calibrated in air in advance to obtain the intrinsic parameters and distortion coefficients of the camera, and can be directly used for underwater camera calibration, with short calibration time, high calibration precision, many calibration parameters and wide application range. BRIEF DESCRIPTION OF DRAWINGS

[0033] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0034] Figure 1 is a flow chart of the underwater camera calibration method based on the forward projection refraction model in a preferred embodiment of the present application;

[0035] Figure 2 is a layout schematic diagram of the underwater camera calibration device based on the forward projection refraction model in a preferred embodiment of the present application;

[0036] Figure 3 is a model principle diagram of the underwater camera calibration method based on the forward projection refraction model in a preferred embodiment of the present application;

[0037] In the figure: 1 is a planar calibration board, 2 is an industrial camera, and 3 is a host computer. DETAILED DESCRIPTION

[0038] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0039] To facilitate understanding of the technical solution of this invention, the three models involved and their interrelationships are first briefly explained. Multi-medium refraction model: A model considering the influence of multiple medium thicknesses, such as in underwater calibration, where the medium thickness is considered, resulting in an air-glass-water multi-medium refraction model. Forward projection refraction model: A refraction model constructed according to the order from the world coordinate system to the pixel coordinate system. Third-order radial and tangential distortion model: A model considering camera distortion. Theoretically, the higher the order of the radial and tangential distortion model, the higher the accuracy of the camera calibration, but it is also more complex and less efficient. The third-order distortion model is a choice that balances calibration accuracy and efficiency, ensuring both. Interrelationships of the three models: The multi-medium refraction model is constructed based on the theory of the forward projection refraction model. It uses a third-order radial and tangential distortion model and the Levenberg-Marquardt method to identify and solve for the final values ​​of the camera's intrinsic and extrinsic parameters, thus achieving camera calibration.

[0040] Based on the above inventive concept, in an embodiment of the present invention, an underwater camera calibration method based on a forward projection refraction model as described in the above embodiments is provided, the specific process of which is as follows: Figure 1 As shown, the underwater camera calibration method includes the following steps:

[0041] S100, initialize the pose of the camera and calibration board;

[0042] After fixing the camera pose, the S200 repeatedly changed the pose of the calibration plate to perform multi-pose calibration underwater.

[0043] S300, based on Zhang Zhengyou's calibration principle, identifies and solves the multi-pose calibration of S200 to obtain the initial values ​​of the camera's intrinsic and extrinsic parameters;

[0044] S400, based on the forward projection refraction model, constructs a multi-medium refraction model that considers the influence of medium thickness and a third-order radial and tangential distortion model. Iteratively optimizes the initial values ​​of the camera's intrinsic and extrinsic parameters in S300, identifies and solves the final values ​​of the camera's intrinsic and extrinsic parameters, and completes the camera calibration.

[0045] This embodiment considers the influence of medium thickness, which makes the refractive model more accurate and the underwater calibration more precise. Specifically, in underwater calibration, if the medium thickness is considered, it is an air-glass-water refractive model; if the medium thickness is not considered, it is an air-water refractive model. Obviously, the former considers more variables, so the model in this embodiment is relatively more accurate.

[0046] In a preferred embodiment of the invention, in step S100, the camera and calibration plate remain underwater throughout the calibration process. In this embodiment, the planar calibration plate 1 can be a circular spot calibration plate. In other embodiments, other calibration plates can also be selected, such as a checkerboard calibration plate. The size of the calibration plate and the specifications of the marker points can be selected based on the effective working distance of the industrial camera during actual calibration.

[0047] In a preferred embodiment of the present invention, step S200 is implemented. Within the effective working distance range of the working camera, the calibration board is moved sequentially to the j-th pose, and the camera captures an image of the calibration board in this pose. The pose of the calibration board is moved repeatedly until the number of poses of the calibration board meets the requirements for the number of poses for multi-pose calibration.

[0048] In this embodiment, the pose of the planar calibration plate 1 can be adjusted using a support. The specific movement path and method can be designed according to actual calibration needs. In other embodiments, other tools can also be used to adjust the pose of the calibration plate.

[0049] In this embodiment, after each change of the calibration plate pose by the bracket, it is necessary to wait a few seconds until the water is basically still before controlling the camera to capture an image of the calibration plate in this pose.

[0050] In a preferred embodiment of the present invention, step S300 is implemented, which, based on Zhang Zhengyou's calibration principle, solves the nonlinear model of the camera. The coordinate values ​​of multiple marker points in each calibration board image from step S200 are iteratively optimized, and the initial values ​​of the camera's intrinsic and extrinsic parameters are identified and solved based on the homography matrix. Specifically:

[0051] S301: For each underwater calibration board image captured by S200, obtain the corresponding coordinate values ​​of multiple marker points on the calibration board in the world coordinate system and the image coordinate system.

[0052] S302 uses the Levenberg-Marquardt method to optimize the initial values ​​of the camera's intrinsic and extrinsic parameters.

[0053] In a preferred embodiment of the present invention, S400 includes two parallel processes: ensuring camera calibration accuracy based on a third-order radial and tangential distortion model; and identifying and solving the final values ​​of the camera's intrinsic and extrinsic parameters using the Levenberg-Marquardt method based on a forward projection refraction model.

[0054] Furthermore, the forward projection refraction model adopts a third-order radial and tangential distortion model, with the following mathematical expressions:

[0055]

[0056] In the formula: (x,y) are the image coordinates before distortion, r 2 =x 2 +y 2 , (x d ,y d P1 is the first-order tangential distortion coefficient of the camera, P2 is the second-order tangential distortion coefficient of the camera, k1 is the first-order radial distortion coefficient of the camera, k2 is the second-order radial distortion coefficient of the camera, and k3 is the third-order radial distortion coefficient of the camera.

[0057] Theoretically, the higher the order of the radial and tangential distortion models, the higher the accuracy of camera calibration, but it is also more complex and less efficient.

[0058] The third-order radial and tangential distortion model is a choice that balances calibration accuracy and calibration efficiency, ensuring both calibration accuracy and calibration efficiency.

[0059] Furthermore, the Levenberg-Marquardt method is used to optimize the solution of the final values ​​of the camera's intrinsic and extrinsic parameters (camera parameters, including camera intrinsic parameters (i.e., f)). x f y (u0, v0, distortion coefficients) and camera extrinsic parameters (i.e., rotation matrix R, translation vector T)).

[0060] In this embodiment, a multi-medium refraction model constructed according to the forward projection refraction model theory is adopted. This model is based on the air-glass-liquid multi-medium refraction model. In other embodiments, other multi-medium refraction models of medium I-medium II-medium III can also be selected according to actual needs.

[0061] Reference Figure 3 The schematic diagram of the underwater camera calibration model shown illustrates how different angles of refraction occur along the direction of light propagation when light passes through different media surfaces. This forward projection refraction model is applicable to both single and double refraction. Figure 3 In the middle, O c It is the light center, d c It is the thickness of medium I (the distance from the camera's optical center to medium II), d m It is the thickness of medium II, P w (x w ,y w ,z w P represents the coordinates of a three-dimensional point in the world coordinate system. c(x c ,y c ,z c ) represents the corresponding 3D point coordinates in the camera coordinate system, v1 is the incident ray of medium I, v2 is the refracted ray of medium II, v3 is the refracted ray of medium III, θ1 is the angle of incidence of medium I, θ2 is the angle of refraction of medium II, θ3 is the angle of refraction of medium III, and O w -X w Y w Z w It is the world coordinate system, O c -X c Y c Z c O is the camera coordinate system, O-XY is the image coordinate system, and δ is the angle between the X-axis of the image coordinate system and the straight line connecting the spatial point to two image points in the image coordinate system, one considering refraction and the other not. (Refer to...) Figure 3 It can be seen that the spatial point P c The coordinate expressions are as follows:

[0062] x c =[d c tanθ0+d m tanθ1+(z c -d c -d m )tanθ2]cosδ

[0063] y c =[d c tanθ0+d m tanθ1+(z c -d c -d m )tanθ2]sinδ

[0064] According to the law of refraction:

[0065] n1 sinθ1=n2 sinθ2=n3 sinθ3

[0066] In the formula, n1 is the refractive index of medium I, n2 is the refractive index of medium II, and n3 is the refractive index of medium III.

[0067] Based on this, the mathematical expression for the forward projection refraction model can be derived as follows:

[0068]

[0069] In the formula: R is a 3x3 rotation transformation matrix, T is a 3x1 translation transformation vector, (u0, v0) are the coordinates of the origin of the image coordinate system in pixel coordinates, (u, v) are the corresponding two-dimensional coordinates in pixel coordinates, f is the camera focal length, (f x ,f y (c) represents the number of pixels per unit distance in the image coordinate system. x ,c y ) is the corresponding ratio, that is, c x =f x / f,c y =f y / f.

[0070] In a preferred embodiment, the mathematical expression for the coefficient factor Δ is as follows:

[0071]

[0072] Where: n A It is the ratio of the refractive indices of medium I and medium II, that is, n A =n1 / n2, n B It is the ratio of the refractive indices of medium I and medium III, that is, n B = n1 / n3.

[0073] It is evident that this model, which considers different medium thicknesses, can improve calibration accuracy compared to existing models.

[0074] The aforementioned forward projection refraction model, from right to left, represents the rigid body transformation from the world coordinate system to the camera coordinate system, the perspective transformation from the camera coordinate system to the image coordinate system, and the quadratic transformation from the image coordinate system to the pixel coordinate system. Specifically, the rigid body transformation of the refraction model has the same form and extrinsic parameters as the pinhole model in air; the perspective transformation of the refraction model has the same form but different coefficients as the pinhole model in air; and the quadratic transformation of the refraction model has the same form and intrinsic parameters as the pinhole model in air.

[0075] In this embodiment, a multi-medium refraction model constructed according to the forward projection refraction model theory is used. A third-order radial and tangential distortion model is employed, and the Levenberg-Marquardt method is used to identify and solve for the final values ​​of the camera's intrinsic and extrinsic parameters, thus achieving camera calibration. The calibration process is time-efficient, highly accurate, covers a wide range of calibration parameters, and has broad applicability.

[0076] Based on the same technical concept described above, another embodiment of the present invention also provides an underwater camera calibration device based on a forward projection refraction model, see [link to relevant documentation]. Figure 2The system includes a planar calibration plate 1, an industrial camera 2, and a host computer 3. The underwater calibration device adopts the underwater camera calibration based on the forward projection refraction model in any of the above embodiments.

[0077] In subsequent underwater applications, one or more industrial cameras can be integrated according to actual needs. After calibration using the underwater camera calibration method based on the forward projection refraction model, they can be applied to various scenarios such as underwater archaeology, underwater robotic gripping, underwater environmental monitoring, underwater vehicle navigation, underwater shape reconstruction, and underwater deformation measurement within the effective working distance range.

[0078] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for calibrating an underwater camera based on a forward projection refraction model, characterized in that, include: Initialize the pose of the camera and calibration board; After fixing the camera pose, the calibration plate pose is changed multiple times to perform multi-pose calibration underwater; Based on Zhang Zhengyou's calibration principle, the multi-pose calibration is identified and solved to obtain the initial values ​​of the camera's intrinsic and extrinsic parameters; Based on the forward projection refraction model, a multi-medium refraction model considering the influence of medium thickness and a third-order radial and tangential distortion model are constructed to iteratively optimize the initial values ​​of the camera's intrinsic and extrinsic parameters, identify and solve the final values ​​of the camera's intrinsic and extrinsic parameters, and complete the camera calibration. The multi-medium refraction model considering the influence of medium thickness, constructed based on the forward projection refraction model, is as follows: ； In the formula: These are the coordinates of a three-dimensional point in the world coordinate system. It is a 3x3 rotation transformation matrix. It is a 3x1 translation transformation vector. These are the coordinates of the origin of the image coordinate system in pixel coordinates. These are the two-dimensional coordinate values ​​in the corresponding pixel coordinate system. It's the camera's focal length. It is the number of pixels per unit distance in the image coordinate system. It is the corresponding ratio, that is, , ; coefficient factor for: ； In the formula: It is the thickness of medium I, that is, the distance from the optical center of the camera to medium II. It is the thickness of medium II. These are the Z-axis coordinates of the corresponding 3D point in the camera coordinate system. It is the refractive index of medium I. It is the refractive index of medium II. It is the refractive index of medium III. It is the ratio of the refractive indices of medium I and medium II, that is, , It is the ratio of the refractive indices of medium I and medium III, that is, ; These are the coordinate values ​​of the distorted image; The camera and calibration plate remained underwater throughout the calibration process.

2. The underwater camera calibration method based on the forward projection refraction model according to claim 1, characterized in that, The process of repeatedly changing the pose of the calibration plate to perform multi-pose calibration underwater includes: Within the effective working distance of the camera, the calibration board is moved sequentially to the j-th pose, and the camera captures an image of the calibration board in this pose. Repeatedly move the calibration board pose until the number of poses of the calibration board meets the number requirement of the multi-pose calibration.

3. The underwater camera calibration method based on the forward projection refraction model according to claim 1, characterized in that, The process of identifying and solving the multi-pose calibration based on Zhang Zhengyou's calibration principle to obtain the initial values ​​of the camera's intrinsic and extrinsic parameters includes: For each underwater calibration board image captured, the corresponding coordinate values ​​of multiple marker points on the calibration board in the world coordinate system and the image coordinate system are obtained respectively. Based on the corresponding coordinates of the multiple marker points in the world coordinate system and the image coordinate system, the initial values ​​of the camera's intrinsic and extrinsic parameters are determined by identifying the homography matrix.

4. The underwater camera calibration method based on the forward projection refraction model according to claim 1, characterized in that, The multi-medium refraction model employs the third-order radial and tangential distortion model, and uses an iterative method to identify and solve for the final values ​​of the camera's intrinsic and extrinsic parameters based on the initial values ​​of the intrinsic and extrinsic parameters.

5. The underwater camera calibration method based on the forward projection refraction model according to claim 4, characterized in that, The third-order radial and tangential distortion models are as follows: ； In the formula: These are the image coordinates before distortion. , These are the distorted image coordinates. It is the camera's first-order tangential distortion coefficient. It is the camera's second-order tangential distortion coefficient. It is the camera's first-order radial distortion coefficient. It is the camera's second-order radial distortion coefficient. It is the camera's third-order radial distortion coefficient.

6. The underwater camera calibration method based on the forward projection refraction model according to claim 1, characterized in that, In the multi-medium refraction model, from right to left, the rigid body transformation from the world coordinate system to the camera coordinate system, the perspective transformation from the camera coordinate system to the image coordinate system, and the quadratic transformation from the image coordinate system to the pixel coordinate system are represented in sequence. The rigid body transformation of the multi-medium refraction model has the same form and external parameter coefficients as the pinhole model in air; the perspective transformation of the multi-medium refraction model has the same form but different coefficients as the pinhole model in air. The second transformation of the multi-medium refraction model has the same form and internal parameter coefficients as the pinhole model in air.

7. An underwater camera calibration device based on a forward projection refraction model, wherein the underwater camera calibration method based on a forward projection refraction model as described in any one of claims 1-6 is used for calibration, characterized in that, This includes a planar calibration board, an industrial camera, and a host computer.

Images