A ray-tracing-based underwater dual-target visual calibration method

Abstract

This invention provides an underwater binocular vision calibration method based on ray tracing. The method involves performing binocular calibration in air to obtain the camera's intrinsic and extrinsic parameters; capturing images of the underwater calibration board to extract the pixel coordinates of the calibration points and calculating the direction vectors of the refracted light rays in the air; simplifying the binocular refraction coplanarity constraint and using the least squares method to solve for the initial value of the normal direction of the refraction plane; calculating the direction vectors of the refracted light rays in the glass and water, and using the binocular refraction coplanarity constraint to solve for the initial values ​​of the refraction distance-related parameters; and performing nonlinear optimization of the initial values ​​of the refraction parameters based on the binocular forward projection to obtain accurate values. This method combines an underwater refraction camera model, using two least squares methods to quickly solve for the initial values ​​of the refraction parameters, and then performing nonlinear optimization of the initial values ​​based on forward projection. This method is applicable to most binocular underwater calibration scenarios, especially sparse calibration in large scenes.

Description

Technical Field

[0001] This invention relates to the field of photogrammetry, and more specifically, to an underwater binocular vision calibration method based on ray tracing. Background Technology

[0002] Underwater imaging is widely used in various fields, such as marine life observation, geological exploration, pipeline inspection, and autonomous underwater navigation robots. However, due to the refraction of light in water, scenes captured by underwater cameras often exhibit severe distortion. Treibitz et al. pointed out that the perspective model of underwater imaging is a non-single-viewpoint model, meaning that light rays emitted from objects no longer converge at a single point. This non-single-viewpoint model is the root cause of underwater imaging distortion. Traditional perspective projection models, calibration methods, reconstruction methods, and various view geometry theories used in the air cannot be applied to underwater cameras, posing a challenge to the accuracy of underwater vision.

[0003] Modeling underwater cameras is generally considered a suitable method for planar refraction. Chari et al. conducted an in-depth study of the geometry of planar refraction using the Plück coordinate system, showing that a straight line is imaged as a quadratic curve after passing through a single layer of planar refraction. Su et al. reconstructed the planar pose by pasting marker points onto the refraction plane; however, this method is not applicable when the camera is placed underwater and too close to the refraction plane. Another approach is to fix the camera's imaging plane in a position parallel to the refraction plane and then measure the distance from the optical center to the refraction plane, but this method is difficult to implement in practice because precise parallelism is hard to achieve. Agrawal et al. introduced the concept of an axial camera to establish planar refraction constraints. The concept of an axial camera is an important innovation and has inspired much subsequent work. However, the algorithm for recovering the layer thickness in Agrawal's method is difficult to optimize; for common two-layer refraction, it requires solving a twelfth-degree equation. Chen et al. provided a concise algorithm for recovering the layer thickness using the condition of binocular coplanar constraints, and they used the Random Sample Consensus (RANSAC) framework to select the optimal solution. This method makes optimization easier to some extent, but it means requiring a large number of corresponding points and is time-consuming. Large-scale underwater observations are widely needed, such as seabed topography observations and surface observations of large underwater components, but dense correspondence points make calibration in large-scale scenarios quite difficult. Therefore, it is necessary to explore a method that can achieve sparse and rapid calibration in large underwater scenarios. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a ray-tracing-based underwater binocular vision calibration method. This method combines an underwater refraction camera model with a double least squares method to quickly solve for the initial values ​​of refraction parameters, and then performs nonlinear optimization based on forward projection on these initial values. This invention is applicable to most binocular underwater calibration scenarios, especially sparse calibration in large scenes.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A ray-tracing-based underwater binocular vision calibration method includes the following steps:

[0007] Step 1: Perform dual-target positioning in the air to obtain the camera's intrinsic and extrinsic parameters;

[0008] Step 2: Take pictures of the underwater calibration plate, extract the pixel coordinates of the calibration points, and calculate the direction vector of the refracted light rays in the air.

[0009] Step 3: Simplify the coplanar constraint of refraction and use the least squares method to solve for the initial value of the direction of the normal to the refraction plane;

[0010] Step 4: Calculate the direction vectors of the refracted light rays in the glass and water, and use the binocular refraction coplanar constraint condition to solve for the initial values ​​of the refraction distance related parameters;

[0011] Step 5: Perform nonlinear optimization on the initial values ​​of the refraction parameters based on double-headed projection to obtain accurate values.

[0012] Furthermore, in step 1, by acquiring the left and right camera images of the calibration plate in the air, the left and right cameras are calibrated using Zhang's calibration method to obtain the intrinsic parameters of the two cameras, and the transformation relationship between the camera coordinate system and the world coordinate system is obtained as the camera extrinsic parameters.

[0013] Furthermore, in step 2, after taking pictures of the underwater calibration plate, the coordinates P of the center of each calibration point on the calibration plate in the object coordinate system are recorded. j Then extract P respectively j Pixel coordinates in the left and right camera images.

[0014] Furthermore, step 2 also includes calculating the direction vector v0 of the light ray corresponding to each pixel in the world coordinate system, and the coordinates C of the camera optical center in the world coordinate system, based on the acquired camera extrinsic parameters.

[0015] Furthermore, step 3 includes: solving for the initial value of the normal direction n of the refraction plane, where n is solved by the monocular refraction coplanar constraint:

[0016] (R P P+t P)·(n×v0)=0,

[0017] Among them, R P and t P Representing the rotation and translation matrix from the object coordinate system to the world coordinate system, the above equation is simplified to a linear equation:

[0018]

[0019] in, This is the symbol for the Kronecker product, E = [n] × R P s = n × t P ,[n] × Let A represent the antisymmetric matrix of vector n, vec represent the column vectorization operation, and A P This corresponds to a calibration point P.

[0020] Furthermore, A, which corresponds to a calibration point P Expanding this into an m×12 matrix corresponding to m calibration points, where m≥11, we use the least squares method to solve for B. The value of E is obtained from B, and then based on n... T The condition E=0 will restore n from E.

[0021] Furthermore, step 4 includes: solving for the distance between the refraction planes. The initial value of d1, where d1 represents the glass thickness, is obtained by directly measuring with vernier calipers. Let represent the distances from the optical centers of the left and right cameras to the air-side glass surfaces, respectively. These distances are solved using the binocular refraction coplanar constraint. The superscript 'l' indicates the relationship with the left camera, and 'r' indicates the relationship with the right camera.

[0022]

[0023] Using u0, u1, and u2 to represent the refractive indices of air, glass, and water, respectively, the direction vector of the refracted light rays in the glass is calculated based on the direction vector v0 of the light ray corresponding to each pixel in the world coordinate system obtained in step 2 and the direction n of the normal to the refractive plane obtained in step 3.

[0024]

[0025] And the direction vector of the refracted light in the water:

[0026]

[0027] The intersection point q2 of the refracted ray and the water-side refraction plane is given by the formula.

[0028] calculate.

[0029] Further derivation of the coplanar constraint conditions for binocular refraction yields information about the unknowns. The linear equation:

[0030]

[0031] in G corresponding to a calibration point P P and H P Extending to a matrix corresponding to m points, the unknown d0 is solved using the least squares method, including... and

[0032] Furthermore, step 5 includes: performing nonlinear optimization on the initial value based on dual-heading projection, and calculating the calibration point P starting from the optical centers of the left and right cameras. j Forward projection coordinates and and It uses pixel coordinates and calculated refraction parameters to calculate the three-dimensional coordinates of the calibration point in the world coordinate system along the refracted ray, and then calculates the difference between the two. Expanding ε, we get:

[0033]

[0034] in, The depth of the object point underwater is represented; the forward projection error function is minimized using the LM algorithm:

[0035]

[0036] n、 The values ​​of all values ​​have been optimized accordingly.

[0037] Beneficial effects:

[0038] The calibration algorithm based on an underwater refraction camera model provided by this invention only requires one image of the planar calibration plate. Moreover, the algorithm is very fast and stable, avoiding the difficulties of existing methods that require dense corresponding points and solving high-order equations. Since this method only requires a few sparse calibration points, and the nonlinear optimization process effectively compensates for the sensitivity to noise during the initial value solution, it is applicable to most binocular underwater calibration scenarios, especially sparse calibration in large-scale scenes. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the system calibration method of the present invention;

[0040] Figure 2 This is a schematic diagram of the underwater binocular refraction model of the present invention;

[0041] Figure 3 This is a schematic diagram showing the placement of the camera and calibration plate in this invention;

[0042] Figure 4 This is an underwater image of the calibration plate used in this invention. Detailed Implementation

[0043] According to embodiments of the present invention, the technical solutions of the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention calculates precise planar refraction parameters by photographing a calibration plate underwater. First, based on the monocular and binocular refraction coplanarity conditions, initial values ​​of the refraction parameters are solved. Then, nonlinear optimization based on binocular forward projection is performed on these initial values ​​to obtain accurate calibration results.

[0045] like Figure 1 As shown, the specific steps of the underwater binocular vision calibration method based on ray tracing of the present invention are as follows:

[0046] Step (1): Calibrate the camera in the air. Acquire images of the left and right cameras on the calibration plate in the air, calibrate the binocular camera using Zhang's calibration method, obtain the intrinsic parameters of the two cameras, and obtain the transformation relationship between the camera coordinate system and the world coordinate system, i.e., the camera extrinsic parameters.

[0047] Step (2): Two cameras are placed in front of a glass tank filled with water to simulate a real underwater shooting scene, such as... Figure 3 As shown. First, take a picture of the underwater calibration board, such as... Figure 4 As shown, record the coordinates P of the center of each calibration point on the calibration plate in the object coordinate system. j Then, the gray-scale centroid method was used to extract P respectively. j The pixel coordinates in the left and right camera images. The underwater calibration board image only needs to be taken once. Finally, based on the camera extrinsic parameters obtained in step (1), the direction vector v0 of the light ray corresponding to each pixel in the world coordinate system, and the coordinates C of the camera optical center in the world coordinate system are calculated. All symbols are in Figure 2 The diagram is shown in the image. The superscript "l" indicates a connection to the left camera, and "r" indicates a connection to the right camera; these will not be explained further later.

[0048] Step (3) solve for the initial value of the normal direction n of the refraction plane. n can be solved by the monocular refraction coplanar constraint:

[0049] (R p P+t P )·(n×v0)=0,

[0050] R P and t P This represents the rotation and translation matrix from the object coordinate system to the world coordinate system. First, simplify the above equation into a linear equation:

[0051]

[0052] in, This is the symbol for the Kronecker product, E = [n] × R P s = n × t P ,[n] × Let represent the antisymmetric matrix of vector n, and vec represent the column vectorization operation. A P For a given calibration point P, in step (2), we extracted m points. Let A... P Expanding this into an m×12 matrix corresponding to m object points, substituting the coordinates of all center points extracted in step (2), and using the least squares method to obtain B. Finally, extracting the first 9 values ​​of B and performing an inverse column vectorization operation, we obtain E, based on n T Given that E = 0, n can be recovered from E.

[0053] Step (4): Solve for the distance between the refraction planes. And the initial value of d1. First, the value of d1 is obtained by measuring it using vernier calipers. The solution can be obtained using the binocular refraction coplanar constraint:

[0054]

[0055] u0, u1, and u2 represent the refractive indices of air, glass, and water, respectively. These three values ​​are usually considered known quantities. First, calculate the direction vector of the refracted light rays in the glass based on v0 obtained in step (2) and n obtained in step (3):

[0056]

[0057] And the direction vector of the refracted light in the water:

[0058]

[0059] The intersection point q2 of the refracted ray and the water-side refraction plane can be expressed by the formula... The parameters mentioned above are all in [the context of the previous sentence]. Figure 2 It is drawn in the middle.

[0060] Combining the above steps, substituting q2 and v2 into the binocular refraction coplanar constraint condition mentioned above, we can further derive the information regarding the unknowns. The linear equation:

[0061]

[0062] in, Similarly, G corresponds to a calibration point P. P and H P It can be easily expanded to m points, forming an m-row matrix or vector. We substitute all the calibration points extracted in step (2) and solve for the unknown d0 using the least squares method. At this point, the initial values ​​of all refraction parameters have been solved.

[0063] Step (5) involves performing nonlinear optimization of the initial value based on dual-heading projection. First, along the optical centers C of the left and right cameras... l and C r Start by calculating the calibration point P. j Forward projection coordinates and Then calculate the difference between the two. Expanding ε, we get:

[0064]

[0065] in, This represents the depth of the object point underwater. Finally, the forward projection error function is minimized using the LM algorithm:

[0066]

[0067] We can get n, The optimized values ​​are obtained. At this point, the precise values ​​for all underwater refraction parameters can be obtained.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for underwater binocular vision calibration based on ray tracing, characterized in that, Includes the following steps: Step 1: Perform dual-target positioning in the air to obtain the camera's intrinsic and extrinsic parameters; Step 2: Take pictures of the underwater calibration board, extract the pixel coordinates of the calibration points, and calculate the direction vector of the refracted light rays in the air; including calculating the direction vector v0 of the light rays corresponding to each pixel in the world coordinate system, and the coordinates C of the camera optical center in the world coordinate system, based on the obtained camera extrinsic parameters. Step 3: Simplify the coplanar constraint of refraction and use the least squares method to solve for the initial value of the normal direction of the refraction plane; including: Find the initial value of the direction n of the normal to the refraction plane. n is solved by the coplanar constraint of monocular refraction: (R P P+t P )·(n×v0)=0, Among them, R P and t P Representing the rotation and translation matrix from the object coordinate system to the world coordinate system, the above equation is simplified to a linear equation: in, This is the symbol for the Kronecker product, E = [n] × R P s = n × t P ,[n] × Let A represent the antisymmetric matrix of vector n, vec represent the column vectorization operation, and A P Corresponding to a calibration point P; A, corresponding to a calibration point P Expanding this into an m×12 matrix corresponding to m calibration points, where m≥11, we use the least squares method to solve for B. The value of E is obtained from B, and then based on n... T The condition E = 0 will restore n from E; Step 4: Calculate the direction vectors of the refracted light rays in the glass and water, and use the binocular refraction coplanar constraint to solve for the initial values ​​of the refraction distance-related parameters; including solving for the distance between the refraction planes. The initial value of d1, where d1 represents the glass thickness, is obtained by directly measuring with vernier calipers. Let represent the distances from the optical centers of the left and right cameras to the air-side glass surfaces, respectively. These distances are solved using the binocular refraction coplanar constraint. The superscript 'l' indicates the relationship with the left camera, and 'r' indicates the relationship with the right camera. Using u0, u1, and u2 to represent the refractive indices of air, glass, and water, respectively, the direction vector of the refracted light rays in the glass is calculated based on the direction vector v0 of the light ray corresponding to each pixel in the world coordinate system obtained in step 2 and the direction n of the normal to the refractive plane obtained in step 3. And the direction vector of the refracted light in the water: The intersection point q2 of the refracted ray and the water-side refraction plane is given by the formula. calculate; Further derivation of the coplanar constraint conditions for binocular refraction yields information about the unknowns. The linear equation: in G corresponding to a calibration point P P and H P Extending to a matrix corresponding to m points, the unknown d0 is solved using the least squares method, including... and Step 5: Perform nonlinear optimization on the initial values ​​of the refraction parameters based on dual-head forward projection to obtain accurate values; this includes: performing nonlinear optimization on the initial values ​​based on dual-head forward projection, and calculating the calibration point P along the optical centers of the left and right cameras. j Forward projection coordinates and and It uses pixel coordinates and calculated refraction parameters to calculate the three-dimensional coordinates of the calibration point in the world coordinate system along the refracted ray, and then calculates the difference between the two. Expanding ε, we get: in, The depth of the object point underwater is represented; the forward projection error function is minimized using the LM algorithm: n、 The values ​​of all values ​​have been optimized accordingly.

2. The underwater binocular vision calibration method based on ray tracing according to claim 1, characterized in that, In step 1, by acquiring the left and right camera images of the calibration plate in the air, the left and right cameras are calibrated using Zhang's calibration method to obtain the intrinsic parameters of the two cameras, and the transformation relationship between the camera coordinate system and the world coordinate system is obtained as the camera extrinsic parameters.

3. The underwater binocular vision calibration method based on ray tracing according to claim 2, characterized in that, In step 2, after taking pictures of the underwater calibration plate, the coordinates P of the center of each calibration point on the calibration plate in the object coordinate system are recorded. j Then extract P respectively j Pixel coordinates in the left and right camera images.

Images