An underwater curved-surface camera calibration method based on a spherical waterproof cover

Abstract

This invention discloses a calibration method for an underwater curved surface camera based on a spherical waterproof cover, belonging to the field of underwater sensing technology. The calibration method includes: establishing a unified camera coordinate system and a dome coordinate system; mechanically adjusting the camera to the center of the dome; collecting calibration data when the two centers coincide in different media; moving the camera to other designated positions inside the glass cover to collect calibration data; processing the collected calibration data; calculating translation compensation between the camera coordinate system and the dome coordinate system; calculating rotation compensation between the camera coordinate system and the dome coordinate system; using Snell's law to calculate the light paths in air, glass, and water media sequentially; and optimizing various calibration parameters. This invention improves the accuracy of curved surface calibration by establishing a unified camera coordinate system and a dome coordinate system for translation and rotation compensation, then using ray tracing technology to derive the expressions for refracted light rays in each medium sequentially, and finally iteratively optimizing to obtain various calibration parameters and reprojection errors.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of underwater sensing, and particularly relates to a method for calibrating an underwater curved-surface camera based on a spherical waterproof cover. BACKGROUND

[0002] As a visual sensor, a camera is widely used in unmanned aerial vehicles, automatic driving, robot applications and the like. The calibration of the camera is mainly to determine the internal and external parameters and distortion coefficients of the camera through the geometric relationship between a three-dimensional space target and pixel values on the imaging plane of the camera. For single-medium camera calibration (in air), the imaging model of the camera is a linear camera model (pinhole model), and the light from an object will not be refracted before reaching the projection center, so it will not affect the imaging effect. However, for multi-medium calibration (air-glass-air / air-glass-water), the refractive index of each medium is different due to different materials, so when the light enters another medium from one medium, it will be refracted to different degrees, which will cause the imaging position to deviate and further cause the picture to be distorted.

[0003] At present, according to the types of the front-end glass of the underwater sealed cabin, the underwater camera calibration is divided into plane calibration and curved-surface calibration. The plane calibration is to place the camera in the sealed cabin with a front-end plane glass, the imaging model of this method is easy to build and low in cost, and has been widely used in the field of underwater imaging. The curved-surface calibration, i.e. the camera takes pictures through the spherical glass cover, can make up for the limited view caused by the plane calibration. According to the research, the view provided by the spherical waterproof cover is about 25% larger than that of the plane waterproof cover. A larger view means that the visual sensing device can extract more visual features from the photographed pictures for subsequent underwater positioning and mapping applications. In addition, the spherical structure can withstand much higher pressure than the plane structure under water, and the streamlined surface can also reduce the resistance when moving in water, so the underwater vehicle using the spherical waterproof cover can work flexibly in deeper water. However, the calibration accuracy of the underwater camera calibration method based on the spherical waterproof cover needs to be further improved. SUMMARY

[0004] In view of the above problems, the application provides a method for calibrating an underwater curved-surface camera based on a spherical waterproof cover, which establishes a unified camera coordinate system and a dome coordinate system for translation and rotation compensation, then uses the ray tracing technology to deduce the refracted light expression in each medium in turn, and finally iteratively optimizes each calibration parameter and the re-projection error to improve the curved-surface calibration accuracy.

[0005] The technical solution adopted by the application to solve the above technical problems is as follows:

[0006] A method for calibrating an underwater curved-surface camera based on a spherical waterproof cover, comprising the following steps:

[0007] Step S1, a unified camera coordinate system and dome coordinate system are established, the camera is mechanically adjusted to the center of the dome of the spherical waterproof cover, and calibration data when two centers coincide in different media are collected; the camera is moved to other specified positions to collect calibration data;

[0008] Step S2, the calibration data collected at different positions in step S1 are processed, and the internal and external parameters of the camera, the two-dimensional pixel values of the corner points and the three-dimensional point coordinates are calibrated;

[0009] Step S3, translation compensation between the camera coordinate system and the dome coordinate system is calculated: the spherical waterproof cover inner and outer hemispherical surface equation is converted to the camera coordinate system to realize the translation compensation between the camera coordinate system and the dome coordinate system;

[0010] Step S4, the rotation compensation between the camera coordinate system and the dome coordinate system is calculated by using the change of the same space vector before and after rotation in different coordinate systems;

[0011] Step S5, the Snell law is used to sequentially calculate the light paths Ray air , Ray glass and Ray water ;

[0012] Step S6, the optimal two-dimensional point is obtained, and a least square function of re-projection error is constructed to obtain the optimized calibration parameters.

[0013] Further, the specific content of the mechanical adjustment of the camera to the center of the dome of the spherical waterproof cover is that when the water is filled to the half position of the spherical glass cover, the focal length and exposure of the camera are adjusted so that the size and clarity of the water and underwater parts of the camera collected pictures are consistent, that is, the camera center is approximately located at the center of the dome.

[0014] Further, the other specified positions in step S1 include X-axis 5 / 10 / -5mm in the dome coordinate system, Y-axis 5 / 10 / -5mm in the dome coordinate system and Z-axis 5 / -5mm in the dome coordinate system.

[0015] Further, the specific content of the processing of the collected calibration data at different positions in step S2 includes: using the calib_gui command of MATLAB, after reading all the to-be-calibrated pictures, clicking Extract grid corners, manually selecting the four corner points of the outermost of the calibration area, setting the chessboard length to 20mm, and then clicking Calibration to calibrate the internal and external parameters of the camera, the two-dimensional pixel values of the corner points and the three-dimensional point coordinates.

[0016] Further, step S3 specifically includes:

[0017] According to the mathematical theory of three-dimensional geometry, assuming that there is a point P on the surface of a unit sphere, the point P should satisfy the constraint condition: P T QP=0, P T is the transpose form of point P; wherein the diagonal matrix is:

[0018]

[0019] And for a sphere with a radius of R in the dome coordinate system, the diagonal matrix should be rewritten as:

[0020]

[0021] In the dome coordinate system, assuming that the camera center coordinates are (Δx, Δy, Δz), the three quantities are the offset between the camera coordinate system and the dome coordinate system, so the translation vector of the point in the dome coordinate system to the camera coordinate system is (-Δx, -Δy, -Δz); in summary, for a sphere with any radius, the constraint diagonal matrix Q" that the point on the sphere surface from the dome coordinate system to the camera coordinate system should satisfy is:

[0022]

[0023] Further, the step S4 specifically comprises:

[0024] According to the Rodrigues rotation formula, the rotation matrix R com between two coordinate systems is:

[0025]

[0026] θ is the angle between C dome P2 and C cam P′2:

[0027]

[0028] C dome P2 and C cam P′2 are the same spatial vectors before and after rotation in the dome coordinate system and the camera coordinate system, respectively, and in the camera coordinate system, C dome P2=(R inner 0 0) T , C cam P′2=Ray air , Ray air is the light vector in the air medium, which can be obtained by pixel coordinate homogenization; P1 and P2 are C dome A and the intersection point of the inner and outer surfaces of the glass cover, C domeA is not only the normal line of the spherical glass cover when the camera center and the dome center coincide, but also the light path when the three-dimensional point A is orthogonally projected onto the imaging plane without refraction, and C dome A will be refracted into A-P1-P'2-C cam ;

[0029] where I is a 3x3 unit matrix, is the antisymmetric matrix in the cross product; C dome P2 and C cam P'2 cross product defines the rotation axis ra:

[0030]

[0031] C dome P2(1), C dome P2(2) and C dome P2(3) represent the first, second and third elements of the vector C dome P2; C cam P'2(1), C cam P'2(2) and C cam P'2(3) represent the first, second and third elements of the vector C cam P'2.

[0032] Further, the step S5 specifically comprises: for the pixel point a recovered after the distortion processing on the pixel plane, the pixel coordinates are back-projected to obtain the corresponding pixel light vector a' in the camera coordinate system:

[0033] a' = K -1 ·a

[0034] K represents the intrinsic matrix of the camera, then the direction vector Ray air of the air part of the light path in the dome glass cover can be homogenized as:

[0035]

[0036] The intersection of the light path in the air and the inner surface must be on the extension line of Ray air , assuming I inner = λ inner ·Ray air , where I inner represents the intersection of the light path and the inner surface of the glass; at the same time, the intersection must be on the inner surface of the glass, so the intersection coordinates are substituted into the spherical surface equation of the inner surface of the glass to solve λ inner :

[0037]

[0038] Q″ is a diagonal matrix representing the constraints that points on a sphere of arbitrary radius must satisfy when transformed from the dome coordinate system to the camera coordinate system. In Q″, R = R inner Let n be the radius of the inner surface of the glass; after finding the intersection point at the air-glass interface, the normal n at the intersection point is... inner It can be done by using intersection point I inner Coordinates minus the center C of the dome dome The coordinates are obtained because there is a rotation between the dome coordinate system and the camera coordinate system, so for the normal n inner The following rotation compensation is used to obtain n′ inner :

[0039] n′ inner =n inner ·R com

[0040] Among them, R com The rotation matrix for compensation;

[0041] Then, according to the vector form of Snell's law, the refracted ray Ray... glass It can be obtained by combining the incident ray and the normal:

[0042] Ray glass =a1Ray air +b1n′ inner

[0043] in In b1, the parameter η1 is defined as:

[0044]

[0045] μ air Let μ be the refractive index of air. glass Let be the refractive index of the glass medium. This is the transpose of the ray vector in the air medium;

[0046] In obtaining Ray glass Then, assume I outer =λ outer Ray glass , where I outer λ is the intersection of the light path and the outer surface of the glass. outer The direction vector coefficient is λ; the intersection point must be on the outer surface of the glass, therefore, substituting the coordinates of the intersection point into the equation of the spherical surface of the outer glass surface can solve for λ. outer :

[0047]

[0048] Q" is a diagonal matrix of the constraint that should be satisfied when converting from the dome coordinate system to the camera coordinate system, in which R = R outer is the radius of the outer surface of the glass; then the normal n outer is rotated as follows to obtain n' outer :

[0049] n' outer = n outer · R com

[0050] where R com is the rotation matrix for compensation;

[0051] According to Snell's law, the refracted light Ray water can be expressed as:

[0052] Ray water = a2Ray glass + b2n' outer

[0053] where In b2, the parameter η2 is defined as:

[0054]

[0055] is the transposed form of the light vector in the glass medium;

[0056] In order to obtain the three-dimensional point corresponding to the pixel point, I outer and Ray water need to be converted to the world coordinate system:

[0057]

[0058]

[0059] I w and Ray w represent the intersection vector of the glass and water medium and the light path vector in the water medium in the world coordinate system; then the light path in the water in the world coordinate system can be expressed according to the point-form equation of a straight line in space; since the checkerboard is located on the XOY plane of the world coordinate system, only the Z coordinate needs to be set to 0 to obtain the corresponding X coordinate and Y coordinate.

[0060] Further, the optimization process in step S6 includes first optimizing the offset and then optimizing the pixel point;

[0061] The cost function for seeking the optimal offset is the least square of the real value of the three-dimensional point collected in the water and the calculated value of the three-dimensional point obtained by back projection:

[0062]

[0063] Wherein, A is the true value of three-dimensional point in water, A' is the calculated value of three-dimensional point, K represents the internal parameter matrix of camera, R and T are the rotation matrix and translation vector in the external parameter matrix of camera respectively, dis represents four distortion parameters (k1, k2, p1, p2), k1 and k2 are radial distortion parameters (distortion distributed along the radius direction of lens, caused by lens quality), p1 and p2 are tangential distortion parameters (caused by lens installation and imaging plane not parallel); in the process of optimizing the offset, (Delta x, Delta y, Delta z) are variables, and other parameters are regarded as constants.

[0064] The cost function of seeking optimal two-dimensional point is:

[0065]

[0066] Wherein, the pixel point is a variable, the initial value of which is the two-dimensional point coordinate extracted in the photo shot in water, and other parameters are constants;

[0067] After obtaining the optimal two-dimensional point, the Euclidean distance between the true value of two-dimensional point in water and the two-dimensional point is subtracted, and the re-projection error can be obtained by optimizing the Euclidean distance:

[0068]

[0069] The technical scheme of the present application can produce the following technical effects:

[0070] 1. The underwater curved surface camera calibration method based on the spherical waterproof cover of the present application establishes a unified camera coordinate system and dome coordinate system, effectively reduces the measurement error caused by misalignment of the two coordinate systems by performing translation and rotation compensation between the camera coordinate system and the dome coordinate system, and improves the calibration accuracy.

[0071] 2. The underwater refraction camera curved surface calibration model established in the underwater curved surface camera calibration process based on the spherical waterproof cover of the present application comprehensively considers the influence of air-glass-water multi-medium refraction on imaging, and the modeling and calibration results are more accurate; secondly, the nested optimization can ensure that the optimization of the offset and the pixel point can be fully iterated to obtain the optimal iteration result, improve the effectiveness of the algorithm, and realize the calibration of camera parameters in underwater complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 It is a flowchart of the underwater curved surface camera calibration method based on the spherical waterproof cover of the present application.

[0073] Figure 2The schematic diagram for derivation of the light path of each medium of the underwater curved surface camera calibration method based on the spherical waterproof cover of the present application.

[0074] Figure 3 The model principle diagram for translation compensation of the underwater curved surface camera calibration method based on the spherical waterproof cover in the embodiment of the present application.

[0075] Figure 4 The model principle diagram for rotation compensation of the underwater curved surface camera calibration method based on the spherical waterproof cover in the embodiment of the present application.

[0076] Figure 5 The two main coordinate systems involved in the curved surface calibration of the present application are described: the camera coordinate system and the dome coordinate system. DETAILED DESCRIPTION

[0077] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0078] The present application provides an underwater curved surface camera calibration method based on a spherical waterproof cover, which performs translation and rotation compensation by establishing a unified camera coordinate system and a dome coordinate system, and then performs Figure 2 derives the refracted light expression in each medium in turn by using the light ray tracing technology, and finally iteratively optimizes each calibration parameter and the re-projection error to improve the curved surface calibration accuracy. As shown in the figure, the implementation overall process of the underwater curved surface camera calibration method based on the spherical waterproof cover of the present application is as follows: Figure 1

[0079] Step one: establish a unified camera coordinate system and a dome coordinate system in a container containing a hemispherical waterproof glass cover on one side, as shown in the figure. Figure 5 Adjust the camera center to the dome center mechanically, and collect the calibration data when the two centers coincide in different media; the specific mechanical adjustment method is to adjust the focal length and exposure of the camera so that the size and clarity of the pictures collected by the camera on the water and underwater are consistent when the water in the container (pool) is filled to the position of half of the spherical glass cover; at the same time, move the camera to other specified positions to collect calibration data.

[0080] When collecting calibration pictures at different positions, since the spherical glass cover is fixed on the side of the pool, it is only necessary to move the camera to the dome center, the X axis 5 / 10 / -5mm under the dome coordinate system, the Y axis 5 / 10 / -5mm under the dome coordinate system and the Z axis 5 / -5mm under the dome coordinate system to shoot the calibration pictures in the corresponding medium, respectively.

[0081] ​Step two, process the calibration data of different positions obtained in step one

[0082] The calibration data is processed using the calib_gui command of MATLAB. After reading all the calibration pictures, click Extract grid corners, manually select the four corner points of the outermost calibration area, set the chessboard length to 20 mm, and then click Calibration to obtain the camera's internal and external parameters, corner two-dimensional pixel value and three-dimensional point coordinates.

[0083] Step three, convert the inner and outer hemispherical surface equations of the dome glass cover to the camera coordinate system to realize the translation compensation Q" between the camera coordinate system and the dome coordinate system:

[0084] Translation compensation is realized by converting the inner and outer spherical surface equations of the glass cover to the camera coordinate system. The model principle diagram is as follows Figure 3 : According to the mathematical theory of three-dimensional geometry, assuming that there is a point P on the surface of a unit sphere, the point should satisfy the constraint condition:

[0085] P T QP=0

[0086] P T The transpose form of point P, where the diagonal matrix is:

[0087]

[0088] And for a sphere with radius R in the dome coordinate system, the diagonal matrix should be rewritten as:

[0089]

[0090] In the dome coordinate system, assume that the camera center coordinates are (Δx, Δy, Δz), and these three quantities are the offset between the camera coordinate system and the dome coordinate system. Therefore, the translation vector of the point in the dome coordinate system to the camera coordinate system is (-Δx, -Δy, -Δz). In summary, for a sphere with any radius, the diagonal matrix of the constraint that the point on the sphere surface from the dome coordinate system to the camera coordinate system should satisfy is:

[0091]

[0092] Step four, calculate the rotation compensation R between the camera coordinate system and the dome coordinate system using the change of the same space vector before and after rotation in different coordinate systems com , the model principle diagram is as follows Figure 4 :

[0093]

[0094] θ is the angle between the camera coordinate system and the dome coordinate system dome P2 and Ccam The angle between P'2 and P2:

[0095]

[0096] C dome P2 and C cam P'2 is the same space vector before and after rotation in two coordinate systems, in the camera coordinate system, C dome P2=(R inner 0 0) T , C cam P'2=Ray air (known); P1 and P2 are respectively C dome A and the intersection point of the inner and outer surfaces of the glass cover, C dome A is not only the normal line of the spherical glass cover when the camera center and the dome center coincide, but also the light path when the three-dimensional point A is orthogonally projected onto the imaging plane when there is no refraction, when the above two centers do not coincide, C dome A will be refracted to A-P1-P'2-C cam ;

[0097] Where I is a 3x3 unit matrix, is the anti-symmetric matrix in the cross product; C dome P2 and C cam P'2 cross product defines the rotation axis ra:

[0098]

[0099] Step five, use Snell's law to calculate the light path Ray air , Ray glass and Ray water

[0100] The present application uses back projection to derive the light path, after considering the unknown rotation and translation, the corresponding space three-dimensional point is derived step by step from the pixel point:

[0101] For a point a on the pixel plane (a is the restored pixel point after de-distortion processing), the pixel coordinates are back projected into the camera coordinate system to obtain the corresponding pixel light vector:

[0102] a'=K -1 ·a

[0103] Then the direction vector of the air part of the light path in the dome glass cover can be homogeneous as:

[0104]

[0105] The intersection point of the light path in the air and the inner surface must be in Rayair On the extension line, assume I inner =λ inner Ray air , where I inner This represents the intersection of the light path and the inner surface of the glass. Since this intersection point must lie on the inner surface of the glass, substituting the coordinates of the intersection point into the equation of the spherical surface of the inner glass surface allows us to solve for λ. inner :

[0106]

[0107] In Q″, R = R inner After finding the intersection point at the air-glass interface, the normal n at the intersection point... inner It can be done by using intersection point I inner Coordinates minus the center C of the dome dome The coordinates are obtained. Because there is rotation between the dome coordinate system and the camera coordinate system, the normal n is... inner Perform the following rotational compensation:

[0108] n′ inner =n inner ·R com

[0109] Then, according to the vector form of Snell's law, the refracted ray Ray... glass It can be obtained by combining the incident ray and the normal:

[0110] Ray glass =a1Ray air +b1n′ inner

[0111] in In b1, the parameter η1 is defined as:

[0112]

[0113] In obtaining Ray glass After, I outer (The intersection of the light path in the glass medium and the outer surface of the dome) and Ray water The solution process is similar to that above.

[0114] Specifically, assuming I outer =λ outer Ray glass , where I outer λ is the intersection of the light path and the outer surface of the glass. outer The direction vector coefficient is λ; the intersection point must be on the outer surface of the glass, therefore, substituting the coordinates of the intersection point into the equation of the spherical surface of the outer glass surface can solve for λ. outer :

[0115]

[0116] Q" is a diagonal matrix of the constraints that should be satisfied when converting from the dome coordinate system to the camera coordinate system, in which R = R outer is the radius of the outer surface of the glass; then the normal n outer is compensated by the following rotation to obtain n' outer :

[0117] n' outer = n outer · R com

[0118] where R com is the compensated rotation matrix;

[0119] According to Snell's law, the refracted ray Ray water can be expressed as:

[0120] Ray water = a2Ray glass + b2n' outer

[0121] where in b2, the parameter η2 is defined as:

[0122]

[0123] is the transposed form of the ray vector in the glass medium;

[0124] In order to obtain the three-dimensional point corresponding to the pixel point, I outer and Ray water need to be converted to the world coordinate system:

[0125]

[0126]

[0127] where I w and Ray w represent the intersection vector of the glass and water media and the light path vector in the water medium in the world coordinate system; then the light path in the water in the world coordinate system can be expressed according to the point-form equation of a straight line in space. Since the checkerboard is located in the XOY plane of the world coordinate system, only the Z coordinate needs to be set to 0 to obtain the corresponding X coordinate and Y coordinate. In summary, the position and direction of the light in each medium can be solved.

[0128] Step six, the optimal two-dimensional point is obtained by using the back projection iterative method, and the least squares function of the re-projection error is constructed to obtain the optimized calibration parameters.

[0129] The optimization process can be divided into two steps: first optimizing the offset and then optimizing the pixel point. The cost function of optimizing the offset is the least square of the real value of the three-dimensional point collected in water and the calculated value of the three-dimensional point obtained by back projection:

[0130]

[0131] Wherein, A is the real value of the three-dimensional point in water, A' is the calculated value of the three-dimensional point, (K, R, T) is the camera internal and external parameters of the air perspective camera model calibration, dis represents four distortion parameters (k1, k2, p1, p2), k1 and k2 are radial distortion parameters, and p1 and p2 are tangential distortion parameters. In this optimization process, (Δx, Δy, Δz) is taken as a variable, and other parameters are taken as constants.

[0132] The cost function of seeking the optimal two-dimensional point is similar to the cost function of seeking the optimal offset, and the main difference is as follows:

[0133]

[0134] Wherein, the pixel point is a variable, the initial value of which is the two-dimensional point coordinate extracted in the photo taken in water, and all other parameters are taken as constants.

[0135] After obtaining the optimal two-dimensional point, the re-projection error can be obtained by subtracting the real value of the two-dimensional point in water from the optimal two-dimensional point and optimizing the Euclidean distance between them:

[0136]

[0137] The experimental simulation content obtained in the embodiment is as follows:

[0138] Based on the underwater refraction camera calibration method of the application, the following device is used to collect data for verification, such as Figure 5 : a 1.2m*0.5m*0.5m water tank made of acrylic material, a circular notch is cut on the 0.5m*0.5m side surface of the water tank, and then a half-sphere glass cover with a thickness of 5mm is adhered to the circular notch, the inner diameter of the half-sphere glass cover is 130mm, and the outer diameter is 140mm; the size of the used chessboard is 12*9, and the size of each chessboard is 15mm*15mm. In the experiment, the refractive index μ air of the air medium is 1.00, the refractive index μ glass of the glass is 1.49, and the refractive index μ water of the water is 1.33.

[0139] Data are collected at the center of the dome, at the X-axis 5 / 10 / -5mm in the dome coordinate system, at the Y-axis 5 / 10 / -5mm in the dome coordinate system, and at the Z-axis 5 / -5mm in the dome coordinate system, and the calibration results are shown in Tables 1 and 2:

[0140] Table 1 Intrinsic parameters of camera when located in the center of the dome

[0141]

[0142] Table 2 Reprojection error of curved surface calibration

[0143]

[0144]

[0145] From the above multiple sets of calibration results, it can be seen that the underwater refraction camera calibration method based on the spherical waterproof cover can accurately solve the camera parameters, and the reprojection error of the corresponding position is 15-25 times smaller than the perspective projection model without considering refraction, proving the effectiveness of the application. Through position compensation between coordinate systems, construction of the imaging model and optimization of the cost function, the camera parameters in the underwater complex environment can be effectively calibrated, and the calibration accuracy is improved.

[0146] The above detailed the preferred embodiments of the application, and does not have any limiting effect on the application. Any person skilled in the art, without departing from the scope of the technical solutions of the application, can make any form of equivalent replacement or modification of the technical solutions and technical contents disclosed by the application, and still belongs to the protection scope of the application.

Claims

1. A method for calibrating an underwater curved camera based on a spherical waterproof cover, characterized in that, The method comprises the following steps: Step S1, a unified camera coordinate system and a dome coordinate system are established, the dome coordinate system takes the center of the dome of the hemispherical waterproof glass cover as the origin, the camera is mechanically adjusted to the center of the dome of the hemispherical waterproof glass cover, and calibration data when the center of the camera and the center of the dome coincide in different media is collected; the camera is moved to positions corresponding to different values of the X axis, the Y axis and the Z axis in the dome coordinate system, and calibration data is collected; Step S2, the calibration data collected at different positions in step S1 is processed, and the internal and external parameters of the camera, the two-dimensional pixel value of the corner point and the three-dimensional point coordinates are calibrated; Step S3, the translation compensation between the camera coordinate system and the dome coordinate system is calculated: the hemispherical waterproof glass cover is converted into the camera coordinate system to realize the translation compensation between the camera coordinate system and the dome coordinate system; The step S3 specifically comprises: According to the mathematical theory of three-dimensional geometry, it is assumed that there is a point P on the surface of a unit sphere, which should satisfy the constraint condition: P T QP = 0, P T is the transpose form of point P; wherein the diagonal matrix: And for the sphere with a radius of R in the dome coordinate system, the diagonal matrix should be rewritten as: In the dome coordinate system, assuming that the camera center coordinates are (Δx, Δy, Δz), the three quantities are the offset between the camera coordinate system and the dome coordinate system, therefore the translation vector of the point in the dome coordinate system converted to the camera coordinate system is (-Δx, -Δy, -Δz); in summary, for any radius of the sphere, the diagonal matrix Q" that should be satisfied by the point on the sphere surface converted from the dome coordinate system to the camera coordinate system is: Step S4, the rotation compensation between the camera coordinate system and the dome coordinate system is calculated by using the change of the same space vector before and after rotation in different coordinate systems; The step S4 specifically comprises: According to the Rodrigues rotation formula, the rotation matrix R between two coordinate systems com is: θ is C dome P2 and C cam the angle between P'2 and C C dome P2 and C cam P'2 are the same spatial vectors before and after rotation in the dome coordinate system and the camera coordinate system, respectively. In the camera coordinate system, C dome P2=(R inner 0 0) T C cam P'2 = Ray air R inner Ray is the inner surface radius of the glass. air The vector of light rays in the air medium is obtained by homogenizing the pixel coordinates; P1 and P2 are C dome A is the intersection of the inner and outer surfaces of the hemispherical waterproof glass cover, and C is the point where these surfaces meet. dome A is not only the normal to the hemispherical waterproof glass dome when the camera center and the dome center coincide, but also the light path when a three-dimensional point A is orthogonally projected onto the imaging plane without refraction. When the two centers do not coincide, C dome A will refract, becoming A-P1-P'2-C. cam ; where I is the 3x3 identity matrix, is the skew-symmetric matrix in the cross product; by C dome P2and C cam P'2cross product defines the rotation axis ra: C dome P2(1), C dome P2(2) and C dome P2(3) represents the vector C dome the first, second and third elements of P2; C cam P'2(1), C cam P'2(2) and C cam P'2(3) represents the vector C cam the first, second and third elements of P'2; Step S5, the light path Ray air , Ray glass and Ray water in air, glass and water medium are calculated in turn using Snell's law. Step S6, the optimal two-dimensional point is obtained, and a least square function of the re-projection error is constructed to obtain the optimized calibration parameters; The optimization process in step S6 comprises optimizing the offset first, and then optimizing the pixel point; The cost function for seeking the optimal offset is the least square of the true value of the three-dimensional point collected in water and the calculated value of the three-dimensional point obtained by back projection: Wherein, A is the true value of the three-dimensional point in water, A' is the calculated value of the three-dimensional point, in the above optimization process of the offset, (Δx, Δy, Δz) is the variable, and other parameters are regarded as constants; (K, R, T) is the camera internal and external parameters calibrated by the perspective camera model in air, wherein K represents the internal parameter matrix of the camera; R and T are respectively the rotation matrix and the translation vector in the camera external parameter matrix; dis represents 4 distortion parameters (k1, k2, p1, p2), k1 and k2 are radial distortion parameters, and p1 and p2 are tangential distortion parameters; The cost function for seeking the optimal two-dimensional point is: Wherein, the pixel point is the variable, the initial value thereof is the two-dimensional point coordinates extracted from the photo taken in water, and other parameters are constants; after the optimal two-dimensional point is obtained, the Euclidean distance between the true value of the two-dimensional point in water and the optimal two-dimensional point is optimized, and the re-projection error is obtained, wherein a is the pixel point recovered after the distortion treatment on the pixel plane, and a' is the pixel light vector obtained by back projecting the pixel coordinates to the camera coordinate system:

2. The method of claim 1, wherein, The specific content of the mechanical adjustment of the camera to the center of the dome of the hemispherical waterproof glass cover is: when the water in the pool is filled to the position of half of the hemispherical waterproof glass cover, the focal length and exposure of the camera are adjusted so that the size and clarity of the water and underwater parts of the pictures collected by the camera are the same, that is, the center of the camera is approximately located at the center of the dome.

3. The method of claim 2, wherein, The positions corresponding to different numerical values of the X-axis, the Y-axis and the Z-axis in step S1 include positions of 5, 10 and -5 mm on the X-axis in the dome coordinate system, positions of 5, 10 and -5 mm on the Y-axis in the dome coordinate system and positions of 5 and -5 mm on the Z-axis in the dome coordinate system.

4. The method of claim 1, wherein, The specific content of the processing of the collected calibration data of different positions in step S2 includes: using the calib_gui command of MATLAB, after reading all the calibration pictures, clicking Extract grid corners, manually selecting the four corner points of the outermost calibration area, setting the length of the checkerboard to 20 mm, and then clicking Calibration to obtain the internal and external parameters of the camera, the two-dimensional pixel value of the corner point and the three-dimensional point coordinates.

5. The method of claim 1, wherein, Step S5 specifically includes: for the pixel point a recovered after the distortion removal on the pixel plane, the pixel coordinates are back-projected to the camera coordinate system to obtain the corresponding pixel ray vector a': a' = K -1 • a K represents the intrinsic matrix of the camera, then the direction vector Ray of the air part of the light path in the hemispherical waterproof glass cover air Homogenization is: The intersection of the optical path in the air and the inner surface must be on the extension line of Ray air , assuming I inner = λ inner • Ray air , where I inner represents the intersection of the optical path and the inner surface of the glass; at the same time, the intersection must be on the inner surface of the glass, so the intersection coordinates are substituted into the spherical equation of the inner surface of the glass to solve λ inner : Q” is a diagonal matrix representing the constraints that points on a sphere of arbitrary radius must satisfy when transformed from the dome coordinate system to the camera coordinate system. In Q”, R = R inner Let n be the radius of the inner surface of the glass; after finding the intersection point at the air-glass interface, the normal n at the intersection point is... inner By using intersection point I inner Coordinates minus the center C of the dome dome The coordinates are obtained because there is a rotation between the dome coordinate system and the camera coordinate system, so for the normal n inner The following rotational compensation is used to obtain n' inner : n' inner = n inner · R com wherein R com is a rotation matrix for compensation; Then according to the vector form of Snell's law, the refracted ray Ray glass From the incident ray and the normal, we get: Ray glass = a1Ray air + b1n' inner wherein In b1, the parameter η1is defined as: μ air n is the refractive index of the air medium, μ glass n is the refractive index of the glass medium, is the transposed form of the light ray vector in the air medium; After obtaining Ray glass , assume I outer = λ outer · Ray glass , where I outer is the intersection of the light path and the outer surface of the glass, and λ outer is a directional vector coefficient; the intersection must be on the outer surface of the glass, so the intersection coordinates are substituted into the spherical equation of the outer surface of the glass to solve for λ outer : Q" is a diagonal matrix of constraints that should be satisfied when converting from the dome coordinate system to the camera coordinate system, in Q", R = R outer is the radius of the outer surface of the glass; then the normal n outer is rotated as follows to obtain n' outer : n' outer = n outer · R com wherein R com is a rotation matrix for compensation; According to Snell's law, the refracted ray Ray water is represented as: Ray water = a2Ray glass + b2n' outer wherein In b2, the parameter η2is defined as: is the transpose of the light ray vector in the glass medium; In order to get the three-dimensional point corresponding to the pixel point, it is necessary to convert I oute r and Ray water into the world coordinate system: I w and Ray w The intersection vector of glass and water medium and the light path vector in water medium under the world coordinate system are represented; then the light path in water under the world coordinate system is represented according to the point vector equation of space straight line; since the chessboard is located on the XOY plane of the world coordinate system, the corresponding X coordinate and Y coordinate are obtained by setting the Z coordinate as 0.

Images