Underwater camera calibration method based on multimedia
By using the refractive relationship of light in an underwater camera to rotate the imaging surface, combined with Shell's law and Opencv function, the problem of internal reference calibration of underwater cameras is solved, and the internal reference of underwater cameras is accurately obtained in the air, improving the calibration efficiency and accuracy.
Patent Information
- Application Number
- CN202210310885.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-28
AI Technical Summary
The existing technology cannot accurately calibrate the internal parameters of underwater cameras, because the refraction of light between different media causes the camera imaging surface to shift, and the existing calibration parameters in the air cannot be applied to underwater scenes, and the manual installation error has a great impact. , less practical.
Through the refractive relationship of light between different media, the camera imaging surface is rotated to make it parallel to the refractive plane, and the relationship between the camera's internal parameters before and after refractive is determined using pixel-by-pixel calibration. The data processing is used using Shell's law and Opencv function to deduce the air Mathematical relationship with the internal parameters of the camera in water.
The influence of manual installation error on calibration is avoided, the efficiency of underwater camera calibration is improved, the accurate internal reference of underwater cameras is achieved in the air, and the cumbersome underwater calibration process is simplified.
Smart Images

Figure CN114663531B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of underwater computer vision, relates to a refraction model of light between different media, and can realize the correction of an underwater camera imaging surface and the calibration of internal parameters. Background Art
[0002] Camera calibration is a popular research field in computer vision. As visual sensors, cameras are widely used in drones, autonomous driving, and robotic applications. Camera calibration mainly determines the camera's internal parameters and distortion coefficients through the geometric relationship between the three-dimensional target in space and the pixel values on the camera imaging plane. In recent years, a series of important research results have been achieved based on the calibration method in air. However, the calibration of underwater cameras faces the problem of light refraction between different media, which makes the calibration parameters in air unable to be applied underwater. Therefore, the research on underwater camera calibration still needs to be deepened.
[0003] In recent years, due to the shortage of land resources and the rich resources in the ocean, underwater cameras are used as visual sensors for collecting data, so the calibration of underwater cameras has received widespread attention from researchers. Based on an accurate underwater imaging model, it is necessary to derive the refraction path of light between each layer of media. This camera calibration method collects the same images underwater and above water by fixing the position between the camera and the calibration plate, and the use conditions are relatively harsh. If the position between the camera and the calibration plate changes, it must be recalibrated, which is less practical. For the refraction of underwater light, it is divided into plane refraction and curved refraction. Curved refraction uses a hemispherical glass cover. If the center of the camera coincides with the center of the hemispherical glass cover, no refraction will occur. This scenario requires high-precision mechanical adjustment. Plane refraction is common. Most of the work first assumes that the imaging surface of the camera is parallel to the refraction surface, and then determines the change relationship of the camera focal length before and after refraction based on the refraction of light. Most of the current underwater camera calibration in China is based on the case where the imaging surface of the camera is parallel to the refraction surface. For multi-media scenes, the refraction of the glass layer is directly ignored.
[0004] For underwater camera calibration, the camera is usually placed in a sealed cabin. Due to installation problems, the imaging plane of the camera is not parallel to the glass at the front of the sealed cabin. Using existing calibration methods, accurate underwater camera intrinsic coefficients cannot be obtained. Since incident light will refract when passing through different media, the camera imaging plane will also shift after refraction, so the camera intrinsic coefficients in the air cannot be applied to underwater scenes. Summary of the invention
[0005] In order to solve the above problems, the present invention provides a multi-media-based underwater camera calibration method. This method first rotates the imaging plane of the camera through the refraction relationship of light between different media, so that the rotated imaging plane is parallel to the refraction plane. Then, pixel-by-pixel calibration is used to determine the relationship between the camera's intrinsic parameters before and after refraction.
[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:
[0007] The present invention is a multimedia-based underwater camera calibration method, which comprises the following steps:
[0008] Step 1: Fix the position between the camera and the calibration plate;
[0009] Step 2: Collect calibration plate data in air and in multiple media respectively;
[0010] Step 3: Calibrate the collected data;
[0011] Step 4: Determine the direction vector of the light refracted between adjacent different media according to Shell's law, denoted as v i ;
[0012] Step 5: The direction vector of the light before and after refraction is v 1 and v 2 , according to the coplanarity constraint of the refracted light, obtain the normal size n of the underwater camera refractive surface;
[0013] Step 6: Based on step 5, calculate the direction vector m of the camera rotation axis;
[0014] Step 7: Based on step 5, calculate the rotation angle of the camera
[0015] Step 8: After considering steps 6 and 7, rotate the camera imaging plane so that the rotated imaging plane is parallel to the refraction plane;
[0016] Step 9: After the camera rotation in step 8, derive the mathematical relationship between the intrinsic parameters of the camera in air and in water.
[0017] A further improvement of the present invention is that in step 1, the distance between the camera and the calibration plate is 55 cm, the relative position between the camera and the calibration plate remains unchanged, and the calibration plate captured by the camera occupies more than 3 / 4 of the camera's field of view.
[0018] A further improvement of the present invention is that step 2 of collecting calibration plate data is specifically as follows: first a set of data is collected in the air, and this set of data is used as a reference, the positions of the camera and the calibration plate remain unchanged, a multi-medium is added between the camera and the calibration plate, and data under the multi-medium is collected.
[0019] A further improvement of the present invention is that step 3 of calibrating the collected data specifically includes the following steps:
[0020] Step 3-1: Use the findChessboardCorners() function in Opencv, set the function parameter image: to the image to be calibrated, set the function parameter patternSize to the number of corner points inside the chessboard to (11,8), and set the function parameter flags to None;
[0021] Step 3-2: Use the calibrateCamera() function in Opencv, and set the function parameter objectPoints to the point in the world coordinate system. The default world coordinate system is on the checkerboard plane, and the Z axis is 0. In addition to outputting the corner point coordinates, step S301 will also extract the coordinates in the world coordinate system and pass them in. The function parameter imagePoints is set to the corner point coordinates output in step 3-1.
[0022] A further improvement of the present invention is that in step 4, the light direction vector is calculated, specifically:
[0023] v i+1 =α i+1 v i +β i+1 n
[0024] where α i+1 =μ i / μ i+1 , n is the normal of the refractive surface, μ i The refractive index of the corresponding medium. For light in air, v 1 , you can directly use Opencv to calibrate and get the direction vector v of the light in the adjacent medium i Obtained by the above formula.
[0025] A further improvement of the present invention is that: in step 5, the direction vector of the light before and after refraction is v 1 and v 2 , according to the coplanarity constraint of the refracted light, obtain the normal size n of the underwater camera refraction surface:
[0026] (v 1 ×v 2 ) T n=0.
[0027] The direction vector m of the camera rotation axis in step 6 is:
[0028]
[0029] Where: a is the direction vector of the camera's optical axis, and n is the interface normal.
[0030] Because the optical axis of the camera is parallel to the z-axis of the camera coordinate system, it may be set to [0,0,1] in practical applications.
[0031] The rotation angle of the optical axis in step 7 is defined as the angle between the optical axis and the interface normal, and the cosine value of the rotation angle is defined as:
[0032]
[0033] Rotate the camera's optical axis Afterwards, the optical axis is parallel to the interface normal, that is, the imaging plane is parallel to the refractive interface.
[0034] A further improvement of the present invention is that: in step 8, the camera imaging plane is rotated so that the rotated imaging plane is parallel to the refraction plane, that is, the imaging point in the original camera coordinate system is rotated so that the plane where the rotated point is located is parallel to the refraction plane:
[0035]
[0036] Where x' is the point after rotation, x 1 =[x,f] T ,x is the point to be rotated, f is the focal length of the camera, is the rotation angle, M is the unit vector of the rotation axis direction vector m.
[0037] A further improvement of the present invention is that in step 9, the position of the real camera is C, the camera is translated along the optical axis, the position of the camera without considering refraction is C1, the camera position ignoring the existence of water is C2, the intersection points of the camera optical axis and the medium surface are A and D respectively, the intersection points of the refracted light and the medium surface are B and E respectively, and the refraction angle of the refracted light and the medium surface is θ i , the refraction between air and glass is projected using perspective projection, and the mathematical relationship between the intrinsic parameters of the camera in air and water is derived as follows:
[0038] Step 9-1: Discuss refraction between air and glass:
[0039] For air-glass, by the principle of similar triangles, we can deduce:
[0040]
[0041]
[0042] For ordinary industrial cameras, because they have a small camera field of view, we can approximate:
[0043] tanθ~sinθ~θ
[0044]
[0045] Where: n 1 is the refractive index in air, the default value is 1; n 2 is the refractive index of glass; n 3 is the refractive index of water;
[0046] Camera C and Camera C ignoring the presence of water 2 With the same field of view, the refraction point B has the same pixel value in both cameras, and its principal point is also the same, so:
[0047]
[0048] Step 9-2: For glass-water, by analyzing step 9-1, we get:
[0049]
[0050] According to the formula and Get f c1 The relationship with f, that is, the relationship between the real camera and the virtual camera:
[0051] f w =n w f a
[0052] Where: f w is the focal length of the camera in water, f a is the focal length of the camera in air, n w is the refractive index of water.
[0053] The beneficial effects of the present invention are as follows: the present invention firstly rotates the imaging plane of the camera through the refraction relationship of light between different media, so that the rotated imaging plane is parallel to the refraction plane, thereby avoiding the influence of the error brought by manually installing the camera on the camera calibration, and then uses pixel-by-pixel calibration to determine the relationship between the camera internal parameters before and after refraction, so that the internal parameters of the underwater camera can be obtained in the air, thereby avoiding the cumbersome underwater calibration process and improving the efficiency of underwater camera calibration.
[0054] The present invention provides the calculation expression of light between different media and the expression of rotation axis and rotation angle obtained according to refracted light. After the imaging surface of the camera is corrected, a new underwater camera model is proposed. Based on the refraction of light and the principle of similar triangles, the transformation relationship of the camera focal length before and after refraction is obtained, that is, the internal parameters of the underwater camera can be obtained in the air. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a schematic diagram of the process of the present invention.
[0056] Figure 2 It is a schematic diagram of the implementation equipment and scenario of the present invention.
[0057] Figure 3 It is the imaging surface correction model of the present invention.
[0058] Figure 4 The model of the imaging points and physical points of the present invention. DETAILED DESCRIPTION
[0059] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.
[0060] like Figure 2 As shown, the camera and the fixed plate of the present invention are fixed by mechanical devices, and the iron rod, base, and iron ring are used to fix the camera and the calibration plate so that the relative positions between them remain unchanged. The calibration plate is directly fixed on the base, and the sealed cabin is fixed on the iron ring. By manually adjusting the distance between the iron ring and the calibration plate on the base, the calibration plate collected by the camera occupies more than 3 / 4 of the camera's field of view. The data cable is used to connect the computer. When the camera is applied underwater, in order to be waterproof, the camera is usually fixed in the sealed cabin through a bracket. The camera is surrounded by air, the front end of the sealed cabin is a glass cover, and the outer end of the glass cover is water. At this moment, the light passes through water, glass and air in turn to reach the imaging center of the camera. Due to the different refractive indices of different media, the path of the light will change every time the light enters a different medium, resulting in the mature calibration method in the air cannot be directly used underwater, so it is necessary to establish a new underwater camera model.
[0061] like Figure 1 , 3 As shown in , the present invention is a multimedia-based underwater camera calibration method, comprising the following steps:
[0062] Step 1: Because the present invention considers the influence of different media on refraction, during the experiment, the position between the camera and the calibration plate is ensured to remain unchanged. The camera and the calibration plate are fixed by a mechanical device such as Figure 2 As shown, the distance between the camera and the calibration plate is set by focusing in the air. The distance in the experiment is 55 cm. At this time, the camera can clearly take pictures of the calibration plate, and the entire calibration plate occupies more than 3 / 4 of the camera's field of view.
[0063] Step 2: Collect calibration plate data in air and in multiple media respectively: First collect a set of data in the air. Taking this set of data as the benchmark, the positions of the camera and the calibration plate remain unchanged. Add multiple media between the camera and the calibration plate, and put the collection device into water. At this time, the multiple media is air-glass-water, and collect data under multiple media.
[0064] Step 3: Calibrate the collected data, which specifically includes the following steps:
[0065] Step 3-1: Use the findChessboardCorners() function in Opencv, set the function parameter image: to the image to be calibrated, set the function parameter patternSize to the number of corner points inside the chessboard to (11,8), and set the function parameter flags to None;
[0066] Step 3-2: Use the calibrateCamera() function in Opencv, and set the function parameter objectPoints to the point in the world coordinate system. The default world coordinate system is on the checkerboard plane, and the Z axis is 0. In addition to outputting the corner point coordinates, step S301 will also extract the coordinates in the world coordinate system and pass them in. The function parameter imagePoints is set to the corner point coordinates output in step 3-1.
[0067] Step 4: The use of light between different media i According to Shell's law, the mathematical representation of light in two adjacent media is:
[0068] v i+1 =α i+1 v i +β i+1 n (1)
[0069] where α i+1 =μ i / μ i+1 , n is the normal of the refractive surface, μ i The refractive index of the corresponding medium. For light in air, v 1 , you can directly use Opencv to calibrate and get the direction vector v of the light in the adjacent medium i Obtained by the above formula.
[0070] Step 5: Based on step 4, the direction vector of the light before and after refraction is v 1 and v 2 , according to the coplanarity constraint of multi-plane refraction, the ray v i On the same plane as the medium normal n, we can get:
[0071] (v1 ×v 2 ) T n=0 (2)
[0072] The normal line of the interface can be obtained by equation (2).
[0073] Step 6: Based on step 5, calculate the direction vector m of the camera rotation axis;
[0074] Due to the problem of camera installation, the camera imaging surface is not parallel to the refraction surface, so the camera needs to be rotated first. Figure 3 As shown, the angle between the optical axis of the camera and the refractive surface is The direction vector of the camera's optical axis is a, and the interface normal is n. The direction vector of the camera's rotation axis can be calculated by the following formula:
[0075]
[0076] Because the optical axis of the camera is parallel to the z-axis of the camera coordinate system, it may be set to [0,0,1] in practical applications.
[0077] Step 7: The rotation angle of the optical axis is defined as the angle between the optical axis and the interface normal. The cosine value of the rotation angle is defined as:
[0078]
[0079] Rotate the camera's optical axis Afterwards, the optical axis is parallel to the interface normal, that is, the imaging plane is parallel to the refractive interface.
[0080] Step 8: Based on the rotation angle and rotation axis obtained in step S6 and step S7, the point in the original camera coordinate system can be converted to an image point on the imaging plane parallel to the refraction surface as follows:
[0081]
[0082] Where x' is the point after rotation, x 1 =[x,f] T ,x is the point to be rotated, f is the focal length of the camera, is the rotation angle, M is the unit vector of the rotation axis direction vector m.
[0083] The rotated image coordinates are:
[0084]
[0085] Through the above step-by-step execution, the points in the original camera coordinate system are rotated to the imaging plane parallel to the refraction interface.
[0086] The above derivation is to correct the camera imaging plane.
[0087] Step 9: Based on the refraction of light between different media and the principle of similar triangles, derive the underwater camera model.
[0088] like Figure 4 As shown, the shell law is used to describe the mathematical relationship between light entering from one medium into another and the intersection of refraction and refractive index, and we get:
[0089] n 1 sinθ 1 =n 2 sinθ 2 =n 3 sinθ 3 (7)
[0090] for Figure 4 The refraction model, the real camera position is C, the camera is translated along the optical axis, the camera position without considering refraction is C1, the camera position ignoring the existence of water is C2, the intersection points of the camera optical axis and the medium surface are A and D respectively, the intersection points of the refracted light and the medium surface are B and E respectively, and the refraction angle of the refracted light and the medium surface is θ i , the refraction between air and glass is projected using perspective. It can be derived in two steps. The first step is to discuss the refraction between air and glass. The second step is to consider the refraction between glass and water. For air-glass, by the principle of similar triangles, we can deduce:
[0091]
[0092]
[0093] For ordinary industrial cameras, because they have a small camera field of view, we can approximate:
[0094] tanθ~sinθ~θ(10)
[0095]
[0096] Camera C and Virtual Camera C 2 With the same field of view, the refraction point B has the same pixel value in both cameras, and its principal point is also the same. So:
[0097]
[0098] For glass-water, through the analysis of the first case, we can similarly obtain:
[0099]
[0100] According to formula (12-13), we can get fc1 The relationship with f, that is, the relationship between the real camera and the virtual camera, can be seen from the derived formula that the proportional coefficient between the two focal lengths is the refractive index of the medium.
[0101] In a real experiment, the value of f can be obtained in air using Zhang Zhengyou's calibration method. Through the above geometric derivation, the focal length f of the underwater camera can be obtained. c1 .
[0102] In order to better verify the effectiveness of our proposed method, we collected multiple sets of pictures with different postures in the air and water, and calibrated the collected data. The experimental results are shown in Table 1.
[0103] Table 1 Calibration results of Zhang Zhengyou calibration method
[0104] medium <![CDATA[f x ]]> <![CDATA[f y ]]> Air 1585.723 1585.636 Air-Glass-Air 1584.731 1584.742 Air-Glass-Water 2148.745 2152.365
[0105] From the above calibration results, it can be seen that for Zhang Zhengyou's calibration method, the ratio factor between the underwater calibration focal length and the air focal length is 1.335. The actual refractive index of clean water is 1.33. The effectiveness of the method proposed in the present invention can be verified through the results of actual experiments.
[0106] The present invention provides the calculation expression of light between different media and the expression of rotation axis and rotation angle obtained according to refracted light. After the imaging surface of the camera is corrected, a new underwater camera model is proposed. Based on the refraction of light and the principle of similar triangles, the transformation relationship of the camera focal length before and after refraction is obtained, that is, the internal parameters of the underwater camera can be obtained in the air.
[0107] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A multimedia-based underwater camera calibration method, characterized in that: The underwater camera calibration method comprises the following steps: Step 1: Fix the position between the camera and the calibration plate; Step 2: Collect calibration plate data in air and in multiple media respectively; Step 3: Calibrate the collected data; Step 4: Determine the direction vector of the light refracted between adjacent different media according to Shell's law, denoted as v i ; Step 5: The direction vectors of the light before and after refraction are v1 and v2. According to the coplanarity constraint of the refracted light, the normal size n of the refractive surface of the underwater camera is obtained; Step 6: Based on step 5, calculate the direction vector m of the camera rotation axis; Step 7: Based on step 5, calculate the rotation angle of the camera Step 8: After considering steps 6 and 7, rotate the camera imaging plane so that the rotated imaging plane is parallel to the refraction plane; Step 9: After rotating the camera in step 8, derive the mathematical relationship between the internal parameters of the camera in air and in water. In step 9, the position of the real camera is C, the camera is translated along the optical axis, the position of the camera without considering refraction is C1, the position of the camera ignoring the existence of water is C2, the intersection points of the camera optical axis and the medium surface are A and D respectively, the intersection points of the refracted light and the medium surface are B and E respectively, and the refraction angle of the refracted light and the medium surface is θ i , the refraction between air and glass is projected using perspective projection, and the mathematical relationship between the intrinsic parameters of the camera in air and water is derived as follows: Step 9-1: Discuss refraction between air and glass: For air-glass, by the principle of similar triangles, we can deduce: For ordinary industrial cameras, we can get the approximate value: tanθ~sinθ~θ Among them: n1 is the refractive index in air, the default value is 1; n2 is the refractive index of glass; n3 is the refractive index of water; Camera C and camera C2, ignoring the presence of water, have the same field of view. The refraction point B has the same pixel value in both cameras, and its principal point is also the same, so: Step 9-2: For glass-water, by analyzing step 9-1, we get: According to the formula and Get f c1 The relationship with f, that is, the relationship between the real camera and the virtual camera: f w =n w f a Where: f w is the focal length of the camera in water, f a is the focal length of the camera in air, n w is the refractive index of water.
2. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: In step 8, the camera imaging plane is rotated so that the rotated imaging plane is parallel to the refraction plane. The imaging point in the original camera coordinate system is rotated so that the plane where the rotated point is located is parallel to the refraction plane: Where x' is the rotated point, x1=[x,f] T ,x is the point to be rotated, f is the focal length of the camera, is the rotation angle, M is the unit vector of the rotation axis direction vector m.
3. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: The direction vectors of the light before and after refraction in step 5 are v1 and v2. According to the coplanarity constraint of the refracted light, the normal size n of the refraction surface of the underwater camera is obtained: (v1×v2) T n=0。 4. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: The direction vector m of the camera rotation axis in step 6 is: Where: a is the direction vector of the camera's optical axis, and n is the interface normal.
5. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: The rotation angle of the optical axis in step 7 is defined as the angle between the optical axis and the interface normal, and the cosine value of the rotation angle is defined as:
6. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: In step 1, the distance between the camera and the calibration plate is 55 cm, the relative position between the camera and the calibration plate remains unchanged, and the calibration plate captured by the camera occupies more than 3 / 4 of the camera's field of view.
7. The multimedia-based underwater camera calibration method according to claim 1, characterized in that: Step 2 of collecting calibration plate data is as follows: first collect a set of data in the air, take this set of data as a reference, keep the position of the camera and the calibration plate unchanged, add multiple media between the camera and the calibration plate, and collect data under the multiple media.
Citation Information
Patent Citations
Underwater camera calibration method
CN105678742A
Underwater stereoscopic vision system calibration method
CN107767420A
Rotating underwater object three-dimensional reconstruction method based on laser triangulation method
CN112991533A
Underwater structured light measurement calibration method and system based on multi-medium refraction imaging
CN113744351A
