An epipolar correction algorithm for a binocular underwater camera system

By establishing the coordinate system of fitting cameras and determining the internal reference of the fitting camera, the problem of insufficient calibration accuracy of the pole line correction algorithm of the binocular underwater camera shooting system is solved, and high-precision pole line correction in the underwater environment is achieved, which is suitable for underwater Zoom job for binocular camera.

CN118691687BActive Publication Date: 2024-11-19INST OF DEEP SEA SCI & ENG CHINESE ACADEMY OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155542.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-19
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

In the prior art, the calibration accuracy of the polar line correction algorithm of the binocular underwater camera shooting system is insufficient, and some system parameters need to be obtained on land, which cannot meet the needs of underwater operations.

Method used

By establishing the fitted camera coordinate system and the real camera coordinate system, determining the calibration parameters and position relationships of the fitted camera components, using the pinhole camera projection model to obtain the fitted camera internal parameters, and applying the polar line correction algorithm to improve the correction accuracy .

Benefits of technology

It improves the accuracy and applicable scenarios of polar line correction, realizes polar line correction in an underwater environment, reduces the dependence on land parameter acquisition, and is suitable for zooming operations of underwater binocular cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691687B_ABST
    Figure CN118691687B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of machine vision technology, and discloses an epipolar correction algorithm for a binocular underwater camera shooting system. The present application introduces the concept of a fitting camera component of a fitting camera coordinate system. For each fitting camera of the fitting camera component based on a real camera coordinate system, the fitting camera position, the fitting camera posture, and the fitting camera internal parameters are determined in sequence, and a fitting camera coordinate system and a pixel mapping relationship are established according to the internal parameters of the fitting camera. Finally, the projection pixels of the real image corresponding to the fitting camera are determined through the pixel mapping relationship, thereby obtaining two fitting camera images. Finally, the fitting camera image and the fitting camera origin coordinates are corrected according to the epipolar correction algorithm, thereby improving the accuracy of the epipolar correction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of machine vision technology, and in particular to an epipolar correction algorithm for a binocular underwater camera shooting system. Background Art

[0002] The epipolar correction method obtained based on the pinhole camera model is difficult to guarantee the correction results of a set of underwater pictures. Specifically, the pinhole camera model does not take into account the image distortion caused by the refraction of light when passing through the medium of water → glass → air → camera, and thus cannot represent the correspondence between three-dimensional points and image points. The epipolar correction based on the pinhole camera model cannot meet the needs of underwater cameras. In fact, in addition to the parameters that can be calibrated by the pinhole camera model, the underwater camera also needs to determine the relative offset between the center of the glass cover and the center of the camera. Only by performing epipolar correction based on the underwater camera model can it better meet the needs of underwater binocular cameras for underwater operations.

[0003] The existing methods have average performance in underwater camera calibration accuracy. The correction methods developed on this basis also have limited effect on the correction of images obtained by underwater cameras. In addition, the existing methods also need to obtain some shooting system parameters on land, which greatly reduces the applicable scenarios of the existing methods. Summary of the invention

[0004] In view of the above problems, an embodiment of the present invention provides an epipolar correction algorithm for a binocular underwater camera shooting system, which is used to solve the technical problems in the prior art that the epipolar correction algorithm for a binocular underwater camera shooting system has insufficient calibration accuracy and needs to obtain some system parameters in advance on land.

[0005] According to one aspect of an embodiment of the present invention, an epipolar line correction algorithm for a binocular underwater camera shooting system is provided, wherein the underwater camera shooting system comprises a first camera, a second camera, a first glass transparent cover corresponding to the first camera, and a second glass transparent cover corresponding to the second camera, including:

[0006] Step S1, establishing a fitting camera coordinate system and a real camera coordinate system, and obtaining calibration parameters of the fitting camera component in the fitting camera coordinate system;

[0007] Step S2, determining the fitting camera origin coordinates of the fitting camera assembly according to the calibration parameters and the underwater camera model;

[0008] Step S3, determining the position and posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system;

[0009] Step S4, determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship;

[0010] Step S5, obtaining the calibration coordinates of at least two calibration points in the world coordinate system;

[0011] Step S6, determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into a pinhole camera projection model to determine the fitting camera intrinsic parameters;

[0012] Step S7, determining a right fitting camera image and a left fitting camera image;

[0013] Step S8: performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an extreme correction algorithm.

[0014] In an optional manner, the step of determining the fitting camera origin coordinates of the fitting camera assembly according to the calibration parameters and the underwater camera model includes:

[0015] Describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix;

[0016] The matrix is ​​solved to determine the fitted camera origin coordinates of the fitted camera assembly.

[0017] In an optional manner, the step of describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix includes:

[0018] The incident light path of the real camera coordinate system Described as formula 1:

[0019] (1)

[0020] in, The incident light path The Euclidean length of each light path is ; The direction cosines corresponding to each light path are , all of these optical path information are located in the real camera coordinate system;

[0021] The linear system represented by Formula 1 is expanded into a matrix:

[0022] (2)

[0023] At this time, the linear system is recorded as:

[0024] (3)

[0025] In the formula, are the three-dimensional point coordinates of each light path passing through the real camera coordinate system; is a size of The specific form of the matrix is:

[0026] (4).

[0027] In an optional manner, the step of solving the matrix to determine the coordinates of the fitting camera origin of the fitting camera assembly includes:

[0028] The matrix The specific form of the vector is as follows:

[0029] (5)

[0030] For the matrix Perform SVD decomposition:

[0031] (6)

[0032] Solve the above vector:

[0033] (7)

[0034] According to formula (5), The first three lines of the vector are the coordinates of the fitted camera origin. :That is, it is recorded as:

[0035] (8).

[0036] In an optional manner, the step of determining the posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system includes:

[0037] Take the center of the fitted camera assembly As the origin, a fitting camera coordinate system is constructed, and the fitting camera coordinate system is located at The camera coordinate system at is parallel, here is the underwater camera number. In the camera coordinate system, the pose of the fitted camera assembly is recorded as , as follows:

[0038] (9)

[0039] here is the origin of the camera coordinate system, is the coordinate of the center of the fitting camera component corresponding to i in the camera coordinate system, so ;

[0040] The pose of a set of fitted camera coordinate systems in their respective camera coordinate systems can be recorded as: left fitting camera, ; Right fitting camera, .

[0041] In an optional manner, the step of determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship includes:

[0042] Take the external parameter acquisition process of the right fitting camera as an example:

[0043] Based on the left camera coordinate system, combined with the calibrated external parameters of the right camera coordinate system , according to the rigid body transformation relationship, obtain the translation vector of the right fitting camera relative to the left fitting camera in the world coordinate system :

[0044] (10)

[0045] In addition, since the two fitting cameras are relative to their respective corresponding camera coordinate system postures are all unit matrices, then the posture of the right fitting camera relative to the left fitting camera is ;

[0046] The external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system are .

[0047] In an optional manner, the step of determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into a pinhole camera projection model to determine the fitting camera intrinsic parameters comprises:

[0048] During the calibration process, the world coordinate system coincides with the plane of the calibration plate. Here, the position and posture of the calibrated camera coordinate system are recorded as ;

[0049] The world coordinates of the 3D point Convert to the camera coordinate system through rigid body transformation:

[0050] (11)

[0051] The coordinates of this point in the camera coordinate system Go to the fitted camera coordinate system Down:

[0052] (12)

[0053] Get the coordinates of the three-dimensional point in the fitting camera coordinate system Finally, the camera intrinsic parameters are determined according to the pinhole camera projection model:

[0054] (13)

[0055] In the formula, point is the pixel coordinate of the projection of the 3D point on the fitted camera image, is the intrinsic parameter matrix of the fitting camera, as follows:

[0056] (14)

[0057] Equation (14) contains four camera internal parameters, including: focal length , principal point coordinates , when determining these internal parameters, no less than two 3D points and the corresponding fitting image points are required.

[0058] In an optional manner, the two three-dimensional points are any two of the four corner points of the largest rectangle on the chessboard calibration plate.

[0059] In an optional manner, the step of determining the right fitting camera image and the left fitting camera image includes:

[0060] Given the pixels of the fitted camera image , combined with formula (11-14), we can find the corresponding coordinates in the camera coordinate system As shown below:

[0061] (15)

[0062] in, is a defined constant used to intercept a 3D point on the ray where the incident light path lies;

[0063] 3D Point The pixel projected onto the real image can be found by the mathematical projection model of the underwater camera ;

[0064] The coordinates of the fitted camera image are obtained by the least squares method. Adjustments:

[0065] (16)

[0066] Fitting the pixels of the camera image The initial coordinates of are used as the initial values ​​of the least squares;

[0067] This adjustment process is repeated for all points in the image area to obtain an image that fits the camera.

[0068] In an optional manner, the step of performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an extreme correction algorithm comprises:

[0069] According to the external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system Determine the external parameters of the right fitting camera relative to the left fitting camera in the left fitting camera coordinate system: ;

[0070] In combination with the Bouguet method, the two fitting camera coordinate systems need to be adjusted to be parallel. First, the rotation matrix Use Rodriguez's formula to convert to angular axis vector In the form of angular axis vector, the characteristic is that the magnitude of the vector represents the rotation angle, and its unit vector is the rotation axis. To make the two fitted camera image planes parallel;

[0071] The angle that the left fitting camera coordinate system needs to rotate is , the angle that the right fitting camera coordinate system needs to rotate is ;

[0072] Use the Rodriguez formula to convert the rotation angle and rotation axis into the corresponding rotation matrix and ;

[0073] Among them, in the left fitting camera coordinate system, the right fitting camera translation vector is:

[0074] (17)

[0075] In the formula, the translation vector is denoted as ;

[0076] Construct a new rotation matrix , so that the baseline of the fitting camera is parallel to the image planes of the two fitting cameras, and its construction process is achieved by translating the vector accomplish;

[0077] (18)

[0078] In formula (18), the translation vector Normalized to get the vector , this vector is the direction vector of the X-axis of the two fitted camera coordinate systems after correction;

[0079] The Y axis of the fitted camera coordinate system after epipolar correction must be perpendicular to the plane XOZ. Then the Y axis direction vector is represented as:

[0080] (19)

[0081] According to the characteristics of the orthogonal base coordinate system, the direction vector of the Z axis of the adjusted fitting camera coordinate system can be obtained. .

[0082] (20)

[0083] Then, the correction matrix for adjusting the two fitted cameras to be coplanar is ;

[0084] The adjusted rotation matrix is and , to achieve epipolar correction of the two fitted cameras.

[0085] The present application introduces the concept of a fitting camera component of a fitting camera coordinate system. For each fitting camera of the fitting camera component based on the real camera coordinate system, the fitting camera position, the fitting camera posture, and the fitting camera internal parameters are determined in sequence, and a fitting camera coordinate system and a pixel mapping relationship are established according to the internal parameters of the fitting camera. Finally, the projection pixels of the real image corresponding to the fitting camera are determined through the pixel mapping relationship, thereby obtaining two fitting camera images. Finally, the fitting camera image and the fitting camera origin coordinates are corrected according to the epipolar correction algorithm, thereby improving the accuracy of the epipolar correction.

[0086] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0088] Figure 1 A schematic diagram showing the flow of an epipolar line correction algorithm for a binocular underwater camera shooting system provided by the present invention is shown;

[0089] Figure 2 The main structural schematic diagram of the underwater camera shooting system provided by the present invention is shown;

[0090] Figure 3 A schematic diagram of the light refraction path of the underwater camera shooting system provided by the present invention is shown;

[0091] Figure 4A schematic diagram of the epipolar geometry of a terrestrial binocular camera is shown;

[0092] Figure 5 A schematic diagram showing the polar plane change of the binocular underwater camera shooting system provided by the present invention is shown;

[0093] Figure 6 A schematic diagram of an underwater camera projection model in a world coordinate system of a binocular underwater camera shooting system provided by the present invention is shown;

[0094] Figure 7 A schematic diagram of the underwater image epipolar correction process of the binocular underwater camera shooting system provided by the present invention is shown;

[0095] Figure 8 A schematic diagram of the underwater fitting camera projection process of the binocular underwater camera shooting system provided by the present invention is shown;

[0096] Fig. 9 A schematic diagram of an underwater fitting binocular camera polar plane of a binocular underwater camera shooting system provided by the present invention is shown. DETAILED DESCRIPTION

[0097] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0098] In the exemplary technology, in a shallow water environment and when the required calibration accuracy is not high, the industry roughly regards the underwater shooting system as a pinhole camera model under an air shooting system. The model is calibrated using the Zhang Zhengyou calibration method. [A flexible new technique for camera calibration, 2000]; after the calibration of the binocular camera system is completed, the Fusiello method is used to perform epipolar correction. [Quasi-euclidean uncalibrated epipolar rectification, 2008]

[0099] Elnashef et al. proposed a calibration scheme that can achieve a rough linear estimate of the initial camera placement position and discussed the axial characteristics of the spherical cover model. The operator is required to adjust the camera position so that it coincides with the center of the dome cover so that the image is not distorted by light path refraction. [Geometry, calibration, and robust centering procedures for refractive dome-port based imaging systems, 2022].

[0100] In addition, the binocular camera system consists of two cameras arranged at different angles, which can simultaneously shoot the target. According to the principle of triangulation, a set of pictures can be used to obtain three-dimensional information of the surface of the scene. In an underwater environment, due to the influence of undercurrents, there are very few static scenes in the water. The process of triangulation using a binocular camera can meet the needs of machine vision in underwater dynamic environments, and the process combines the reliability of the application with the feasibility of implementation. In addition, binocular camera equipment also has the advantages of simple operation and low operating cost.

[0101] Epipolar correction is a unique technology of binocular machine vision. Figure 4 As shown in the figure, epipolar correction technology can use mathematical methods to make the projection of a 3D point on the pictures taken by two cameras lie on the same straight line. When triangulating a 3D point, it is necessary to find the projection of the point in two perspectives. When searching for projection points, epipolar correction technology can narrow the search area, thereby improving the efficiency and accuracy of finding projection point pairs. When binocular machine vision is used underwater, it is necessary to apply underwater cameras, and the special structure of underwater cameras raises new problems for epipolar correction technology.

[0102] To protect the camera equipment, the camera is usually installed in a pressure-resistant shell when the shooting system is working underwater. The camera shoots the outside through the glass cover. Figure 3 As shown, when shooting, the light goes through the process of water → glass → air → camera.

[0103] Underwater cameras are generally equipped with hemispherical glass covers. This type of glass cover has better mechanical properties, allowing the camera to have a larger field of view. When the center of the camera and the center of the cover do not coincide, light will refract when passing through different media, causing distortion in the pictures taken by the underwater camera. The accuracy of the method of finding the epipolar line on land is poor for distorted underwater pictures, resulting in the average performance of the epipolar line correction method of the land binocular system.

[0104] The present application proposes an epipolar correction algorithm for a binocular underwater camera shooting system, which is used to solve the technical problems in the prior art that the epipolar correction algorithm for a binocular underwater camera shooting system has insufficient calibration accuracy and needs to obtain some system parameters in advance on land.

[0105] In an alternative embodiment, referring to Figure 1 , Figure 2 as well as Figure 6 As shown in Figure 1, an epipolar correction algorithm for a binocular underwater camera system is used. Figure 2 As shown, the underwater camera shooting system includes a first camera, a second camera, a first glass cover corresponding to the first camera, and a second glass cover corresponding to the second camera, including:

[0106] Step S1, establishing a fitting camera coordinate system and a real camera coordinate system, and obtaining calibration parameters of the fitting camera component in the fitting camera coordinate system;

[0107] Step S2, determining the fitting camera origin coordinates of the fitting camera assembly according to the calibration parameters and the underwater camera model;

[0108] Step S3, determining the position and posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system;

[0109] Step S4, determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship;

[0110] Step S5, obtaining the calibration coordinates of at least two calibration points in the world coordinate system;

[0111] Step S6, determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into a pinhole camera projection model to determine the fitting camera intrinsic parameters;

[0112] Step S7, determining a right fitting camera image and a left fitting camera image;

[0113] Step S8: performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an extreme correction algorithm.

[0114] This application introduces the concept of fitting camera, and establishes a fitting camera coordinate system and a pixel mapping relationship by successively determining the fitting camera position, fitting camera posture, fitting camera intrinsic parameters and the fitting camera intrinsic parameters. Finally, the projection pixels of the real image corresponding to the fitting camera are determined through the pixel mapping relationship, thereby obtaining two fitting camera images. Finally, the fitting camera image and the fitting camera origin coordinates are corrected according to the epipolar correction algorithm, thereby improving the accuracy of the epipolar correction.

[0115] Optionally, the step of determining the fitting camera origin coordinates of the fitting camera assembly according to the calibration parameters and the underwater camera model comprises:

[0116] Describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix;

[0117] The matrix is ​​solved to determine the fitted camera origin coordinates of the fitted camera assembly.

[0118] In an alternative embodiment, referring to Figure 4 As shown, the step of describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix includes:

[0119] The incident light path of the real camera coordinate system Described as formula 1:

[0120] (1)

[0121] in, The incident light path The Euclidean length of each light path is ; The direction cosines corresponding to each light path are , all of these optical path information are located in the real camera coordinate system;

[0122] The linear system represented by Formula 1 is expanded into a matrix:

[0123] (2)

[0124] At this time, the linear system is recorded as:

[0125] (3)

[0126] In the formula, are the three-dimensional point coordinates of each light path passing through the real camera coordinate system; is a size of The specific form of the matrix is:

[0127] (4).

[0128] Optionally, the step of solving the matrix to determine the coordinates of the fitting camera origin of the fitting camera assembly comprises:

[0129] The matrix The specific form of the vector is as follows:

[0130] (5)

[0131] For the matrix Perform SVD decomposition:

[0132] (6)

[0133] Solve the above vector:

[0134] (7)

[0135] According to formula (5), The first three lines of the vector are the coordinates of the fitted camera origin. :That is, it is recorded as:

[0136] (8).

[0137] At this time, the incident light outside the glass cover is determined by the given underwater camera model. The linear system of the incident light is combined with the matrix method to find the position of the fitting camera center.

[0138] Optionally, the step of determining the posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system comprises:

[0139] Take the center of the fitted camera assembly As the origin, a fitting camera coordinate system is constructed, and the fitting camera coordinate system is located at The camera coordinate system at is parallel, here is the underwater camera number. In the camera coordinate system, the pose of the fitted camera assembly is recorded as , as follows:

[0140] (9)

[0141] here is the origin of the camera coordinate system, is the coordinate of the center of the fitting camera component corresponding to i in the camera coordinate system, so ;

[0142] The pose of a set of fitted camera coordinate systems in their respective camera coordinate systems can be recorded as: left fitting camera, ; Right fitting camera, .

[0143] It should be noted that the fitting camera origin coordinates are equivalent to the fitting camera center coordinates. According to the coordinate system, they are called the fitting camera origin coordinates, and according to the physical position of the camera, they are called the fitting camera center coordinates. Generally, the origin of the left fitting camera is set as the fitting camera origin coordinates in the world coordinate system.

[0144] Optionally, refer to Figure 5 , 7 As shown, the step of determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship includes:

[0145] Take the external parameter acquisition process of the right fitting camera as an example:

[0146] Based on the left camera coordinate system, combined with the calibrated external parameters of the right camera coordinate system , according to the rigid body transformation relationship, obtain the translation vector of the right fitting camera relative to the left fitting camera in the world coordinate system :

[0147] (10)

[0148] In addition, since the two fitting cameras are relative to their respective corresponding camera coordinate system postures are all unit matrices, then the posture of the right fitting camera relative to the left fitting camera is ;

[0149] The external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system are .

[0150] In the above process, after determining the positions of the two fitting cameras in their respective camera coordinate systems, the pose relationship of the two fitting cameras in the camera coordinate system is obtained. This pose relationship is used as the external parameter of the fitting binocular camera. Then, the internal parameters of the two fitting cameras are obtained through the pinhole camera projection model.

[0151] In an optional embodiment, the step of determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into a pinhole camera projection model to determine the fitting camera intrinsic parameters comprises:

[0152] During the calibration process, the world coordinate system coincides with the plane of the calibration plate. Here, the position and posture of the calibrated camera coordinate system are recorded as ;

[0153] The world coordinates of the 3D point Convert to the camera coordinate system through rigid body transformation:

[0154] (11)

[0155] The coordinates of this point in the camera coordinate system Go to the fitted camera coordinate system Down:

[0156] (12)

[0157] Get the coordinates of the three-dimensional point in the fitting camera coordinate system Finally, the camera intrinsic parameters are determined according to the pinhole camera projection model:

[0158] (13)

[0159] In the formula, point is the pixel coordinate of the projection of the 3D point on the fitted camera image, is the intrinsic parameter matrix of the fitted camera, as follows:

[0160] (14)

[0161] Equation (14) contains four camera internal parameters, including: focal length , principal point coordinates , at least two 3D points and the corresponding fitting image points are required to determine these intrinsic parameters.

[0162] Optionally, the two three-dimensional points are any two of the four corner points of the largest rectangle on the chessboard calibration plate.

[0163] Optionally, the step of determining a right fitting camera image and a left fitting camera image includes:

[0164] Given the pixels of the fitted camera image , combined with formula (11-14), we can find the corresponding coordinates in the camera coordinate system As shown below:

[0165] (15)

[0166] in, is a defined constant used to intercept a 3D point on the ray where the incident light path lies;

[0167] 3D Point The pixel projected onto the real image can be found by the mathematical projection model of the underwater camera ;

[0168] The coordinates of the fitted camera image are obtained by the least squares method. Adjustments:

[0169] (16)

[0170] Fitting the pixels of the camera image The initial coordinates of are used as the initial values ​​of the least squares;

[0171] This adjustment process is repeated for all points in the image area to obtain an image that fits the camera.

[0172] Among them, according to the underwater camera projection model and the internal and external parameters of the fitting camera, the position relationship between the pixels on the fitting image and the pixels on the real image can be established. According to this position relationship, the least square method is used to adjust the pixel position of the fitting image. By traversing the pixel points of the entire image using this method, the image of the fitting camera is obtained.

[0173] Optionally, refer to Figure 8 As shown, the step of performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an extreme correction algorithm includes:

[0174] According to the external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system Determine the external parameters of the right fitting camera relative to the left fitting camera in the left fitting camera coordinate system: ;

[0175] In combination with the Bouguet method, the two fitting camera coordinate systems need to be adjusted to be parallel. First, the rotation matrix Use Rodriguez's formula to convert to angular axis vector In the form of angular axis vector, the characteristic is that the magnitude of the vector represents the rotation angle, and its unit vector is the rotation axis. To make the two fitted camera image planes parallel;

[0176] The angle that the left fitting camera coordinate system needs to rotate is , the angle that the right fitting camera coordinate system needs to rotate is ;

[0177] Use the Rodriguez formula to convert the rotation angle and rotation axis into the corresponding rotation matrix and ;

[0178] Among them, in the left fitting camera coordinate system, the right fitting camera translation vector is:

[0179] (17)

[0180] In the formula, the translation vector is denoted as ;

[0181] Construct a new rotation matrix , so that the baseline of the fitting camera is parallel to the image planes of the two fitting cameras, and its construction process is achieved by translating the vector accomplish;

[0182] (18)

[0183] In formula (18), the translation vector Normalized to get the vector , this vector is the direction vector of the X-axis of the two fitted camera coordinate systems after correction;

[0184] The Y axis of the fitted camera coordinate system after epipolar correction must be perpendicular to the plane XOZ. Then the Y axis direction vector is represented as:

[0185] (19)

[0186] According to the characteristics of the orthogonal base coordinate system, the direction vector of the Z axis of the adjusted fitting camera coordinate system can be obtained. .

[0187] (20)

[0188] Then, the correction matrix for adjusting the two fitted cameras to be coplanar is , the adjusted rotation matrix is and , to achieve epipolar correction of the two fitted cameras.

[0189] In the above process, the present invention uses the fitting camera internal parameters determined by the above process As the common internal parameter matrix of the binocular system. According to the three-dimensional point coordinates, the pinhole camera model is used to construct the pixel mapping relationship between the fitting camera image and the image after epipolar correction, thereby realizing the epipolar correction process of the fitting camera image. Among them, the adjusted fitting camera image and the center position relationship of the two fitting cameras can meet the definition of the epipolar geometry of the binocular camera. The epipolar correction process can be completed using the internal and external parameters of the two fitting cameras.

[0190] The present invention uses numerical methods to verify its actual performance. It is necessary to triangulate the corner points of the underwater chessboard image after epipolar correction, and then compare the measurement results with the three-dimensional coordinates of the corner points. The present invention uses multiple sets of underwater images to perform triangulation, and the average measurement error obtained is about 1.3697mm, and the maximum deviation from the mean does not exceed 1.4mm.

[0191] The present invention solves the problem of insufficient accuracy when performing epipolar correction after the currently commonly used pinhole camera model on land is applied to an underwater binocular camera system. For an underwater binocular camera shooting system equipped with a hemispherical glass cover, the present invention can implement the epipolar correction process of the shooting system in an underwater environment; because the epipolar correction process of the present invention is combined with the mathematical projection model of the underwater camera, the accuracy of triangulation using the adjusted image of the present invention is higher. In addition, compared with other camera correction processes that require obtaining some camera parameters on land, the present invention can achieve the entire correction process underwater. The present invention is suitable for scenarios where an underwater binocular camera shooting system needs to perform zoom operations. It should be noted that after adjusting the fitting camera image, the part of the present invention that combines the Bouguet method can also be combined with other methods of epipolar correction on land.

[0192] In the solution of the present invention, the key technical points are:

[0193] 1. Determination of the position of the fitting camera. The present invention obtains the position of the fitting camera by a matrix method. By giving a mathematical model of the underwater camera, the incident light path outside the hemispherical glass cover corresponding to the pixel on the underwater image is determined. First, the external incident light path is expressed according to formula (1) using point and direction cosine; then, the incident light path is expanded into a linear system according to formula (2), and the linear system is abstracted into the form of formula (3), and its specific parameters are formulas (4, 5); finally, formula (3) is solved according to the process of formulas (6-8) to obtain the coordinates of the center of the fitting camera.

[0194] 2. Determination of the pose of the fitting camera. The present invention sets the fitting camera coordinate system and the camera coordinate system to be parallel, and uses formula (9) to determine the pose and position of the fitting camera relative to the real camera. In the left camera coordinate system, the pose relationship between the right fitting camera and the left fitting camera is determined by formula (10).

[0195] 3. Determination of the intrinsic parameters of the fitting camera. The present invention transforms the three-dimensional points of the world coordinate system to the camera coordinate system by rigid body transformation using formula (11), and then switches to the fitting camera coordinate system using formula (12). Given the image points of the fitting camera and the corresponding three-dimensional point coordinates, the intrinsic parameters of the fitting camera are obtained according to formulas (13, 14) through the pinhole camera projection model.

[0196] 4. Acquisition of the fitted camera image. The present invention sets the size of the fitted camera image to be consistent with that of the original image. Given a pixel of the fitted image, the coordinates of the corresponding three-dimensional point in the camera coordinate system are found using formula (15). The coordinates are then used to find the projection pixels of the real image using the underwater camera projection model, and the pixel coordinates of the fitted camera are adjusted using the least squares method of formula (16). The above process is repeated for each point on the graph, and finally the image of the fitted camera is obtained.

[0197] 5. Epipolar correction of the fitted camera image. The present invention combines the obtained fitted camera image with the Bouguet method, and first adjusts the two fitted camera coordinate systems to be parallel by formula (17), and then adjusts them to be coplanar by using formulas (18-20).

[0198] For an underwater binocular camera shooting system equipped with a hemispherical glass cover, the present invention can realize an epipolar line correction process for the shooting system in an underwater environment; because the epipolar line correction process of the present invention is combined with a mathematical projection model of the underwater camera, the accuracy of triangulation using the image adjusted by the present invention is higher.

[0199] In the description provided herein, a large number of specific details are described. However, it can be understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Among them, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention. It can be understood by those skilled in the art that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiment. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive. It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the attached claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the existence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limiting the order of execution.

Claims

1. A method for correcting the epipolar line of a binocular underwater camera shooting system, characterized in that: The underwater camera shooting system includes a first camera, a second camera, a first glass cover corresponding to the first camera, and a second glass cover corresponding to the second camera, including: Step S1, establishing a fitting camera coordinate system and a real camera coordinate system, and obtaining calibration parameters of the fitting camera component in the fitting camera coordinate system; Step S2, determining the fitting camera origin coordinates of the fitting camera assembly according to the calibration parameters and the underwater camera model; Step S3, determining the position and posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system; Step S4, determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship; Step S5, obtaining the calibration coordinates of at least two calibration points in the world coordinate system; Step S6, determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into a pinhole camera projection model to determine the fitting camera intrinsic parameters; Step S7, determining a right fitting camera image and a left fitting camera image; Step S8: performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an epipolar correction method.

2. The epipolar line correction method for a binocular underwater camera shooting system according to claim 1, characterized in that: The step of determining the coordinates of the fitting camera origin of the fitting camera assembly according to the calibration parameters and the underwater camera model comprises: Describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix; The matrix is ​​solved to determine the fitted camera origin coordinates of the fitted camera assembly.

3. The epipolar line correction method of the binocular underwater camera shooting system according to claim 2, characterized in that: The step of describing the incident light path of the real camera coordinate system by point coordinates and writing it into a matrix comprises: The incident light path of the real camera coordinate system Described as formula 1: (1) in, The incident light path The Euclidean length of each light path is ; The direction cosines corresponding to each light path are , all of these optical path information are located in the real camera coordinate system; The linear system represented by Formula 1 is expanded into a matrix: (2) At this time, the linear system is recorded as: GN=S (3) In the formula, is the three-dimensional point coordinate of each light path passing through the real camera coordinate system; N is the vector in the matrix obtained by expanding the linear system; is a size of The specific form of the matrix is: (4)。 4. The epipolar line correction method for a binocular underwater camera shooting system as claimed in claim 3, characterized in that: The step of solving the matrix to determine the coordinates of the fitting camera origin of the fitting camera assembly comprises: The specific form of the N-vector of the matrix obtained by expanding the linear system is as follows: (5) For the matrix Perform SVD decomposition: (6) Solving for the N vector: (7) According to formula (5), the first three rows of the N vector are the coordinates of the fitting camera origin. :That is, it is recorded as: (8)。 5. The epipolar line correction method for a binocular underwater camera shooting system as claimed in claim 4, characterized in that: The step of determining the posture relationship between the right fitting camera and the left fitting camera of the fitting camera assembly according to the fitting camera origin coordinates, the fitting camera coordinate system and the real camera coordinate system comprises: Take the center of the fitted camera assembly As the origin, a fitting camera coordinate system is constructed, and the fitting camera coordinate system is located at The camera coordinate system at is parallel, here is the coordinate system number of the underwater camera. In the camera coordinate system, the pose of the fitted camera component is recorded as , as follows: (9) here is the origin of the camera coordinate system, is the coordinate of the center of the fitting camera component corresponding to i in the camera coordinate system, so ; The pose of a set of fitted camera coordinate systems in their respective camera coordinate systems can be recorded as: left fitting camera, ; Right fitting camera, .

6. The epipolar line correction method for a binocular underwater camera shooting system as claimed in claim 5, characterized in that: The step of determining the external parameters of the right fitting camera or the external parameters of the left fitting camera according to the posture relationship comprises: Take the external parameter acquisition process of the right fitting camera as an example: Based on the left camera coordinate system, combined with the calibrated external parameters of the right camera coordinate system , according to the rigid body transformation relationship, obtain the translation vector of the right fitting camera relative to the left fitting camera in the world coordinate system : (10) In addition, since the two fitting cameras are relative to their respective corresponding camera coordinate system postures are all unit matrices, then the posture of the right fitting camera relative to the left fitting camera is ; The external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system are .

7. The epipolar line correction method for a binocular underwater camera shooting system as claimed in claim 6, characterized in that: The step of determining the fitting coordinates in the fitting camera coordinate system corresponding to the calibration coordinates, and inputting the determined fitting coordinates into the pinhole camera projection model to determine the fitting camera intrinsic parameters comprises: During the calibration process, the world coordinate system coincides with the plane of the calibration plate. Here, the position and posture of the calibrated camera coordinate system are recorded as ; The world coordinates of the 3D point Through rigid body transformation to the camera coordinate system, the coordinates of the three-dimensional point in the camera coordinate system are obtained: (11) The coordinates of the three-dimensional point in the camera coordinate system of this point Go to the coordinate system numbered In the fitted camera coordinate system: (12) Get the pixel where the 3D point is projected on the fitted camera image Finally, the camera intrinsic parameters are determined according to the pinhole camera projection model: (13) In the formula, ; is the pixel projected by the 3D point on the fitted camera image, and It's pixels The coordinates of is the intrinsic parameter matrix of the fitted camera, as follows: (14) Equation (14) contains four camera internal parameters, including: focal length , principal point coordinates , at least two 3D points and the corresponding fitting image points are required to determine these intrinsic parameters.

8. The epipolar line correction method for a binocular underwater camera shooting system as claimed in claim 7, characterized in that: The two three-dimensional points are any two of the four corner points of the largest rectangle on the chessboard calibration plate.

9. The epipolar line correction method of a binocular underwater camera shooting system as claimed in claim 7, characterized in that: The step of determining the right fitting camera image and the left fitting camera image comprises: The pixel where a given 3D point is projected onto the fitted camera image , combined with formula (11-14), we can find the coordinates of the corresponding three-dimensional point in the camera coordinate system As shown below: (15) in, is a defined constant used to intercept a 3D point on the ray where the incident light path lies; Coordinates of the 3D point in the camera coordinate system The pixel projected onto the real image can be found by the mathematical projection model of the underwater camera ; The least squares method is used to realize the pixel projection of the three-dimensional point on the fitted camera image Adjustments: (16) The pixel where the 3D point is projected on the fitted camera image The initial coordinates of are used as the initial values ​​of the least squares; This adjustment process is repeated for all points in the image area to obtain an image that fits the camera.

10. The epipolar line correction method of a binocular underwater camera shooting system according to claim 7, characterized in that: The step of performing epipolar correction on the right fitting camera image, the left fitting camera image and the fitting camera origin coordinates by using an epipolar correction method comprises: According to the external parameters of the right fitting camera relative to the left fitting camera in the left camera coordinate system Determine the external parameters of the right fitting camera relative to the left fitting camera in the left fitting camera coordinate system: ; Combined with the Bouguet method, the two fitting camera coordinate systems are adjusted to be parallel. First, the rotation matrix Use Rodriguez's formula to convert to angular axis vector In the form, the characteristic of the angular axis vector is that the magnitude of the vector represents the rotation angle, and its unit vector is the rotation axis, so that the image planes of the two fitting cameras are parallel; The angle that the left fitting camera coordinate system needs to rotate is , the angle that the right fitting camera coordinate system needs to rotate is ; Use the Rodriguez formula to convert the rotation angle and rotation axis into the corresponding rotation matrix and ; Among them, in the left fitting camera coordinate system, the right fitting camera translation vector is: (17) In the formula, the translation vector is denoted as ; Construct a new rotation matrix , so that the baseline of the fitting camera is parallel to the image planes of the two fitting cameras, and its construction process is achieved by translating the vector accomplish; (18) In formula (18), the translation vector Normalized to get the vector , this vector is the direction vector of the X-axis of the two fitted camera coordinate systems after correction; The Y axis of the fitted camera coordinate system after epipolar correction must be perpendicular to the plane XOZ, so the Y axis direction vector is represented as: (19) According to the characteristics of the orthogonal base coordinate system, the direction vector of the Z axis of the adjusted fitting camera coordinate system can be obtained. ; (20) Then, the correction matrix for adjusting the two fitted cameras to be coplanar is ; The adjusted rotation matrix is and , to achieve epipolar correction of the two fitted cameras.