A Stereo Calibration Method for Camera Lens Distortion

Abstract

The present invention discloses a stereo rectification method for camera lens distortion. First, using the calibration parameters of the camera and the stereo calibration parameters, stereo rectification is performed on the image to obtain the rectified camera parameters. However, due to the inherent perspective distortion of the optical lens, the collected image has distortion, resulting in data loss in some areas of the rectified image. It is necessary to crop the image and recalculate the camera parameters after cropping. Finally, using the finally obtained camera parameters, the stereo rectification process is implemented. Compared with the traditional stereo rectification method, the present invention proposes a stereo rectification strategy of cropping and rectification, which maximally retains the visible area of the rectified image, thereby achieving high-precision stereo matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and particularly relates to a stereo calibration method for camera lens distortion. Background Art

[0002] Stereo vision is an important branch of computer vision. It is a multi-disciplinary field involving in-depth research in many disciplines such as visual physiology, psychology, physics, mathematics, and computer science. Its application fields are expanding rapidly and have broad application prospects in modern industries, medicine, space technology, etc. The research on stereo vision not only has important theoretical significance but also has very important practical value. A computer binocular stereo vision system mainly includes: hardware system configuration and composition, system calibration, extraction, matching, and three-dimensional reconstruction of image features.

[0003] The method adopted in binocular stereo vision is to use two cameras to obtain two original digital images of the target object and the surrounding scenery from different angles, or use a single camera to obtain two original digital images of the surrounding scenery from different angles at different time periods. Based on the positional relationship between the two cameras, the three-dimensional spatial information of the target object is restored, and the spatial position and three-dimensional shape of the target object are reconstructed.

[0004] The University of Washington and Microsoft Corporation cooperated to develop the Mars satellite "Pathfinder" and developed a wide-baseline binocular stereo vision system, enabling "Pathfinder" to accurately position and navigate the terrain within several kilometers it was about to cross on Mars. The system uses the same camera to collect image pairs at different positions on "Pathfinder". The greater the collection interval, the wider the baseline, and the farther the terrain that can be observed. The system uses non-linear optimization to obtain the relatively accurate positions of the cameras when collecting images twice, uses the robust maximum likelihood probability method combined with efficient stereo search for image matching to obtain sub-pixel accuracy parallax, and calculates the three-dimensional coordinates of each point in the image pair based on this parallax. Compared with traditional binocular stereo vision systems, it can draw the terrain around "Pathfinder" more accurately and observe farther terrain with higher accuracy.

[0005] Southeast University, based on binocular stereo vision, innovatively proposed a stereo matching method for minimizing the absolute value of gray multi-peak parallax to non-contact precisely measure the three-dimensional spatial coordinates of a three-dimensional irregular object deflection coil.

[0006] In the process of three-dimensional reconstruction of binocular stereo vision, it is necessary to match the projection points of points in space in two images, and then restore the depth value of the measurement scene according to the triangulation relationship, and further combine the camera parameters to restore the three-dimensional information of the scene. However, matching corresponding points in the two-dimensional space is very time-consuming. In order to reduce the matching search range, it is necessary to perform stereo calibration of the binocular camera, strictly align the two images after removing distortion, and use the epipolar constraint to make the epipolar lines of the two image pairs exactly on the same horizontal line, reducing the two-dimensional matching search to one dimension.

[0007] Stereo image calibration is a preliminary work for stereo image matching. Using it can improve the matching speed, so it is an effective method to achieve fast stereo matching. Since the accuracy of stereo image calibration is directly related to the quality of image pair matching, the calibration process is an important aspect of the research work on stereo image matching. The calibration of stereo images is to perform a projective transformation on the images to restore the distorted images. However, currently, no stereo calibration method takes into account the distortion problem of the camera lens itself of the captured images, which will cause the captured images to be distorted as well. Therefore, for stereo calibration methods, there is currently a lack of a stereo calibration method that can eliminate the distortion of the test images caused by the camera lens distortion. Summary of the Invention

[0008] To solve the above technical deficiencies in the prior art, the present invention proposes a stereo calibration method for camera lens distortion, which realizes correct stereo calibration by cropping and scaling transformation of the image pair.

[0009] The technical solution to achieve the object of the present invention is: a stereo calibration method for camera lens distortion, and the specific steps are as follows:

[0010] Step 1: Use a stereo vision binocular camera to collect target images, obtain left and right test image pairs, and calculate the parameters for stereo calibration of the test image pairs collected by the camera;

[0011] Step 2: Perform stereo calibration on the test images, align the two images row by row, and use the epipolar constraint to make the epipolar lines of the two image pairs on the same horizontal line;

[0012] Step 3: Through image cropping and scaling transformation, obtain the scale factor before and after cropping and the positions of the four vertices for cropping the original image;

[0013] Step 4: According to the scale factor, recalculate the parameters of the camera after cropping;

[0014] Step 5: Perform stereo calibration on the collected images according to the final camera parameters to obtain the finally stereo-calibrated images.

[0015] Preferably, parameter calculation for stereo rectification of the test image pairs collected by the camera is performed, specifically including calculating the focal length and principal point coordinates of the camera.

[0016] Preferably, the specific calculation formulas for the focal length and principal point coordinates of the camera are as follows:

[0017] The focal lengths of the left and right cameras after rectification can be obtained by the following formula:

[0018] f y_new = min(f left_y_new , f right_y_new )

[0019] f left_new = f right_new = ([f y_new , f y_new ) round

[0020] where (·) round denotes rounding; f le f t_new and f right_new are the focal lengths of the left and right cameras after rectification, f left_y_new is the focal length of the left camera in the y direction, and f right_y_new is the focal length of the right camera in the y direction;

[0021] The principal point coordinates of the left and right cameras after rectification can be obtained by the following formula:

[0022]

[0023] In the formula, W and H are the width and height of the image respectively, A left , A right are the internal parameter matrices of the left and right cameras respectively, c left_new and c right_new are the recalculated principal point coordinates of the camera, and p i is the vertex coordinate value.

[0024] Preferably, the specific steps for image stereo rectification are as follows:

[0025] The pixel coordinate systems of the two images are respectively transformed into the camera coordinate system through a common internal parameter matrix;

[0026] The two camera coordinate systems are respectively rotated to obtain new camera coordinate systems, and the two camera coordinate systems are respectively left-multiplied by the rotation matrices and

[0027] Undistortion operations for the left and right cameras are respectively performed on the new camera coordinates;

[0028] The left and right camera coordinate systems are respectively re - transformed into the left and right image pixel coordinate systems using the internal parameter matrices of the left and right cameras;

[0029] The pixel points of the new left and right images are interpolated using the pixel values of the left and right source images respectively.

[0030] Preferably, the calculation method of the rotation matrices of the left and right cameras is as follows:

[0031] Use the Rodriguez transformation to convert the rotation matrix obtained from the stereo calibration of the cameras into the form of a rotation vector;

[0032] Rotate the two cameras so that they face the same direction. After unit - ization, the translated vector t after rotation is the projection r1 of the OX axis of the corrected coordinate system in the camera coordinate system, and we get:

[0033] r l =r r ’;

[0034] t=r r ×T;

[0035] r1=t / ||t||;

[0036] where r l is the rotation vector for the left camera coordinate system to rotate half along the positive direction of the rotation vector R, r r is the rotation vector for the right camera coordinate system to rotate half along the reverse direction of the rotation vector R, T is the translation vector, and t is the translated vector after rotation;

[0037] Perform horizontal correction, and respectively find the projections r2 and r3 of the OY and OZ axes of the corrected coordinate system in the camera coordinates:

[0038]

[0039] Obtain the correction matrix:

[0040] R rect =(r1'; r2'; r3')

[0041] The rotation matrices of the left and right cameras are obtained as:

[0042] R L =R rect ×r l

[0043] R R =R rect ×r r .

[0044] Preferably, select to calculate the proportionality coefficient before and after cropping by cropping in the direction with larger directional distortion. Specifically:

[0045]

[0046] In the formula, h1 is the cropping amount above the image in the Y direction, h2 is the cropping amount below the image in the Y direction, and H is the length of the image in the Y direction.

[0047] Preferably, according to the obtained cropping ratio coefficient s and the cropping positions h1 and h2 in the Y direction, the corresponding cropping amounts in the other direction are obtained, specifically:

[0048] w1 = [s×h1] floor

[0049] w2 = [s×h2] floor

[0050] In the formula, w1 is the cropping amount to the left of the image in the X direction, w2 is the cropping amount to the right of the image in the X direction, and [·] floor represents rounding down.

[0051] Preferably, the four vertices p of the original image cropping i The specific positions are:

[0052] p i = [(w1, h1) (W - w2, h1) (w1, H - h2) (W - w2, H - h2)]

[0053] In the formula, H is the length of the image in the Y direction, W is the length of the image in the X direction, and i is the index of the four coordinate points.

[0054] Preferably, according to the proportionality coefficient, the parameters of the camera after cropping are recalculated, specifically:

[0055] According to the proportionality coefficient s and the four vertex coordinates p after cropping i Recalculate the focal length and principal point coordinates of the camera:

[0056] f Left = f left_new ×s

[0057] f Right = f right_new ×s

[0058]

[0059] Among them, f left_new is the focal length of the corrected left camera, f right_new is the focal length of the corrected right camera, c Left is the principal point of the corrected left camera, c Right is the principal point of the corrected right camera, p i are the four vertex coordinates of the cropping, A Leftis the left camera intrinsic matrix, A Right is the right camera intrinsic matrix, and W and H are the width and height of the image.

[0060] Compared with the prior art, the present invention has the following remarkable advantages: The present invention proposes a robust and high-precision calibration method. After the first calibration, the method performs cropping and scaling transformation on the calibrated image, recalculates the camera parameters after cropping using the cropping coefficient, and then performs stereo calibration on the acquired image, eliminating the partial data loss caused by distortion at the image edge and restoring it to the original image resolution. It successfully solves the image distortion problem caused by camera lens distortion, thereby obtaining an image with a small degree of distortion and achieving high-precision stereo calibration.

[0061] Other features and advantages of the present invention will be described in the following specification, and some of them will become obvious from the specification or be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings are only for the purpose of illustrating specific embodiments and are not considered as a limitation to the present invention. Throughout the drawings, the same reference signs represent the same components.

[0063] Figure 1 is a schematic diagram of an image acquisition system for a stereo calibration method for camera lens distortion.

[0064] Figure 2 is a schematic flow diagram of the present invention.

[0065] Figure 3 is a schematic diagram of the cropping process in the stereo calibration of the present invention.

[0066] Figure 4 is the cropped image and four vertices. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] It is easy to understand that according to the technical solution of the present invention, without changing the essence of the present invention, those of ordinary skill in the art can imagine various embodiments of the present invention. Therefore, the following detailed embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as all of the present invention or as a limitation or restriction of the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention will be specifically described below with reference to the drawings, where the drawings form a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0068] The inventive concept is a stereo correction method for camera lens distortion, and the specific steps are as follows:

[0069] Step 1: First, use a stereo vision binocular camera to collect a target image. As shown in the system, Figure 1 left and right test images are obtained. Then, calculate the parameters for stereo correction of the test image pair collected by the camera, and calculate the focal length and principal point coordinates of the camera;

[0070] Focal length of the left camera in the y direction (the vertical focal lengths of the two images must be the same):

[0071]

[0072] Focal length of the right camera in the y direction:

[0073]

[0074] Among them, f left and f right are the original focal lengths of the left and right cameras respectively; k left and k right are the distortion coefficients of the left and right cameras respectively; f left_y_new and f right_y_new are the focal lengths of the left and right cameras in the y direction after correction.

[0075] Finally, the focal lengths of the left and right cameras after correction can be obtained by the following formula:

[0076]

[0077] Among them, (·) round represents rounding; f left_new and f right_new are the focal lengths of the left and right cameras after correction.

[0078] The principal point coordinates of the left and right cameras after correction can be obtained by the following formula:

[0079]

[0080] Among them, W and H are the width and height of the image respectively, A = [f, c], which is the internal parameter matrix. c left_new and c right_new are the recalculated principal point coordinates of the camera, and p i is the vertex coordinate value.

[0081] It is generally considered that there is no distortion and rotation between the left and right images after correction, so it can be considered that the distortion coefficients of the left and right cameras:

[0082] k left = k right = [0; 0; 0; 0; 0](5)

[0083] Step 2: Perform stereo rectification on the test images to strictly align the two images. Using the epipolar constraint, the epipolar lines of the two image pairs are exactly on the same horizontal line. In this way, any point on one image and its corresponding point on the other image will necessarily have the same row number. When performing subsequent stereo matching, only a one-dimensional search needs to be carried out in this row to match the corresponding points.

[0084] In a further embodiment, the specific steps of image stereo rectification are as follows:

[0085] 1. Respectively transform the pixel coordinate systems of the two images to the camera coordinate system through a common internal parameter matrix;

[0086] 2. Rotate the two camera coordinate systems respectively to obtain new camera coordinate systems. The two camera coordinate systems are respectively left-multiplied by the rotation matrices and (Epipolar constraint)

[0087] 3. Perform de-distortion operations on the left and right cameras for the new camera coordinates respectively; (Distortion correction)

[0088] 4. After the de-distortion operation, respectively use the internal parameter matrices of the left and right cameras to re-transform the left and right camera coordinate systems to the left and right image pixel coordinate systems;

[0089] 5. And respectively interpolate the pixel points of the new left and right images with the pixel values of the left and right source images.

[0090] Among them, the calculation method of the rotation matrices of the left and right cameras is as follows: First, use the Rodriguez transformation to convert the rotation matrix obtained by camera stereo calibration into the form of a rotation vector. Rotate the two cameras so that they face the same direction. The unitized translation vector t is the projection of the OX axis of the calibration coordinate system in the camera coordinate system, and can be obtained:

[0091]

[0092] Among them, r l is the rotation vector of the left camera coordinate system rotating half of the positive direction along the rotation vector R, and r r is the rotation vector of the right camera coordinate system rotating half of the negative direction along the rotation vector R.

[0093] Since the stereo cameras are usually placed horizontally, taking the horizontal placement as an example in the present invention, the translation vectors of the two cameras are aligned with the X axis, and the translation vector u of the camera is [0; 0; 1], that is, perform horizontal correction, and the projections r2 and r3 of the calibration coordinate system OY and OZ in the camera coordinate can be obtained respectively:

[0094]

[0095] Next, the calibration matrix can be obtained:

[0096] R rect =(r1'; r2'; r3') (8)

[0097] Finally, the rotation matrices of the left and right cameras are obtained as:

[0098]

[0099] Step 3: Since there are distortions in the camera itself, which result in data loss in some areas of the image after stereo calibration. To maximize the retention of the visible area of the calibrated image and reduce its data loss area, an image cropping process is implemented, and then a scaling transformation is performed to obtain the scaling factor before and after cropping;

[0100] The cropping process is as Figure 2 shown. By observing the calibrated image, it can be seen that the distortion in the y - direction is larger, resulting in greater data loss in the y - direction area than in the x - direction. Therefore, the cropping positions h1 and h2 are selected according to the y - direction. Let H be the length of the direction with larger image loss. The scaling factor for cropping is calculated based on the y - direction dimensions of the image before and after cropping:

[0101]

[0102] According to the obtained cropping scaling factor s and the cropping positions h1 and h2 in the y - direction, the corresponding cropping amounts w1 and w2 in the x - direction can be obtained:

[0103]

[0104] where, [·] floor represents rounding down.

[0105] Since cropping will cause corresponding height or width loss in the original image, a scaling transformation is required to restore the test image to its original resolution.

[0106] Step 4: According to the scaling factor, recalculate the parameters of the cropped camera: focal length and principal point coordinates.

[0107] The vertex coordinate p i is located as Figure 3 shown. According to the scaling factor s and the four vertex coordinates after cropping, recalculate the focal length and principal point coordinates of the camera:

[0108]

[0109]

[0110] where, f left_newis the focal length of the corrected left camera, f right_new is the focal length of the corrected right camera, c Left is the principal point of the corrected left camera, c Right is the principal point of the corrected right camera, p i are the coordinates of the four vertices of the cropping, A Left is the internal parameter matrix of the left camera, A Right is the internal parameter matrix of the right camera, W and H are the width and height of the image.

[0111] Step 5: Perform stereo rectification on the acquired images according to the final camera parameters to obtain the finally stereo-rectified images.

[0112] In a further embodiment, the method of stereo rectification is the same as that in Step 3.

[0113] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0114] It should be understood that, in order to streamline the present invention and help those skilled in the art understand various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as meaning that the features included in the exemplary embodiments are all essential technical features of the patent claims of the present invention.

[0115] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be disposed in a device different from that of the embodiment. The different modules, units or components included in the device of the embodiment can be combined into one module, unit or component, or they can be divided into multiple sub-modules, sub-units or sub-components.

Claims

1. A stereoscopic correction method for camera lens distortion, characterized in that, The specific steps are as follows: Step 1: Use a stereo vision binocular camera to collect the target image, obtain the left and right test image pairs, and calculate the parameters for stereo rectification of the test image pairs collected by the camera; specifically, calculate the focal length and principal point coordinates of the camera. The specific calculation formulas for the focal length and principal point coordinates of the camera are as follows: The corrected left and right camera focal lengths can be obtained by the following formula: f y_new = min(f left_y_new , f right_y_new ) f left_new = f right_new = ([f y_new , f y_new ) round Among them, (·) round represents rounding; f left_new and f right_new are the focal lengths of the left and right cameras after calibration, f left_y_new is the focal length of the left camera in the y direction, f right_y_new is the focal length of the right camera in the y direction; The corrected left and right camera principal point coordinates are obtained by the following formula: Where W and H are the width and height of the image, respectively, and A left , A right are the left and right camera intrinsic matrices, respectively, c left_new and c right_new are the recomputed camera principal point coordinates, and p i is the vertex coordinate value; Step 2: Perform stereo rectification on the test images, align the rows of the two images, and use the epipolar constraint to make the epipolar lines of the two image pairs on the same horizontal line; Step 3: Through image cropping and scaling transformation, obtain the scale factor before and after cropping and the positions of the four vertices for cropping the original image; Step 4: According to the scale factor, recalculate the parameters of the camera after cropping; Step 5: Perform stereo rectification on the collected images according to the final camera parameters to obtain the finally stereo-rectified images.

2. The stereo calibration method for camera lens distortion according to claim 1, wherein The specific steps of image stereo rectification: Respectively transform the pixel coordinate systems of the two images to the camera coordinate system through a common internal parameter matrix; The two camera coordinate systems are respectively rotated to obtain new camera coordinate systems, and the two camera coordinate systems are respectively left-multiplied by the rotation matrices and Perform de-distortion operations on the left and right cameras respectively for the new camera coordinates; Respectively use the internal parameter matrices of the left and right cameras to re-transform the left and right camera coordinate systems to the left and right image pixel coordinate systems; Respectively interpolate the pixel points of the new left and right images with the pixel values of the left and right source images.

3. The stereoscopic correction method for camera lens distortion according to claim 2, wherein The calculation method of the rotation matrix of the left and right cameras is as follows: Use the Rodriguez transformation to convert the rotation matrix obtained from camera stereo calibration into the form of a rotation vector; Rotate the two cameras so that they face the same direction. After unitization, the translated vector t after rotation is the projection r1 of the OX axis of the corrected coordinate system in the camera coordinate system, and is obtained as: t = r r ×T; r1 = t / ||t||; where r l is the rotation vector obtained by rotating the left camera coordinate system by half along the positive direction of the rotation vector R, and r r is the rotation vector obtained by rotating the right camera coordinate system by half along the negative direction of the rotation vector R, T is the translation vector, and t is the translated vector after rotation; Perform horizontal correction, and respectively find the projections r2 and r3 of the OY and OZ axes of the corrected coordinate system in the camera coordinates: Obtain the correction matrix: The rotation matrices of the left and right cameras are obtained as: R L = R rect × r l R R = R rect × r r .

4. The stereoscopic correction method for camera lens distortion according to claim 1, characterized in that Select to calculate the scale factor before and after cropping by cropping in the direction where the direction distortion is greater than the set threshold. Specifically: In the formula, h1 is the cropping amount above the Y direction of the image, h2 is the cropping amount below the Y direction of the image, and H is the length of the Y direction of the image.

5. The stereo calibration method for camera lens distortion according to claim 4, characterized in that, According to the obtained cropping scale factor s and the cropping positions h1 and h2 in the Y direction, find the corresponding cropping amounts in the other direction. Specifically: where \(w1\) is the cropping amount on the left side of the image in the X direction, and \(w2\) is the cropping amount on the right side of the image in the X direction, [·] floor represents rounding down.

6. The stereoscopic correction method for camera lens distortion according to claim 5, characterized in that, The four vertices p of the original image cropping i The specific positions are as follows: p i = [(w1, h1) (W - w2, h1) (w1, H - h2) (W - w2, H - h2)] In the formula, H is the length of the Y direction of the image, W is the length of the X direction of the image, and i is the index of the four coordinate points.

7. The stereo calibration method for camera lens distortion according to claim 1, characterized in that According to the scale factor, recalculate the parameters of the camera after cropping. Specifically: According to the scale factor s and the coordinates p of the four vertices after cropping i Recalculate the focal length and principal point coordinates of the camera: f Left = f left_new × s f Right = f right_new × s Among them, f left_new is the calibrated left camera focal length, f right_new is the calibrated right camera focal length, c Left is the calibrated left camera principal point, c Right is the calibrated right camera principal point, p i are the four vertex coordinates of the cropping, A Left is the left camera internal parameter matrix, A Right is the right camera internal parameter matrix, and W and H are the width and height of the image.

Images