A method for calibrating left and right camera parameters for UAV binocular photogrammetry

Abstract

This invention relates to the field of dynamic photogrammetry technology, and in particular to a method for calibrating the parameters of the left and right cameras in binocular photogrammetry for UAVs. This method only requires one photograph taken by the left and right cameras to achieve calibration. Compared with existing methods that require multiple standard checkerboard patterns and structural features, it has a wider range of applicability. This method enables the testing of three-dimensional vibration response of engineering structures using UAVs, and is particularly suitable for testing areas that are difficult for field personnel to reach. It can effectively improve work efficiency and does not have strict restrictions on the relative positions of the left and right cameras, thus having greater mobility, flexibility and practicality. In this way, it solves the technical problem in the prior art that it is difficult to effectively calibrate the relative position parameters of the left and right lenses when performing binocular photogrammetry for UAVs.

Description

Technical Field

[0001] This invention relates to the field of dynamic photogrammetry technology, and in particular to a method for calibrating the parameters of the left and right cameras for binocular photogrammetry of unmanned aerial vehicles. Background Technology

[0002] With economic development, engineering structures such as high-rise buildings, bridges, and transmission towers operate in increasingly complex environments. Therefore, testing the actual mechanical properties of these structures and analyzing their actual state is crucial for ensuring their safe operation. Traditional structural vibration monitoring methods require embedding accelerometers and other equipment in the structure, along with corresponding data acquisition and transmission devices. This results in a large number of devices and high maintenance costs. Consequently, non-contact vibration testing has gained attention, especially photogrammetry based on structural images or video streams, which offers significant advantages such as simple hardware and convenient testing.

[0003] Based on the number of lenses used, existing photogrammetry techniques can be broadly categorized into monocular photogrammetry and binocular photogrammetry. Monocular photogrammetry requires only one camera but can only test the motion and vibration response of a moving point within a plane. Binocular photogrammetry, on the other hand, utilizes two lenses simultaneously to capture the vibration of an engineering structure, enabling the testing of the motion of a moving point in three-dimensional space and the three-dimensional vibration response of the structure. Binocular photogrammetry can be performed using two lenses with fixed relative positions, or it can use two lenses with non-fixed relative positions, such as two cameras or two drones.

[0004] However, extracting the vibration response of a moving point using binocular photogrammetry requires pre-estimating the relative positional relationship between the left and right lenses, i.e., the translation and rotation matrices of the right camera relative to the left camera. Existing camera calibration methods generally include traditional calibration methods, self-calibration methods, active visual calibration methods, and Zhang Zhengyou calibration methods. While traditional calibration methods yield high-precision parameters, they require sophisticated processing of the calibration materials. Self-calibration methods utilize images captured by the camera and internal camera specifications, but their accuracy is low. Active vision calibration methods capture multiple images from different positions during motion, process the information in the images, and use linear transformations to obtain the camera's internal parameters. The Zhang Zhengyou calibration method, similar to self-calibration, obtains calibration parameters by moving a checkerboard calibration board. It balances the advantages and disadvantages of traditional and self-calibration methods, offering high precision while being easy to operate, making it widely used in camera calibration. Therefore, current binocular photogrammetry often employs the Zhang Zhengyou calibration method for relative parameter calibration of the left and right cameras. However, calibration is still required before testing after the left and right cameras are set up, and recalibration is necessary when the camera positions change, significantly limiting the application scenarios of binocular photogrammetry. The reason is that for remote areas such as power transmission lines and railway bridges, it is difficult for technicians to reach the location of the structure to be observed, or it may take a long time. Therefore, the use of binocular photogrammetry devices with fixed relative positions of left and right cameras has significant limitations. While drones are more convenient to use, it is difficult to calibrate the relative position parameters of the left and right lenses in advance when two drones are used for measurement. Moreover, the engineering structures vary, making it difficult to calibrate the parameters of the left and right cameras using the characteristics of the structure.

[0005] In summary, existing technologies for performing binocular photogrammetry on drones present the problem of difficulty in effectively calibrating the relative position parameters of the left and right lenses. Summary of the Invention

[0006] The purpose of this invention is to provide a method for calibrating the parameters of the left and right cameras in binocular photogrammetry for UAVs, aiming to solve the technical problem that the relative position parameters of the left and right lenses are difficult to calibrate effectively when performing binocular photogrammetry on UAVs in the prior art.

[0007] To achieve the above objectives, the present invention employs a method for calibrating the parameters of the left and right cameras suitable for binocular photogrammetry of unmanned aerial vehicles, comprising the following steps:

[0008] Step 1: Based on the engineering structure to be tested, place multiple three-dimensional markers on it, which are composed of two standard checkerboard squares;

[0009] Step 2: Apply environmental or artificial excitation to the engineering structure and use two cameras (left and right) to capture the vibration process of the engineering structure at 60 frames per second.

[0010] Step 3: Segment the checkerboard pattern in the first frame images from the left and right cameras;

[0011] Step 4: calibrate the relative position parameters of the left and right cameras during the initial frame capture and export the calibration results;

[0012] Step 5: Using the first frame image captured by the left camera as a reference, perform homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames of the left camera to the shooting position of the first frame image. Using the first frame image captured by the right camera as a reference, perform homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames of the right camera to the shooting position of the first frame image.

[0013] Step 6: Based on the relative position parameters of the left and right cameras when shooting the initial frame, perform stereo matching on each frame of images captured by the left and right cameras so that the projection points of the same scene point in the images captured by the left and right cameras are located on parallel scan lines.

[0014] Step 7: Using the key feature points in the images after stereo matching of the left and right lenses, form matching point pairs, calculate the disparity map, and perform time-series matching on the left and right images from frame 1 to frame N to accurately track the dynamic changes of the structure throughout the time series and obtain the three-dimensional vibration response of the engineering structure.

[0015] The specific method for segmenting each checkerboard grid in the first frame images of the left and right cameras is as follows:

[0016] Select the first frame image from either the left or right camera and define it as image A0. Extract the positions of all chessboard squares in image A0 and number all chessboard squares from 1 to M.

[0017] Then, construct M images A1-AM with the same size and resolution as image A0. Each image contains only a single standard checkerboard pattern image. For example, image A1 contains only the standard checkerboard pattern image labeled 1, image A2 contains only the standard checkerboard pattern image labeled 2, and so on. Image AM contains only the standard checkerboard pattern image labeled M.

[0018] The method of using the first frame captured by the left camera as a reference and performing a homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames of the left camera to the shooting position of the first frame is the same as the method of using the first frame captured by the right camera as a reference and performing a homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames of the right camera to the shooting position of the first frame. The specific methods are as follows:

[0019] The homography matrix H can be extracted by identifying the pixel coordinates of the corner points within a specified checkerboard grid in different frames of the image. Then, the identified homography matrix H and the following formula are used to transform the images from the second frame to the last frame.

[0020]

[0021] Where s = h 31 u'+h 32 v'+h 33 u' and v' are the pixel coordinates of the measurement points before correction, and u and v are the pixel coordinates of the measurement points after correction.

[0022] In step four, the relative position parameters of the left and right cameras during the initial frame capture are calibrated using a calibration program, and the calibration results are exported.

[0023] The present invention provides a method for calibrating the left and right camera parameters for binocular photogrammetry of unmanned aerial vehicles. The method involves performing homography matrix correction on each frame of images captured by the left and right cameras, with the first frame of images captured by each camera as the reference for correction. Then, using the calibrated intrinsic and extrinsic parameters of the left and right cameras in the initial frame, the method performs stereo matching on each frame of images captured by the left and right cameras to identify the three-dimensional vibration response of the structure.

[0024] This method requires only one photograph taken by the left and right cameras for calibration, which is more widely applicable than existing methods that require multiple standard checkerboard patterns and structural features. This method enables the testing of three-dimensional vibration response of engineering structures using UAVs, and is especially suitable for testing areas that are difficult for field personnel to reach. It can effectively improve work efficiency and does not have strict restrictions on the relative positions of the left and right cameras, thus having greater mobility, flexibility and practicality. This method solves the technical problem in existing technologies where it is difficult to effectively calibrate the relative position parameters of the left and right lenses when performing binocular photogrammetry on UAVs. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the three-dimensional marking layout of the engineering structure of the present invention.

[0027] Figure 2 This is a schematic diagram showing the disassembly of eight standard chessboard grids in an embodiment of the present invention.

[0028] Figure 3 This is a flowchart illustrating the steps of the left and right camera parameter calibration method for binocular photogrammetry of unmanned aerial vehicles (UAVs) according to the present invention. Detailed Implementation

[0029] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0030] Please see Figures 1-3 ,in Figure 1 This is a schematic diagram of the three-dimensional marking layout of the engineering structure of the present invention. Figure 2 This is a schematic diagram showing the disassembly of eight standard chessboard squares in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the steps of the left and right camera parameter calibration method for binocular photogrammetry of unmanned aerial vehicles (UAVs) according to the present invention.

[0031] This invention provides a method for calibrating the parameters of the left and right cameras in binocular photogrammetry for unmanned aerial vehicles (UAVs), comprising the following steps:

[0032] Step 1: Based on the engineering structure to be tested, place multiple three-dimensional markers on it, which are composed of two standard checkerboard squares;

[0033] In this specific embodiment, the engineering structure is composed of multiple small steel sheets, each with a width of 30mm and a thickness of 1mm, a length of 200mm, a width of 200mm, and a total height of 600mm.

[0034] Step 2: Apply environmental or artificial excitation to the engineering structure and use two cameras (left and right) to capture the vibration process of the engineering structure at 60 frames per second.

[0035] Step 3: Segment the checkerboard pattern in the first frame images from the left and right cameras;

[0036] In this specific implementation, the method for segmenting each checkerboard grid in the first frame images of the left and right cameras is as follows:

[0037] Select the first frame image from either the left or right camera and define it as image A0. Extract the positions of all chessboard squares in image A0 and number all chessboard squares from 1 to M.

[0038] Then, construct M images A1-AM with the same size and resolution as image A0. Each image contains only a single standard checkerboard pattern image, such as image A1 containing only the standard checkerboard pattern image labeled 1.

[0039] Step 4: calibrate the relative position parameters of the left and right cameras during the initial frame capture and export the calibration results;

[0040] In this specific implementation, a calibration program is used to calibrate the relative position parameters of the left and right cameras during the initial frame capture, and the calibration results are exported.

[0041] Step 5: Using the first frame image captured by the left camera as a reference, perform homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames of the left camera to the shooting position of the first frame image. Using the first frame image captured by the right camera as a reference, perform homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames of the right camera to the shooting position of the first frame image.

[0042] In this specific implementation, the method of using the first frame image captured by the left camera as a reference, and performing homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames of the left camera to the shooting position of the first frame image, is the same as the method of using the first frame image captured by the right camera as a reference, and performing homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames of the right camera to the shooting position of the first frame image. The specific method is as follows:

[0043] The homography matrix H can be extracted by identifying the pixel coordinates of the corner points within a specified checkerboard grid in different frames of the image. Then, the identified homography matrix H and the following formula are used to transform the images from the second frame to the last frame.

[0044]

[0045] Where s = h 31 u'+h 32 v'+h 33 u' and v' are the pixel coordinates of the measurement points before correction, and u and v are the pixel coordinates of the measurement points after correction.

[0046] Step 6: Based on the relative position parameters of the left and right cameras when shooting the initial frame, perform stereo matching on each frame of images captured by the left and right cameras so that the projection points of the same scene point in the images captured by the left and right cameras are located on parallel scan lines.

[0047] Step 7: Using the key feature points in the images after stereo matching of the left and right lenses, form matching point pairs, calculate the disparity map, and perform time-series matching on the left and right images from frame 1 to frame N to accurately track the dynamic changes of the structure throughout the time series and obtain the three-dimensional vibration response of the engineering structure.

[0048] Using the left and right camera parameter calibration method applicable to UAV binocular photogrammetry in this embodiment, homography matrix correction is performed on each frame of images captured by the left and right cameras respectively, with the first frame of images captured by each left and right camera as the reference for correction; then, using the endo- ...

[0049] This method requires only one photograph taken by the left and right cameras for calibration, which is more widely applicable than existing methods that require multiple standard checkerboard patterns and structural features. This method enables the testing of three-dimensional vibration response of engineering structures using UAVs, and is especially suitable for testing areas that are difficult for field personnel to reach. It can effectively improve work efficiency and does not have strict restrictions on the relative positions of the left and right cameras, thus having greater mobility, flexibility and practicality. This method solves the technical problem in existing technologies where it is difficult to effectively calibrate the relative position parameters of the left and right lenses when performing binocular photogrammetry on UAVs.

[0050] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for calibrating the parameters of the left and right cameras in UAV binocular photogrammetry. Its features are, Includes the following steps: Step 1: Based on the engineering structure to be tested, place multiple three-dimensional markers on it, which are composed of two standard checkerboard squares; Step 2: Apply environmental or artificial excitation to the engineering structure and use two cameras (left and right) to capture the vibration process of the engineering structure at 60 frames per second. Step 3: Segment the checkerboard pattern in the first frame images from the left and right cameras; Step 4: calibrate the relative position parameters of the left and right cameras during the initial frame capture and export the calibration results; Step 5: Using the first frame image captured by the left camera as a reference, perform homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames of the left camera to the shooting position of the first frame image. Using the first frame image captured by the right camera as a reference, perform homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames of the right camera to the shooting position of the first frame image. Step 6: Based on the relative position parameters of the left and right cameras when shooting the initial frame, perform stereo matching on each frame of images captured by the left and right cameras so that the projection points of the same scene point in the images captured by the left and right cameras are located on parallel scan lines. Step 7: Using the key feature points in the images after stereo matching of the left and right lenses, form matching point pairs, calculate the disparity map, and perform time-series matching on the left and right images from frame 1 to frame N to accurately track the dynamic changes of the structure throughout the time series and obtain the three-dimensional vibration response of the engineering structure.

2. The method for calibrating left and right camera parameters for UAV binocular photogrammetry as described in claim 1, characterized in that, The specific method for segmenting each checkerboard grid in the first frame images of the left and right cameras is as follows: Select the first frame image from either the left or right camera and define it as image A0. Extract the positions of all chessboard squares in image A0 and number all chessboard squares from 1 to M. Then, construct M images A1-AM with the same size and resolution as image A0, each containing only a single standard checkerboard image.

3. The method for calibrating left and right camera parameters for UAV binocular photogrammetry as described in claim 2, characterized in that, The method of using the first frame captured by the left camera as a reference, and then performing a homography matrix transformation on the second to last frames captured by the left camera to unify the shooting position of different frames from the left camera to the shooting position of the first frame, is the same as the method of using the first frame captured by the right camera as a reference, and then performing a homography matrix transformation on the second to last frames captured by the right camera to unify the shooting position of different frames from the right camera to the shooting position of the first frame. The specific method is as follows: The homography matrix H is extracted by identifying the pixel coordinates of the corner points within a specified checkerboard grid in different frames. Then, the identified homography matrix H and the following formula are used to transform the images from the second to the last frame. Where s = h 31 u'+h 32 v'+h 33 u' and v' are the pixel coordinates of the measurement points before correction, and u and v are the pixel coordinates of the measurement points after correction.

4. The method for calibrating left and right camera parameters for UAV binocular photogrammetry as described in claim 3, characterized in that, In step four, the relative position parameters of the left and right cameras during the initial frame capture are calibrated using a calibration program, and the calibration results are exported.

Images