Left and right camera parameter calibration method suitable for binocular photogrammetry of unmanned aerial vehicle

Abstract

The invention relates to the technical field of dynamic photogrammetry, in particular to a left and right camera parameter calibration method suitable for binocular photogrammetry of an unmanned aerial vehicle, calibration can be realized only by taking one picture by a left camera and a right camera, and compared with an existing method which needs a plurality of standard checkerboards and structural characteristics, the method has wider applicability and is more convenient to operate. According to the method, the three-dimensional vibration response of the engineering structure is tested by using the unmanned aerial vehicle, the method is particularly suitable for a test area where field personnel are difficult to reach, the working efficiency can be effectively improved, the relative positions of the left camera and the right camera are not strictly limited, and the method has higher maneuverability, flexibility and practicability and is suitable for popularization and application. In this way, the technical problem that in the prior art, when binocular photogrammetry is carried out on the unmanned aerial vehicle, the relative position parameters of the left lens and the right lens are difficult to effectively calibrate is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic photogrammetry, and in particular to a left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry. Background Art

[0002] With the development of economy, the environment of engineering structures such as high-rise buildings, bridges, transmission towers and other structures is becoming more and more complex. Therefore, testing the actual mechanical properties of engineering structures and analyzing the actual state of the structures are extremely important for ensuring the safe operation of engineering structures. The traditional structural vibration monitoring method requires the installation of equipment such as accelerometers on the engineering structures, and the configuration of corresponding data acquisition equipment and data transmission devices. The equipment types are numerous, the quantity is large, and the maintenance cost is high. Therefore, non-contact vibration testing has been paid attention to, especially the photogrammetry technology based on structural images or video streams, which has outstanding advantages such as simple hardware equipment and convenient testing.

[0003] According to the number of lenses used, the existing photogrammetry technology can be roughly divided into monocular photogrammetry and binocular photogrammetry. Monocular photogrammetry only requires one camera, but can only test the motion and vibration response of the moving point in the plane; while binocular photogrammetry technology uses two left and right lenses to simultaneously shoot the vibration of the engineering structure, which can test the motion of the moving point in three-dimensional space and the three-dimensional vibration response of the structure. Binocular photogrammetry can be performed using two lenses with fixed relative positions of the left and right cameras, or two lenses with non-fixed relative positions, such as two cameras or two drones.

[0004] However, to extract the vibration response of moving points by binocular photogrammetry, it is necessary to estimate the relative position relationship between the left and right lenses in advance, that is, the translation matrix and rotation matrix of the right camera relative to the left camera. Existing camera calibration methods generally include traditional calibration method, self-calibration method, active vision calibration method, Zhang Zhengyou calibration method, etc. Although the parameters obtained by the traditional calibration method are highly accurate, it has high requirements for the processing of its calibration objects; the principle of the self-calibration method is to use the images taken by the camera and make agreements inside the camera, but its calibration accuracy is not high; the active vision calibration method processes the information on the image by shooting multiple images at different positions during the movement process, and obtains the internal parameters of the camera using linear transformation; the Zhang Zhengyou calibration method is similar to the image obtained by the self-calibration method. The calibration parameters can be obtained by moving the checkerboard calibration board. The Zhang Zhengyou calibration method balances the advantages and disadvantages of the traditional calibration method and the self-calibration method. It has high accuracy and is easy to operate. Therefore, it is widely used in camera calibration. Therefore, the existing binocular photogrammetry left and right camera relative parameter calibration work mostly adopts the Zhang Zhengyou calibration method, but it still needs to be calibrated after the left and right cameras are set up and before testing, and when the positions of the left and right cameras change, the calibration work needs to be re-performed, which greatly limits the use of binocular photogrammetry scenarios. The reason is that for remote areas such as power transmission lines and railway bridges, it is very difficult for technicians to reach the location of the structure to be observed, or it takes a long time. Therefore, the use of binocular photogrammetry devices with fixed relative positions of left and right cameras has great limitations. The use of drones is more convenient, but when two drones are used together for measurement, it is difficult to calibrate the relative position parameters of the left and right lenses in advance. In addition, the engineering structures are different, so it is difficult to use the characteristics of the structure to calibrate the parameters of the left and right cameras.

[0005] In summary, in the prior art, when binocular photogrammetry is performed on UAVs, there is a problem that the relative position parameters of the left and right lenses are difficult to effectively calibrate. Summary of the invention

[0006] The purpose of the present invention is to provide a left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry, aiming to solve the technical problem in the prior art that the relative position parameters of the left and right lenses are difficult to effectively calibrate when binocular photogrammetry is performed on unmanned aerial vehicles.

[0007] To achieve the above object, the present invention adopts a left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry, comprising the following steps:

[0008] Step 1: Based on the engineering structure to be tested, a plurality of three-dimensional markers composed of two standard checkerboards are placed on it;

[0009] Step 2: Perform environmental or artificial excitation on the engineering structure, use the left and right cameras to shoot the vibration process of the engineering structure, and collect 60 frames of images per second;

[0010] Step 3: segment each chessboard grid in the first frame of the left camera and the right camera;

[0011] Step 4: calibrate the relative position parameters of the left and right cameras when shooting the initial frame, and export the calibration results;

[0012] Step 5: Based on the first frame of the picture taken by the left camera, the second frame to the last frame of the picture taken by the left camera are transformed by homography matrix to realize that the shooting positions of different frames of the left camera are unified to the shooting position of the first frame of the picture; based on the first frame of the picture taken by the right camera, the second frame to the last frame of the picture taken by the right camera are transformed by homography matrix to realize that the shooting positions of different frames of the right camera are unified to the shooting position of the first frame of the picture;

[0013] Step 6: Based on the calibrated relative position parameters of the left and right cameras when shooting the initial frame, stereo matching is performed on each frame of images shot by the left and right cameras, so that the projection points of the same scene point in the images shot by the left and right cameras are located on parallel scanning lines;

[0014] Step 7: Use the key feature points in the image after stereo matching of the left and right lenses to form matching point pairs, calculate the disparity map, perform time-series matching on the left and right images from the 1st frame to the Nth frame, accurately track the dynamic changes of the structure in the entire time series, and obtain the three-dimensional vibration response of the engineering structure.

[0015] The specific method of segmenting each chessboard grid in the first frame of the left camera and the right camera is as follows:

[0016] Select the first frame of the left camera or the right camera, defined as picture A0, extract the positions of all chessboard squares in picture A0, and number all chessboard squares from 1 to M;

[0017] Then, M images A1-AM are constructed with the same size and resolution as image A0. Each image contains only a single standard checkerboard image. For example, image A1 contains only the standard checkerboard image labeled 1, image A2 contains only the standard checkerboard image labeled 2, and so on. Image AM contains only the standard checkerboard image labeled M.

[0018] Among them, taking the first frame of the picture taken by the left camera as a reference, the second frame to the last frame of the picture taken by the left camera is transformed by homography matrix to realize the method of unifying the shooting positions of different frames of the left camera to the shooting position of the first frame of the picture. The method of taking the first frame of the picture taken by the right camera as a reference, the second frame to the last frame of the picture taken by the right camera is transformed by homography matrix to realize the shooting positions of different frames of the right camera to be unified to the shooting position of the first frame of the picture. The specific method is as follows:

[0019] The extraction of the homography matrix H can be performed by identifying the pixel coordinate positions of the corner points in the specified chessboard in different frame images, and then the second frame to the last frame image is transformed using the identified homography matrix H and the following formula:

[0020]

[0021] Where s = h 31 u'+h 32 v'+h 33 , u', v' are the pixel coordinates of the measurement point before correction, and u, v are the pixel coordinates of the measurement point after correction.

[0022] Among them, in step 4, the relative position parameters of the left and right cameras when the initial frame is captured are calibrated using a calibration program, and the calibration results are exported.

[0023] The present invention provides a left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry, which performs homography matrix correction on each frame of pictures taken by the left and right cameras respectively, and the correction basis is the first frame of pictures taken by the left and right cameras respectively; then, the internal and external parameters of the calibrated initial frame of the left and right cameras are used to perform stereo matching on each frame of images taken by the left and right cameras, and the three-dimensional vibration response of the structure is identified;

[0024] This method only needs one photo taken by the left and right cameras to achieve calibration. Compared with the existing method that requires multiple standard checkerboards and structural features, this method has wider applicability. This method realizes the test of the three-dimensional vibration response of engineering structures using drones, and is particularly suitable for test areas that are difficult for field personnel to reach. It can effectively improve work efficiency and has no strict restrictions on the relative positions of the left and right cameras. It has stronger maneuverability and practicality. In this way, the technical problem in the existing technology that the relative position parameters of the left and right lenses are difficult to effectively calibrate when implementing binocular photogrammetry for drones is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of the three-dimensional marking layout of the engineering structure of the present invention.

[0027] Figure 2 It is a schematic diagram of disassembling 8 standard chessboard grids in an embodiment of the present invention.

[0028] Figure 3 It is a flow chart of the steps of the left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0030] See also Figure 1 to Figure 3 ,in Figure 1 is a schematic diagram of the three-dimensional marking layout of the engineering structure of the present invention, Figure 2 is a schematic diagram of disassembling eight standard chessboards in an embodiment of the present invention, Figure 3 It is a flow chart of the steps of the left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry of the present invention.

[0031] The present invention provides a left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry, comprising the following steps:

[0032] Step 1: Based on the engineering structure to be tested, a plurality of three-dimensional markers composed of two standard checkerboards are placed on it;

[0033] According to this specific implementation, the engineering structure is composed of a plurality of small steel sheets, the width and thickness of the small steel sheets are 30 mm and 1 mm respectively, and the length is 200 mm. The width of the engineering structure is 200 mm, and the total height is 600 mm.

[0034] Step 2: Perform environmental or artificial excitation on the engineering structure, use the left and right cameras to shoot the vibration process of the engineering structure, and collect 60 frames of images per second;

[0035] Step 3: segment each chessboard grid in the first frame of the left camera and the right camera;

[0036] According to this specific implementation, the specific method of segmenting each chessboard in the first frame of the left camera and the right camera is:

[0037] Select the first frame of the left camera or the right camera, defined as picture A0, extract the positions of all chessboard squares in picture A0, and number all chessboard squares from 1 to M;

[0038] Then, M images A1-AM are constructed with the same size and resolution as image A0. Each image contains only a single standard checkerboard image. For example, image A1 contains only a standard checkerboard image numbered 1.

[0039] Step 4: calibrate the relative position parameters of the left and right cameras when shooting the initial frame, and export the calibration results;

[0040] According to this specific implementation, a calibration program is used to calibrate the relative position parameters of the left and right cameras when the initial frame is captured, and the calibration results are exported.

[0041] Step 5: Based on the first frame of the picture taken by the left camera, the second frame to the last frame of the picture taken by the left camera are transformed by homography matrix to realize that the shooting positions of different frames of the left camera are unified to the shooting position of the first frame of the picture; based on the first frame of the picture taken by the right camera, the second frame to the last frame of the picture taken by the right camera are transformed by homography matrix to realize that the shooting positions of different frames of the right camera are unified to the shooting position of the first frame of the picture;

[0042] For this specific implementation, taking the first frame of the picture taken by the left camera as a reference, the second frame to the last frame of the picture taken by the left camera is transformed by homography matrix to realize the method of unifying the shooting positions of different frames of the left camera to the shooting position of the first frame of the picture. The method of taking the first frame of the picture taken by the right camera as a reference, the second frame to the last frame of the picture taken by the right camera is transformed by homography matrix to realize the method of unifying the shooting positions of different frames of the right camera to the shooting position of the first frame of the picture. The specific method is as follows:

[0043] The extraction of the homography matrix H can be performed by identifying the pixel coordinate positions of the corner points in the specified chessboard in different frame images, and then the second frame to the last frame image is transformed using the identified homography matrix H and the following formula:

[0044]

[0045] Where s = h 31 u'+h 32 v'+h 33 , u', v' are the pixel coordinates of the measurement point before correction, and u, v are the pixel coordinates of the measurement point after correction.

[0046] Step 6: Based on the calibrated relative position parameters of the left and right cameras when shooting the initial frame, stereo matching is performed on each frame of images shot by the left and right cameras, so that the projection points of the same scene point in the images shot by the left and right cameras are located on parallel scanning lines;

[0047] Step 7: Use the key feature points in the image after stereo matching of the left and right lenses to form matching point pairs, calculate the disparity map, perform time-series matching on the left and right images from the 1st frame to the Nth frame, accurately track the dynamic changes of the structure in the entire time series, and obtain the three-dimensional vibration response of the engineering structure.

[0048] A left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry of the present embodiment is used to perform homography matrix correction on each frame of the picture taken by the left and right cameras, and the correction basis is the first frame of the picture taken by the left and right cameras respectively; then, the internal and external parameters of the calibrated initial frame of the left and right cameras are used to perform stereo matching on each frame of the image taken by the left and right cameras, and the three-dimensional vibration response of the engineering structure is identified;

[0049] This method only needs one photo taken by the left and right cameras to achieve calibration. Compared with the existing method that requires multiple standard checkerboards and structural features, this method has wider applicability. This method realizes the test of the three-dimensional vibration response of engineering structures using drones, and is particularly suitable for test areas that are difficult for field personnel to reach. It can effectively improve work efficiency and has no strict restrictions on the relative positions of the left and right cameras. It has stronger maneuverability and practicality. In this way, the technical problem in the existing technology that the relative position parameters of the left and right lenses are difficult to effectively calibrate when implementing binocular photogrammetry for drones is solved.

[0050] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for calibrating left and right camera parameters for UAV binocular photogrammetry. It is characterized in that The following steps are involved: Step 1: Based on the engineering structure to be tested, a plurality of three-dimensional markers composed of two standard checkerboards are placed on it; Step 2: Perform environmental or artificial excitation on the engineering structure, and use the left and right cameras to shoot the vibration process of the engineering structure, collecting 60 frames of images per second; Step 3: segment each chessboard grid in the first frame of the left camera and the right camera; Step 4: calibrate the relative position parameters of the left and right cameras when shooting the initial frame, and export the calibration results; Step 5: Based on the first frame of the picture taken by the left camera, the second frame to the last frame of the picture taken by the left camera are transformed by homography matrix to realize that the shooting positions of different frames of the left camera are unified to the shooting position of the first frame of the picture; based on the first frame of the picture taken by the right camera, the second frame to the last frame of the picture taken by the right camera are transformed by homography matrix to realize that the shooting positions of different frames of the right camera are unified to the shooting position of the first frame of the picture; Step 6: Based on the calibrated relative position parameters of the left and right cameras when shooting the initial frame, stereo matching is performed on each frame of images shot by the left and right cameras, so that the projection points of the same scene point in the images shot by the left and right cameras are located on parallel scanning lines; Step 7: Use the key feature points in the image after stereo matching of the left and right lenses to form matching point pairs, calculate the disparity map, perform time-series matching on the left and right images from the 1st frame to the Nth frame, accurately track the dynamic changes of the structure in the entire time series, and obtain the three-dimensional vibration response of the engineering structure.

2. The left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry according to claim 1, characterized in that: The specific method of segmenting each chessboard grid in the first frame of the left camera and the right camera is as follows: Select the first frame of the left camera or the right camera, defined as picture A0, extract the positions of all chessboard squares in picture A0, and number all chessboard squares from 1 to M; Then, M images A1-AM are constructed with the same size and resolution as image A0. Each image contains only a single standard checkerboard image. For example, image A1 contains only the standard checkerboard image labeled 1, image A2 contains only the standard checkerboard image labeled 2, and so on. Image AM contains only the standard checkerboard image labeled M.

3. The left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry according to claim 2, characterized in that: Taking the first frame of the picture taken by the left camera as a reference, the second frame to the last frame of the picture taken by the left camera is transformed by homography matrix to realize the method of unifying the shooting positions of different frames of the left camera to the shooting position of the first frame of the picture. The method of taking the first frame of the picture taken by the right camera as a reference, the second frame to the last frame of the picture taken by the right camera is transformed by homography matrix to realize the method of unifying the shooting positions of different frames of the right camera to the shooting position of the first frame of the picture. The specific method is as follows: The extraction of the homography matrix H can be performed by identifying the pixel coordinate positions of the corner points in the specified chessboard in different frame images, and then the second frame to the last frame image is transformed using the identified homography matrix H and the following formula: Where s = h 31 u'+h 32 v'+h 33 , u', v' are the pixel coordinates of the measurement point before correction, and u, v are the pixel coordinates of the measurement point after correction.

4. The left and right camera parameter calibration method suitable for unmanned aerial vehicle binocular photogrammetry according to claim 3, characterized in that: In step 4, the relative position parameters of the left and right cameras when the initial frame is captured are calibrated using a calibration program, and the calibration results are exported.

Images