Binocular camera external parameter self-calibration method and system during unmanned aerial vehicle flight

Abstract

The application discloses a binocular camera external parameter self-calibration method and system in the flight process of a UAV, and the method comprises the following steps: collecting a monocular image sequence when the UAV flies along the optical axis direction of a binocular camera to be calibrated; performing initial feature alignment on the latest collected adjacent two frames of monocular images; extracting a region of interest in the aligned images according to the flight speed of the UAV; performing feature matching and denoising in the region of interest; calculating a homography matrix of the two frames of monocular images; and then calculating the distance between the UAV and an obstacle; comparing the distance between the UAV and the obstacle with a set threshold value; if the distance is greater than the set threshold value, the rotation amount of the external parameter is calculated; if the distance is less than the set threshold value, the UAV is controlled to fly at a reduced speed until the distance is less than or equal to a preset distance, the UAV is controlled to hover, and the translation amount of the external parameter is calculated. The application realizes the binocular camera external parameter self-calibration of the UAV in the flight process.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for self-calibrating the extrinsic parameters of a binocular camera during the flight of an unmanned aerial vehicle (UAV). Background Technology

[0002] With the development of semiconductor technology, visual sensors are being used more and more widely in small drones. For environmental perception of drones, binocular vision sensors offer advantages over sensors such as lidar and millimeter-wave radar, including lower cost, lighter weight, and richer sensory information. On one hand, binocular vision can acquire environmental depth information over a large field of view, enabling the calculation of obstacle distances in various directions; on the other hand, it provides rich image information for advanced applications such as target recognition and segmentation; furthermore, the combination of binocular vision sensors and inertial navigation sensors allows for convenient estimation of drone motion states, providing navigation information.

[0003] However, the practical application of binocular vision in drones still faces the following challenges: To ensure the accuracy of stereo imaging, the rigidity of the structural connection between the two binocular cameras is crucial. Even with a rigid connection, deformation can occur due to prolonged vibration and extreme temperature changes during drone operation, causing the previously calibrated extrinsic parameters to become invalid and reducing the accuracy of stereo ranging. When the extrinsic parameters of the binocular cameras fail, recalibration is necessary. Traditional offline calibration methods require specialized calibration boards and involve complex procedures, making them difficult for ordinary drone operators to implement.

[0004] To address the aforementioned challenges, various online self-calibration methods have been proposed in current research on binocular online self-calibration for UAVs. Among these, several methods are based on epipolar constraint equations, iteratively estimating camera extrinsic parameters through filtering / optimization methods. Additionally, some methods fuse IMU (Inertial Measurement Unit) inertial navigation data and utilize SLAM (Simultaneous Localization and Mapping) methods to estimate camera extrinsic parameters by minimizing the vision-inertial fusion error.

[0005] However, the existing methods mentioned above have certain limitations for the actual application scenarios of UAVs: for the method based on epipolar constraint optimization, the epipolar constraint degrades when the distance to obstacles is far during high-altitude flight; for the calibration method that integrates inertial navigation, it does not directly model the two cameras, but instead calculates the extrinsic parameters of the two cameras indirectly by calculating the two relative attitudes between the camera and the IMU after minimizing the visual-inertial observation error, which has relatively low accuracy. Summary of the Invention

[0006] To solve the above problems of the prior art, the application provides a binocular camera external parameter self-calibration method and system in the flight process of a UAV, which combines with the POS (Positioning System Data) information of the UAV, estimates the distance of an obstacle by means of feature matching of images collected at different times, and divides two different calibration stages according to the distance, so as to calculate the rotation and translation of the binocular camera external parameters respectively, and realizes the binocular camera external parameter self-calibration of the UAV in the flight process. The self-calibration method can make the UAV realize online double target calibration through natural scene features without scene limitation, and solve the problem of poor calibration accuracy in the prior art.

[0007] In a first aspect, the application provides a binocular camera external parameter self-calibration method in the flight process of a UAV.

[0008] A binocular camera external parameter self-calibration method in the flight process of a UAV, comprising:

[0009] Collecting a monocular image sequence when the UAV flies straight along the optical axis direction of a binocular camera to be calibrated, and obtaining the flight speed of the UAV;

[0010] Aligning the initial features of the latest collected two adjacent monocular images, extracting a region of interest in the aligned images according to the flight speed of the UAV, performing feature matching and denoising in the region of interest, calculating the homography matrix of the two monocular images, and then calculating the distance between the UAV and the obstacle;

[0011] Comparing the distance between the UAV and the obstacle with a set threshold value, if the distance is greater than the set threshold value, performing feature matching and denoising on the latest collected binocular left and right images, constructing a residual term, and calculating the rotation of the external parameters; if the distance is less than the set threshold value, controlling the UAV to fly at a reduced speed until the distance is less than or equal to a preset distance, controlling the UAV to hover, and performing feature matching and denoising on the latest collected binocular left and right images, constructing a residual term, and calculating the translation of the external parameters.

[0012] In a second aspect, the application provides a binocular camera external parameter self-calibration system in the flight process of a UAV.

[0013] A binocular camera external parameter self-calibration system in the flight process of a UAV, comprising:

[0014] A data acquisition module for collecting a monocular image sequence when the UAV flies straight along the optical axis direction of a binocular camera to be calibrated, and obtaining the flight speed of the UAV;

[0015] The distance calculation module is configured to perform initial feature alignment on the newly collected two adjacent monocular images, extract a region of interest in the aligned images according to the flight speed of the unmanned aerial vehicle, perform feature matching and denoising in the region of interest, calculate a homography matrix of the two monocular images, and further calculate the distance of the unmanned aerial vehicle from the obstacle.

[0016] The external parameter self-calibration module is configured to compare the distance of the unmanned aerial vehicle from the obstacle with a set threshold value, perform feature matching and denoising on the newly collected left and right binocular images, construct a residual term, and calculate the rotation amount of the external parameter if the distance is greater than the set threshold value; control the unmanned aerial vehicle to fly at a reduced speed until the distance is less than or equal to a preset distance, control the unmanned aerial vehicle to hover, and perform feature matching and denoising on the newly collected left and right binocular images, construct a residual term, and calculate the translation amount of the external parameter if the distance is less than the set threshold value.

[0017] In a third aspect, the present disclosure further provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of the first aspect are completed.

[0018] In a fourth aspect, the present disclosure further provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the steps of the method of the first aspect are completed.

[0019] The above one or more technical solutions have the following beneficial effects:

[0020] 1. The present application provides a binocular camera external parameter self-calibration method and system in the flight process of an unmanned aerial vehicle, which utilizes image processing and multi-view geometry related technologies. The unmanned aerial vehicle first collects monocular image sequences at different time points through a mounted binocular camera, estimates the distance of an obstacle in real time through image feature matching, then divides the calibration process into two stages according to the distance, constructs geometric constraint information of the scene in the two calibration stages, constructs different residual equations through binocular left and right image feature matching pairs, and calculates the rotation and translation of the camera external parameter respectively, thereby realizing binocular camera external parameter self-calibration in the flight process of the unmanned aerial vehicle.

[0021] 2. The binocular self-calibration method in the flight process proposed in the present application divides the process of solving external parameters into two stages, applies different constraint conditions to solve optimization problems in each stage, and can improve the accuracy and robustness of calibration. Moreover, the entire flight and calibration process does not require personnel operation and has no scene restrictions, which is more flexible and convenient. This method has strong practicality for the unmanned aerial vehicle platform and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of the application, and their

[0023] Figure 1 The overall flow chart of the binocular camera extrinsic parameter self-calibration method in the flight process of the unmanned aerial vehicle according to the embodiments of the application is shown in the figure;

[0024] Figure 2 The initial alignment of the front and rear frame monocular images according to the embodiments of the application is shown in the figure;

[0025] Figure 3 The feature matching of the front and rear frame monocular images according to the embodiments of the application is shown in the figure;

[0026] Figure 4 The feature matching before and after the optimization in the rotation amount self-calibration process of the binocular extrinsic parameter according to the embodiments of the application is shown in the figure;

[0027] Figure 5 The feature matching of the left and right images of the binocular in the translation amount self-calibration process of the binocular extrinsic parameter according to the embodiments of the application is shown in the figure;

[0028] Figure 6 The disparity map comparison before and after the self-calibration according to the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed description is merely exemplary and is intended to provide further description of the application, and is not intended to limit the exemplary embodiments according to the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0030] Embodiment one

[0031] To solve the problems in the prior art, the embodiment provides a binocular camera extrinsic parameter self-calibration method in the flight process of an unmanned aerial vehicle. The method can be applied to the actual flight scene of the unmanned aerial vehicle, and can conveniently and quickly complete the binocular camera extrinsic parameter calibration. First, the following reasonable assumptions are made in the calculation process:

[0032] 1) After the binocular is calibrated by the factory, the intrinsic parameters of the binocular usually do not change with the deformation of the binocular structure, so in the entire calibration process, it is considered that each camera intrinsic parameter (such as focal length, optical center, distortion coefficient, etc.) of the binocular is unchanged;

[0033] 2) The baseline distance of the binocular is difficult to change with the structural deformation, and even if it changes, the error only linearly affects the depth information, and the error of the baseline does not affect the epipolar constraint relationship, so the baseline length of the binocular is considered unchanged in the whole calibration process.

[0034] Under the above assumptions, the binocular online calibration problem is designed as a 5-tof extrinsic parameter calibration problem, in which 3 degrees of freedom are rotation and 2 degrees of freedom are translation. In order to facilitate the calculation of the extrinsic parameters, it is assumed that the left and right images of the binocular have been initially corrected according to the intrinsic and extrinsic parameters before calibration and through the Bouguet method, so in the present application, the calculated extrinsic parameters are relative values, and the relative values of the extrinsic parameters are superimposed with the extrinsic rotation and translation before calibration to obtain the new extrinsic parameters after calibration.

[0035] In the method provided in the embodiment, firstly, the geometric constraint information of two different scenes is considered. When the distance of the scene in the field of view of the camera is far enough, the translation between the two cameras can be ignored relative to the distance value, at this time, the two imaging planes can be considered to be in a homographic relationship, and the extrinsic parameters between the two cameras only exist in the rotation; when the scene in the field of view of the camera is close enough, the extrinsic parameters between the two cameras constitute an essential matrix, at this time, the feature points of the two imaging planes can be associated through the essential matrix, and the translation is solved by means of the constraints of the rotation and the baseline length. Relative to the existing method of constructing an essential matrix to solve the rotation and translation, the solving process is divided into two stages of solving the rotation first and then solving the translation in the embodiment, and different constraint conditions are applied to solve the optimization problem in each stage, which is less likely to fall into local optimization, can effectively improve the accuracy and robustness of the extrinsic parameter calculation, and thus improve the long-distance ranging accuracy and the depth map quality.

[0036] The embodiment provides a binocular camera extrinsic parameter self-calibration method in the flight process of a UAV. By means of image processing and multi-view geometry related technology, the UAV first collects monocular (left eye or right eye, the left eye is used in the embodiment) image sequences at different time points through the binocular camera carried thereon, and estimates the distance of the obstacle in real time through image feature matching; then, according to the distance, two different calibration stages are divided, geometric constraint information of the scene is constructed in the two calibration stages, different residual error equations are constructed through binocular left and right image feature matching pairs, the rotation and translation of the camera extrinsic parameters are calculated respectively, and the self-calibration of the UAV in the flight process is realized. As shown in Figure 1 , the method comprises the following steps:

[0037] Collecting monocular image sequences when the UAV flies along the optical axis direction of the binocular camera to be calibrated in a straight line, and obtaining the flight speed of the UAV;

[0038] The initial feature alignment is performed on the newly collected two adjacent frames of monocular images, the region of interest is extracted in the aligned images according to the flight speed of the unmanned aerial vehicle, the feature matching and denoising are performed in the region of interest, the homography matrix of the two frames of monocular images is calculated, and then the distance between the unmanned aerial vehicle and the obstacle is calculated.

[0039] The distance between the unmanned aerial vehicle and the obstacle is compared with the set threshold value, if the distance is greater than the set threshold value, the feature matching and denoising are performed on the newly collected binocular left and right images, the residual term is constructed, and the rotation amount of the external parameter is calculated; if the distance is less than the set threshold value, the unmanned aerial vehicle is controlled to fly at a reduced speed until the distance is less than or equal to the preset distance, the unmanned aerial vehicle is controlled to hover, and the feature matching and denoising are performed on the newly collected binocular left and right images, the residual term is constructed, and the translation amount of the external parameter is calculated.

[0040] In this embodiment, the motion information and the attitude information of the unmanned aerial vehicle are combined, the distance between the unmanned aerial vehicle and the obstacle along the flight direction is determined by image processing on the monocular sequence images, and according to the distance information, the external parameter estimation is divided into two stages of rotation amount estimation and translation amount estimation. In each stage, the feature matching, the residual construction and the optimization calculation are performed on the binocular left and right images, so as to obtain the rotation amount and the translation amount of the external parameter respectively.

[0041] The binocular camera external parameter self-calibration method for the unmanned aerial vehicle in the flight process proposed in this embodiment is described in more detail through the following contents.

[0042] Step S1, set the flight task of the unmanned aerial vehicle, make the unmanned aerial vehicle fly straight along the optical axis direction of the binocular camera to be calibrated, and in the flight process, real-time collect the left and right monocular image sequences of the binocular camera, and obtain the flight speed of the unmanned aerial vehicle. The time interval between the two adjacent frames of monocular images in the monocular image sequence is automatically set based on the flight speed of the unmanned aerial vehicle.

[0043] Step S2, combine the POS information of the unmanned aerial vehicle, and perform initial feature alignment on the newly collected two adjacent frames of monocular images. In the two frames of monocular images after initial feature alignment, according to the flight speed vector of the unmanned aerial vehicle, the region of interest (ROI) is extracted, the feature matching and denoising are performed in the ROI, the homography matrix of the two frames of monocular images is calculated, and the distance between the unmanned aerial vehicle and the obstacle is calculated.

[0044] Specifically, the distance between the unmanned aerial vehicle and the obstacle in the flight direction is determined by image processing on the monocular sequence images, including the following steps:

[0045] Step S2.1, perform initial feature alignment on the two adjacent frames of monocular images, and align the two frames of monocular images. First, based on the flight speed and the positioning information of the unmanned aerial vehicle, the sampling time interval of the monocular image sequence is determined. Taking the fixed flight distance in the two frames of collection period as the reference, the sampling time interval is:

[0046]

[0047] wherein d is a fixed flight distance in a collection period, and v is a flight speed.

[0048] Then, combined with the unmanned aerial vehicle posture when collecting two frames of monocular images, a homography matrix H1 between the two frames of monocular images is calculated, and the formula is:

[0049]

[0050]

[0051] wherein subscript n is a NED (North East Down) navigation coordinate system, and c is a camera coordinate system, wherein c0 is a camera coordinate system corresponding to a front frame of monocular images, and c1 is a camera coordinate system corresponding to a rear frame of monocular images, both of which are left-eye cameras of binoculars; K l is an internal parameter matrix of the camera (left-eye camera of binoculars); considering that the translation of the camera-vehicle coordinate system parameter can be ignored, the rotation matrix of the camera and the NED navigation coordinate system can be calculated by the rotation matrix of the camera-vehicle coordinate system parameter (obtained by camera-IMU calibration tool, which is a fixed value) and the rotation matrix of the vehicle-NED navigation coordinate system (obtained by calculating the pitch, roll and yaw angles of the unmanned aerial vehicle at the image collection time), that is, (subscript b is a vehicle coordinate system).

[0052] After H1 is calculated, the front frame of monocular images (referred to as front frame image) is subjected to homographic transformation to make it initially aligned with the rear frame of monocular images (referred to as rear frame image), and the formula of the homographic transformation is:

[0053] M′1=H1.M1

[0054] wherein M1 is the original image of the front frame image, and M′1 is the image after the homographic transformation. The schematic diagram before and after alignment is shown in Figure 2 After the homographic alignment, the optical axis direction of the front frame image is consistent with that of the rear frame image.

[0055] Step S2.2, based on the aligned front and rear frames of monocular images, an area scaling coefficient is calculated according to the flight speed vector of the unmanned aerial vehicle.

[0056] Firstly, the flight speed vector V n in the NED navigation coordinate system is mapped into the pixel coordinate system of the rear frame image, and the formula is:

[0057]

[0058] where (u_speed, v_speed) is the flight velocity vector V n where (u_speed, v_speed) is the flight velocity vector V

[0059] Then, in the latter frame image, a region with a proper radius is selected with the mapping point as the center, and the formula is as follows:

[0060]

[0061] where f is the focal length (unit: pixel), θ fov is the field of view angle of the range of interest (may be 10°-20°), and radius is the region radius (unit: pixel).

[0062] Through the above region radius, the region of interest is determined, and then a feature point set is extracted in the region of interest. For the extraction of the feature points and the construction of the descriptors, a feature extraction algorithm such as ORB / SURF algorithm can be used.

[0063] Next, the feature point matching is performed on the front and rear monocular images. In order to improve the matching accuracy, on the one hand, the matching search range is limited on the basis of the angle alignment; on the other hand, a simple loop matching mechanism is designed, that is, the feature points of the rear frame image are matched to the front frame image to determine the feature points of the front frame image, and then the feature points of the front frame image are matched to the rear frame image again to determine the feature points of the rear frame image, and then in the rear frame image, the Euclidean distance between the pixel coordinates of the matched feature points of the two rear frame images is calculated, and if the distance is lower than a threshold, it is considered that the matching is successful.

[0064] Then, based on the plurality of matched points of the front and rear monocular images, a homography matrix is calculated, and the RANSAC (Random Sample Consensus) method is used to filter out the mismatched points. A schematic diagram of the feature matching of the front and rear frame images is shown in Figure 3 In Figure 3 , the feature matching of the front and rear frame images is performed based on the SURF method and denoising, and the feature matching results are superimposed in the rear frame image. The feature points of the rear frame image are identified by hollow circles, and the feature matching pairs with the front frame image are identified by line segments; in addition, the projection points of the velocity vectors are identified by solid circle points, and the regions of interest are represented by rectangular frames.

[0065] Finally, since the front and rear frame images have been initially aligned, after the alignment, the homography matrix of the two frame images is approximately degenerated into a similarity transformation matrix, and therefore, only the determinant of the rotation matrix in the upper left corner of the homography matrix needs to be calculated, and then the area scaling factor S of the front and rear frame images can be obtained.

[0066] Step S2.3, based on the flight distance of the UAV between the two frames of monocular images before and after acquisition and the area scaling factor of the two frames of monocular images, the distance between the UAV and the obstacle is calculated.

[0067] According to the pinhole imaging model, combined with the flight distance d of the UAV between the two frames of monocular images before and after acquisition (obtained based on the UAV positioning information), the following formula is obtained:

[0068]

[0069] Wherein, D is the obstacle distance, and S is the area scaling factor obtained previously.

[0070] Step S3, the distance of the UAV from the obstacle is compared with the set threshold value, and it is determined whether the distance of the UAV from the obstacle is greater than the set threshold value, if it is far enough (i.e. the distance is greater than the set threshold value), the rotation parameter calibration condition is met, at this time the rotation amount of the external parameter is calculated, that is, the feature matching and denoising of the binocular left and right images are performed, and the residual term is constructed to calculate the rotation amount of the external parameter; if the distance is less than the set threshold value and the rotation amount has not been calculated, the flight direction needs to be changed, and the above steps are re-executed to start the calibration again.

[0071] In this embodiment, after obtaining the distance of the UAV from the obstacle, it is determined whether it is greater than the set threshold value (such as the set threshold value is 100m), when it is greater than the set threshold value, it is considered that the UAV is far enough from the obstacle. For the left and right images after binocular correction, the feature point pairs of the left and right images should satisfy the following homography relationship:

[0072]

[0073] Wherein, (u_feature, v_feature) is the pixel coordinate of the feature point, s is the scale factor, K b is the binocular intrinsic matrix after binocular correction (calculated by Bouguet algorithm, the left and right camera intrinsic parameters are consistent); subscript i1 is the image coordinate system of the left camera of the rear frame, c1 is the camera coordinate system of the left camera of the rear frame; i1' is the image coordinate system of the right camera of the rear frame, c1' is the camera coordinate system of the right camera of the rear frame, is the rotation external parameter matrix between the two camera coordinate systems of the binocular.

[0074] In the rotation matrix calculation process of the external parameter, the feature matching and denoising of the newly collected binocular left and right images are performed, the residual term is constructed, and the rotation amount is obtained by continuous iteration optimization, including:

[0075] Firstly, a set of Shi-Tomasi corner points is extracted in the region of interest of the left image (using the Shi-Tomasi corner detection algorithm), and the feature point matching is performed on the left and right images by the LK optical flow algorithm (Lucas-Kanade optical flow algorithm, a two-frame difference optical flow estimation algorithm). Similar to the feature matching processing of the above-mentioned front and rear monocular images, the cyclic matching mechanism is used to improve the matching accuracy.

[0076] Then, for a plurality of matching points of the left and right images, a homography matrix between the left and right images is calculated and filtered and denoised based on the RANSAC method, to obtain a matching inlier set;

[0077] Next, for each pair of matching points in the matching inlier set, a residual term is constructed according to the homographic relationship. In order to construct an unconstrained optimization problem, the rotation amount is represented by using Lie algebra , and the residual expression is:

[0078]

[0079] wherein subscript k is the index of the matching point pair. According to the above residual expression, the optimization solving problem is constructed as:

[0080]

[0081] wherein m is the number of matching point pairs, and ||2 represents the two-norm of a vector.

[0082] Finally, a perturbation is applied to the rotation amount in the Lie algebra manifold space, the derivative of the residual with respect to the rotation amount is constructed, and the rotation amount is iteratively optimized and solved in the Lie algebra manifold space by the Gauss-Newton method. When the residual converges to a sufficiently small value, the calculation is completed.

[0083] Further, since the rotation amount is a relative value, multiplying the initial extrinsic parameter rotation matrix can obtain the self-calibrated extrinsic parameter rotation matrix. The left and right image feature matching diagrams before and after the rotation amount optimization are shown in Figure 4 It can be seen from Figure 4 that the matching features of the left and right images before optimization have obvious distance errors (i.e. re-projection errors), and after optimization, the errors converge to a sufficiently small value, and the feature points basically coincide.

[0084] Step S4, when the calculation of the rotation amount is completed, the unmanned aerial vehicle is controlled to continue flying until the distance is less than a set threshold, at this time the unmanned aerial vehicle is controlled to fly at a reduced speed until the distance is less than or equal to a preset distance, and in the process of flying at a reduced speed, the monocular and binocular obstacle distance calculations are cooperated to ensure the safety of the unmanned aerial vehicle flight; when the unmanned aerial vehicle flies to a distance less than or equal to a preset distance, the unmanned aerial vehicle is controlled to hover and the feature matching and denoising are performed on the latest collected binocular left and right images, a residual term is constructed, and the translation amount of the extrinsic parameter is calculated.

[0085] In the translation amount calculation process of the external parameter, the latest collected binocular left and right images are matched and denoised, a residual term is constructed, and the translation amount of the external parameter is calculated, including the following steps:

[0086] First, the shi-tomasi corner set is extracted in the region of interest of the left eye image, and the feature point matching is performed on the left and right eye images by LK optical flow. Similar to the feature matching process of the above-mentioned front and rear monocular images, the cyclic matching mechanism is adopted to improve the matching accuracy.

[0087] Then, based on the 7-point method or the 8-point method, the fundamental matrix is calculated for a plurality of matching points of the left and right images, and the RANSAC method is used for denoising. By calculating the fundamental matrix and combining RANSAC, the matching inlier set is obtained. The schematic diagram of the feature matching inlier set of the left and right images is shown in FIG. 2. Figure 5

[0088] Next, for each pair of matching points in the inlier set, according to the epipolar constraint relationship, the residual term (or residual equation) is obtained as follows:

[0089]

[0090] Where t x , t y , t z are the components of the external parameter translation to be solved in the x, y, and z directions (based on the left eye camera coordinate system), is the anti-symmetric matrix of the external parameter translation to be solved. Since the binocular image has been binocularly corrected and the external parameter rotation has been calibrated in the first stage, it is considered that the optical axes of the left and right cameras are parallel at this time, so R is a unit matrix. By introducing it into the residual term, the following equation is obtained:

[0091]

[0092] Wherein, is the normalized coordinate of the feature point of the right eye image, is the normalized coordinate of the feature point of the left eye image. Since the translation lacks a true scale, in order to calculate t y , t z , the constraint of the translation t x is the baseline length of the initial calibration, and after expanding, the following equation is obtained:

[0093]

[0094] By n pairs of feature matching points (n≥3), the overdetermined equation is constructed as follows:

[0095] ​

[0096] The above overdetermined equations are in the form of Ax = b, where x is The linear least squares solution is calculated directly by the following formula is:

[0097] x = (A T A) -1 A T b

[0098] The translation obtained here is calculated in the binocular camera coordinate system after correction using the original external parameters, and is an increment relative to the original calibration external parameter translation. Therefore, t y , t z and the original calibration external parameter t y , t z are superimposed to obtain a new external parameter translation, thereby completing the entire self-calibration process. Relative to before calibration, the binocular disparity map quality is significantly improved. An example of an indoor scene is shown in Figure 6 .

[0099] In this embodiment, in order to complete the second stage of calibration calculation, the unmanned aerial vehicle needs to continuously fly close enough to the obstacle. In this stage of unmanned aerial vehicle flight, the flight safety of the unmanned aerial vehicle needs to be ensured, and the specific method includes:

[0100] In this stage, the flight speed of the unmanned aerial vehicle is reduced, and monocular and binocular obstacle perception methods are used to calculate the distance of the obstacle.

[0101] For the monocular obstacle perception method, first, a sequence of monocular front and rear frame images is collected, and the distance information of the obstacle is calculated using the method of step S2 described above. Considering that when the unmanned aerial vehicle is close to the obstacle scene, the complex geometric structure of the scene usually easily leads to homography degradation, therefore, in the process of calculating the homography matrix, the RANSAC algorithm is used for iterative denoising, and the ratio of the number of inliers to outliers is calculated to judge the quality of the homography matrix. When the ratio of the inlier and outlier sets is lower than a threshold, it is considered that the scene does not satisfy the homography, and binocular vision is used to calculate the obstacle; for the binocular obstacle perception method, in the calculation process, the rotation obtained in the first stage and the original external parameter translation are used as the external parameters, the binocular images are corrected based on the Bouguet method, and the SGBM (simi-global matching) method is used to calculate the disparity map; then, in the disparity map, the flight speed vector is mapped to the pixel coordinate system of the disparity map image to determine the ROI range, and the disparity value disparity of each pixel in the ROI is mapped to the depth, and the formula is:

[0102]

[0103] where stereo_ is the baseline length of the binocular, focal is the focal length of the camera.

[0104] After mapping, the histogram of depth values in the ROI is counted, in which the horizontal coordinate is a plurality of depth value intervals (with 1 m as the step interval), and the vertical coordinate is the pixel number distribution of each depth interval. For the set of depth intervals exceeding the threshold pixel number, the minimum value of the depth interval is calculated, and the average value of the upper and lower limits of the depth interval is taken as the distance value of the nearest obstacle.

[0105] For the above two methods, monocular sequence image matching is preferred to calculate obstacle distance information, and binocular matching is used to calculate obstacle distance information in the case of invalid scene homography. Although the translation parameters of the binocular are not calibrated, the generated disparity map is not of the best quality, but the SGBM method has a certain error matching tolerance, and in the case of slow flight speed, the quality of the disparity map is also sufficient to meet the accuracy requirements of short distance ranging by setting appropriate binocular block matching parameters such as window size, confidence, etc.

[0106] Considering that the factory calibration parameters of the unmanned aerial vehicle in the binocular vision system are prone to failure due to long-term vibration and high-low temperature changes, thereby causing errors in binocular matching, for ordinary operators, it is tedious and time-consuming to perform offline binocular calibration again, therefore, under the premise of reasonable assumption of invariable binocular internal parameters and baseline, based on image processing, multi-view geometry, optimization and other knowledge, combined with flight POS information, the feature matching of natural scenes is performed, on the one hand, the distance from the obstacle is calculated, the flight safety is ensured, and the calibration is divided into two stages according to the distance; on the other hand, feature matching is performed on the left and right images in each stage, and different residual error equations are constructed, and the rotation and translation parameters of the external parameters are solved by different methods. Finally, the method proposed in the embodiment can complete the online calibration of the binocular external parameters only by relying on a period of continuous flight.

[0107] The binocular self-calibration method in the flight process proposed in the embodiment divides the process of solving external parameters into two stages, and solves the optimization problem by applying different constraints in each stage, which can improve the accuracy and robustness of the calibration; and in the whole flight and calibration process, there is no need for personnel operation, there is no scene restriction, and it is more flexible and convenient. For the unmanned aerial vehicle platform, this method is practical and has a wide application prospect.

[0108] Embodiment Two

[0109] The embodiment provides a binocular camera external parameter self-calibration system in the flight process of an unmanned aerial vehicle, which comprises:

[0110] The data acquisition module is configured to acquire a monocular image sequence when the UAV flies along a straight line in the direction of the optical axis of the binocular camera to be calibrated, and obtain a flight speed of the UAV.

[0111] The distance calculation module is configured to perform initial feature alignment on the newly-acquired adjacent two frames of monocular images, extract a region of interest in the aligned images according to the flight speed of the UAV, perform feature matching and denoising in the region of interest, calculate a homography matrix of the two frames of monocular images, and further calculate a distance of the UAV from the obstacle.

[0112] The external parameter self-calibration module is configured to compare the distance of the UAV from the obstacle with a set threshold value, perform feature matching and denoising on the newly-acquired left and right images of the binocular camera if the distance is greater than the set threshold value, construct a residual term, and calculate a rotation amount of the external parameter; control the UAV to fly at a reduced speed until the distance is less than or equal to a preset distance, control the UAV to hover, and perform feature matching and denoising on the newly-acquired left and right images of the binocular camera if the distance is less than the set threshold value, construct a residual term, and calculate a translation amount of the external parameter.

[0113] Embodiment Three

[0114] The embodiment provides an electronic device, which comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, and when the computer instructions are run by the processor, the steps in the method for self-calibrating external parameters of a binocular camera in a UAV flight process are completed.

[0115] Embodiment Four

[0116] The embodiment also provides a computer-readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the steps in the method for self-calibrating external parameters of a binocular camera in a UAV flight process are completed.

[0117] The steps involved in the above embodiments two to four correspond to the method embodiment one, and the specific implementation can be referred to the related description part of the embodiment one. The term "computer-readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; and should also be understood as including any medium capable of storing, encoding or carrying an instruction set for execution by a processor and causing the processor to perform any method in the present application.

[0118] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0119] The above only describes the preferred embodiments of the present application, and the specific embodiments of the present application are described in conjunction with the drawings, but are not limited to the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A method for self-calibrating the extrinsic parameters of a binocular camera during UAV flight, characterized in that, include: Acquire monocular image sequences of the UAV while it flies in a straight line along the optical axis of the binocular camera to be calibrated, and obtain the UAV's flight speed; Initial feature alignment is performed on the two consecutive monocular images acquired recently. Based on the flight speed of the UAV, the region of interest is extracted from the aligned image. Feature matching and denoising are performed in the region of interest. The homography matrix of the two monocular images is calculated, and then the distance between the UAV and the obstacle is calculated. The distance between the drone and the obstacle is compared with a set threshold. If the distance is greater than the set threshold, feature matching and denoising are performed on the latest acquired binocular left and right images to construct residual terms and calculate the rotation amount of the extrinsic parameters. If the distance is less than the set threshold, the drone is controlled to decelerate until the distance is less than or equal to the preset distance. The drone is then controlled to hover and feature matching and denoising are performed on the latest acquired binocular left and right images to construct residual terms and calculate the translation amount of the extrinsic parameters. Calculate the rotation of the extrinsic parameters, including: Extract the shi-tomasi corner point set from the region of interest in the left eye image, and perform feature point matching between the left and right eye images using the LK optical flow algorithm; For multiple matching points in the left and right eye images, the homography matrix between the left and right eye images is calculated and filtered and denoised based on the RANSAC method to obtain the set of matching intra-points; For each pair of matching points in the matching inner point set, construct the residual equation based on the homography relation; The rotation is perturbed in the Lie algebra manifold space, and the derivative of the residual with respect to the rotation is constructed. The rotation is solved iteratively in the Lie algebra manifold space using the Gauss-Newton method. When the residual converges to the set value, the rotation of the extrinsic parameter is calculated. Multiply the calculated rotation amount of the extrinsic parameters by the initial extrinsic parameter rotation matrix to obtain the self-calibrated extrinsic parameter rotation matrix; Calculate the translation of the extrinsic parameters, including: Extract the shi-tomasi corner point set from the region of interest in the left eye image, and perform feature point matching between the left and right eye images using LK optical flow; For multiple matching points in the left and right images, the fundamental matrix is ​​calculated based on the 7-point or 8-point method, and then filtered and denoised using the RANSAC method to obtain the set of matching inner points. For each pair of matching points in the matching interior point set, an overdetermined equation is constructed based on the epipolar constraint relationship and using the updated extrinsic rotation matrix as a constraint. For the calculation of extrinsic translation x The constraint of the directional component is the initial calibration baseline length. The translation amount of the extrinsic parameters is obtained by solving the linear least squares solution. The calculated extrinsic parameter translation is superimposed with the initial extrinsic parameter translation to obtain the final extrinsic parameter translation.

2. The method for self-calibrating the extrinsic parameters of a binocular camera during UAV flight as described in claim 1, characterized in that, Initial feature alignment is performed on two consecutive monocular images. Align the two consecutive monocular images, including: Based on the UAV's flight speed and positioning information, the sampling time interval of the monocular image sequence is determined; Based on the sampling time interval, the attitude of the UAV is acquired when two frames of monocular images are collected, and the homography matrix between the two frames of monocular images is calculated. Based on the homography matrix between two monocular images, a homography transformation is performed on the previous monocular image to initially align it with the subsequent monocular image.

3. The method for self-calibrating the extrinsic parameters of a binocular camera during UAV flight as described in claim 1, characterized in that, Calculating the distance between the drone and the obstacle includes: Based on the two consecutive monocular images after initial alignment, the area scaling factor is calculated according to the UAV's flight velocity vector. The distance between the drone and the obstacle is calculated based on the drone's flight distance between two consecutive monocular images and the area scaling factor of the two consecutive monocular images.

4. The method for self-calibrating the extrinsic parameters of a binocular camera during UAV flight as described in claim 3, characterized in that, Calculate the area scaling factor, including: The flight velocity vector in the NED navigation coordinate system is mapped to the pixel coordinate system of the subsequent frame image. In the subsequent frame image, the region of interest is determined with the mapped point as the center, and the feature point set is extracted within the region of interest. Feature point matching is performed on the monocular images of the previous and next frames. Based on multiple matching points of the two monocular images, the homography matrix is ​​calculated. Based on the determinant of the 2*2 rotation matrix at the top left corner of the homography matrix, the area scaling factor of the monocular images of the preceding and following frames is calculated.

5. A binocular camera extrinsic parameter self-calibration system during UAV flight, characterized in that, include: The data acquisition module is used to acquire monocular image sequences when the UAV flies in a straight line along the optical axis of the binocular camera to be calibrated, and to obtain the flight speed of the UAV. The distance calculation module is used to perform initial feature alignment on two consecutive monocular images acquired recently. Based on the flight speed of the UAV, it extracts the region of interest in the aligned image, performs feature matching and denoising in the region of interest, calculates the homography matrix of the two monocular images, and then calculates the distance between the UAV and the obstacle. The extrinsic parameter self-calibration module compares the distance between the UAV and the obstacle with a set threshold. If the distance is greater than the set threshold, feature matching and denoising are performed on the latest acquired binocular left and right images to construct residual terms and calculate the rotation amount of the extrinsic parameters. If the distance is less than the set threshold, the UAV is controlled to decelerate until the distance is less than or equal to the preset distance. The UAV is then controlled to hover and feature matching and denoising are performed on the latest acquired binocular left and right images to construct residual terms and calculate the translation amount of the extrinsic parameters. Calculate the rotation of the extrinsic parameters, including: Extract the shi-tomasi corner point set from the region of interest in the left eye image, and perform feature point matching between the left and right eye images using the LK optical flow algorithm; For multiple matching points in the left and right eye images, the homography matrix between the left and right eye images is calculated and filtered and denoised based on the RANSAC method to obtain the set of matching intra-points; For each pair of matching points in the matching inner point set, construct the residual equation based on the homography relation; The rotation is perturbed in the Lie algebra manifold space, and the derivative of the residual with respect to the rotation is constructed. The rotation is solved iteratively in the Lie algebra manifold space using the Gauss-Newton method. When the residual converges to the set value, the rotation of the extrinsic parameter is calculated. Multiply the calculated rotation amount of the extrinsic parameters by the initial extrinsic parameter rotation matrix to obtain the self-calibrated extrinsic parameter rotation matrix; Calculate the translation of the extrinsic parameters, including: Extract the shi-tomasi corner point set from the region of interest in the left eye image, and perform feature point matching between the left and right eye images using LK optical flow; For multiple matching points in the left and right images, the fundamental matrix is ​​calculated based on the 7-point or 8-point method, and then filtered and denoised using the RANSAC method to obtain the set of matching inner points. For each pair of matching points in the matching interior point set, an overdetermined equation is constructed based on the epipolar constraint relationship and using the updated extrinsic rotation matrix as a constraint. For the calculation of extrinsic translation x The constraint of the directional component is the initial calibration baseline length. The translation amount of the extrinsic parameters is obtained by solving the linear least squares solution. The calculated extrinsic parameter translation is superimposed with the initial extrinsic parameter translation to obtain the final extrinsic parameter translation.

6. The binocular camera extrinsic parameter self-calibration system during UAV flight as described in claim 5, characterized in that, Initial feature alignment is performed on two consecutive monocular images. Align the two consecutive monocular images, including: Based on the UAV's flight speed and positioning information, the sampling time interval of the monocular image sequence is determined; Based on the sampling time interval, the attitude of the UAV is acquired when two frames of monocular images are collected, and the homography matrix between the two frames of monocular images is calculated. Based on the homography matrix between two monocular images, a homography transformation is performed on the previous monocular image to initially align it with the subsequent monocular image.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps of a binocular camera extrinsic parameter self-calibration method during UAV flight as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of a binocular camera extrinsic parameter self-calibration method during UAV flight as described in any one of claims 1-4.

Images