A method for depth estimation of narrow-baseline stereo images of a drone

Abstract

The application is a kind of unmanned aerial vehicle narrow baseline stereo image depth estimation method, belongs to the field of computer stereo vision. When the unmanned aerial vehicle flies high enough, the local area of the continuous two frames satisfies the translation transformation relationship, and the depth information can be directly solved by the phase correlation method, but when the unmanned aerial vehicle does not fly high enough or the unmanned aerial vehicle lens is not parallel to the terrain, the local area of the image no longer satisfies the translation transformation relationship. The application proposes a more general depth estimation algorithm, which does not limit the flight height of the unmanned aerial vehicle, based on the fact that the local area of the continuous two frames of the unmanned aerial vehicle satisfies the affine transformation relationship, the depth of the local area of the continuous two frames can be approximated as a plane, and the normal vector of the plane can be calculated by calculating the affine transformation parameter information, and then the depth information of the image is solved.

Description

A Depth Estimation Method for Narrow Baseline Stereo Images from UAVs Technical Field

[0001] This invention designs a depth estimation method for narrow baseline stereo images of UAVs. This method is a depth estimation method based on calculating affine transformation parameters. Background Technology

[0002] With a wide baseline, there are large occlusion areas and significant geometric distortion between images. Furthermore, the long interval between two imaging processes allows for significant changes in illumination conditions. With a narrow baseline, the occlusion area between images is smaller, and geometric distortion is limited. A very narrow baseline minimizes occlusion, thus making each image view geometrically more similar. Obtaining depth values ​​from stereo images is easier with narrow-baseline stereo configurations than with wide-baseline stereo configurations, but requires very precise sub-pixel measurements. This invention proposes a high-precision depth estimation method suitable for narrow-baseline stereo images from UAVs.

[0003] Although depth estimation methods have been extensively studied, the accuracy of depth estimation can still be improved. Many existing methods follow the typical process of patch-matching stereo images. These methods initialize a random plane for each pixel of two images and consider how to find the optimal patch match for each pixel with the minimum matching cost. However, this invention does not find the best-fitting 3D plane by calculating the matching cost of each pixel, but directly calculates the parameter information of the plane. Summary of the Invention

[0004] When a drone flies at high altitude, the depth of a local region in two consecutive frames is approximately the same. Therefore, the local regions in two consecutive frames satisfy a translation transformation, and the depth information can be directly calculated using a phase correlation algorithm. However, when the drone's flight altitude is insufficient or the drone is not parallel to the terrain, the depth of the local regions in two consecutive frames cannot be considered approximately the same. In other words, the algorithm for solving depth information by calculating the translation vector is no longer applicable. When the drone's altitude is insufficient or the drone is not parallel to the terrain, the local regions in two consecutive frames satisfy an affine transformation relationship. This invention proposes a more general depth estimation algorithm.

[0005] A novel depth estimation method for narrow baseline stereo images from UAVs is proposed. This method is based on the following assumptions: the depth of a local region in two consecutive frames of UAV data is approximately a plane, and the local region satisfies an affine transformation relationship. Therefore, the normal vector information of this plane can be calculated by solving the affine transformation parameters, and thus the depth information can be estimated. Detailed Implementation

[0006] The specific content of this invention can be divided into five steps. The affine transformation parameters are decomposed, and the parameters of the affine transformation satisfied by local regions of two consecutive frames are expressed as follows:

[0007]

[0008] in and These are the coordinates of image point 1 and image point 2, respectively. These are the camera center coordinates. It is the normal vector of the depth approximate plane, and its value is unknown and needs to be solved. It's the camera's focal length. It is the offset of the principal point.

[0009] To calculate the linear components of the affine parameters, this invention employs the Fourier moment matching algorithm implemented by Su et al. [Su HR, Lai SH. Non-rigid registration of images with geometric and photometric deformation by using local affine Fourier-moment matching[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 2874-2882.], which is used to calculate the linear part of the affine transformation parameters. The specific steps are as follows:

[0010] image and images satisfy Relationship, Fourier transform express, Fourier transform It indicates that the Fourier spectrum The Step moment is , If the value is 1 or 2, considering the linear components of the affine parameter, the first moment of the Fourier spectrum can be expressed as: ,in The second moment of the Fourier spectrum can be expressed as: ,in affine parameters It can be determined by the total constraint error It was estimated that, Indicates the order of the moment. These are the weights related to the Fourier moment constraints.

[0011] To calculate the translation amount, this invention uses the phase correlation algorithm implemented by Wan et al. [Wan X, Liu JG, LiS, et al. Phase correlation decomposition: The impact of illumination variation for robust subpixel remotely sensed image matching[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9): 6710-6725.] to calculate the translation vector. The specific steps are as follows:

[0012] After obtaining the linear part of the affine transformation parameters using the Fourier moment algorithm, the transformed image can be obtained. Its Fourier spectrum is , and The normalized cross power spectrum is , Indicates complex conjugation. This represents the phase correlation matrix affected only by illumination. This represents the phase correlation matrix that represents only image translation. This represents the phase correlation matrix between two images, including the difference in illumination and image displacement. The translation is calculated using the following formula: ,in It is the result function of the fitting. It is the root mean square error. It is the least squares fitting function. and These are the fitting variables for the main singular vectors. We begin by using 30% of the points from the dominant singular vector data in the initial fit, and then gradually increase the number of points in the fit. The formula for increasing the number of points in the fit is as follows: ,in The value is set to 0.1. The fitting ends when the root mean square error exceeds the given threshold of 0.1.

[0013] Calculate the normal vector of the local region's depth approximation plane. Combining the values ​​obtained from the Fourier moment algorithm and the phase correlation algorithm, the normal vector is obtained using the following formula:

[0014]

[0015] Depth estimation is achieved by combining the following formulas to obtain the depth value:

[0016]

[0017] in Image points The corresponding three-dimensional points, For depth value, It is a camera intrinsic parameter. Given that a 3D point lies on a plane with approximate depth in a local region, the depth value is calculated by combining the coordinates of the 3D point and the normal vector information.

Claims

1. A depth estimation method for narrow baseline stereo images from unmanned aerial vehicles, characterized in that... Based on the fact that two consecutive frames of local regions from the UAV satisfy an affine transformation relationship, the depth of these two consecutive frames of local regions approximates a plane. By calculating the affine transformation parameters of the two consecutive frames of local regions, the normal vector information of this plane is obtained, and thus the image depth information is obtained. The specific steps for depth estimation are as follows: 1) The parameters of the affine transformation satisfied by the two consecutive frames of local regions are expressed as follows: in and These are the coordinates of image point 1 and image point 2, respectively. These are the camera center coordinates. It is the normal vector of the approximate depth plane of the local region, and its value is unknown and needs to be solved. It's the camera's focal length. 1) The offset of the principal point; 2) Solve for the linear part of the affine transformation parameters using the Fourier moment matching algorithm. 3) Calculate the translation vector in the affine transformation parameters using the phase correlation algorithm. 4) Combining the values ​​obtained in steps 2 and 3, the normal vector information of the approximate plane of the local region is obtained using the following formula: 5) Combining the normal vector information of the approximate plane in the local region, the depth value is obtained using the following formula: in Image points The corresponding three-dimensional points, For depth value, It is a camera intrinsic parameter. Given that a 3D point lies on a plane with approximate depth in a local region, the depth value is calculated by combining the coordinates of the 3D point and the normal vector information.

Images