A method for UAV terrain-following flight based on monocular vision

Abstract

This invention discloses a method for UAV terrain-following flight based on monocular vision, specifically including: S3, in non-keyframes, tracking and matching the image region block where the reference coordinates are located, and calculating the odometry information relative to the keyframe by combining prior information including ground altitude data, attitude data, and GNSS altitude; S4, calculating the scale ratio based on the visual odometry information and GNSS odometry data, thereby correcting the ground altitude. This invention, based on monocular vision odometry, sets a prior ground altitude when the actual ground altitude is lacking. It employs a keyframe-based visual odometry calculation method, using observations of the same landmark point in two image frames, combining attitude and altitude data to calculate the visual displacement odometry information between the two frames, and using real GNSS displacement data to correct the ground altitude, thus achieving UAV ground altitude estimation and enabling terrain-following flight of the UAV relying solely on monocular vision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and artificial intelligence, and particularly relates to a method for ground-following flight of a UAV based on monocular vision. BACKGROUND

[0002] During the process of surveying and mapping, inspection and other tasks of a UAV, the UAV may face a scene of undulating terrain such as a hillside, which increases the risk of the task process and may affect the quality of data (such as image coverage rate of a surveying scene) in the task process.

[0003] Generally, the UAV can increase sensor devices for detecting the ground, such as a millimeter wave radar, a laser radar and a binocular camera, to measure the distance to the ground and thus control the UAV to always fly at a certain height above the ground during the task process. However, the above-mentioned sensors have high volume, weight and cost, and have a limited detection distance. In the field of surveying, a monocular camera and SFM technology can be used to realize three-dimensional mapping of a scene and thus obtain point cloud information of the scene and realize ground-following flight. However, the technology usually faces the following difficulties:

[0004] 1) Three-dimensional mapping needs feature extraction and matching calculation operations between multiple images, which consumes a large amount of computing resources and has a high requirement for camera resolution and scene feature texture;

[0005] 2) The monocular SFM method needs a certain camera panning operation to complete initialization, and in order to reduce error accumulation in actual application, initialization may be performed more frequently, which reduces the practicability of the method. SUMMARY

[0006] The application provides a method for ground-following flight of a UAV based on monocular vision, which adopts a visual odometry calculation method based on observation of a single landmark point, and only needs a low-cost and low-resolution monocular camera, combined with data such as UAV attitude, height and GNSS displacement, to realize measurement of the height above the ground without complicated three-dimensional sparse / dense mapping steps, thereby solving the technical difficulties in the above.

[0007] The application adopts the following technical scheme:

[0008] A method for ground-following flight of a UAV based on monocular vision, comprising the following steps:

[0009] S1. In a vision processor, accurately synchronize monocular collected images and attitude data, GNSS height and GNSS coordinate data transmitted by a flight control;

[0010] S2. Adopt a visual odometry calculation method based on a key frame, calculate pixel coordinates of a normal projection point of the UAV in the key frame as a reference coordinate, and collect image features around the reference coordinate.

[0011] S3. In non-key frames, track and match the image region block where the reference coordinates are located, and combine prior information including ground height data, attitude data, and GNSS height to calculate the mileage information relative to the key frame.

[0012] S4. Based on visual mileage information and combined with GNSS mileage data, calculate the scale ratio to correct the ground altitude.

[0013] Preferably, the UAV orthophoto projection point is calculated in the key frame, and the projection point is tracked by feature matching in the non-key frame. The missing scale visual odometer information is calculated by relying on the prior ground altitude information, synchronized attitude and altitude data, and the ground altitude information is corrected by relying on the real GNSS odometer information, so as to control the UAV to maintain a certain ground altitude.

[0014] Preferably, a keyframe-based visual odometry method is used, which calculates the orthogonal projection point of the UAV in the keyframe, performs feature matching and tracking on the projection point in non-keyframe, and calculates visual odometry information at missing scales based on prior ground altitude information, synchronized attitude and altitude data.

[0015] Assuming the camera and the drone are rigidly connected, with their x and y axes overlapping the drone's x and y axes respectively, and ψ, γ, and θ representing the drone's yaw, roll, and pitch angles, respectively. First, in the keyframe, using the attitude information, the pixel coordinates (uv) of the drone's orthographic projection point on the ground are obtained using the following calculation method;

[0016] u = cx + f.tanγ

[0017] v = cy + f.tanθ / cosγ

[0018] Where pitch / roll are the drone's pitch / roll angles, respectively, and cx / cy / f are the camera's intrinsic parameters.

[0019] In keyframes, a 100*100 image block centered at the orthographic projection pixel coordinates is selected as the image feature block to be matched and tracked. Then, in non-keyframes, the image feature block of the keyframe is tracked using a correlation filtering tracking method to obtain its center pixel coordinates (rx, ry), and the displacement relative to the keyframe in the navigation coordinate system is calculated based on the pose data.

[0020]

[0021] In the above calculations, λ is a non-zero scale factor. The ground altitude is determined through keyframe priors. The GNSS altitude change Δh between non-keyframes and keyframes, as a function of X d By eliminating the scale factor through constraints, X can be solved. n and X e :

[0022]

[0023] because The calculated X is based on prior ground altitude information, not the actual ground altitude. n and X e It is not displacement at a real scale.

[0024] Preferred methods for keyframe updates include:

[0025] The following strategy is used to evaluate the current frame. If any of the following conditions are met, the current frame is immediately designated as the new keyframe, i.e., the keyframe is updated:

[0026] 1) The tracking quality of the current frame is assessed by the PSR (Peak-to-Sidelobe Ratio) parameter of the correlation filter. When it falls below a fixed threshold, it means that tracking is about to be lost, and the key frame is updated.

[0027] 2) Disparity of the current frame relative to the previous keyframe. Based on the principle of disparity, we calculate the ratio of the horizontal displacement of the current frame relative to the keyframe to the prior ground height. When it exceeds a certain preset threshold, it means that the feature disparity is too large, and the keyframe is updated at this time.

[0028] 3) The pose of the current frame relative to the previous keyframe. If the pitch or roll angle of the current frame relative to the keyframe exceeds the threshold, it means that the tracked features are about to move out of the field of view and be lost. At this time, the keyframe is updated.

[0029] Methods for correcting ground elevation based on visual odometry information, combined with GNSS odometry data, to calculate scale ratios include:

[0030] Let the obtained visual odometry displacements in the navigation coordinate system be X... n and X e The corresponding GNSS displacements are D n and D e It can be seen that since the two odometers differ by a scale factor s, the values ​​satisfy the following condition:

[0031]

[0032] Can be made by X n and D n By solving for s, the actual ground altitude at the keyframe moment can be obtained as follows:

[0033] h = sXd -Δh

[0034] Technical effects and advantages of the present invention: The ground-following flight method for unmanned aerial vehicles based on monocular vision proposed in this invention has the following advantages compared with the prior art:

[0035] 1. Low computational complexity, low requirements for computing equipment, reducing equipment cost, size and weight;

[0036] 2. Robust and reliable, with low requirements for image resolution and environmental texture features;

[0037] 3. Supports higher flight mission altitudes. Attached Figure Description

[0038] Fig. 1 This is a schematic diagram of the terrain-following flight mission of the present invention;

[0039] Fig. 2 This is a schematic diagram of orthophoto projection points and feature blocks in the keyframe of this invention;

[0040] Fig. 3 This is a schematic diagram of the displacement mileage calculation model for key frames and non-key frames in this invention. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example

[0043] This invention utilizes image processing and video analysis technologies, and uses a ground-oriented monocular camera mounted on a UAV to acquire, analyze, and calculate images of the ground scene. At the same time, it uses attitude and GNSS data provided by the flight controller to calculate the altitude above the ground, ultimately enabling the UAV to fly in a terrain-following manner.

[0044] A method for UAV terrain-following flight based on monocular vision includes the following steps:

[0045] S1. In the vision processor, the monocular acquired images and the attitude data, GNSS altitude, and GNSS coordinate data transmitted from the flight control are precisely synchronized.

[0046] S2. A keyframe-based visual odometry calculation method is adopted. In the keyframe, the pixel coordinates of the UAV orthophoto projection point are calculated as the reference coordinates, and image features around the reference coordinates are collected.

[0047] S3. In non-key frames, track and match the image region block where the reference coordinates are located, and combine prior information including ground height data, attitude data, and GNSS height to calculate the mileage information relative to the key frame.

[0048] S4. Based on visual mileage information and combined with GNSS mileage data, calculate the scale ratio to correct the ground altitude.

[0049] By calculating the orthophoto point of the UAV in keyframes, performing feature matching and tracking on the projection point in non-keyframes, and relying on prior ground altitude information, synchronized attitude and altitude data to calculate visual odometry information at missing scales, and relying on real GNSS odometry information to correct the ground altitude information, the UAV is controlled to maintain a certain ground altitude during flight.

[0050] A keyframe-based visual odometry method is employed, which calculates the orthogonal projection point of the UAV in keyframes, performs feature matching and tracking on the projection point in non-keyframes, and calculates visual odometry information at missing scales based on prior ground altitude information, synchronized attitude and altitude data.

[0051] Assuming the camera and the drone are rigidly connected, with their x and y axes overlapping the drone's x and y axes respectively, and ψ, γ, and θ representing the drone's yaw, roll, and pitch angles, respectively. First, in the keyframe, using the attitude information, the pixel coordinates (uv) of the drone's orthographic projection point on the ground are obtained using the following calculation method;

[0052] u = cx + f.tanγ

[0053] v = cy + f.tanθ / cosγ

[0054] Where pitch / roll are the drone's pitch / roll angles, respectively, and cx / cy / f are the camera's intrinsic parameters.

[0055] In keyframes, a 100*100 image block centered at the orthographic projection pixel coordinates is selected as the image feature block to be matched and tracked. Then, in non-keyframes, the image feature block of the keyframe is tracked using a correlation filtering tracking method to obtain its center pixel coordinates (rx, ry), and the displacement relative to the keyframe in the navigation coordinate system is calculated based on the pose data.

[0056]

[0057] In the above calculations, λ is a non-zero scale factor. The ground altitude is determined through keyframe priors. The GNSS altitude change Δh between non-keyframes and keyframes, as a function of Xd By eliminating the scale factor through constraints, X can be solved. n and X e :

[0058]

[0059] because The calculated X is based on prior ground altitude information, not the actual ground altitude. n and X e It is not displacement at a real scale.

[0060] Keyframe update methods include:

[0061] The following strategy is used to evaluate the current frame. If any of the following conditions are met, the current frame is immediately designated as the new keyframe, i.e., the keyframe is updated:

[0062] 1) The tracking quality of the current frame is determined by the PSR (Peak-to-Side Lobe Ratio) parameter of the correlation filter. When it falls below a fixed threshold, it means that tracking is about to be lost, and the key frame is updated at this time.

[0063] 2) Disparity of the current frame relative to the previous keyframe. Based on the principle of disparity, we calculate the ratio of the horizontal displacement of the current frame relative to the keyframe to the prior ground height. When it exceeds a certain preset threshold, it means that the feature disparity is too large, and the keyframe is updated at this time.

[0064] 3) The pose of the current frame relative to the previous keyframe. If the pitch or roll angle of the current frame relative to the keyframe exceeds the threshold, it means that the tracked features are about to move out of the field of view and be lost. At this time, the keyframe is updated.

[0065] Methods for correcting ground elevation based on visual odometry information, combined with GNSS odometry data, to calculate scale ratios include:

[0066] Let the obtained visual odometry displacements in the navigation coordinate system be X... n and X e The corresponding GNSS displacements are D n and D e It can be seen that since the two odometers differ by a scale factor s, the values ​​satisfy the following condition:

[0067]

[0068] Can be made by X n and D n By solving for s, the actual ground altitude at the keyframe moment can be obtained as follows:

[0069] h = sX d -Δh

[0070] Please see Figs. 1-3 As shown, this invention provides a method for UAV terrain-following flight based on monocular vision, including the following steps:

[0071] First, the attitude data from the inertial navigation system and the image data from the camera are strictly synchronized. For each frame of image, sensor information that is as synchronized as possible needs to be acquired. We have set up an IO pin in the processor, connected to the flight controller and the camera, which is triggered by a level edge to control image capture and acquire synchronized attitude, GNSS position, and altitude data.

[0072] Secondly, the first frame of the system during operation (when the UAV initiates its terrain-following flight mission) is used as the keyframe. Within this keyframe, orthophoto points are used as landmarks. Based on the synchronized attitude data, the pixel coordinates of the orthophoto points in the image are calculated, and a feature block (a positive-direction region centered on the orthophoto point with a side length of 100 pixels) is generated. See [link to relevant documentation]. Fig. 2 ;

[0073] Then, in subsequent non-key frames, the following calculations are performed:

[0074] 1) By relying on the correlation filtering (CF) method, the feature blocks in the key frame are tracked to obtain the pixel coordinates of the landmarks in the current frame;

[0075] 2) Based on the landmark pixel coordinates, synchronized attitude data, and GNSS altitude changes, calculate the relative displacement information between the current frame and the key frame in the navigation coordinate system. A schematic diagram of the geometric model of the current frame versus the key frame is shown below. Fig. 3 1 represents the camera pose during keyframe capture, and the calculated orthophoto point is used as the landmark point; 2 represents non-keyframes, in which the landmark points are tracked and the position of the landmark points under NED is obtained, thereby obtaining the displacement information relative to the keyframe.

[0076] 3) Does the keyframe need to be updated?

[0077] Finally, by combining the relative displacement information from GNSS, the accurate ground altitude information is recovered and corrected. The flight controller then uses this ground altitude information to control the drone's altitude, ensuring it maintains a specific altitude for flight.

[0078] Method principle: For the terrain-following flight function of UAVs, sensors such as lidar, millimeter-wave radar, and vision can be used.

[0079] This invention relies on a low-resolution monocular vision sensor, rather than employing traditional methods like SFM mapping. Instead, it takes a different approach, using visual odometry based on observations of a single landmark. A fast correlation filtering tracking method is used to track landmarks between frames, eliminating the need for landmark feature extraction and matching calculations, thus significantly improving computational efficiency. Simultaneously, visual odometry information, combined with GNSS information, is used to recover and correct ground altitude information. Finally, the flight control system uses this ground altitude to control the UAV's terrain-following flight. This terrain-following flight method uses a monocular camera sensor, which, compared to methods using lidar or millimeter-wave radar sensors, reduces cost, size, and weight, while increasing the operating altitude. Furthermore, the visual odometry method employed in this invention is computationally efficient compared to traditional visual 3D reconstruction methods, has lower requirements for image resolution and environmental texture features, and eliminates the need for complex translation initialization operations, making it a promising solution for UAV platforms.

[0080] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for UAV terrain-following flight based on monocular vision, characterized in that, Includes the following steps: S1. In the vision processor, the monocular acquired images and the attitude data, GNSS altitude, and GNSS coordinate data transmitted from the flight control are precisely synchronized. S2. A keyframe-based visual odometry calculation method is adopted. In the keyframe, the pixel coordinates of the UAV orthophoto projection point are calculated as the reference coordinates, and image features around the reference coordinates are collected. S3. In non-key frames, track and match the image region block where the reference coordinates are located, and calculate the mileage information relative to the key frame based on information including ground altitude data, attitude data, and GNSS altitude. S4. Based on visual mileage information and combined with GNSS mileage data, calculate the scale ratio to correct the ground altitude. Let the obtained visual odometry displacements in the navigation coordinate system be X... n and X e The corresponding GNSS displacements are D n and D e Since the two odometers differ by a scale s, the values ​​satisfy the following condition: Can be made by X n and D n By solving for s, the actual ground altitude at the keyframe moment can be obtained as follows: h = sX d -Δh, where Δh is the GNSS altitude change of a non-key frame relative to a key frame.

2. The method for UAV terrain-following flight based on monocular vision according to claim 1, characterized in that: The orthophoto projection point of the UAV is calculated in the key frame, and feature matching and tracking of the projection point are performed in the non-key frame. The missing scale visual odometry information is calculated based on the prior ground altitude information, synchronized attitude, and GNSS altitude data. The ground altitude information is corrected based on the real GNSS odometry information, thereby controlling the UAV to maintain a certain ground altitude.

3. The method for UAV terrain-following flight based on monocular vision according to claim 2, characterized in that: Visual odometry information at missing scales is calculated by relying on prior ground altitude information, synchronized attitude data, and GNSS altitude data. Specific methods include the following: Assuming the camera and the drone are rigidly connected, and their x and y coordinate axes overlap with the x and y coordinate axes of the drone body, respectively, ψ, γ, and θ are the drone's yaw, roll, and pitch angles, respectively. First, in the keyframe, using the attitude information, the pixel coordinates (u, v) of the orthophoto projection point of the UAV on the ground are obtained by the following calculation method; u = cx + f.tanγ v = cy + f.tanθ / cosγ Where cx / cy / f are camera intrinsic parameters; In keyframes, a 100*100 image block centered at the orthographic projection pixel coordinates is selected as the image feature block to be matched and tracked. Then, in non-keyframes, the image feature block of the keyframe is tracked using a correlation filtering tracking method to obtain its center pixel coordinates (rx, ry), and the displacement relative to the keyframe in the navigation coordinate system is calculated based on the pose data. In the above calculations, λ is a non-zero scaling factor; Ground altitude based on keyframe priors The GNSS altitude change Δh between non-keyframes and keyframes, as a function of X d By eliminating the scale factor through constraints, X can be solved. n and X e : because The calculated X is based on prior ground altitude information, not the actual ground altitude. n and X e It is not displacement at a real scale.

4. The method for UAV terrain-following flight based on monocular vision according to claim 2, characterized in that: Keyframe update methods include: The following strategy is used to evaluate the current frame. If any of the following conditions are met, the current frame is immediately designated as the new keyframe, i.e., the keyframe is updated: 1) The tracking quality of the current frame is determined by the PSR parameter of the correlation filter. When it falls below a fixed threshold, it means that tracking is about to be lost, and the key frame is updated at this time. 2) The disparity of the current frame relative to the previous key frame: Based on the principle of disparity, the ratio of the horizontal displacement of the current frame relative to the key frame to the prior ground height is calculated. When it exceeds a certain preset threshold, it means that the feature disparity is too large, and the key frame is updated at this time. 3) The pose of the current frame relative to the previous keyframe. If the pitch or roll angle of the current frame relative to the keyframe exceeds the threshold, it means that the tracked features are about to move out of the field of view and be lost. At this time, the keyframe is updated.

Images