A Monocular Visual Odometry Method for Unmanned Aerial Vehicles

Abstract

The present invention discloses a method for a drone monocular visual odometer, which includes the following steps: S1. Precisely synchronize the monocular captured images and the attitude data transmitted by the flight control; S2. In the key frames, calculate the pixel coordinates of the orthographic projection points of the drone as the reference coordinates, and collect the image feature blocks around the reference coordinates; S3. In the non-key frames, calculate the pixel coordinates of the current orthographic projection points of the drone, and calculate the pixel distance between the orthographic projection points and the reference coordinates; S4. Use the true distance at the moment of collecting the reference coordinates as the motion mileage; The present invention calculates the pixel distance between the orthographic projection points in the current frame and the orthographic projection points in the reference frame, and restores the true scale according to the synchronized attitude data and the height above the ground, and converts the mileage data into the NED coordinate system, so as to obtain the true mileage data, enabling the drone to more accurately restore the odometer scale in the high-altitude scenario.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing and video analysis, and particularly relates to a method for a monocular visual odometer of an unmanned aerial vehicle (UAV). Background Art

[0002] UAVs are frequently used in border areas for patrol, reconnaissance, etc. After the UAV loses the GNSS signal, theoretically, only the camera can be used as the information acquisition source. By analyzing and processing the video, the estimation of its own positioning position can be realized and provided to the flight control to control the UAV, and finally hovering or even returning under interference conditions can be achieved. In recent years, with the development of visual SLAM technology, some open-source visual odometer solutions have emerged in the academic community. Among them, ORBSLAM is a mainstream visual odometer solution that supports mono and stereo systems. It can calculate the H / F matrix by relying on the matching of feature points, and solve the rotation and translation parameters from it to realize odometry calculation; while VINS is a mainstream visual + inertial odometer solution that relies on the tight coupling of mono + IMU to realize odometry calculation.

[0003] However, in the prior art, there are some problems when applying mono and stereo systems to high-altitude and strong-vibration industrial UAV scenarios:

[0004] 1. For ORBSLAM, it degrades to a monocular system at high altitude, resulting in scale loss and difficulty in initialization;

[0005] 2. VINS requires a tightly coupled visual + IMU system, and its IMU is easily affected by strong vibration, which also leads to difficulty in initialization, and the fused data is prone to divergence or even failure.

[0006] Therefore, we propose a method for a monocular visual odometer of an unmanned aerial vehicle to solve the problems existing in the prior art, so that the UAV can achieve reliable hovering and positioning whether at low altitude or high altitude under interference conditions. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for a monocular visual odometer of an unmanned aerial vehicle to solve the problems in the prior art that for ORBSLAM, it degrades to a monocular system at high altitude, resulting in scale loss and difficulty in initialization, and that VINS requires a tightly coupled visual + IMU system, and its IMU is easily affected by strong vibration, which also leads to difficulty in initialization, and the fused data is prone to divergence or even failure.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] A method for a monocular visual odometer of an unmanned aerial vehicle includes the following steps:

[0010] S1. Precisely synchronize the monocular captured image and the attitude data transmitted by the flight control system.

[0011] S2. In the key frame, calculate the pixel coordinates of the orthographic projection point of the UAV as the reference coordinates, and collect the image feature blocks around the reference coordinates.

[0012] S3. In the non-key frame, calculate the pixel coordinates of the current orthographic projection point of the UAV, simultaneously track and match the image region block where the reference coordinates are located, and calculate the pixel distance between the orthographic projection point and the reference coordinates.

[0013] S4. According to the attitude, ground height data, and geometric model, calculate the real distance of the UAV relative to the moment when the reference coordinates are collected as the motion mileage.

[0014] Preferably, the prerequisite for precisely synchronizing the monocular captured image and the attitude data transmitted by the flight control system in step 1 is that when there is GNSS interference, the heading angle data of the UAV is not affected. The heading angle data is collected by the magnetic compass sensor, and the area directly below the UAV is a local plane.

[0015] Preferably, when precisely synchronizing the monocular captured image and the attitude data transmitted by the flight control system in step 1, the UAV relies on the heading data of the magnetic compass, while the vision system only provides the horizontal displacement of the UAV. During the calculation of the displacement, by calculating the pixel distance between the current frame orthographic projection point and the orthographic projection point in the reference frame, and according to the synchronized attitude data and ground height, the real scale is restored, and the mileage data is converted to the NED coordinate system to obtain the real mileage data.

[0016] Preferably, in steps 2 and 3, by calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV, and with the help of the ground height data and synchronized attitude data, the real motion distance is restored to estimate the motion mileage of the UAV.

[0017] Preferably, the method for calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV includes:

[0018] Correct each frame of the image. For the corrected image (meeting the pinhole imaging model), combined with the camera internal parameters (cx, cy) and pitch, roll angles, calculate the pixel coordinates of its orthographic projection point according to the following formula:

[0019] px = cx(1 + tan(roll))

[0020] py = cy(1 + tan(pitch))

[0021] In the key frame, select an image block centered at the orthographic projection point with a size of 100*100 as the subsequent image feature block to be matched.

[0022] In the non-key frame, use the tracking method of correlation filtering to track the image feature block of the key frame, obtain its central pixel coordinates (rx, ry), and calculate the differences rx - px and ry - py between the x-y two directions and the orthographic projection point respectively, which are the pixel coordinate distances in the x-y two directions.

[0023] Preferably, the method for obtaining the ground height data in step 4 is to rigidly connect a ground laser module to the bottom of the UAV fuselage. This module sends the ground ranging value to the UAV at a frequency of 1 Hz, and the UAV flight control combines the real-time attitude to compensate this ranging value and calculate the ground height.

[0024] Preferably, the method for obtaining the attitude data in step 4 is that the embedded processing platform generates a photo-taking trigger signal, which is simultaneously connected to the photo-taking trigger pin of the camera and the interrupt trigger pin of the inertial navigation module. According to the chip manual of the camera CMOS, determine the trigger-exposure delay time t1; and in the inertial navigation module, perform higher-frequency linear interpolation on the inertial navigation data at 200 Hz, and determine the output attitude data according to the t1 value; the theoretical synchronization error of this method can reach 1 ms.

[0025] Preferably, the estimation of the motion mileage in step 4 is to recover the real motion distance by means of the ground height data and the synchronized attitude data.

[0026] Preferably, the method for estimating the motion mileage includes:

[0027] A1. In the first frame after the system runs, as the key frame, initialize and clear the odometer.

[0028] A2. In the non-key frame, through the values of rx - px and ry - py and the pitch and roll angles, deduce according to the camera model and geometric model, and finally simplify it into the following formula to calculate its real displacement relative to the key frame:

[0029]

[0030]

[0031] Among them, when rx - px or ry - py exceeds a certain threshold, update the key frame, accumulate the mileage, and recalculate the reference coordinates and collect the image feature blocks around the reference coordinates.

[0032] A method for a UAV monocular vision odometer proposed by the present invention has the following advantages compared with the prior art:

[0033] 1. The present invention calculates the pixel distance between the orthographic projection point of the current frame and the orthographic projection point in the reference frame, restores the true scale according to the synchronized attitude data and the height above the ground, and converts the odometry data into the NED coordinate system to obtain the true odometry data, so that the drone can more accurately restore the odometer scale in the high-altitude scenario;

[0034] 2. The present invention calculates the pixel coordinate distance between the current orthographic projection point of the drone and the reference projection point, and restores the true movement distance by means of the height above the ground data and the synchronized attitude data. By directly obtaining the synchronized data provided by the height above the ground data and the synchronized attitude data, the influence of vibration is avoided, and the fused data is stable and will not diverge and fail. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow block diagram of the present invention;

[0036] Figure 2 is a timing diagram of the present invention;

[0037] Figure 3 is a geometric model diagram of the present invention taking the pitch direction as an example. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] The present invention provides a monocular vision odometry method for drones, including the following steps:

[0040] S1. Precisely synchronize the monocular captured images and the attitude data transmitted by the flight control;

[0041] S2. In the key frame, calculate the pixel coordinates of the orthographic projection point of the drone as the reference coordinates, and collect the image feature blocks around the reference coordinates;

[0042] S3. In the non-key frame, calculate the pixel coordinates of the current orthographic projection point of the drone, simultaneously track and match the image region block where the reference coordinates are located, and calculate the pixel distance between the orthographic projection point and the reference coordinates;

[0043] S4. According to the attitude, height above the ground data, and geometric model, calculate the true distance of the drone relative to the moment when the reference coordinates are collected as the motion mileage;

[0044] Among them, the prerequisite for the precise synchronization of the monocular captured image and the attitude data transmitted by the flight controller in step 1 is that when there is GNSS interference, the heading angle data of the UAV is not affected. The heading angle data is collected by a magnetic compass sensor, and the area directly below the UAV is a local plane. The precise synchronization is to strictly synchronize the attitude data of the inertial navigation and the image data of the camera;

[0045] Among them, when the monocular captured image and the attitude data transmitted by the flight controller are precisely synchronized in step 1, the UAV relies on the heading data of the magnetic compass, and the vision system only provides the displacement of the UAV in the horizontal direction. During the calculation of the displacement, the pixel distance between the orthographic projection point of the current frame and the orthographic projection point in the reference frame is calculated, and the true scale is restored according to the synchronized attitude data and the height above the ground, and the odometry data is converted into the NED coordinate system to obtain the true odometry data;

[0046] Among them, in steps 2 and 3, the true moving distance is restored by calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV, and by means of the height above the ground data and the synchronized attitude data, so as to estimate the moving odometry of the UAV;

[0047] Among them, the method for calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV includes:

[0048] Each frame of image is corrected. For the corrected image (satisfying the pinhole imaging model), combined with the internal parameters of the camera (cx, cy) and the pitch and roll angles, the pixel coordinates of its orthographic projection point are calculated according to the following formula:

[0049] px = cx(1 + tan(roll))

[0050] py = cy(1 + tan(pitch))

[0051] In the key frame, an image block with a size of 100*100 centered on the orthographic projection point is selected as the subsequent image feature block to be matched;

[0052] In the non-key frame, through the tracking method of correlation filtering, the image feature block of the key frame is tracked to obtain its central pixel coordinates (rx, ry), and the differences rx - px and ry - py in the x-y two directions from the orthographic projection point are calculated respectively, which are the pixel coordinate distances in the x-y two directions;

[0053] The specific calculation method is described as follows: The first frame when the system is working (the system is turned on when the UAV detects interference) is used as the key frame; in the key frame, according to the synchronized pitch and roll angles, calculate the position of the orthographic projection point and the subsequent feature blocks to be matched and tracked. Then, in the subsequent non-key frames, perform the following calculations:

[0054] 1) According to the synchronized pitch and roll angles, calculate the position of the orthographic projection point;

[0055] 2) Rely on the correlation filtering (CF) method to track the feature blocks in the key frame and calculate the pixel distance between the current orthographic projection point and the orthographic projection point in the key frame;

[0056] 3) According to the internal parameters of the camera, the synchronized attitude data, and the height above the ground, calculate the real distance between the current orthographic projection point and the orthographic projection point in the key frame. The geometric schematic diagram is shown in Figure 3 , where ab is the imaging plane and the dotted line is the auxiliary line;

[0057] 4) According to the pixel distance obtained in 2), judge whether it is necessary to update the key frame;

[0058] For the odometry data, convert it to the odometry in the NED coordinate system according to the heading angle, and perform filtering and denoising through the one-dimensional Kalman filtering method;

[0059] Among them, the method for obtaining the height above the ground data in step 4 is to rigidly connect a ground laser module to the bottom of the UAV fuselage. This module sends the ground ranging value to the UAV at a frequency of 1 Hz, and the UAV flight control combines the real-time attitude to compensate this ranging value and calculate the height above the ground;

[0060] Among them, the method for obtaining the attitude data in step 4 is that the embedded processing platform generates a photo shooting trigger signal, which is simultaneously connected to the photo shooting trigger pin of the camera and the interrupt trigger pin of the inertial navigation module. According to the chip manual of the camera CMOS, determine the trigger-exposure delay time t1; and in the inertial navigation module, perform higher-frequency linear interpolation (1000 Hz) on the inertial navigation data of 200 Hz, and determine the output attitude data according to the t1 value. The timing diagram is as shown in Figure 2 shown, and the theoretical synchronization error of this method can reach 1 ms;

[0061] Among them, the estimation of the motion odometry in step 4 is to recover the real motion distance by means of the height above the ground data and the synchronized attitude data;

[0062] Among them, the methods for estimating the motion odometry include:

[0063] A1. In the first frame after the system runs, use it as the key frame and initialize and clear the odometer;

[0064] A2. In non-key frames, based on the values of rx - px and ry - py, as well as the pitch and roll angles, deductions are made according to the camera model and geometric model, and finally simplified into the following formula to calculate its true displacement relative to the key frame:

[0065]

[0066]

[0067] Among them, when rx - px or ry - py exceeds a certain threshold, the key frame is updated, the mileage is accumulated, and the reference coordinates are recalculated, and the image feature blocks around the reference coordinates are collected;

[0068] The specific method of deduction according to the camera model and geometric model is as follows:

[0069] Taking the pitch direction as an example, according to Figure 3 the geometric model, similar triangles Δoab~ΔoAB are constructed, and the following equation can be obtained:

[0070]

[0071] Substitute dx = rx - px and dy = ry - py, and finally obtain the following formula:

[0072]

[0073]

[0074] Among them, γ pitch and γ roll are the influence coefficients of the pitch angle and roll angle on Dx and Dy respectively, and their values are:

[0075]

[0076]

[0077] Working principle: During calculation, by calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV, and with the help of ground height data and synchronized attitude data, the real motion distance is restored, so as to estimate the motion mileage of the UAV, and thus realize the reliable hovering and positioning of the UAV under the state of GNSS signal interference; when the GNSS of the UAV is interfered, visual displacement calculation is carried out. During the displacement calculation, assuming that the heading angle is not affected (in the actual scenario, the heading angle is extremely difficult to be interfered), the orthographic projection point is calculated relying on vision, strictly synchronized attitude data and ground height data, and at the same time, the orthographic projection point in the previous key frame is tracked, the real displacement between the two orthographic projection points is calculated, and the data is filtered and smoothed by means of the one-dimensional Kalman filtering method. Finally, the flight control relies on the displacement and the speed value obtained by its differentiation to control the UAV to hover, so that the UAV can more accurately restore the odometer scale in the high-altitude scenario;

[0078] By calculating the pixel coordinate distance between the current orthographic projection point and the reference projection point of the UAV, and with the help of ground height data and synchronized attitude data to restore the real motion distance, the synchronized data provided by directly obtaining the ground height data and synchronized attitude data is used, avoiding the influence of vibration, and the fused data is stable and will not diverge and fail.

[0079] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for a single - vision odometer of an unmanned aerial vehicle, characterized in that: it includes the following steps: Precisely synchronize the monocular - acquired images and the attitude data transmitted by the flight control. The prerequisite for precisely synchronizing the monocular - acquired images and the attitude data transmitted by the flight control is that when there is GNSS interference, the heading - angle data of the unmanned aerial vehicle is not affected. The heading - angle data is collected by a magnetic - compass sensor, and the area directly below the unmanned aerial vehicle is a local plane; In key frames, calculate the pixel coordinates of the orthographic - projection point of the unmanned aerial vehicle as the reference coordinates, and collect the image - feature blocks around the reference coordinates. In non - key frames, calculate the pixel coordinates of the current orthographic - projection point of the unmanned aerial vehicle, and at the same time, track and match the image - area block where the reference coordinates are located, and calculate the pixel distance between the orthographic - projection point and the reference coordinates. Among them, the method for calculating the pixel distance between the orthographic - projection point and the reference coordinates includes: correct each frame of the image. For the corrected image that satisfies the pinhole - imaging model, combined with the camera internal parameters (cx, cy) and the pitch and roll angles, calculate the pixel coordinates of its orthographic - projection point according to the following formula: px = cx(1 + tan(roll)) py = cy(1 + tan(pitch)) In key frames, select an image block centered on the orthographic - projection point with a size of 100*100 as the subsequent image - feature block to be matched. In non - key frames, use the tracking method of correlation filtering to track the image - feature block of the key frame, obtain its central - pixel coordinates (rx, ry), and calculate the differences rx - px and ry - py in the x - y directions between the central - pixel coordinates and the orthographic - projection point respectively, which are the pixel - coordinate distances in the x - y directions; According to the attitude, ground - to - height data, and geometric model, calculate the real - distance of the unmanned aerial vehicle relative to the moment when the reference coordinates are collected as the motion odometer; The method for estimating the motion odometer includes: in the first frame after the system runs, as the key frame, initialize and clear the odometer. In non - key frames, through the values of rx - px and ry - py and the pitch and roll angles, deduce according to the camera model and geometric model, and finally simplify it to the following formula to calculate its real displacement relative to the key frame: When rx - px or ry - py exceeds a certain threshold, the key frame is updated, the mileage is accumulated, and the reference coordinates are recalculated, and the image feature blocks around the reference coordinates are collected; where γ p i tc h and γ roll are the influence coefficients of the pitch angle and the roll angle on Dx and Dy respectively, and their values are:

2. The method for a single - vision odometer of an unmanned aerial vehicle according to claim 1, characterized in that: When precisely synchronizing the monocular - acquired images and the attitude data transmitted by the flight control, the unmanned aerial vehicle relies on the heading data of the magnetic compass, and the vision system only provides the displacement of the unmanned aerial vehicle in the horizontal direction. During the calculation of the displacement, calculate the pixel distance between the current - frame orthographic - projection point and the orthographic - projection point in the reference frame, and restore the real scale according to the synchronized attitude data and ground - to - height, and convert the odometer data to the NED coordinate system to obtain the real odometer data.

3. The method for a single - vision odometer of an unmanned aerial vehicle according to claim 2, characterized in that: The method for obtaining the ground height data is to rigidly connect a ground laser module to the bottom of the UAV fuselage. This module sends the ground ranging value to the UAV at a frequency of 1 Hz. The UAV flight controller combines the real-time attitude and compensates the ranging value to calculate the ground height.

4. According to the method of a UAV monocular visual odometer described in claim 3, it is characterized in that: The method for obtaining the attitude data is that the embedded processing platform generates a photo-taking trigger signal, which is simultaneously connected to the photo-taking trigger pin of the camera and the interrupt trigger pin of the inertial navigation module. According to the chip manual of the camera CMOS, the trigger-exposure delay time t1 is determined; And in the inertial navigation module, linear interpolation of the inertial navigation data at 200 Hz is performed at a higher frequency, and the output attitude data is determined according to the t1 value; the theoretical synchronization error of this method can reach 1 ms.

Images