Aircraft visual odometry scale estimation method based on average scene depth

Abstract

This invention discloses a method for estimating the scale of an aircraft visual odometry system based on average scene depth. This method belongs to the field of aircraft positioning technology and includes the following steps: S1, detecting and matching feature points between an observation image obtained by an aircraft equipped with a visual odometry system and a satellite map, and calculating the aircraft's altitude based on these features; S2, calculating the average scene depth of the visual odometry system, and using the aircraft's altitude and the average scene depth to calculate the scale transformation between the visual odometry system and the real world, thereby implementing the application of visual odometry in aircraft visual positioning. This invention enables the direct application of visual odometry to aircraft positioning and improves the accuracy of aircraft altitude estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft positioning, and more specifically, to an aircraft visual odometry scale estimation method based on average scene depth. Background Art

[0002] When performing a mission, an aircraft (such as a drone) must first locate its position. This positioning task relies on GPS. However, if the GPS signal encounters obstructions or, in military applications, is jammed by the enemy, the aircraft's positioning system may fail. In the event of a temporary GPS failure, the aircraft can continue to locate itself using visual odometry.

[0003] There are currently many open-source visual odometry systems, but these systems cannot estimate the scale of motion trajectories without an inertial measurement unit (IMU). In other words, the relative scale transformation between the estimated trajectory and the actual trajectory is unknown, making it difficult to directly apply visual odometry to aircraft positioning. Furthermore, the estimation accuracy of aircraft altitude is low. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an aircraft visual odometry scale estimation method based on average scene depth, so that the visual odometry can be directly applied to aircraft positioning and improve the accuracy of aircraft flight altitude estimation.

[0005] The object of the present invention is achieved through the following solutions:

[0006] A method for estimating the scale of an aircraft visual odometry based on average scene depth comprises the following steps:

[0007] S1, detects and matches feature points between the observation image obtained by the aircraft equipped with the visual odometry system and the satellite map, and calculates the aircraft altitude based on this;

[0008] S2 calculates the average scene depth of the visual odometry, and calculates the scale transformation between the visual odometry and the real world through the aircraft altitude and the average scene depth, realizing the application of visual odometry in aircraft visual positioning.

[0009] Furthermore, in step S1, the feature point detection and matching includes the following sub-steps:

[0010] S11, approximate the depth z of the feature point in the camera coordinate system of the visual odometry system to the height of the aircraft in the visual odometry system coordinate system, and calculate the average depth of N points in the field of view, where N is a positive integer, called the average scene depth:

[0011]

[0012] S12, matching the feature points of the observed image with the satellite map;

[0013] S13: After obtaining the feature point matching between the observed image and the satellite map, first obtain the distance between a feature point on the satellite map and the optical axis, recorded as X. Then, based on the pixel distance of the matching feature point on the camera observed image and the camera pixel size, obtain the distance between the pixel point and the optical axis, recorded as x. The focal length f of the camera is obtained through calibration. According to the principle of similar triangles, the height h of the camera is calculated according to the following relationship:

[0014]

[0015] Furthermore, step S1 includes the following sub-steps: using multiple feature points and calculating the height h of the camera respectively, and then obtaining an accurate camera height by least squares processing, which is recorded as H.

[0016] Furthermore, in step S2, the average scene depth z is used to represent the height of the aircraft in the visual odometry coordinates, and the actual camera precise height H is calculated using satellite map matching. The scale transformation of the visual odometry relative to the real world is expressed as:

[0017]

[0018] Furthermore, the camera is installed in a direction vertically downward with the optical center.

[0019] Furthermore, the aircraft infers the flight trajectory through the visual odometry system and can also build a three-dimensional point cloud map of the observation scene.

[0020] Furthermore, in step S12, the actual coordinates of the feature point of the selected satellite map are obtained according to its pixel position, and then its distance relative to the camera optical axis is obtained.

[0021] Furthermore, the aircraft includes a drone.

[0022] The beneficial effects of the present invention are:

[0023] In an embodiment of the present invention, a method is provided for estimating the relative scale transformation from visual odometry to the real world in an aircraft visual positioning application, based on the average scene depth of a visual odometry point cloud map and matching of satellite map feature points. This method enables visual odometry to be directly applied to aircraft positioning. In particular, after the GPS of an aircraft fails, in the absence of an IMU, the method provided by the present invention can directly use the visual odometry to continue positioning.

[0024] In an embodiment of the present invention, a method for estimating the flight altitude of an aircraft based on matching of feature points on a satellite map is provided, thereby improving the accuracy of altitude estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 Schematic diagram of feature point matching between an observation image and a satellite map according to an embodiment of the present invention;

[0027] Figure 2 A flowchart of the method steps of an embodiment of the present invention;

[0028] In the figure, 1 is the feature point on the satellite map, 2 is the pixel point projected on the image plane by the feature point on the satellite map, 3 is the optical center of the camera, 4 is the distance from the feature point on the satellite map to the optical center of the camera, 5 is the distance from the pixel point to the optical axis, 6 is the actual height of the camera on the aircraft relative to the ground, and 7 is the focal length of the camera. DETAILED DESCRIPTION

[0029] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0030] The following is based on the attached Figures 1 and 2 , further detailed description of the technical problems solved by the present invention, technical concepts and working process.

[0031] The present invention solves at least two technical problems raised in the background technology. It estimates the scale transformation of the visual odometry to the real world based on the average scene depth, so that the aircraft can continue to use the visual odometry for positioning after the GPS fails and in the absence of an IMU.

[0032] The technical concept of the present invention is as follows: by detecting and matching feature points between aircraft observation images and satellite maps, the actual flight altitude of the aircraft is calculated, and then the average scene depth of the visual odometry is calculated. The scale transformation between the visual odometry and the real world is calculated based on the aircraft altitude and the average scene depth, thereby realizing the application of visual odometry in aircraft visual positioning.

[0033] The working principle and process of the present invention are as follows: When the aircraft is in flight, the camera is installed in a direction vertically downward along the optical center. The aircraft can infer the flight trajectory through the visual odometry system and build a three-dimensional point cloud map of the observation scene. However, the scale of the map built by the visual odometry is unknown relative to the real world. If the aircraft's flight altitude is high enough, the feature points on the ground can be assumed to be on a plane. At this time, the depth z of the feature point in the camera coordinate system in the visual odometry can be approximated as the height of the aircraft in the visual odometry coordinate system. In order to make the height estimation more accurate and robust, the present invention calculates the average depth of all N points in the field of view, which is called the average scene depth:

[0034]

[0035] Then the present invention estimates the actual flight altitude of the aircraft by matching the feature points between the observation image and the satellite map. The altitude calculation principle is as follows: Figure 1 shown. Figure 1 In the figure, 1 represents a feature point on a satellite map. The present invention can obtain its actual coordinates based on its pixel position, and then obtain its distance relative to the optical axis of the camera; 2 represents the pixel point obtained by projecting feature point 1 on the satellite map onto the image plane; 3 represents the optical center of the camera; 4 represents the distance from feature point 1 on the satellite map to the optical center 3 of the camera; 5 represents the distance from pixel point 2 obtained by projecting feature point 1 on the satellite map onto the image plane to the optical axis; 6 represents the actual height of the camera on the aircraft relative to the ground; and 7 represents the focal length of the camera. After obtaining the matching between the camera image and the feature point of the satellite map, the distance between the feature point on the satellite map and the optical axis is first obtained, denoted as X. Then, based on the pixel distance of the matching feature point on the camera observation image and the camera pixel size, the distance between the pixel point and the optical axis is obtained, denoted as x. The focal length f of the camera can be obtained by calibration. According to the principle of similar triangles, the height h of the camera can be obtained according to the following relationship:

[0036]

[0037] The camera height obtained based on only one point is not accurate enough. The present invention uses multiple points and obtains a more accurate camera height by the least square method, which is recorded as H.

[0038] Use average scene depth Represents the height of the aircraft in the visual odometry coordinates. The actual camera height H is calculated using satellite map matching. The scale transformation of the visual odometry relative to the real world is then expressed as:

[0039]

[0040] The technical solution of the present invention is applicable not only to drones, but also to other aircraft, such as spacecraft, etc., which involve scenarios involving estimation of flight altitude and application of visual odometer systems.

[0041] Example 1: Figure 2 As shown, a method for estimating the scale of an aircraft visual odometry based on average scene depth includes the following steps:

[0042] S1, detects and matches feature points between the observation image obtained by the aircraft equipped with the visual odometry system and the satellite map, and calculates the aircraft altitude based on this;

[0043] S2 calculates the average scene depth of the visual odometry, and calculates the scale transformation between the visual odometry and the real world through the aircraft altitude and the average scene depth, realizing the application of visual odometry in aircraft visual positioning.

[0044] Example 2: Based on Example 1, in step S1, the feature point detection and matching includes the following sub-steps:

[0045] S11, approximate the depth z of the feature point in the camera coordinate system of the visual odometry system to the height of the aircraft in the visual odometry system coordinate system, and calculate the average depth of N points in the field of view, where N is a positive integer, called the average scene depth:

[0046]

[0047] S12, matching the feature points of the observed image with the satellite map;

[0048] S13: After obtaining the feature point matching between the observed image and the satellite map, first obtain the distance between a feature point on the satellite map and the optical axis, recorded as X. Then, based on the pixel distance of the matching feature point on the camera observed image and the camera pixel size, obtain the distance between the pixel point and the optical axis, recorded as x. The focal length f of the camera is obtained through calibration. According to the principle of similar triangles, the height h of the camera is calculated according to the following relationship:

[0049]

[0050] Example 3: Based on Example 2, in step S1, the sub-steps are included: using multiple feature points and calculating the height h of the camera respectively, and then obtaining an accurate camera height by least squares processing, recorded as H.

[0051] Example 4: Based on Example 3, in step S2, the average scene depth z is used to represent the height of the aircraft in the visual odometry coordinates, and the actual camera precise height H is calculated using satellite map matching. The scale transformation of the visual odometry relative to the real world is expressed as:

[0052]

[0053] In actual application, the camera is installed in a vertical downward direction along the optical center.

[0054] In actual applications, the aircraft uses the visual odometry system to infer the flight trajectory and at the same time build a three-dimensional point cloud map of the observation scene.

[0055] In actual application, in step S12, the actual coordinates of the feature point selected in the satellite map are obtained according to its pixel position, and then its distance relative to the camera optical axis is obtained.

[0056] In practical applications, the aircraft includes a drone.

[0057] If the functions of the present invention are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, and all or part of the steps of the methods described in each embodiment of the present invention are executed in a computer device (which can be a personal computer, server, or network device, etc.) and corresponding software. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, or an optical disk. The test or actual data in the program implementation is stored in a read-only memory (RAM), a random access memory (RAM), etc.

Claims

1. A method for estimating aircraft visual odometry scale based on average scene depth, characterized in that: Including steps: S1, detects and matches feature points between the observation image obtained by the aircraft equipped with the visual odometry system and the satellite map, and calculates the aircraft altitude based on this; S2 calculates the average scene depth of the visual odometry, and calculates the scale transformation between the visual odometry and the real world through the aircraft altitude and the average scene depth, realizing the application of visual odometry in aircraft visual positioning; In step S1, the feature point detection and matching includes the following sub-steps: S11, the depth of the feature point in the camera coordinate system of the visual odometry system It is approximated as the height of the aircraft in the visual odometry system coordinates, and the average depth of N points in the field of view is calculated, where N is a positive integer, called the average scene depth: S12, matching the feature points of the observed image with the satellite map; S13: After obtaining the feature point matching between the observed image and the satellite map, first obtain the distance between a feature point on the satellite map and the optical axis, recorded as X. Then, based on the pixel distance of the matching feature point on the camera observed image and the camera pixel size, obtain the distance between the pixel point and the optical axis, recorded as x. The focal length f of the camera is obtained through calibration. According to the principle of similar triangles, the height h of the camera is calculated according to the following relationship: 。 2. The aircraft visual odometry scale estimation method based on average scene depth according to claim 1, characterized in that: In step S1, the sub-steps are as follows: using multiple feature points and calculating the height h of the camera respectively, and then obtaining an accurate camera height by least squares processing, which is recorded as H.

3. The aircraft visual odometry scale estimation method based on average scene depth according to claim 2, characterized in that: In step S2, the average scene depth Represents the height of the aircraft in the visual odometry coordinates. The actual camera precise height H is calculated using satellite map matching. The scale transformation of the visual odometry relative to the real world is expressed as: 。 4. The aircraft visual odometry scale estimation method based on average scene depth according to claim 1, characterized in that: The camera is installed with the optical center facing vertically downward.

5. The method for estimating aircraft visual odometry scale at average scene depth according to claim 1, wherein: The aircraft infers the flight trajectory through the visual odometry system and can also build a three-dimensional point cloud map of the observation scene.

6. The method for estimating aircraft visual odometry scale at average scene depth according to claim 1, wherein: In step S12, the actual coordinates of the feature point of the selected satellite map are obtained according to its pixel position, and then its distance relative to the camera optical axis is obtained.

7. The method for estimating aircraft visual odometry scale at average scene depth according to any one of claims 1 to 6, wherein: The aircraft includes a drone.

Images