A method for acquiring elevation of feature points of remote sensing images based on multiple sensors
Through multi-sensor combinations with SLAM technology and barometer, the problem of lack of elevation information in remote sensing maps is solved, and the conversion of three-dimensional information of drone position and flight scenes is realized and the acquisition of elevation information is obtained, which has enhanced the application value of remote sensing maps.
Patent Information
- Application Number
- CN202111087831.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-16
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-09-16
AI Technical Summary
Remote sensing maps lack elevation information and cannot directly add the three-dimensional information in the drone SLAM map to the remote sensing map, which limits its application value.
Multi-sensor combined with SLAM technology is used to generate a three-dimensional sparse point cloud map of drone position and flight scene using airborne cameras and IMU information, and restore the elevation information of feature points in the remote sensing image through a barometer to realize coordinate conversion and elevation information. Getting.
It adds elevation information to remote sensing maps, provides more reference information, and enhances its use value and application potential.
Smart Images

Figure CN113807435B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of autonomous navigation and remote sensing images, and in particular to a method for acquiring the elevation of remote sensing image feature points based on multiple sensors. Background Art
[0002] Remote sensing maps are visual maps in which the corresponding latitude and longitude information of each pixel is known, which can provide a certain reference for users. However, remote sensing maps lack elevation information of each point and cannot provide reference for users' further judgment and use. Therefore, after using multiple sensors to add elevation information to feature points in the map, remote sensing maps will contain more information and provide more valuable reference.
[0003] The multi-sensor system includes an onboard camera, an IMU (inertial sensor) and an altimeter. By fusing the information from the onboard camera and the IMU using the SLAM method, a SLAM map containing a three-dimensional sparse point cloud of the drone's position and flight scene can be generated. However, there is a lack of corresponding coordinate conversion between the SLAM map and the remote sensing map, and the three-dimensional information cannot be directly added to the remote sensing map. Summary of the invention
[0004] In order to solve the above problems, the present invention provides a method for acquiring the elevation of feature points of remote sensing images based on multiple sensors. The method is based on the image information of the airborne camera and the IMU information, realizes the autonomous positioning of the UAV and the reconstruction of the sparse point cloud map of the UAV flight scene through the UAV SLAM technology, and uses the barometric altimeter and the geometric relationship in the map to restore the elevation information of the feature points in the remote sensing image, so as to provide more reference information for the users of remote sensing maps:
[0005] To achieve the above purpose, the technical solution adopted by the present invention is:
[0006] A method for acquiring elevation of feature points of remote sensing images based on multiple sensors comprises the following steps:
[0007] Step 1: load the remote sensing image and calculate the location information and descriptor information of the feature points in the remote sensing map, set the UAV flight area according to the shooting range of the remote sensing image, and control the UAV to take off and cruise in the area;
[0008] Step 2: Use the SLAM method of the airborne camera and inertial sensor combination to estimate the posture of the UAV, and restore the feature points in the camera image to three-dimensional points to establish a SLAM map containing the UAV posture and scene three-dimensional point cloud information;
[0009] Step 3, obtain the altitude information of the drone at each moment through the barometric altimeter, use the timestamp information to obtain the actual altitude of each position of the drone in the SLAM coordinate system, perform feature matching on the camera image and the remote sensing image, and obtain the coordinates of the matching points in the SLAM coordinate system;
[0010] Step 4: According to the information of the SLAM coordinate system, the vector from the drone to the feature point in the SLAM coordinate system is obtained, and then the projection length of the vector in the vertical direction is obtained. The projection length is subtracted from the drone height to obtain the elevation information of the feature point.
[0011] Furthermore, the specific method of step 2 is:
[0012] Set the origin and direction of the drone SLAM coordinate system, determine the position transformation relationship between the camera, inertial sensor and drone, and unify the postures of the three as the drone posture;
[0013] Perform feature detection on the camera image sequence to obtain the location information and descriptor information of the feature points, and obtain the location of the same feature point in different camera images by feature tracking;
[0014] The pose transformation between different camera images is calculated by multi-view geometry method, and the feature points in the camera image are restored into a three-dimensional point cloud by triangulation method;
[0015] The UAV posture and 3D point cloud coordinates are optimized using the bundle adjustment method;
[0016] According to the calculated UAV posture information and the data output by the inertial sensor, the parameters of the inertial sensor are calculated and optimized, and the pre-integrated quantity of the inertial sensor is calculated;
[0017] Fuse visual and inertial sensor information to build a UAV SLAM map with scale information.
[0018] Furthermore, the specific method of step 3 is:
[0019] The altitude information of the drone is output in real time through the barometric altimeter, and the position of the drone at each moment is output through the SLAM method. The two data are aligned according to the timestamp information to obtain the actual altitude of the drone in the SLAM coordinate system;
[0020] According to the descriptor information of the feature points, the feature points of the camera image and the feature points of the remote sensing image are matched to establish the feature matching point pair relationship;
[0021] According to the mapping relationship between the camera image feature points and the three-dimensional points in the SLAM coordinate system, the three-dimensional coordinate points corresponding to the feature points in the remote sensing image in the SLAM coordinate system are found.
[0022] Furthermore, the specific method of step 4 is:
[0023] Using the information of the SLAM coordinate system, the vector from the drone to the feature point is calculated;
[0024] According to the definition of the SLAM coordinate system, the projection distance of the vector in the vertical direction is obtained;
[0025] Subtract the projection distance from the UAV cruising altitude to obtain the elevation information of the feature points in the remote sensing image;
[0026] When a feature point in the remote sensing map matches feature points in multiple camera images, the elevations are calculated and averaged respectively, and the average is used as the elevation information of the feature point in the remote sensing image;
[0027] Add elevation information to the information of feature point locations on remote sensing maps.
[0028] Compared with the existing method, the beneficial effects of the present invention are:
[0029] 1. The method of the present invention controls the UAV to cruise in the remote sensing image area, and adopts the SLAM method of the combination of an airborne camera and an IMU to establish a SLAM map containing the UAV posture information and the sparse point cloud information of the UAV flight scene. The altitude of the UAV at each moment can be obtained by the barometric altimeter, and the remote sensing image and the airborne camera image are feature matched to obtain the corresponding points of the feature points in the remote sensing image in the airborne camera image, and the three-dimensional coordinates of the point in the SLAM coordinate system are obtained, and the three-dimensional vector from the UAV to the feature point in the SLAM coordinate system is calculated, and then the distance between the UAV and the point in the vertical direction is obtained. The elevation information of the point can be obtained by subtracting the distance from the UAV altitude, and finally the elevation information is added to the remote sensing image.
[0030] 2. Compared with existing remote sensing images, the present invention adds elevation information of feature points in remote sensing maps, which can provide more references for remote sensing map users. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0032] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. Specific implementation methods
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0035] A method for acquiring elevation of feature points of remote sensing images based on multiple sensors comprises the following steps:
[0036] Step 1: load the remote sensing image and calculate the location information and descriptor information of the feature points in the remote sensing map, set the UAV flight area according to the shooting range of the remote sensing image, and control the UAV to take off and cruise in the area;
[0037] Step 2: Use the SLAM method of the airborne camera and inertial sensor combination to estimate the position and posture of the UAV, and restore the feature points in the camera image to three-dimensional points to establish a SLAM map containing the UAV position and the scene three-dimensional point cloud information (i.e., a map generated based on the SLAM method);
[0038] Step 3, obtain the altitude information of the drone at each moment through the barometric altimeter, use the timestamp information to obtain the actual altitude of each position of the drone in the SLAM coordinate system (i.e., the coordinate system established based on the SLAM method), perform feature matching on the camera image and the remote sensing image, and obtain the coordinates of the matching points in the SLAM coordinate system;
[0039] Step 4: According to the information of the SLAM coordinate system, the vector from the drone to the feature point in the SLAM coordinate system is obtained, and then the projection length of the vector in the vertical direction is obtained. The projection length is subtracted from the drone height to obtain the elevation information of the feature point.
[0040] Furthermore, the specific method of step 2 is:
[0041] Set the origin and direction of the drone SLAM coordinate system, determine the position transformation relationship between the camera, inertial sensor and drone, and unify the postures of the three as the drone posture;
[0042] Perform feature detection on the camera image sequence to obtain the location information and descriptor information of the feature points, and obtain the location of the same feature point in different camera images by feature tracking;
[0043] The pose transformation between different camera images is calculated by multi-view geometry method, and the feature points in the camera image are restored into a three-dimensional point cloud by triangulation method;
[0044] The UAV posture and 3D point cloud coordinates are optimized using the bundle adjustment method;
[0045] According to the calculated UAV posture information and the data output by the inertial sensor, the parameters of the inertial sensor are calculated and optimized, and the pre-integrated quantity of the inertial sensor is calculated;
[0046] Fuse visual and inertial sensor information to build a UAV SLAM map with scale information.
[0047] Furthermore, the specific method of step 3 is:
[0048] The altitude information of the drone is output in real time through the barometric altimeter, and the position of the drone at each moment is output through the SLAM method. The two data are aligned according to the timestamp information to obtain the actual altitude of the drone in the SLAM coordinate system;
[0049] According to the descriptor information of the feature points, the feature points of the camera image and the feature points of the remote sensing image are matched to establish the feature matching point pair relationship;
[0050] According to the mapping relationship between the camera image feature points and the three-dimensional points in the SLAM coordinate system, the three-dimensional coordinate points corresponding to the feature points in the remote sensing image in the SLAM coordinate system are found.
[0051] Furthermore, the specific method of step 4 is:
[0052] Using the information of the SLAM coordinate system, the vector from the drone to the feature point is calculated;
[0053] According to the definition of the SLAM coordinate system, the projection distance of the vector in the vertical direction is obtained;
[0054] Subtract the projection distance from the UAV cruising altitude to obtain the elevation information of the feature points in the remote sensing image;
[0055] When a feature point in the remote sensing map matches feature points in multiple camera images, the elevations are calculated and averaged respectively, and the average is used as the elevation information of the feature point in the remote sensing image;
[0056] Add elevation information to the information of feature point locations on remote sensing maps.
[0057] A system for acquiring elevation of feature points in remote sensing images based on multiple sensors, comprising:
[0058] The remote sensing map module is used to load remote sensing images and calculate the location information and descriptor information of feature points in the remote sensing map, set the UAV flight area according to the shooting range of the remote sensing image, and control the UAV to take off and cruise in the area;
[0059] The UAV SLAM module is used to estimate the UAV's posture using the SLAM method of the airborne camera and inertial sensor combination, and restore the feature points in the camera image to three-dimensional points to establish a SLAM map containing the UAV's posture and scene three-dimensional point cloud information;
[0060] The information matching module is used to obtain the altitude information of the drone at each moment through the barometric altimeter, obtain the actual altitude of each position of the drone in the SLAM coordinate system using the timestamp information, perform feature matching on the camera image and the remote sensing image, and obtain the coordinates of the matching points in the SLAM coordinate system;
[0061] The elevation calculation module is used to obtain the vector from the drone to the feature point in the SLAM coordinate system according to the information of the SLAM coordinate system, and then obtain the projection length of the vector in the vertical direction, and subtract the projection length from the drone height to obtain the elevation information of the feature point.
[0062] Here are more specific examples:
[0063] A method for obtaining the elevation of feature points in remote sensing images based on multiple sensors, such as Figure 1 As shown, the following steps are included:
[0064] S1: Load the remote sensing image and calculate the location information and descriptor information of the feature points in the remote sensing map, set the UAV flight area according to the shooting range of the remote sensing image, and control the UAV to take off and cruise in the area;
[0065] S2: Use the SLAM method of the airborne camera and IMU combination to estimate the UAV's posture, restore the feature points in the camera image to three-dimensional points, and establish a SLAM map containing the UAV's posture and scene three-dimensional point cloud information;
[0066] S3: The altitude information of the drone at each moment can be obtained through the barometric altimeter. The actual altitude of each position of the drone in the SLAM coordinate system can be obtained by using the timestamp information. The airborne image and the remote sensing image are feature matched to obtain the coordinates of the matching points in the SLAM coordinate system.
[0067] S4: According to the information of the SLAM coordinate system, the vector from the drone to the feature point is obtained, and then the projection length of the vector in the vertical direction is obtained. The elevation information of the feature point can be obtained by subtracting the distance from the drone height. Finally, the elevation information is added to the remote sensing map.
[0068] The method controls the UAV to cruise in the remote sensing image area, and uses the SLAM method of the airborne camera and IMU combination to establish a SLAM map containing the UAV posture information and the sparse point cloud information of the UAV flight scene. The altitude of the UAV at each moment can be obtained through the barometric altimeter. The remote sensing image and the airborne camera image are feature matched to obtain the corresponding points of the feature points in the remote sensing image in the airborne camera image, and the three-dimensional coordinates of the point in the SLAM coordinate system are obtained. According to the coordinate transformation, the distance from the point to the camera plane can be obtained. According to the posture of the camera in the SLAM coordinate system, the vertical distance between the UAV and the point is obtained. The elevation information of the point can be obtained by subtracting the distance from the altitude of the UAV. Finally, the elevation information is added to the remote sensing image. Compared with the existing remote sensing images, this method adds the elevation information of the feature points in the remote sensing map, which can provide more references for remote sensing map users.
[0069] For step S1, the remote sensing image is loaded and the location information and descriptor information of the feature points in the remote sensing map are calculated, the flight area of the UAV is set according to the shooting range of the remote sensing image, and the UAV is controlled to take off and cruise in the area, including:
[0070] S11: Load the remote sensing image and use the feature detection algorithm to calculate the location information and descriptor information of the feature points in the remote sensing image.
[0071] The feature detection algorithm is a feature extraction algorithm based on ORB features, a feature extraction algorithm based on SIFT features, a feature extraction algorithm based on SURF features, and the like.
[0072] The feature points in the remote sensing image are recorded as Where y represents the yth remote sensing image, and N represents the total number of visual features in a remote sensing image.
[0073] According to the information of the pixels around the feature point, the descriptor information of the corresponding feature point is obtained through SIFT features, SURF features, ORB features and other methods, and recorded as
[0074] S12: Setting a UAV flight area according to the shooting range of the remote sensing image, and controlling the UAV to take off and cruise in the area.
[0075] For step S2, the SLAM method of the combination of the onboard camera and the IMU is used to estimate the posture of the UAV, and the feature points in the camera image are restored as three-dimensional points to establish a SLAM map containing the posture of the UAV and the three-dimensional point cloud information of the scene, including:
[0076] S21: Set the origin and direction of the drone SLAM coordinate system, determine the position transformation relationship between the camera, IMU and drone, and then the postures of the three will be uniformly expressed as the drone posture.
[0077] The position of the drone SLAM initialization is set as the origin of the SLAM coordinate system, and the XYZ coordinate axes point to the front, left, and top of the drone respectively. According to the drone structure, the position relationship between the drone, camera, and IMU is determined, and the postures of the three are uniformly represented by the drone posture after transformation.
[0078] S22: Perform feature detection on the airborne camera image sequence to obtain location information and descriptor information of feature points, and obtain the location of the same feature point in different camera images by feature tracking.
[0079] The feature detection algorithm is a feature extraction algorithm based on ORB features, a feature extraction algorithm based on SIFT features, a feature extraction algorithm based on SURF features, and the like.
[0080] The feature points in the airborne camera image are recorded as Where y represents the yth airborne camera image, and N represents the total number of visual features in a certain airborne camera image.
[0081] According to the information of the pixels around the feature point, the descriptor information of the corresponding feature point is obtained through SIFT features, SURF features, ORB features and other methods, and recorded as
[0082] Feature tracking uses existing methods, such as KLT sparse optical flow tracking method, dense optical flow tracking method, feature matching method, etc.
[0083] S23: Calculate the pose transformation between different camera images through multi-view geometry method, and restore the feature points in the camera image into a three-dimensional point cloud through triangulation method.
[0084] Multi-view geometry methods include the PNP method, the method for solving the basic matrix F between images, the method for solving the homography matrix H between images, and so on.
[0085] S24: Use bundle adjustment method to optimize the UAV pose and 3D point cloud coordinates.
[0086] S25: According to the calculated UAV posture information and the data output by the IMU, various IMU parameters are calculated and optimized, and the pre-integrated quantity of the IMU is calculated.
[0087] S26: Fusion of visual and IMU information to build a UAV SLAM map with scale information.
[0088] For step S3, the altitude information of the drone at each moment can be obtained through the barometric altimeter, and the actual altitude of each position of the drone in the SLAM coordinate system can be obtained by using the timestamp information. The airborne image and the remote sensing image are feature matched to obtain the coordinates of the matching points in the SLAM coordinate system, including:
[0089] S31: The barometric altimeter can output the altitude information of the drone in real time Among them, u indicates that this is the data of the drone, and the number in the lower right corner is the sequence number of the height arranged according to time. The position of the drone at each moment can be output in SLAM Among them, u indicates that this is the data of the drone, and i is the serial number of the drone coordinates arranged according to time in the SLAM coordinate system. According to the timestamp information, the two data can be aligned to obtain the actual altitude of the drone in the SLAM coordinate system;
[0090] S32: Describe the feature points of the airborne image according to the descriptor information of the feature points. and remote sensing image feature points Perform matching and establish feature matching point pair relationships.
[0091] S33: According to the mapping relationship between the feature points of the airborne image and the three-dimensional points in the SLAM coordinate system, the three-dimensional coordinate points corresponding to the feature points in the remote sensing image in the SLAM map are found, and their coordinates are marked as Where y represents the yth remote sensing image, and i is the serial number of the feature points of the point in the remote sensing image.
[0092] For step S4, according to the information of the SLAM coordinate system, the vector from the drone to the feature point is obtained, and then the projection length of the vector in the vertical direction is obtained. The elevation information of the feature point can be obtained by subtracting the distance from the height of the drone. Finally, the elevation information is added to the remote sensing map, including:
[0093] S41: Using the information of the SLAM coordinate system, the vector l from the drone position to the feature point is calculated;
[0094] S42: According to the definition of the drone coordinate system, the projection distance of the vector in the vertical direction is obtained, which is the modulus h of the component of the vector in the Z-axis direction;
[0095] S43: UAV cruising altitude h u Subtracting the above distance h, we can get the elevation information h of the feature point in the remote sensing image. p ;
[0096] S44: When a feature point in the remote sensing map matches feature points in multiple airborne images, the above calculation process is repeated and the average value is calculated at the end. Add it to the information of the location of feature points in the remote sensing map.
Claims
1. A method for acquiring elevation of feature points of remote sensing images based on multiple sensors, characterized in that: The following steps are involved: Step 1: load the remote sensing image and calculate the location information and descriptor information of the feature points in the remote sensing map, set the UAV flight area according to the shooting range of the remote sensing image, and control the UAV to take off and cruise in the area; Step 2: Use the SLAM method of the airborne camera and inertial sensor combination to estimate the posture of the UAV, and restore the feature points in the camera image to three-dimensional points to establish a SLAM map containing the UAV posture and scene three-dimensional point cloud information; the specific method is: Set the origin and direction of the drone SLAM coordinate system, determine the position transformation relationship between the camera, inertial sensor and drone, and unify the postures of the three as the drone posture; Perform feature detection on the camera image sequence to obtain the location information and descriptor information of the feature points, and obtain the location of the same feature point in different camera images by feature tracking; The pose transformation between different camera images is calculated by multi-view geometry method, and the feature points in the camera image are restored into a three-dimensional point cloud by triangulation method; The UAV posture and 3D point cloud coordinates are optimized using the bundle adjustment method; According to the calculated UAV posture information and the data output by the inertial sensor, the parameters of the inertial sensor are calculated and optimized, and the pre-integrated quantity of the inertial sensor is calculated; Fuse visual and inertial sensor information to build a UAV SLAM map with scale information; Step 3: Obtain the altitude information of the drone at each moment through the barometric altimeter, use the timestamp information to obtain the actual altitude of each position of the drone in the SLAM coordinate system, perform feature matching on the camera image and the remote sensing image, and obtain the coordinates of the matching points in the SLAM coordinate system; the specific method is: The altitude information of the drone is output in real time through the barometric altimeter, and the position of the drone at each moment is output through the SLAM method. The two data are aligned according to the timestamp information to obtain the actual altitude of the drone in the SLAM coordinate system; According to the descriptor information of the feature points, the feature points of the camera image and the feature points of the remote sensing image are matched to establish the feature matching point pair relationship; According to the mapping relationship between the camera image feature points and the three-dimensional points in the SLAM coordinate system, find the three-dimensional coordinate points corresponding to the feature points in the remote sensing image in the SLAM coordinate system; Step 4: According to the information of the SLAM coordinate system, the vector from the drone to the feature point in the SLAM coordinate system is obtained, and then the projection length of the vector in the vertical direction is obtained. The elevation information of the feature point is obtained by subtracting the projection length from the drone height. The specific method is: Using the information of the SLAM coordinate system, the vector from the drone to the feature point is calculated; According to the definition of the SLAM coordinate system, the projection distance of the vector in the vertical direction is obtained; Subtract the projection distance from the UAV cruising altitude to obtain the elevation information of the feature points in the remote sensing image; When a feature point in the remote sensing map matches feature points in multiple camera images, the elevations are calculated and averaged respectively, and the average is used as the elevation information of the feature point in the remote sensing image; Add elevation information to the information of feature point locations on remote sensing maps.
