An unmanned aerial vehicle perspective target geographic positioning method based on point cloud modeling

Abstract

The application belongs to the technical field of unmanned aerial vehicle target positioning, and discloses a kind of unmanned aerial vehicle perspective target geographic positioning method based on point cloud modeling, through the digital elevation model (Digital Elevation Model, DEM) and digital satellite image data obtained, point cloud modeling is carried out in offline mode, and is bound to the unmanned aerial vehicle cache before the unmanned aerial vehicle is launched.Target geographic positioning stage, according to the projection of the incidence angle and azimuth angle of the unmanned aerial vehicle, obtain the projection map, and the real-time visible light / infrared image of the target area obtained by the unmanned aerial vehicle, through image matching algorithm, obtain the coordinate mapping relationship between the target area and the digital satellite image, further calculate the corresponding coordinate geographic positioning information (latitude, longitude, elevation information).The application utilizes point cloud modeling and image matching algorithm, so that the unmanned aerial vehicle can obtain the elevation information of the target, obtain more accurate geographic position information of the target, which helps to improve the aircraft hit accuracy.

Description

Technical Field

[0001] This invention belongs to the field of UAV target localization technology, and relates to a UAV-based method for geographic localization of targets from a point cloud modeling perspective. Background Technology

[0002] With the development of drone technology, achieving target geolocation from a drone's perspective has broad application prospects in situations where global navigation satellite systems are interrupted or unavailable, and can be widely used in various industrial and agricultural practices. The mainstream approach is to combine drone-view images with satellite images and use computer vision methods to calculate the target's location. The challenge lies in the significant difference in perspective between the drone and satellite images, which affects the accuracy of image matching and consequently reduces the accuracy of geolocation.

[0003] Existing positioning technologies mainly fall into two categories: one approach is to match the target area image with satellite images containing raw geographic location information to obtain the target area's geographic location information. However, this method cannot obtain the target area's elevation information, and the target area may experience occlusion between objects due to elevation angles, while satellite images, being top-down views, do not have this occlusion problem, which also affects the matching accuracy. The other approach is to directly match the target area image with DEM data. The advantage of this approach is that it can obtain the target's latitude, longitude, and elevation information. The disadvantage is that there is a significant semantic gap between DEM data and the target area image. DEM data can only provide shape gradient information, while the target area image is a natural image with rich texture and shape information, resulting in insufficient matching accuracy and affecting the accuracy of the information obtained. However, this type of method is greatly affected by the image matching accuracy.

[0004] With the development of remote sensing technology, the difficulty of acquiring DEM data and high-resolution satellite imagery has been significantly reduced. Point cloud modeling methods can combine terrain information from DEM data and texture information from digital satellite imagery to accurately reconstruct the environment in which the UAV is located. Furthermore, by combining point cloud projection and image matching techniques, the limitation of existing target geolocation methods in obtaining target elevation information can be addressed. Summary of the Invention

[0005] (I) Purpose of the Invention

[0006] The purpose of this invention is to address the problem of low geolocation accuracy and inability to obtain target elevation information by UAVs, and to provide a UAV-based target geolocation method based on point cloud modeling. This method uses offline processing and online matching, combining accurate geographic location information from DEMs with rich texture features from satellite images, and employs 3D modeling and image matching techniques to solve the problem.

[0007] (II) Technical Solution

[0008] To address the challenge of obtaining target elevation information, this invention provides a UAV-based target geolocation method based on point cloud modeling. It employs an offline processing + online matching approach, combining precise geographic location information from DEMs with rich texture features from satellite images, and utilizes 3D modeling and image matching techniques to solve the problem.

[0009] The method of the present invention includes the following steps:

[0010] Step 1: Using the digital satellite image as a reference image, resample the DEM using linear interpolation to ensure consistent resolution. Then, intersect the digital satellite image and the resampled DEM along their latitude and longitude to obtain an aligned digital satellite image and an aligned DEM.

[0011] Step 2: Convert the aligned DEM into distance coordinates (x, lat) in meters based on latitude and longitude coordinates (lon, lat). meter ,y meter The modeling process involves preserving the elevation information (z) of each point and adding the pixel values ​​(R, G, B) of the corresponding locations in the aligned digital satellite image to the point's color information. The modeling result is then saved as a point cloud. Before launch, the point cloud data is bound to the UAV's cache.

[0012] Step 3: Target geolocation stage. Based on the azimuth and pitch angles provided by the UAV sensors, the constructed point cloud model is projected, and the projected image is used for image matching algorithms.

[0013] Step 4: Calculate the SIFT feature descriptors in the field-of-view image acquired by the UAV and the point cloud projection image, respectively, and detect the corresponding keypoint sets. Further, use the FLANN algorithm to match corresponding points in the two feature point sets, and obtain the transformation matrix (M1) between the two images. Similarly, calculate the transformation matrix (M2) corresponding to the projected image and the aligned digital satellite image.

[0014] Step 5: Detect targets in the UAV's field of view using a target detection algorithm, and return the corresponding area of ​​the target in the field of view in the form of coordinates.

[0015] Step 6: Based on the target coordinates (x, y) obtained in Step 5 object ,y object 1) First, perform matrix multiplication with the matrix M1 obtained in step 4 to obtain the coordinates (x, y) of this position in the projected image. projection ,y projection ,1); then the coordinates (x, y) in the projected image projection ,y projection,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position (x) in the digital satellite image. satellite ,y satellite ,1), and based on this location, query the corresponding latitude, longitude and elevation information in the point cloud data.

[0016] Steps 1 and 2 involve large computational loads and require significant storage resources, so they are primarily processed offline on a computer to achieve textured 3D point cloud modeling. Steps 3, 4, 5, and 6 are processed online on a drone to perform calculations, primarily for image projection, matching, target detection, and extraction of localization information.

[0017] (III) Beneficial Effects

[0018] The UAV-based target geolocation method based on point cloud modeling provided by the above technical solution has the following beneficial effects:

[0019] (1) This method obtains a more accurate target scene by combining three-dimensional modeling with projection, and obtains a slanted view of the digital satellite image through projection, which reduces the difference in perspective between the digital satellite image and the target image and helps to improve the accuracy of image matching.

[0020] (2) This method uses three-dimensional point cloud modeling, which helps to improve the accuracy of image matching and can achieve more accurate geographic positioning.

[0021] (3) This method can obtain the elevation information of the target by combining DEM data, which is impossible for passive geolocation methods based on two-dimensional images. Passive geolocation methods based on two-dimensional images can only obtain the latitude and longitude information of the target through image matching. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention.

[0023] Figure 2 This is a schematic diagram of three-dimensional geographic image modeling and projection in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of image matching in an embodiment of the present invention, wherein image a shows the image matching between the seeker's field of view and the projection image, and image b shows the image matching between the projection image and the satellite image. Detailed Implementation

[0025] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0026] This embodiment addresses the application of UAV-based target geolocation in situations where the Global Navigation Satellite System (GNSS) is interrupted or unavailable. It discloses a UAV-based target geolocation method based on point cloud modeling, which helps improve the UAV's target location capabilities under complex conditions.

[0027] Reference Figure 1 and Figure 2 As shown, the UAV-based target geolocation method based on point cloud modeling in this embodiment includes the following steps:

[0028] Step 1: Using the digital satellite image as a reference image, resample the DEM using a linear interpolation method. Then, take the intersection of the digital satellite image and the resampled DEM according to their latitude and longitude to obtain the aligned digital satellite image and the aligned DEM.

[0029] In this step, both the digital satellite imagery and DEM data used are in geotiff format, not the typical TIFF format. The difference lies in that geotiff data includes geographic coordinate system data, which can be used to calculate the latitude and longitude information of a point. The accuracy of these two types of data significantly affects the accuracy of subsequent geolocation. They are primarily obtained through open-source platforms. The resolution of the digital satellite imagery is 0.15 meters, and the resolution of the DEM data is 5 meters. There may be slight differences between the geographic areas corresponding to the satellite imagery and the DEM data, and their resolutions may also differ. This step uses resampling with the digital satellite imagery as a reference image, and bilinear interpolation is used to process the DEM data to ensure that its resolution is consistent with that of the digital satellite imagery. Figure 2 To ensure alignment between the digital satellite image and the DEM, the intersection of the digital satellite image and the resampled DEM data is further calculated according to their latitude and longitude to obtain the aligned digital satellite image and the aligned DEM. This step is a preprocessing step and is completed offline on a PC.

[0030] Step 2: Convert the aligned DEM into distance coordinates (x, lat) in meters based on latitude and longitude coordinates (lon, lat). meter ,y meter The modeling process involves preserving the elevation information (z) of each point and adding the pixel values ​​(R, G, B) of the corresponding locations in the aligned digital satellite image to the point's color information. The modeling result is then saved as a point cloud. Before launch, the point cloud data is bound to the UAV's cache.

[0031] The purpose of this step is to present the scene in more detail through 3D modeling, and then obtain a more refined scene image through projection operations, which helps improve subsequent image matching. During modeling, the latitude and longitude coordinates (lon, lat) are first converted to distance coordinates (x, lat) in meters.meter ,y meter Meanwhile, the elevation information z of each point is retained, and the modeling results are saved in point cloud form, as follows:

[0032] (i,j,longitude,latitude,x meter ,y meter ,z meter ),

[0033] Where i,j represent the row and column information of the current point in the digital satellite image and DEM data, longitude,latitude represent the longitude and latitude of the point, and x meter ,y meter Represents coordinate information in meters, z meter This provides the elevation information corresponding to the point. Furthermore, the pixel values ​​(R, G, B) at the corresponding location in the aligned digital satellite image are added to the point's color information. Finally, the modeling result is saved as a point cloud (txt format). Before spacecraft launch, the point cloud data needs to be bound to the image seeker.

[0034] Step 3: Target geolocation stage. Based on the azimuth and pitch angles provided by the UAV sensors, the constructed point cloud model is projected, and the projected image is used for image matching algorithms.

[0035] Figure 2 The image shows the results of projection with an azimuth angle of -20 degrees and an elevation angle of -20 degrees.

[0036] Step 4: Using an image matching algorithm, map the coordinates in the UAV's field of view to the coordinates in the digital satellite image, facilitating the subsequent extraction of geolocation information for the corresponding area. The specific process is as follows: First, calculate the SIFT feature descriptors in both the UAV-acquired field-of-view image and the point cloud projection image, and detect the corresponding keypoint sets. Then, use FLANN to match corresponding points in the two feature point sets, obtaining the transformation matrix M1 between the two images. Similarly, calculate the transformation matrix M2 corresponding to the projected image and the aligned digital satellite image; the matching process is as follows... Figure 3 As shown. This step needs to be completed online in real time by the drone.

[0037] Step 5: Detect targets in the UAV's field of view using a target detection algorithm, and return the corresponding area of ​​the target in the field of view in the form of coordinates.

[0038] The purpose of this step is to detect targets within the UAV's field of view using target detection algorithms, providing corresponding coordinates for target geolocation. Specific algorithms include deep learning algorithms (YOLO or Fast R-CNN, etc.) and traditional machine learning algorithms (template matching, etc.), and their output results are as follows: Figure 3 (a) The red box in the left field of view. This step is not part of the present invention; it is only used to provide target coordinates and is mainly completed online in real time by the UAV.

[0039] Step 6: Using the two transformation matrices obtained in Step 4 and the target coordinates obtained in Step 5, geographic location information is acquired. The specific process is as follows: Based on the coordinates (x, y) obtained from target detection... object ,y object 1) First, perform matrix multiplication with matrix M1 to obtain the coordinates (x, y) of this position in the projected image. projection ,y projection ,1); then the coordinates (x, y) in the projected image projection ,y projection ,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position (x) in the digital satellite image. satellite ,y satellite ,1), and query the corresponding location in the point cloud data (i,j,longitude,latitude,x) ​​based on this location. meter ,y meter ,z meter Query latitude, longitude and elevation information in (x) satellite ,y satellite (Matching with i and j). This step needs to be completed online in real time on the drone.

[0040] It is important to note that the accuracy of the final geolocation is affected by a variety of factors. These are as follows:

[0041] The accuracy and time of the acquired DEM: Specifically, the accuracy of the acquired DEM is much lower than that of digital satellite imagery. To maintain consistent resolution when constructing the point cloud model, an interpolation algorithm is needed to resample the DEM data to the same resolution. Therefore, the final location information may be generated by the interpolation algorithm and may differ from the actual geographic information. When the acquired DEM is too old, significant terrain changes can also lead to larger geolocation errors.

[0042] Image matching accuracy: This invention uses an image matching algorithm to map the target coordinates in the UAV's field of view to the target coordinates in a digital satellite image. This process mainly relies on calculating the SIFT descriptor of the image. When the images differ significantly, the image matching accuracy is greatly affected. For example, if the acquired digital satellite image is too old, resulting in huge differences in image texture, the image matching accuracy may decrease.

[0043] In the above technical solutions, the final accuracy of geolocation is affected by a variety of factors, mainly the following two aspects:

[0044] (1) DEM accuracy and time: Specifically, the accuracy of the acquired DEM is much lower than that of digital satellite imagery. To maintain consistent resolution when constructing the point cloud model, an interpolation algorithm is needed to resample the DEM data to the same resolution. Therefore, the final location information may be generated by the interpolation algorithm and may differ from the actual geographic information. When the acquired DEM is too old, significant terrain changes can also lead to larger geolocation errors.

[0045] (2) Image matching accuracy: This invention uses an image matching algorithm to map the target coordinates in the seeker's field of view to the target coordinates in the digital satellite image. This process mainly relies on calculating the SIFT descriptor of the image. When the image differences are large, the image matching accuracy is significantly affected. For example, when the seeker uses visible light images while the digital satellite image is infrared; or when the seeker samples infrared imaging while the digital satellite image uses visible light imaging; or when the acquired digital satellite image is too old, resulting in huge differences in image texture, the image matching accuracy may decrease.

[0046] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for geolocation of a target from a UAV perspective based on point cloud modeling, characterized in that, Includes the following steps: Step 1: Using the digital satellite image as a reference image, resample the DEM using a linear interpolation method. Then, take the intersection of the digital satellite image and the resampled DEM according to their latitude and longitude to obtain the aligned digital satellite image and the aligned DEM. Step 2: Convert the aligned DEM into distance coordinates (x, lat) in meters based on latitude and longitude coordinates (lon, lat). meter ,y meter The modeling process involves preserving the elevation information of each point, adding the pixel values ​​(R, G, B) of the corresponding locations in the aligned digital satellite image to the color information of the point, and finally saving the modeling results in the form of a point cloud. Before the UAV is launched, the point cloud data is bound to the UAV cache. Step 3: Target geolocation stage. Based on the azimuth and pitch angles provided by the UAV sensors, the constructed point cloud model is projected, and the projected image is used for image matching algorithms. Step 4: Calculate the SIFT feature descriptors in the field-of-view image and the point cloud projection image acquired by the UAV, and detect the corresponding feature point sets; further use the FLANN algorithm to match the corresponding points of the two feature point sets, and obtain the transformation matrix M1 between the two images; similarly, calculate the transformation matrix M2 corresponding to the projected image and the aligned digital satellite image. Step 5: Implement target detection in the UAV's field of view using target detection algorithms (x object ,y object ,1), and return the corresponding area of ​​the target in the field of view in coordinate form; Step 6: Obtain geographic location information by using the two transformation matrices obtained in Step 4 and the target coordinates obtained in Step 5.

2. The UAV-based target geolocation method based on point cloud modeling as described in claim 1, characterized in that, In step 1, both the digital satellite imagery and the DEM data are in geotiff format. Geotiff format data contains geographic coordinate system data, which is used to calculate the latitude and longitude information for a given point.

3. The UAV-based target geolocation method based on point cloud modeling as described in claim 2, characterized in that, In step 2, the modeling results are saved in point cloud format, as follows: (i,j,longitude,latitude,x meter ,y meter ,z meter ), Where i,j represent the row and column information of the current point in the digital satellite image and DEM data, longitude,latitude represent the longitude and latitude of the point, and x meter ,y meter Represents coordinate information in meters, z meter This provides the elevation information corresponding to that point.

4. The UAV-based target geolocation method based on point cloud modeling as described in claim 3, characterized in that, Step 4 is completed online in real time on the drone.

5. The UAV-based target geolocation method based on point cloud modeling as described in claim 3, characterized in that, In step 5, the target detection algorithm uses a deep learning algorithm.

6. The UAV-based target geolocation method based on point cloud modeling as described in claim 5, characterized in that, Step 5 is completed online in real time on the drone.

7. The UAV-based target geolocation method based on point cloud modeling as described in claim 3, characterized in that, In step 6, based on the target coordinates (x, y) obtained in step 5... object ,y object 1) First, perform matrix multiplication with the matrix M1 obtained in step 4 to obtain the coordinates (x, y) of this position in the projected image. projection ,y projection ,1); then the coordinates (x, y) in the projected image projection ,y projection ,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position (x) in the digital satellite image. satellite ,y satellite ,1), and based on this location, query the corresponding latitude, longitude and elevation information in the point cloud data.

8. The UAV-based target geolocation method based on point cloud modeling as described in claim 7, characterized in that, In step 6, in the point cloud data (i,j,longitude,latitude,x) meter ,y meter ,z meter To query latitude, longitude, and elevation information, use x in the query. satellite ,y satellite Match with i and j.

9. The UAV-based target geolocation method based on point cloud modeling as described in claim 8, characterized in that, Step 6 is completed online in real time on the drone.

10. The UAV-based target geolocation method based on point cloud modeling as described in claim 3, characterized in that, Steps 1 and 2 are processed offline and completed on a computer.

Images