Point cloud modeling-based unmanned aerial vehicle visual angle target geographic positioning method

Abstract

The invention belongs to the technical field of unmanned aerial vehicle target positioning, and discloses an unmanned aerial vehicle visual angle target geographic positioning method based on point cloud modeling, and the method comprises the steps: carrying out the point cloud modeling in an offline mode through an obtained digital elevation model (DEM) and digital satellite image data, and binding the data to an unmanned aerial vehicle for caching before the unmanned aerial vehicle is launched. In the target geographic positioning stage, projection is performed according to the incidence angle and azimuth angle of the unmanned aerial vehicle to obtain a projection drawing, the real-time visible light / infrared image of the target area is obtained by the unmanned aerial vehicle, and the coordinate mapping relation between the target area and the digital satellite image is obtained through an image matching algorithm; and further calculating geographic positioning information (longitude and latitude and elevation information) of the corresponding coordinates. According to the invention, by using the point cloud modeling and image matching algorithm, the unmanned aerial vehicle can obtain the elevation information of the target, more accurate geographic position information of the target is obtained, and improvement of the hit precision of the unmanned aerial vehicle is facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle target positioning, and relates to a method for geographical positioning of unmanned aerial vehicle perspective targets based on point cloud modeling. Background Art

[0002] With the development of drone technology, in the case of interruption or unavailability of the global navigation satellite system, achieving target geolocation from the drone's perspective has broad application prospects and is widely used in various industrial and agricultural practices. Combining drone-perspective images with satellite images and calculating the position of the target through computer vision methods is the mainstream solution. The difficulty lies in the huge perspective difference between drone-perspective images and satellite images, which affects the accuracy of image matching and thus reduces the accuracy of geolocation.

[0003] There are two main types of existing positioning technologies: one is to match the target area image with the satellite image with the original geographic location information to obtain the geographic location information of the target area, but this solution cannot obtain the elevation information of the target area, and the target area will be blocked by objects due to the pitch angle, while the satellite image is a bird's-eye view and there is no occlusion problem, which will also affect the matching accuracy; the other is to directly match the target area image with the DEM data. The advantage of this solution is that the target latitude and longitude information and elevation information can be obtained, but the disadvantage is that there is a huge semantic gap between the DEM data and the target area image. Only shape gradient information can be obtained in the DEM, while the target area image is a natural image with rich texture and shape information, resulting in insufficient matching accuracy and affecting the accuracy of information acquisition. However, this type of method is greatly affected by the image matching accuracy.

[0004] With the development of remote sensing technology, the difficulty of obtaining DEM data and high-resolution satellite images has been significantly reduced. The point cloud modeling method can combine the terrain information in the DEM data and the texture information in the digital satellite image to accurately restore the environment in which the drone is located. At the same time, combined with point cloud projection and image matching technology, it can solve the defect that the existing target geographic positioning cannot obtain the target elevation information. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to provide a method for geo-locating targets from a UAV perspective based on point cloud modeling, in order to solve the problem that the UAV geo-locating accuracy is low and the target elevation information cannot be obtained. The method adopts the method of offline processing + online matching, combines the precise geographic location information of DEM and the rich texture feature information of satellite images, and uses three-dimensional modeling and image matching technologies to solve the problem.

[0007] (II) Technical solution

[0008] In order to solve the problem of being unable to obtain target elevation information, the present invention provides a method for geographical positioning of targets from a UAV perspective based on point cloud modeling. It adopts an offline processing + online matching approach, combines the precise geographical location information of DEM and the rich texture feature information of satellite images, and uses three-dimensional modeling and image matching technologies to solve the problem.

[0009] The method of the present invention comprises the following steps:

[0010] Step 1: Using the digital satellite image as the reference image, the DEM is resampled using the linear interpolation method to ensure that the resolution of the two is consistent. The digital satellite image and the resampled DEM are further intersected according to their longitude and latitude to obtain the aligned digital satellite image and aligned DEM.

[0011] Step 2: Convert the aligned DEM into distance coordinates (x, y) in meters based on the latitude and longitude coordinates (lon, lat). meter ,y meter ), while retaining the elevation information z of each point, and adding the pixel value information (R, G, B) at the corresponding position in the aligned digital satellite image to the color information of the point, and finally saving the modeling results in the form of point cloud. Before the drone is launched, the point cloud data is bound to the drone cache.

[0012] Step 3: In the target geolocation stage, the constructed point cloud model is projected according to the azimuth and pitch provided by the drone sensor, and the projected image is used for the image matching algorithm.

[0013] Step 4: Calculate the SIFT feature descriptors in the field of view image acquired by the drone and the point cloud projection image, and detect the corresponding key point set. The FLANN algorithm is further used to match the corresponding points of the two feature point sets, and the transformation matrix (M1) between the two images is obtained. Similarly, the transformation matrix (M2) corresponding to the projected image and the aligned digital satellite image is calculated.

[0014] Step 5: Detect the target in the drone’s field of view through the target detection algorithm, and return the corresponding area of ​​the target in the field of view in the form of coordinates.

[0015] Step 6: According to the target coordinates (x object ,y object ,1) First, perform matrix multiplication with the matrix M1 obtained in step 4 to obtain the coordinates (x projection ,y projection ,1); then the coordinates (x projection ,y projection,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position in the digital satellite image (x satellite ,y satellite ,1), and based on this location query, the corresponding location is calculated in the point cloud data for the corresponding latitude, longitude and elevation information.

[0016] Among them, steps 1 and 2 are computationally intensive and occupy a large amount of storage resources. They are mainly processed offline and completed on the computer side to realize the textured 3D point cloud modeling part; steps 3, 4, 5, and 6 are processed online and completed on the drone side to mainly realize image projection, matching, target detection, and positioning information extraction processes.

[0017] (III) Beneficial effects

[0018] The above technical solution provides a method for geo-locating a target from a drone's perspective based on point cloud modeling, which has the following beneficial effects:

[0019] (1) This method combines three-dimensional modeling with projection to obtain a more accurate target scene, and obtains an oblique view of the digital satellite image through the projection method, which reduces the perspective difference between the target image and helps to improve the accuracy of image matching.

[0020] (2) This method helps improve the accuracy of image matching through three-dimensional point cloud modeling, which can achieve more accurate geographic positioning.

[0021] (3) This method can obtain the target's elevation information by combining DEM data, which is impossible to achieve with passive geolocation methods based on two-dimensional images. Passive geolocation methods based on two-dimensional images can only obtain the target's latitude and longitude information through image matching methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flow chart of the method of the present invention.

[0023] Figure 2 It is a schematic diagram of three-dimensional geographic image modeling and projection in an embodiment of the present invention.

[0024] Figure 3 Schematic diagram of image matching in an embodiment of the present invention, wherein Figure a is the image matching between the seeker field of view and the projection image, and Figure b is the image matching between the projection image and the satellite image. DETAILED DESCRIPTION

[0026] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0027] This embodiment is aimed at the application of target geolocation from the perspective of a drone when the Global Navigation Satellite System (GNSS) is interrupted or unavailable, and discloses a method for geolocation of targets from the perspective of a drone based on point cloud modeling, which helps to improve the target positioning capability of the drone under complex conditions.

[0028] Reference Figure 1 and Figure 2 As shown, the method for geographical positioning of a target from a drone's perspective based on point cloud modeling in this embodiment includes the following steps:

[0029] Step 1: Using the digital satellite image as the reference image, the DEM is resampled using the linear interpolation method. The digital satellite image and the resampled DEM are further intersected according to their longitude and latitude to obtain the aligned digital satellite image and aligned DEM.

[0030] In this step, the digital satellite images and DEM data used are both in geotiff format data, rather than general tiff format data. The difference is that geotiff data contains data in the geographic coordinate system, which can be used to calculate the longitude and latitude information of a point. The accuracy of these two types of data greatly affects the accuracy of subsequent geolocation. They are mainly obtained through open source platforms. The resolution of digital satellite images is 0.15 meters, and the accuracy of DEM data is 5 meters. There may be slight differences in the geographical areas corresponding to satellite images and DEM data, and their resolutions may also be different. This step uses resampling to use digital satellite images as reference images, and uses bilinear interpolation to process the DEM data to keep it consistent with the resolution of the digital satellite images ( Figure 2 ). To ensure the alignment between the digital satellite image and the DEM, the digital satellite image and the resampled DEM data are further intersected according to their longitude and latitude to obtain the aligned digital satellite image and aligned DEM. This step belongs to the preprocessing process and is completed offline on the PC.

[0031] Step 2: Convert the aligned DEM into distance coordinates (x, y) in meters based on the latitude and longitude coordinates (lon, lat). meter ,y meter ), while retaining the elevation information z of each point, and adding the pixel value information (R, G, B) at the corresponding position in the aligned digital satellite image to the color information of the point, and finally saving the modeling results in the form of point cloud. Before the drone is launched, the point cloud data is bound to the drone cache.

[0032] The purpose of this step is to display the scene in more detail through 3D modeling, and then obtain a more detailed scene map through projection operation, which will help improve subsequent image matching. When modeling, first convert the longitude and latitude coordinates (lon, lat) into distance coordinates in meters (xmeter ,y meter ), while retaining the elevation information z of each point, and saving the modeling results in the form of point cloud. The specific form is as follows:

[0033] (i,j,longitude,latitude,x meter ,y meter ,z meter ),

[0034] Among them, i, j represent the row and column information corresponding to the current point in the digital satellite image and DEM data, longitude, latitude represent the longitude and latitude of the point, x meter ,y meter Represents coordinate information in meters, z meter is the elevation information corresponding to the point. In addition, the pixel value information (R, G, B) at the corresponding position in the aligned digital satellite image is added to the color information of the point, and finally the modeling result is saved in the form of point cloud (txt format). Before the aircraft is launched, the point cloud data needs to be bound to the image seeker.

[0035] Step 3: In the target geolocation stage, the constructed point cloud model is projected according to the azimuth and pitch provided by the drone sensor, and the projected image is used for the image matching algorithm.

[0036] Figure 2 The result of projection at an azimuth angle of -20 degrees and an elevation angle of -20 degrees is shown in FIG.

[0037] Step 4: Use the image matching algorithm to map the coordinates in the drone’s field of view to the coordinates in the digital satellite image, so as to facilitate the subsequent extraction of the geographic positioning information of the corresponding area. The specific process is as follows: First, calculate the SIFT feature descriptors in the field of view image acquired by the drone and the point cloud projection image, and detect the corresponding key point set. Further use FLANN to match the corresponding points of the two feature point sets, and obtain the transformation matrix M1 between the two images. Similarly, calculate the transformation matrix M2 corresponding to the projected image and the aligned digital satellite image. The matching process is as follows: Figure 3 This step needs to be completed in real time online on the drone.

[0038] Step 5: Detect the target in the drone’s field of view through the target detection algorithm, and return the corresponding area of ​​the target in the field of view in the form of coordinates.

[0039] The purpose of this step is to detect the target in the drone's field of view through the target detection algorithm and provide the corresponding coordinates for the target's geographic positioning. The specific algorithms include deep learning algorithms (Yolo algorithm or FastRCNN, etc.) and traditional machine learning algorithms (template matching, etc.). The output results are as follows: Figure 3 (a) The red box in the left field of view. This step does not belong to the content of the present invention, is only used to provide target coordinates, and is mainly completed by the drone in real time online.

[0040] Step 6: Acquire geographic positioning information through the two transformation matrices obtained in step 4 and the target coordinates obtained in step 5. The specific process is: according to the coordinates (x object ,y object ,1) First, perform matrix multiplication with matrix M1 to obtain the coordinates of the position in the projected image (x projection ,y projection ,1); then the coordinates (x projection ,y projection ,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position in the digital satellite image (x satellite ,y satellite ,1), and query the corresponding position in the point cloud data (i,j,longitude,latitude,x meter ,y meter ,z meter ) to query the latitude, longitude and elevation information (replace x satellite ,y satellite This step needs to be completed in real time online on the drone.

[0041] It should be noted that the accuracy of the final geolocation is affected by many factors. They are as follows:

[0042] DEM accuracy and time of acquisition: Specifically, the accuracy of the acquired DEM is much lower than that of digital satellite images. When building a point cloud model, the resolution of the two must be kept consistent. Therefore, it is necessary to use a difference algorithm to resample the DEM data to the same resolution. Therefore, the final positioning information may be generated by the difference algorithm, which may be partially different from the actual geographic information. When the DEM is acquired too long ago, the terrain changes greatly, which will also lead to large geographic positioning errors.

[0043] Image matching accuracy: The present invention realizes the mapping of the target coordinates in the drone field of view and the target coordinates in the digital satellite image through the image matching algorithm. This process mainly relies on the calculation of the SIFT descriptor of the image. When the image difference is large, the image matching accuracy is greatly affected. For example: when the acquired digital satellite image is too old, resulting in huge differences in image texture, etc., the image matching accuracy may be reduced.

[0044] In the above technical solution, the accuracy of the final geolocation is affected by many factors, mainly the following two aspects:

[0045] (1) DEM accuracy and time of acquisition: Specifically, the accuracy of the acquired DEM is much lower than that of the digital satellite image. When building the point cloud model, the resolution of the two images must be kept consistent. Therefore, the DEM data needs to be resampled to the same resolution using a difference algorithm. Therefore, the final positioning information may be generated by the difference algorithm, which may be partially different from the actual geographic information. When the DEM is acquired too long ago, the terrain changes greatly, which will also lead to large geographic positioning errors.

[0046] (2) Image matching accuracy: The present invention realizes the mapping of the target coordinates in the field of view of the seeker and the target coordinates in the digital satellite image through an image matching algorithm. This process mainly relies on the calculation of the SIFT descriptor of the image. When the image difference is large, the image matching accuracy is greatly affected. For example: when the seeker uses visible light images, and the digital satellite image is infrared imaging; or the seeker samples infrared imaging, and the digital satellite image uses visible light imaging; or when the acquired digital satellite image is too old, resulting in huge differences in image texture, etc., the image matching accuracy may be reduced.

[0047] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for geographical positioning of targets from an unmanned aerial vehicle perspective based on point cloud modeling, characterized in that: The following steps are involved: Step 1: Using the digital satellite image as the reference image, the DEM is resampled using the linear interpolation method, and the digital satellite image and the resampled DEM are intersected according to their longitude and latitude to obtain the aligned digital satellite image and aligned DEM; Step 2: Convert the aligned DEM into distance coordinates (x, y) in meters based on the latitude and longitude coordinates (lon, lat). meter ,y meter ), while retaining the elevation information of each point, and adding the pixel value information (R, G, B) at the corresponding position in the aligned digital satellite image to the color information of the point, and finally saving the modeling result in the form of point cloud; before the drone is launched, the point cloud data is bound to the drone cache; Step 3: In the target geolocation phase, the constructed point cloud model is projected according to the azimuth and pitch angles provided by the drone sensor, and the projected image is used in the image matching algorithm; Step 4: Calculate the SIFT feature descriptors in the field of view image acquired by the drone and the point cloud projection image respectively, and detect the corresponding feature point set. Further use the FLANN algorithm to match the corresponding points of the two feature point sets, and obtain the transformation matrix M1 between the two images; similarly, calculate the transformation matrix M2 corresponding to the projected image and the aligned digital satellite image; Step 5: Detect objects in the drone’s field of view using the object detection algorithm (x object ,y object ,1), and returns the corresponding area of ​​the target in the field of view in the form of coordinates; Step 6: Acquire geographic positioning information through the two transformation matrices obtained in step 4 and the target coordinates obtained in step 5.

2. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 1, characterized in that: In step 1, the digital satellite image and DEM data are both in geotiff format data, which contains data in a geographic coordinate system and is used to calculate the longitude and latitude information of a point.

3. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 2, characterized in that: In step 2, the modeling results are saved in the form of point cloud, as follows: (i,j,longitude,latitude,x meter ,y meter ,z meter ), Among them, i, j represent the row and column information corresponding to the current point in the digital satellite image and DEM data, longitude, latitude represent the longitude and latitude of the point, x meter ,y meter Represents coordinate information in meters, z meter It is the elevation information corresponding to this point.

4. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 3, characterized in that: Step 4 is completed online in real time on the drone.

5. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 3, characterized in that: In step 5, the target detection algorithm adopts a deep learning algorithm.

6. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 5, characterized in that: Step 5 is completed in real time online on the drone.

7. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 3, characterized in that: In step 6, according to the target coordinates (x object ,y object ,1) First, perform matrix multiplication with the matrix M1 obtained in step 4 to obtain the coordinates (x projection ,y projection ,1); then the coordinates (x projection ,y projection ,1) Perform matrix multiplication with the transformation matrix M2 to obtain the corresponding position in the digital satellite image (x satellite ,y satellite ,1), and based on this location query, the corresponding location is calculated in the point cloud data for the corresponding latitude, longitude and elevation information.

8. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 7, characterized in that: In step 6, in the point cloud data (i, j, longitude, latitude, x meter ,y meter ,z meter ) to query the latitude, longitude and elevation information. When querying, replace x satellite ,y satellite Match with i,j.

9. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 8, characterized in that: Step 6 is completed in real time online on the drone.

10. The method for geographical positioning of a target from an unmanned aerial vehicle perspective based on point cloud modeling as claimed in claim 3, characterized in that: Step 1 and step 2 are processed offline and completed on the computer.

Images