A three-dimensional spatial target positioning method for drone inspection

Abstract

The present invention relates to a three-dimensional space target positioning method for drone inspection. It solves the problem existing in the prior art that the higher the altitude of the drone relative to the ground, the larger the tilt angle during shooting, and the more the object is at the edge of the image, and this gap is often larger. The present invention includes the following steps: obtaining the EXIF parameter information of the drone photo through an image parsing tool; finding the corresponding DEM data block from the terrain DEM database; generating a three-dimensional surface grid model; using an artificial intelligence detection method to detect the target object from the drone photo and obtaining the two-dimensional coordinates of the object in the photo; constructing an intersection ray in three-dimensional space according to the two-dimensional coordinates of the target object; efficiently calculating the three-dimensional space position of the target object, and finally realizing the three-dimensional space positioning of the inspected object. The advantages of the present invention are: low cost and high efficiency.

Description

Technical Field

[0001] The present invention relates to a three-dimensional space target positioning method for drone inspection, especially for a target object detected in a drone photo. By using photo parameters and terrain data, the three-dimensional position of the target object in the earth coordinate system is calculated. Background Art

[0002] With the popularization of drones, drone inspection is a technology for low-altitude detection of target objects. This technology can not only greatly make up for the deficiency of low resolution of high-altitude satellite images, but also combine with advanced image-based artificial intelligence object recognition technology to automatically detect target objects from drone photos, playing an increasingly important role in the fields of electric power, transportation, security, etc.

[0003] However, object recognition based on drone images can only obtain which position (in a two-dimensional image coordinate system) of which photo has a target object, and cannot obtain the longitude and latitude coordinates of the target object. Due to the tilt angle during drone shooting, or the target object not being at the center of the image, there is often a certain gap between the GPS information carried by the drone image and the actual position of the target object in the image. The higher the drone is relative to the ground, the larger the tilt angle during shooting, and the more the object is at the edge of the image, the greater this gap tends to be. Therefore, there is an urgent need for a method that can calculate the three-dimensional space position of objects in drone images, which can quickly achieve the rapid positioning of inspection target objects, thus supporting the implementation of the spatial application of drone inspection. Summary of the Invention

[0004] For the three-dimensional space positioning of drone inspection target objects, the present invention provides a three-dimensional space target positioning method for drone inspection. This method makes full use of the camera parameters embedded in drone photos, combines with global terrain DEM data to generate a three-dimensional surface grid, and locates the results of image-based artificial intelligence object detection in the three-dimensional space through the method of three-dimensional space intersection, realizing the automatic three-dimensional positioning of drone inspection target objects.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] Step 1: Obtain the EXIF parameter information of the drone photo through an image parsing tool, mainly obtaining the position information, orientation information, and internal parameters of the drone camera when taking the photo.

[0007] Step 2: Find the corresponding DEM data block from the terrain DEM database according to the camera position information.

[0008] Step 3: Generate a three-dimensional surface grid model with the DEM data block;

[0009] Step 4: Use the artificial intelligence detection method to detect the target object from the drone photos and obtain the two-dimensional coordinates of the object in the photo;

[0010] Step 5: Based on the external parameters such as the position and orientation of the drone and its internal parameters, construct the intersection rays in the three-dimensional space according to the two-dimensional coordinates of the target object;

[0011] Step 6: Through the three-dimensional intersection operation based on hierarchical progression, efficiently calculate the three-dimensional space position of the target object, and finally realize the three-dimensional space positioning of the inspected object. Brief Description of the Drawings

[0012] Figure 1 is the overall flowchart of the three-dimensional space target positioning algorithm for drone inspection

[0013] Figure 2 is a schematic diagram of two triangular patches for constructing the ground grid with a pixel of the DEM image. The left subfigure is the DEM image, the middle subfigure is a small part of the DEM image, and the right subfigure is a schematic diagram of the two triangular patch planes corresponding to a pixel in the DEM

[0014] Figure 3 is a schematic diagram of the drone photo and its detected target. The detected target is marked with a red rectangular frame

[0015] Figure 4 is a schematic diagram of the position of the inspection target on the map. The detected target is marked with a position marker symbol, mainly presenting the longitude and latitude information of the three-dimensional positioning, rather than the height information

[0016] Figure 5 is a schematic diagram of the DEM image segmentation for accelerating intersection Detailed Embodiment

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with specific implementation embodiments and with reference to the accompanying drawings.

[0018] Figure 1 The overall technical flowchart of the present invention is shown as follows. The steps of using the present invention are:

[0019] Step 1: Obtain the EXIF parameter information of the drone photos through an image parsing tool. When the drone camera takes a photo, it usually records the camera lens parameters and shooting parameters into the EXIF (Exchangeable Image File Format). By using the picture metadata extraction tool Exiftool (https: / / exiftool.org / ) to extract the EXIF information, these parameters can be obtained from the drone images, mainly including the position information, orientation information, and internal parameters of the drone camera. The internal parameters include focal length, field of view angle, lens sensor size, etc.

[0020] Step 2: According to the camera position information, i.e., the longitude and latitude coordinates, find the corresponding DEM data block from the terrain DEM database. The terrain DEM database is mainly constructed with publicly available low-precision DEM data, such as the 30M-precision global DEM data provided by NASA on its website. These DEM data are often stored in the form of tiled images, and each DEM data block has a longitude and latitude coverage range. According to this range, it can be quickly detected which DEM data block the camera position of the drone is within.

[0021] Step 3: Generate a three-dimensional surface grid model with the DEM data block. Use the DEM data block to construct 2*W*H triangular patches, where W and H are the length and width of the DEM data block respectively. The DEM data block is usually a digital image in TIFF format, and the GDAL library (Geospatial Data Abstraction Library) is used to align and parse the image data, hereinafter referred to as the DEM image; each pixel of the DEM image corresponds to two small triangular patches. The first patch is composed of the upper left corner, upper right corner, and lower left corner of the pixel, and the second patch is composed of the upper right corner, lower right corner, and lower left corner of the pixel; the X and Y coordinates of the vertices of these two patches are determined according to the two-dimensional coordinates of this pixel in the image, the longitude and latitude information, and the scale of this DEM image, while the height of the vertices of these two patches is determined by the value of the DEM image at this pixel; denote the set of all triangular patches of the three-dimensional surface grid constructed from the DEM data block as {T i}.

[0022] Step 4: Use the artificial intelligence detection method to detect the target object from the drone photos and obtain the two-dimensional coordinates of the object in the photo. There are many artificial intelligence object detection libraries that can be used. This invention mainly uses YOLO V5 to detect the target object, which can obtain a rectangular box surrounding the target object in the input drone photos. This invention takes the center point of this rectangular box as the two-dimensional coordinates of the object.

[0023] Step 5: Combine the two-dimensional coordinates of the target object with parameters such as the position and orientation of the drone to construct an intersection ray in three-dimensional space. Taking the position of the camera (including longitude, latitude, and altitude) as the starting point, use the camera orientation, focal length, and field of view angle of the camera to construct a frustum of a cone, and then use the two-dimensional coordinates of the target object in the photo to obtain a ray in this frustum of a cone, denoted as L. This ray passes through both the camera position and the corresponding position of the object's two-dimensional coordinates in the frustum of a cone.

[0024] Step 6: Through a hierarchical progressive three-dimensional intersection operation, efficiently calculate the three-dimensional spatial position of the target object and finally achieve the three-dimensional spatial positioning of the inspected object. That is, intersect L with {T iPerform intersection operations separately to find the patch T' with intersections, and record the intersection point of the ray L and the patch T' as the three-dimensional spatial position of the target object.

[0025] Further, the hierarchical progressive three-dimensional intersection operation method in step 6 is as Figure 5 shown. This method divides the entire intersection process into three layers: First, divide the entire DEM image into squares of size 256*256. The original DEM image of size 4096*4096 is divided into 4096*4096 / (256*256) = 256 squares, generating a total of 2*256 = 512 triangular patches, denoted as {Mi}; then, find the intersection points of the ray L and {Mi}, and the number of intersection operations is 512; assume that the ray L intersects K1 patches {Mk}, and then divide the 256*256 squares of these K1 {Mk} triangular patches into 16*16 squares, also divided into 256 small squares, forming 512 smaller triangular patches, denoted as {Ni}, and find the intersection points of the ray L and {Ni}, and the number of intersection operations for each patch is also 512. Since there are a total of K1 {Mk} triangular patches, the total number of intersection operations is 512*K1; finally, assume that the ray L intersects K2 of the patches {Nk}. Since the 16*16 squares are already relatively small, there is no need to divide further. Use the 16*16 squares and 512 smallest triangular patches {Pi} to intersect with the ray L, and record the intersection points as {Pk}; because the drone takes pictures from top to bottom, the coordinates of the one with the largest elevation among {Pk} are the three-dimensional spatial position of the target object, and the number of intersection operations is 512*K1*K2. Therefore, the total number of intersection operations for the entire hierarchical progression is: 512 + 512*K1 + 512*K1*K2.

[0026] Adopting the hierarchical progressive algorithm only adds a small amount of DEM block division work, and the impact on efficiency can be almost ignored. This algorithm greatly improves the calculation efficiency. Without using the hierarchical progressive algorithm, it takes about 1 second to calculate 1 target point, while after using the hierarchical progressive algorithm, it only takes about 0.5 seconds to calculate 100 target points. Moreover, the memory requirement of the optimization method calculation is also greatly reduced because not all DEM data needs to be loaded after optimization.

[0027] The intersection in three-dimensional space involves a large amount of computation. Among all the computations, calculating the intersection point of a ray and a spatial triangle and determining whether the intersection point is inside the triangle account for more than 98% of the computation. For DEM images with sizes of 4096 and 4096, the total number of triangular patches is 2 * 4096 * 4096 times. Traversing these triangular patches to find intersections with ray L respectively has low efficiency. The method adopted in the present invention is a hierarchical progressive acceleration intersection method, and the total number of intersection calculations is: 512 + 512 * K1 + 512 * K1 * K2, where the coefficients K1 and K2 are related to the flight altitude of the unmanned aerial vehicle, the rate of change of the DEM elevation, and the tilt angle of the target point relative to the unmanned aerial vehicle. The average values are obtained through many experiments. The average value of K1 is 2.5, and the average value of K2 is 1.2. Therefore, the average total number of intersection calculations is = 512 + 512 * 2.5 + 512 * 2.5 * 1.2 = 3328 times, which is less than 1% of the direct calculation of the number of intersections.

[0028] Currently, the inspection methods for unmanned aerial vehicle photos mainly aim to detect relevant objects in the photos and mark the detected targets in the form of dots or boxes on the photos as the inspection results. However, since the unmanned aerial vehicle takes photos in the air and the shooting perspective usually has a certain inclination, such results cannot accurately indicate the accurate position of the object, especially the accurate longitude and latitude data on the map. In view of this problem, the present invention designs a three-dimensional space target positioning method for unmanned aerial vehicle inspection, obtains and utilizes publicly available DEM terrain data to construct a three-dimensional surface grid near the shooting position of the unmanned aerial vehicle, and performs three-dimensional space intersection with the shooting perspective ray to achieve accurate three-dimensional positioning of the target object. The present invention has the advantages of low cost and high efficiency, and the calculated longitude and latitude data of the target object can play an important role in various applications based on unmanned aerial vehicle inspection.

[0029] The content described in the embodiments of this specification is only a list of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A three-dimensional spatial target positioning method for UAV inspection, comprising the following steps: Step 1: Obtain the EXIF parameter information of the UAV photo through an image analysis tool, mainly obtaining the position information, orientation information, and internal parameters of the UAV camera when the photo was taken; Step 2: Find the DEM data block where the UAV photo was taken from the terrain DEM database according to the camera position information; Step 3: Generate a three-dimensional surface grid model using this DEM data block; Step 4: Use an artificial intelligence detection method to detect the target object from the UAV photo and obtain the two-dimensional coordinates of the object in the photo; Step 5: Construct an intersection ray in three-dimensional space based on the two-dimensional coordinates of the target object through the external parameters of the UAV position and orientation and its internal parameters; Step 6: Through a three-dimensional intersection operation based on hierarchical progression, efficiently calculate the three-dimensional spatial position of the target object, and finally achieve the three-dimensional spatial positioning of the inspected object; Specifically, in Step 3, 2*W*H triangular patches are constructed using this DEM data block, where W and H are the length and width of the DEM data block respectively. The DEM data block is a TIFF format digital image, and the GDAL library (Geospatial Data Abstraction Library) is used to align and parse the image data, hereinafter referred to as the DEM image; each pixel of the DEM image corresponds to two small triangular patches. The first patch is composed of the upper left corner, upper right corner, and lower left corner of the pixel, and the second patch is composed of the upper right corner, lower right corner, and lower left corner of the pixel; the X and Y coordinates of the vertices of these two patches are determined according to the two-dimensional coordinates of this pixel in the image, the longitude and latitude information, and the scale of this DEM image, while the height of the vertices of these two patches is determined by the value of the DEM image at this pixel; Denote the set of all triangular patches of the three-dimensional surface grid constructed from this DEM data block as {T i}; these triangular patches provide a low-cost three-dimensional positioning data basis for three-dimensional spatial target positioning; Step 6 specifically divides the entire intersection process into three layers: First, the entire DEM image is divided into squares of size 256*256. The original DEM image with a size of 4096*4096 is divided into 4096*4096 / (256*256) = 256 squares, generating a total of 2*256 = 512 triangular patches, denoted as {Mi}; then, the intersection points of the ray L and {Mi} are found, and the number of intersection calculations is 512; assume that the ray L intersects K1 patches {Mk}. Then, the 256*256 squares of these K1 {Mk} triangular patches are further divided into 16*16 squares, also resulting in 256 small squares, forming 512 smaller triangular patches, denoted as {Ni}. The intersection points of the ray L and {Ni} are found, and the number of intersection calculations for each patch is also 512. Since there are K1 {Mk} triangular patches in total, the total number of intersection calculations is 512*K1; finally, assume that the ray L intersects K2 of the patches {Nk}. The 16*16 squares and 512 smallest triangular patches {Pi} are used to find the intersection with the ray L, and the intersection points are denoted as {Pk}; the coordinates of the point with the highest elevation among {Pk} are the three-dimensional spatial position of the target object, and the number of intersection calculations is 512*K1*K2. Therefore, the total number of intersection calculations in this hierarchical and progressive process is: 512 + 512*K1 + 512*K1*K2.

Images