Real-time positioning method of unmanned aerial vehicle oblique image fused with onboard LiDAR point cloud

Abstract

The application discloses a kind of real-time positioning methods of unmanned aerial vehicle oblique image of fusion airborne LiDAR point cloud, will unmanned aerial vehicle airborne LiDAR point cloud data rasterization and image processing, generate digital surface model;With the real-time POS data of unmanned aerial vehicle oblique image, after unmanned aerial vehicle oblique image is corrected to horizontal image, using iterative intersection method, realize the geometric fusion of unmanned aerial vehicle oblique image and airborne LiDAR point cloud data image, realize the real-time positioning of unmanned aerial vehicle oblique image.The application has the advantages that by rasterization and image of unmanned aerial vehicle airborne LiDAR point cloud data, generate digital surface model;With the real-time POS data of unmanned aerial vehicle, the geometric fusion of unmanned aerial vehicle oblique image and airborne LiDAR point cloud is realized;By pixel-by-pixel solution, realize the whole frame positioning of unmanned aerial vehicle oblique image.Compared with existing technical method, the application is high in real-time, need not to lay out ground control point, also need not to construct solid image pair, avoid time-consuming image feature matching and optical flow matching etc.

Description

Technical Field

[0001] This invention relates to the field of image positioning technology, and in particular to a real-time positioning method for UAV tilted images based on fused airborne LiDAR point clouds. Background Technology

[0002] In recent years, some regions have experienced severe geological disasters such as floods, landslides, mudslides, and ground subsidence, causing significant losses to people's lives and property. Drones, acting as "safety sentinels," can conduct safety inspections, hazard assessments, emergency mapping, early warnings, and evaluations in areas prone to geological disasters, and will play an increasingly important role with technological advancements. During operations, drones can acquire, identify, and locate key target areas by capturing real-time images.

[0003] Currently, existing image localization methods mainly include the following four types: photogrammetry, binocular vision, optical flow, and point cloud and image feature matching.

[0004] (1) Photogrammetry first involves using a UAV to capture a sequence of aerial images with a certain degree of overlap; then, using collinear / coplanar equations with the aid of image matching technology, relative positioning between adjacent aerial images is achieved; finally, using network adjustment technology with the aid of a certain number of ground control points, absolute positioning of the aerial images is completed. Currently, photogrammetry can automatically map large-scale topographic maps with an accuracy of 5cm. However, photogrammetry is a post-processing method, which is difficult to meet the real-time requirements of applications such as UAV safety inspection or emergency mapping.

[0005] (2) The binocular vision method uses existing calibration methods (DLT, PnP, and PST methods, etc.) to calibrate a fixed binocular camera to determine its precise position and orientation. Then, image matching technology is used to perform feature matching on two overlapping images captured by the binocular camera. Finally, using the principle of similar triangles, the three-dimensional coordinates of the object points corresponding to the same image points on the two images are intersected, thus achieving absolute positioning of the image. Currently, the binocular vision method can achieve real-time absolute positioning of images. However, the binocular vision method requires the position and orientation of the binocular camera to be strictly fixed, resulting in a limited field of view, which cannot meet the motion requirements and large-scene requirements of UAV safety inspection or emergency mapping.

[0006] (3) Optical flow method uses a video camera to acquire a sequence of highly overlapping video image frames; then, under certain assumptions (small displacement, consistent brightness, and consistent regional motion, etc.), an optical flow model is constructed; finally, the constructed optical flow model is used to calculate or track the displacement of specific points (regions) in the video frames, thereby achieving relative positioning between adjacent images. Currently, optical flow method has been widely used in motion tracking, image stitching, medical modeling and other fields. However, optical flow method is a relative positioning method and cannot meet the absolute positioning requirements of UAV safety inspection or emergency mapping.

[0007] (4) The feature matching method between point cloud and aerial imagery is to acquire point cloud data and aerial imagery data of the target survey area; use information such as the height and reflection intensity of the point cloud as grayscale values ​​to convert the disordered three-dimensional point cloud data into an ordered two-dimensional point cloud image; and then use image matching technology to achieve feature matching between the point cloud imagery and the aerial imagery, thereby realizing the positioning of the aerial imagery. At present, the feature matching method between point cloud and aerial imagery is still in the preliminary research stage, and the matching accuracy and matching stability cannot meet the reliability requirements of UAV safety inspection or emergency mapping.

[0008] In summary, existing image positioning methods are insufficient to meet the real-time, absolute, motion-dependent, and reliable requirements of image positioning in application scenarios such as UAV safety inspection or emergency mapping. Summary of the Invention

[0009] The purpose of this invention is to provide a real-time positioning method for UAV tilted images by fusing airborne LiDAR point clouds to address the problem of real-time "positioning" of UAV tilted images.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: The present invention discloses a real-time positioning method for UAV oblique imagery fused with airborne LiDAR point cloud. The method involves rasterizing and image-processing the UAV airborne LiDAR point cloud data to generate a digital terrain model. Using real-time POS data of the UAV oblique imagery, the UAV oblique imagery is corrected to a horizontal image. Then, using an iterative intersection method, the geometric fusion of the UAV oblique imagery and the airborne LiDAR point cloud data imagery is achieved, thereby realizing real-time positioning of the UAV oblique imagery.

[0011] Furthermore, the rasterization of the airborne LiDAR point cloud data specifically involves: comprehensively considering the complexity of the terrain, the density of the point cloud, and the positioning accuracy to determine the row width and column width of a single raster, and establishing a regular two-dimensional raster network; determining the raster corresponding to each point cloud data based on the horizontal coordinate values ​​of the airborne LiDAR point cloud data, and storing the data in the corresponding raster.

[0012] Furthermore, the image processing specifically involves determining the grayscale value of each raster based on the cloud point data surrounding each raster to be imaged, specifically including: For a raster to be imaged containing 0 point cloud data, take the raster to be imaged as the center raster and search for point cloud data in all rasteres in 8 directions around the center raster in turn; calculate the expected value and standard error of the gray value of the center raster based on the search results, and then determine the gray value of the center raster. For a raster to be imaged that contains two or fewer point cloud data points, the search is performed around the central raster. Point cloud data in all grids within the range; if the total number of point cloud data found is less than the set threshold, then the gray value of the grid to be imaged is determined according to the step that the grid to be imaged contains 0 point cloud data; otherwise, the gray value of the central grid is determined by calculating the expected gray value of the central grid and the mean square error according to the two-dimensional Gaussian function. For a raster to be imaged that contains more than two point cloud data, all point cloud data in the raster to be imaged are sorted by elevation value, and the median elevation value in the sequence of all point cloud data elevation values ​​is taken as the gray value of the raster to be imaged.

[0013] Furthermore, the search results specifically refer to the following: if a raster containing more than two point cloud data points is found in the search direction, the search is stopped and the raster containing more than two point cloud data points is used as the point cloud data in the search direction; if no raster containing more than two point cloud data points is found in the search direction, the point cloud data in the search direction is empty; if more than four search directions have empty point cloud data points, the central raster is the map edge, and it is not necessary to determine the grayscale value of the central raster.

[0014] Furthermore, the geometric fusion of UAV oblique imagery and airborne LiDAR point cloud data imagery using the iterative intersection method specifically involves: S1, using real-time POS data from the UAV tilted image, obtain the coordinates of the center point S of the tilted image captured by the UAV and the principal point O of the corrected horizontal image. N The coordinates; S2, starting from point S and passing through point O N Draw a perpendicular line that intersects the digital surface model at point N, and determine the horizontal coordinates of point N through projection relationships; S3, determine the gray value Z of the grid corresponding to point N based on the horizontal coordinate of point N. N ; Calculate the length H of the perpendicular line SN SN ; S4, select any point a on the drone's tilted image. t Determine point a t The corresponding point a in the corrected horizontal imagev The coordinates; S5, taking the shooting center S as the starting point, passing through point a t Draw a ray that intersects the digital surface model at point A; S6. Draw a horizontal line starting from point N, intersecting the ray at point A0. Determine the coordinates of point A0 based on the principle of similar triangles. S7. Draw a perpendicular line from point A0 to the digital surface model, intersecting at point G1. Determine the horizontal coordinates of point G1 through projection relationship. S8, draw a horizontal line starting from point G1, intersecting the ray at point A1, and intersecting the extension of the perpendicular line SN at point N1; S9, determine the gray value Z of the grid corresponding to point G1 based on the horizontal coordinate of point G1. G1 ; Calculate the length H of the perpendicular line SN1 SN1 ; S10, Determine the coordinates of point A1 based on the principle of similar triangles; S11, Repeat steps S7-S10 until point A. i With point A i-1 If the height difference is less than the preset threshold, then A will be... i The coordinates are considered as the coordinates of point A.

[0015] The advantages of this invention lie in generating a digital terrain model by rasterizing and image-encoding UAV-borne LiDAR point cloud data; achieving geometric fusion of UAV oblique imagery and onboard LiDAR point cloud using real-time UAV POS data; and realizing whole-frame localization of UAV oblique imagery through pixel-by-pixel calculation. Compared to existing technologies, this invention offers high real-time performance, eliminates the need for ground control points and stereo image pairs, and avoids time-consuming operations such as image feature matching and optical flow matching.

[0016] Meanwhile, this invention boasts high stability and real-time positioning accuracy, fully considering processes such as point cloud data denoising, point cloud fitting, and oblique image correction, thus avoiding matching errors caused by factors like lighting, motion, occlusion, and similar ground features present in computer vision algorithms. In UAV safety inspections and emergency mapping in hilly areas, urban areas, river channels, and various areas with significant elevation differences, the calculation results are stable and reliable, achieving UAV oblique imagery with an accuracy of 0.5m within 0.38s. Full-frame positioning (pixels).

[0017] Finally, this invention does not rely on any pre- or post-processing software and can be entirely independently written on the C++ platform according to the method described herein, exhibiting strong extensibility and scalability. It can quickly meet the application needs of different related scenarios. Attached Figure Description

[0018] Figure 1 It is a rasterization schematic diagram of airborne LiDAR point cloud in the method described in the present invention.

[0019] Figure 2 It is a schematic diagram of searching along the "rice" shape in the method described in the present invention.

[0020] Figure 3 It is a raster schematic diagram to be imaged for the point cloud data containing 2 or less in the method described in the present invention.

[0021] Figure 4 It is a principle schematic diagram of the iterative intersection method in the method described in the present invention.

[0022] Figure 5 It is a specific implementation flowchart in the method described in the present invention.

[0023] Figure 6 It is a schematic diagram of the imaging application effect of airborne LiDAR point cloud in the method described in the present invention.

[0024] Figure 7 It is a schematic diagram of the application effect of the method described in the present invention in hilly areas.

[0025] Figure 8 It is a schematic diagram of the application effect of the method described in the present invention in urban areas.

[0026] Figure 9 It is a schematic diagram of the application effect of the method described in the present invention in river areas.

[0027] Figure 10 It is a schematic diagram of the application effect of the method described in the present invention in terrain with large height differences. Specific Embodiments

[0028] Next, the technical solutions in the embodiments of the present invention will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] Embodiment 1: Detailed Introduction of the UAV Oblique Image Real-Time Positioning Method Integrating Airborne LiDAR Point Cloud Described in the Present Invention This invention discloses a real-time localization method for UAV oblique imagery fused with airborne LiDAR point clouds. The method involves rasterizing and image-processing the UAV-borne LiDAR point cloud data to generate a digital terrain model. Using real-time POS data from the UAV oblique imagery, the oblique imagery is corrected to a horizontal image. Then, using an iterative intersection method, geometric fusion of the UAV oblique imagery and the airborne LiDAR point cloud data is achieved, enabling real-time localization of the UAV oblique imagery. Specifically: First, the unordered airborne LiDAR point cloud needs to be rasterized for storage to facilitate organization and management. For example... Figure 1 As shown, the algorithm steps are as follows: First, establish a regular two-dimensional grid. Among these, The origin of the coordinate system for the two-dimensional grid; Indicates the row number. Indicates the number of columns; the row width of a single grid is defined as... Column width is defined as The values ​​for the row width and column width of a single raster can be set by comprehensively considering factors such as the complexity of the terrain, the density of the point cloud, and the accuracy requirements of the product.

[0030] Second, rasterize and store the airborne LiDAR point cloud. Based on the horizontal coordinate values ​​of the airborne LiDAR point cloud... Determine the raster row and column numbers corresponding to each point, and store them sequentially in the corresponding raster. The raster row and column numbers corresponding to each point are determined by equation (1), i.e. (1) in, Indicating the first point cloud The row and column numbers of each point , This represents the total number of points in the point cloud; Indicating the first point cloud The planar coordinates of each point; Indicating point cloud data Minimum value in direction, Indicating point cloud data The maximum value in the direction; This indicates the column width of a single grid cell. This indicates the row width of a single grid cell.

[0031] Secondly, the UAV-borne LiDAR point cloud data is imaged, that is, based on the cloud point data surrounding each raster to be imaged, the grayscale value of each raster is determined, and a digital land surface model is generated; the specific algorithm steps are as follows: Due to the deficiencies of the airborne LiDAR system itself and the interference of the external environment, the acquired original point cloud data has disorder and non-uniformity, and contains a small number of noise points. Therefore, the rasterized point cloud data still cannot meet the application requirements and needs to be further imaged.

[0032] According to the distribution of the point cloud data in the grid network and considering the requirements of denoising and fitting, the present invention proposes three types of imaging methods for raster point clouds, namely: the number of point cloud data in a single raster to be imaged is 0, the number of point clouds in a single raster to be imaged is 2 or less, including two scenarios of 1 and 2; the number of point clouds in a single raster to be imaged is more than 2.

[0033] Due to reasons such as the non-uniform distribution of point clouds, the refraction of water surfaces, and the absorption of artificial materials, there will be many grids in the grid network that do not contain point cloud data.

[0034] First, for the raster to be imaged that contains 0 point cloud data, the present invention proposes a "cross" - shaped search and denoising method to image the raster that does not contain point cloud data, as Figure 2 shown.

[0035] Taking the raster to be imaged as the central raster, sequentially search for the point cloud data in all grids in 8 directions around the central raster along the "cross" - shaped; calculate the expected value of the gray value and the mean square error of the central raster according to the search results, and then determine the gray value of the central raster; During the search process, if a grid containing more than 2 point cloud data is searched in a certain search direction, stop the search and use the grid containing more than 2 point cloud data as the point cloud data in that search direction; if a grid containing more than 2 point cloud data is not searched in a certain search direction, the point cloud data in that search direction is empty; if the point cloud data in more than 4 search directions is empty, the central raster is the map edge and there is no need to determine the gray value of the central raster.

[0036] Calculate the expected value of the gray value of the central raster, and the formula is as follows: (2) Where, represents the expected value of the gray value of the central raster; represents the search direction, that is, 8 directions of the "cross" - shaped; represents the direction the number of point clouds searched in the direction; represents the elevation of the th point in the point cloud set searched in the direction; represents the weight value of the point in the point cloud set searched in the direction, ; Indicates the right of direction. express The row and column numbers of the raster containing the point cloud found in the direction search. Indicates the row and column number of the center grid; Indicates quantity weights, Indicates direction The number of point clouds found in the search.

[0037] The standard error is calculated using the following formula: (3) The formula for calculating the grayscale value of the center grid is as follows: (4) in, This represents the grayscale value of the center grid, which is also the elevation fit value of the center grid. This indicates a constraint, meaning that only those conditions are met. Participate in the calculation of the grayscale value of the center grid.

[0038] Second, for raster cells to be imaged that contain two or fewer point cloud data points, i.e., a single raster cell to be imaged contains one or two point cloud data points, the specific method is as follows: Figure 3 As shown.

[0039] Using the raster to be imaged as the central raster, search the area surrounding the central raster. Point cloud data in all grids within the range; if the total number of point cloud data found is less than the set threshold, then the gray value of the grid to be imaged is determined according to the step that the grid to be imaged contains 0 point cloud data; otherwise, the gray value of the central grid is determined by calculating the expected gray value of the central grid and the mean square error according to the two-dimensional Gaussian function. The expected gray value of the center grid is calculated using a two-dimensional Gaussian function, as shown in the following formula: (5) in, Indicates the row and column number of the center grid; Indicates the search scope, which can be customized. It can be set to... ; Indicates the row and column numbers of the raster within the search range; Indicates the first Line 1 The number of point clouds in the raster of a column; Indicates the first Line 1 The point cloud set searched in the column of the raster is the first Elevation of each point; Represents a two-dimensional Gaussian function. , The variance of a two-dimensional Gaussian function can be expressed as: .

[0040] The standard error is calculated using the following formula: (6) The formula for calculating the grayscale value of the center grid is as follows: (7) in, This represents the grayscale value of the center grid, which is also the elevation fit value of the center grid. This indicates a constraint, meaning that only those conditions are met. Participate in the calculation of the grayscale value of the center grid.

[0041] Third, for raster cells to be imaged that contain more than two point cloud data points, i.e., a single raster cell to be imaged contains three or more point cloud data points.

[0042] When a single raster to be imaged contains three or more point cloud data points, in order to express the elevation of the ground more quickly, and considering the needs of point cloud denoising and fitting, this invention uses median filtering to image the raster. Specifically, all point cloud data within the raster to be imaged are sorted by elevation value, and the median of the elevation values ​​in the sequence of all point cloud data elevation values ​​is taken as the grayscale value of the raster to be imaged, as expressed by the following formula.

[0043] (8) in, This represents the grayscale value of the center grid, which is also the elevation fit value of the center grid. Represents the elevation median filter function; Indicates the row and column numbers of the raster; This represents the set of point cloud elevations within a raster.

[0044] Finally, using real-time POS data from the UAV oblique imagery, the UAV oblique imagery is corrected to a horizontal image. Then, using an iterative intersection method, geometric fusion of the UAV oblique imagery and the airborne LiDAR point cloud data is achieved, enabling real-time localization of the UAV oblique imagery. Specifically: Using real-time POS data from drone imagery, the tilted images captured by the drone are corrected to be horizontal, as expressed by the formula: (9) in, This represents the image plane coordinates of an image point on an oblique image. This represents the image plane coordinates of the corresponding image point on the horizontal image; Indicates the camera's focal length; These represent pitch angle, roll angle, and yaw angle, respectively, provided by real-time POS data from UAV imagery; , , ; Expanding equation (9), we get (10) in, This allows us to calculate the image plane coordinates of the horizontal image corresponding to the image plane coordinates of the tilted image of the UAV, thus enabling the correction of the tilted image of the UAV into a horizontal image.

[0045] The specific principle of geometric fusion of UAV oblique imagery and airborne LiDAR point cloud data using the iterative intersection method is as follows: Figure 4 As shown. Here, Ground represents the digital terrain model, provided by point cloud imagery. Point S represents the center of the oblique image captured by the drone, and its three-dimensional coordinates are... This can be provided by real-time POS data from drone tilted imagery. t P represents a tilted image taken by a drone. v This represents the horizontal image corresponding to the tilted image captured by the drone; through the process described above of correcting the tilted image captured by the drone to a horizontal image, the tilted image P can be obtained. t The horizontal image coordinates corresponding to any point.

[0046] Figure 4 Midpoint O represents the principal point of the oblique image, located at P. t Above, and the straight line SO⊥P t The corresponding point of point O on the horizontal image is point O. N Point O N Located at P v Above, and the straight line SO N ⊥P v Point O can be determined based on its three-dimensional coordinates. N The three-dimensional coordinates. Simultaneously, using UAV oblique imagery P... t any image point a on t The three-dimensional coordinates can also determine point a t In horizontal image P v The corresponding image point a on v The three-dimensional coordinates of the UAV tilted imagery. Geometric fusion of UAV tilted imagery and airborne LiDAR point cloud data aims to determine the three-dimensional coordinates of the UAV tilted imagery P. t any image point a on t The coordinates of point A corresponding to the digital land model, and points S and a v a tPoints A, A, and B are collinear.

[0047] The iterative intersection method used in this invention determines the UAV tilt image P. t any image point a on t The coordinates of point A corresponding to the digital land model, starting from point S and passing through point O. N Draw a perpendicular line, intersecting the digital surface model at point N. Based on the relationship of vertical projection, the X-axis and Y-axis coordinates of point N can be determined, i.e., ... Simultaneously, starting from point S, the image P from the UAV tilted viewpoint... t any image point a on t As a ray Sa t Then, draw a horizontal line starting from point N, intersecting the ray at point A0. Determine the grayscale value Z of the corresponding grid cell using the X-axis and Y-axis coordinates of point N. N Calculate the length of the perpendicular line SN. Then, as follows: Figure 4 As shown, according to triangle SO N N and triangle Sa v Based on the principle of similarity to A0, the three-dimensional coordinates of A0 are calculated. Then, a perpendicular line is drawn from point A0, intersecting the digital surface model at point G1. The horizontal coordinates of point G1 are determined through projection relationships. Next, a horizontal line is drawn from point G1, intersecting ray Sa... t The lines intersect at point A1 and the extension of the perpendicular line SN at point N1. Since G1 and N1 are on the same horizontal line, the horizontal coordinates of N1 can be determined based on the horizontal coordinates of G1. Therefore, the gray value Z of the grid corresponding to point N1 can be determined by the horizontal coordinates of point N1. N1 Calculate the length of the perpendicular SN1. Continue according to triangle SO. N N1 and triangle Sa v Using the principle of similarity at A0, the 3D coordinates of A1 are calculated. This process is iterated repeatedly until the coordinates of point A in two consecutive calculations are found. i With point A i-1 After the height difference is less than the preset threshold, A i The coordinates are considered as the tilted image P of the UAV. t any image point a on t The coordinates of point A corresponding to the digital land model.

[0048] The specific steps are described below: S1, using real-time POS data from the drone's oblique image, obtain the coordinates of the center point S of the oblique image captured by the drone. and the principal point O of the corrected horizontal image N The coordinates; S2, starting from point S and passing through point O NDraw a perpendicular line, intersecting the digital surface model at point N, and determine the horizontal coordinates of point N through projection relationships. The actual coordinates are... ; S3, determine the gray value Z of the grid corresponding to point N based on the horizontal coordinate of point N. N ; Calculate the length H of the perpendicular line SN SN ; (11) This represents the elevation retrieval function, which retrieves the grayscale (elevation) value of the corresponding raster in a point cloud image based on the horizontal coordinates.

[0049] S4, select any point a on the drone's tilted image. t Determine point a t The corresponding point a in the corrected horizontal image v The coordinates; S5, taking the shooting center S as the starting point, passing through point a t Draw a ray that intersects the digital surface model at point A; S6. Draw a horizontal line starting from point N, intersecting the ray at point A0. Determine the coordinates of point A0 based on the principle of similar triangles; that is... (12) in, Point 3D coordinates; Indicates horizontal image On point The image plane coordinates can be obtained according to equation (10); The focal length of the camera; Indicates the initial elevation difference. , It can be obtained from equation (11); For the photography center 3D coordinates; The meaning of reference formula (11).

[0050] S7. Draw a perpendicular line from point A0 to the digital surface model, intersecting at point G1. Determine the horizontal coordinates of point G1 through projection relationship. S8, draw a horizontal line starting from point G1, intersecting the ray at point A1, and intersecting the extension of the perpendicular line SN at point N1; S9, determine the gray value Z of the grid corresponding to point G1 based on the horizontal coordinate of point G1. G1 ; Calculate the length H of the perpendicular line SN1 SN1 ; S10, based on the principle of similar triangles, determine the coordinates of point A1; that is... (13) in, Point 3D coordinates; Indicates horizontal image On point The image plane coordinates can be obtained according to equation (10); The focal length of the camera; Indicates the iterative elevation difference. ; For the photography center 3D coordinates; Transition point The horizontal coordinates are obtained from equation (12); The meaning of reference formula (11).

[0051] S11, Repeat steps S7-S10 until point A. i With point A i-1 If the height difference is less than the preset threshold, then A will be... i The coordinates are considered as the coordinates of point A.

[0052] By performing step-by-step calculations on all pixels in the UAV tilted image, the three-dimensional coordinates of the ground point corresponding to each pixel in the UAV tilted image can be obtained, thereby achieving whole-frame positioning of the UAV tilted image.

[0053] Example 2: Implementation process of the real-time localization method for UAV tilted images fused with airborne LiDAR point clouds as described in this invention like Figure 5 As shown, in step 501, the airborne LiDAR point cloud data is rasterized and stored. First, a regular two-dimensional grid is constructed; then, the airborne LiDAR point cloud data is rasterized and stored, that is, the airborne LiDAR point cloud is stored into the corresponding grid according to its horizontal coordinates.

[0054] In step 502, the airborne LiDAR point cloud data stored in raster form is imaged. The raster point cloud is imaged in three cases: the number of points in a single raster is 0, the number of points in a single raster is 1 or 2, and the number of points in a single raster is 3 or more.

[0055] In step 503, the tilted image of the UAV is corrected to a horizontal image using real-time POS data.

[0056] In step 504, real-time POS data is used to achieve geometric fusion of UAV tilted images and airborne LiDAR point cloud images.

[0057] In step 505, the three-dimensional coordinates of the ground points corresponding to the unmanned tilted image are calculated pixel by pixel, thereby realizing the whole-frame positioning of the UAV tilted image.

[0058] In step 506, it is determined whether to perform real-time positioning calculation on the new UAV tilt image; if yes, proceed to step 503; otherwise, end.

[0059] Example 3: Specific application of the UAV tilt image real-time localization method based on airborne LiDAR point cloud as described in this invention Based on steps 501 and 502, the airborne LiDAR point cloud of a certain survey area is rasterized and imaged, such as... Figure 6 As shown in the figure, the black area represents the edge of the survey area and does not contain point cloud data; the gray area represents the interior of the survey area, contains point cloud data, and the brightness of the gray area represents the terrain elevation. Figure 6 (a) is a schematic diagram of the rasterization result of the airborne LiDAR point cloud, which contains a large number of empty areas that do not contain point clouds; Figure 6 (b) is a schematic diagram of the image of the airborne LiDAR point cloud. There are no empty areas in the image, and the overall terrain shows a smooth trend, which is in line with objective laws.

[0060] Based on steps 503, 504, and 505, geometric fusion is performed on the UAV tilted image and the airborne LiDAR point cloud image. The whole-frame localization result is as follows: Figure 7 , Figure 8 , Figure 9 and Figure 10 As shown.

[0061] Figure 7 It is a hilly area. Among them, Figure 7 (a) is an oblique image taken by a drone; Figure 7 (b) Use existing high-precision DOM imagery as a reference ground truth to evaluate the positioning accuracy of UAV tilt imagery; Figure 7 (c) is the overlay effect of the localized UAV tilted image and the high-precision DOM image, that is, the effect of overlaying the localized tilted image (gray area) and the high-precision DOM image (colored area) together according to coordinates in ArcGIS. Measurements show that the positional error of the corresponding points between the localized UAV tilted image and the high-precision DOM is approximately 0.5m, and the entire frame ( The resolution time (per pixel) is approximately 0.38 seconds.

[0062] Figure 8 Urban areas. Among them, Figure 8 (a) is an oblique image taken by a drone; Figure 8(b) Use existing high-precision DOM imagery as a reference ground truth to evaluate the positioning accuracy of UAV tilt imagery; Figure 8 (c) is an overlay image of the UAV tilted image and the high-precision DOM image after localization. Measurements show that the positional error between the corresponding points in the localized UAV tilted image and the high-precision DOM is approximately 0.5m, and the UAV tilted image ( The full frame resolution time (pixels) is approximately 0.37 seconds.

[0063] Figure 9 This is a riverine area. Among them, Figure 9 (a) is an oblique image taken by a drone; Figure 9 (b) Use existing high-precision DOM imagery as a reference ground truth to evaluate the positioning accuracy of UAV tilt imagery; Figure 9 (c) is a superimposed image of the localized UAV tilted image and the high-precision DOM image. Measurements show that the positional error between the corresponding points in the localized UAV tilted image and the high-precision DOM is approximately 0.5m, and the entire frame ( The resolution time (per pixel) is approximately 0.39 seconds.

[0064] Figure 10 It is a region with significant elevation differences. Among them, Figure 10 (a) is an oblique image taken by a drone; Figure 10 (b) Use existing high-precision DOM imagery as a reference ground truth to evaluate the positioning accuracy of UAV tilt imagery; Figure 10 (c) is a superimposed image of the localized UAV tilted image and the high-precision DOM image. Measurements show that the positional error between the corresponding points in the localized UAV tilted image and the high-precision DOM is approximately 0.5m, and the entire frame ( The resolution time (per pixel) is approximately 0.38 seconds.

[0065] The above experiments demonstrate that the present invention can robustly reproduce single-frame UAV tilted images with an accuracy of approximately 0.5m in any region (hilly areas, towns, river channels, and areas with large elevation differences). The system performs real-time (approximately 0.38s) positioning using pixels.

Claims

1. A real-time localization method for UAV oblique imagery fused with airborne LiDAR point clouds, characterized in that: The UAV-borne LiDAR point cloud data is rasterized and imaged to generate a digital land model. Using real-time POS data from the UAV oblique imagery, the UAV oblique imagery is corrected to a horizontal image. Then, using the iterative intersection method, the geometric fusion of the UAV oblique imagery and the airborne LiDAR point cloud data imagery is achieved, enabling real-time positioning of the UAV oblique imagery. The specific steps of the airborne LiDAR point cloud data rasterization are as follows: determining the row width and column width of a single raster, establishing a regular two-dimensional raster network; determining the raster corresponding to each point cloud data based on the horizontal coordinate values ​​of the airborne LiDAR point cloud data, and storing the data in the corresponding raster. The image processing specifically involves determining the grayscale value of each raster based on the point cloud data surrounding each raster to be imaged, specifically including: For a raster to be imaged containing 0 point cloud data, take the raster to be imaged as the center raster and search for point cloud data in all rasteres in 8 directions around the center raster in turn; calculate the expected value and standard error of the gray value of the center raster based on the search results, and then determine the gray value of the center raster. For a raster to be imaged containing one or two point cloud data points, the search is performed around the raster to be imaged, with the raster to be imaged as the central raster. Point cloud data in all grids within the range; if the total number of point cloud data found is less than the set threshold, then the gray value of the grid to be imaged is determined according to the step that the grid to be imaged contains 0 point cloud data; otherwise, the gray value of the central grid is determined by calculating the expected gray value of the central grid and the mean square error according to the two-dimensional Gaussian function. For a raster to be imaged that contains three or more point cloud data, all point cloud data in the raster to be imaged are sorted by elevation value, and the median elevation value in the elevation value sequence of all point cloud data is taken as the gray value of the raster to be imaged. The specific method for geometrically fusing UAV oblique imagery and airborne LiDAR point cloud data using the iterative intersection method is as follows: S1, using real-time POS data of the unmanned aerial vehicle oblique image, obtaining coordinates of a center point S of the unmanned aerial vehicle oblique image and an image principal point O of the corrected horizontal image N ; S2, passing through point O with point S as the starting point N drawing a perpendicular line to intersect the digital terrain model at point N, and determining the horizontal coordinates of point N through a projection relationship; S3, determine the gray value Z of the grid corresponding to point N based on the horizontal coordinate of point N. N ; Calculate the length H of the perpendicular line SN SN ; S4, select any point a on the drone's tilted image. t Determine point a t The corresponding point a in the corrected horizontal image v The coordinates; S5, taking the shooting center S as the starting point, passing through point a t Draw a ray that intersects the digital surface model at point A; S6. Draw a horizontal line starting from point N, intersecting the ray at point A0. Determine the coordinates of point A0 based on the principle of similar triangles. S7. Draw a perpendicular line from point A0 to the digital surface model, intersecting at point G1. Determine the horizontal coordinates of point G1 through projection relationship. S8, draw a horizontal line starting from point G1, intersecting the ray at point A1, and intersecting the extension of the perpendicular line SN at point N1; S9, determine the gray value Z of the grid corresponding to point G1 based on the horizontal coordinate of point G1. G1 ; Calculate the length H of the perpendicular line SN1 SN1 ; S10, Determine the coordinates of point A1 based on the principle of similar triangles; S11, Repeat steps S7-S10 until point A. i With point A i-1 If the height difference is less than the preset threshold, then A will be... i The coordinates are considered as the coordinates of point A.

2. The method for real-time localization of UAV tilted images fused with airborne LiDAR point clouds according to claim 1, characterized in that: Specifically, the search results are as follows: if a grid containing more than two point cloud data points is found in the search direction, the search stops and the grid containing more than two point cloud data points is used as the point cloud data in the search direction; if no grid containing more than two point cloud data points is found in the search direction, the point cloud data in the search direction is empty; if more than four search directions have empty point cloud data points, the central grid is the map edge, and there is no need to determine the grayscale value of the central grid.

Images