A real-time incremental splicing method for unmanned aerial vehicle aerial images

Abstract

The application discloses a kind of unmanned aerial vehicle aerial image real-time incremental splicing methods, comprising: creating new map template;Listen to unmanned aerial vehicle aerial image storage folder, real-time update image splicing sequence;Setting splicing route parameters;Image splicing;New map generation;Center offset correction.The application can be realized in unknown area environment fast, real-time, incremental, robust, efficient long distance, multiple band unmanned aerial vehicle aerial image splicing.

Description

Technical Field

[0001] This invention relates to unmanned aerial vehicle (UAV) technology, specifically to a method for real-time incremental stitching of UAV aerial images. Background Technology

[0002] Rapidly stitching together aerial image sequences captured by UAVs to construct panoramic images has significant application value and research implications. It can be used in both military (e.g., aerial reconnaissance, border patrol) and civilian (e.g., aerial mapping, aerial traffic surveillance, disaster reconnaissance). In many specific tasks, constructing panoramic images through rapid image stitching technology is crucial for expanding the field of view and obtaining more comprehensive information. Therefore, image stitching technology based on UAV aerial image sequences has become a major research hotspot.

[0003] The process of stitching together drone aerial images mainly involves four steps: image geometric correction, image preprocessing, image registration, and image fusion.

[0004] Image geometric correction. Due to the small size and poor self-stability of drones, tilting and shaking are inevitable during the shooting process. Therefore, the first step is to perform geometric correction on the aerial images captured by the drone to correct various distortions such as edge distortion and center offset that occurred during the imaging process, and generate a new image that meets the stitching requirements.

[0005] Image preprocessing. Image preprocessing refers to unifying the images to be stitched into the established coordinate system.

[0006] Image registration. Image registration refers to extracting matching information between images, finding the best matching pair from the extracted information according to the corresponding matching strategy, and using this as the basis for aligning and unifying the images.

[0007] Image fusion. Image fusion refers to the process of mosaicking images after image registration and smoothing the boundaries to create a natural transition between the images.

[0008] Image geometric correction is particularly difficult. If the image distortion is severe, it will affect the image stitching effect. The key to successful image stitching is image registration.

[0009] Currently, the mainstream image stitching algorithms are mainly based on feature extraction and matching methods. They use deep learning algorithms or feature descriptors to detect and extract key feature points of the image, and then match the key feature points between the images according to a specific matching strategy. The corresponding homography matrix is ​​calculated, thereby transforming multiple images into the same field of view and realizing image stitching.

[0010] Numerous studies have been conducted both domestically and internationally on drone aerial image stitching technology. One image registration algorithm is based on Fourier-Mellin transform and feature point matching. It uses Fourier-Mellin transform to calculate initial registration parameters, then maps the feature points extracted between images according to the initial registration parameters to complete the point matching and optimization process, obtaining one-to-one matching point pairs, and finally uses the least squares method to calculate the final registration parameters.

[0011] The ORB feature descriptor operator is used for feature extraction. A fast and robust feature matching strategy based on descriptor similarity is proposed. The spatial affine transformation between image sequences captured by UAV is modeled and a robust Bayesian framework is introduced to estimate the transformation. At the same time, a gradual fading method is proposed for image fusion.

[0012] A VGG-style regression Homography network has been proposed to learn the homography between two images. It uses a neural network to obtain an intermediate variable, Homography, and then uses Homography for registration, without having to separate feature point detection and transformation estimation.

[0013] A method for fast stitching of aerial image sequences taken by small UAVs has been proposed. This method uses bundle adjustment of a small number of ground control points to estimate the homography transformation between the image and the ground, calculates orthorectification parameters, and then corrects and merges the images based on the calculated orthorectification parameters to generate a stitched panoramic image of the entire scene.

[0014] Drone aerial image sequences are characterized by their large quantity, multiple strips, small phase amplitude between adjacent frames, and high overlap. Furthermore, due to the inevitable tilting and shaking during drone flight, the captured images exhibit affine distortion compared to the actual scene. Most existing stitching technologies are ill-suited to the demands of stitching drone aerial image sequences. As the flight distance increases and the number of images to be stitched grows, the stitching time skyrockets. Moreover, with the accumulation of errors, the subsequent stitching results become far from ideal, exhibiting severe distortion and failing to simultaneously meet the requirements of real-time performance, incremental processing, robustness, and accuracy. Summary of the Invention

[0015] The main objective of this invention is to provide a real-time incremental stitching method for UAV aerial images, thereby solving the problems existing in current UAV aerial image sequence stitching technology.

[0016] The technical solution adopted in this invention is: a method for real-time incremental stitching of UAV aerial images, comprising:

[0017] Create a new map template;

[0018] Monitor the folder where drone aerial images are stored and update the image stitching sequence in real time;

[0019] Set the splicing route parameters;

[0020] Image stitching;

[0021] New map generated;

[0022] Center offset correction.

[0023] Furthermore, the creation of the new map template includes:

[0024] Using the WGS 84 geographic coordinate system, based on the original map information or the given scanned area information, the affine transformation parameters of the new map are obtained: the latitude and longitude coordinates of the top left corner vertex of the new map and the latitude and longitude span parameters of the new map (different map accuracy levels can be set by adjusting the latitude and longitude span parameters as needed). Through the affine transformation parameters, the mutual conversion between pixel coordinates and latitude and longitude coordinates on the image can be realized. At the same time, the R, G, and B band matrices of the map are initialized with all black (0).

[0025] The latitude and longitude coordinates are converted to pixel coordinates as shown in the following formula:

[0026] col = (longitude - transform[0]) / trans[1]

[0027] row = (latitude - transform[3]) / trans[5]

[0028] Where col is the column coordinate of the pixel, row is the row coordinate of the pixel, longitude is the longitude value of the pixel, latitude is the latitude value of the pixel, and transform is the affine transformation parameter (transform[0] represents the longitude value of the upper left corner, transform[1] represents the horizontal resolution of the image, i.e., the longitude span of the pixel, transform[2] represents the rotation angle, which is 0 when facing due north, transform[3] represents the latitude value of the upper left corner, transform[4] represents the rotation angle, which is 0 when facing due north, and transform[5] represents the vertical resolution of the image, i.e., the latitude span of the pixel). In the above formula, the image faces due north, i.e., the values ​​of transform[2] and transform[4] are both 0.

[0029] Furthermore, the setting of the splicing route parameters includes:

[0030] The flight path angle parameters, adjacent frame movement distance parameters, and stitching image span parameters are set. For the current image to be stitched, the flight control data of the image and the previous two frames are combined to determine whether the image is currently flying in a straight line at a constant speed along the predetermined route. If so, the stitching is performed; otherwise, it is skipped, and invalid interference images of the drone before stitching and during turning are removed.

[0031] Furthermore, the image stitching includes:

[0032] Image rotation correction: The orientation of the images to be stitched is not always due north, but along the current flight path. This can be obtained from the uav_yaw parameter in the flight control data. Based on the uav_yaw angle, a rotation matrix is ​​constructed with the image center point as the rotation center. The images to be stitched are then losslessly corrected to face due north, and the missing background is filled with black (0,0,0).

[0033] The rotation matrix is ​​constructed as shown in the following equation:

[0034]

[0035] in, This is the rotation angle (clockwise is positive). Center of rotation;

[0036] Geographic information determination: SURF feature point calculation and matching are performed between the current image to be stitched and the previous image. KnnMatch matching strategy is adopted. According to the characteristics of UAV aerial image sequence, the lines between correctly matched feature points should be approximately parallel. RANSAC algorithm is used to remove incorrect matching points. Homography matrix is ​​calculated. The center point coordinates of the previous image are transformed into the current image to be stitched. Based on the pixel coordinates and latitude and longitude coordinates of the transformed points and the center point of the image to be stitched, the latitude and longitude span parameters of the image to be stitched are calculated.

[0037] The latitude and longitude span parameters are calculated as follows:

[0038]

[0039] in, The center point of the image to be stitched together Center of the image to be stitched

[0040] The latitude and longitude values ​​of the point To match the center point of the image for features, To match the latitude and longitude values ​​of the image center point for feature matching;

[0041] Image mosaicking: Based on the latitude and longitude span parameters of the image to be mosaicked and the pixel coordinates and latitude and longitude coordinates of the image center point, calculate the specific location of the image to be mosaicked on the new map, update the R, G, and B band matrix information of the region, and remove the mosaic seams.

[0042] Remove the seam as shown in the following formula:

[0043]

[0044] If pixel If the pixel is part of the stitching area, update the pixel value; otherwise, leave it unchanged.

[0045] Repeat the above steps until all images are stitched together.

[0046] Furthermore, the generation of the new map includes:

[0047] After all images are stitched together, the R, G, and B band matrix data are written into a new map file to generate the final mosaic result in GeoTIFF format.

[0048] Furthermore, the center offset correction includes:

[0049] During the mapping process, the drone should maintain flight stability, that is, the deviation of pitch and roll angle should be as small as possible, and the maximum deviation should not exceed 3°; finally, uniform offset correction should be performed according to the strip.

[0050] Advantages of this invention:

[0051] This invention enables rapid, real-time, incremental, robust, and efficient long-distance, multi-band UAV aerial image stitching in unknown environments.

[0052] The algorithm is modular, and the stitching effect can be improved by subsequently improving the feature extraction method or feature matching strategy without redesigning the algorithm process and other modules.

[0053] The algorithm can automatically remove invalid and interfering images based on the set stitching route parameters.

[0054] The image stitching effect is only related to adjacent frames, and will not introduce cumulative errors. Local areas do not affect each other.

[0055] The generated map file has a higher level of accuracy compared to the original map and can be used for other subsequent tasks.

[0056] The algorithm can correct the center offset to a certain extent, further improving the robustness of the algorithm.

[0057] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0058] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0059] Figure 1 This is an example diagram of image input according to the present invention;

[0060] Figure 2 This is a schematic diagram of the original map of the present invention;

[0061] Figure 3 This is the overall flowchart of the algorithm of this invention;

[0062] Figure 4 This is an example diagram of image rotation correction according to the present invention;

[0063] Figure 5 This is a diagram showing the SURF feature point calculation and matching results of the present invention;

[0064] Figure 6 This is the image mosaic of the present invention;

[0065] Figure 7 This is the image stitching result of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] The input required for this algorithm is a sequence of drone aerial images and their corresponding flight control data, such as... Figure 1 As shown. Every 0.5 seconds, a frame is extracted, and the aerial image transmitted back by the drone in real time is stored in the corresponding file path along with its corresponding flight control data, with consistent naming and alignment. The flight control data must include the latitude and longitude coordinates of the drone at the time of capture (i.e., the latitude and longitude coordinates of the center point of the aerial image), the camera's yaw angle, pitch angle, and roll angle at the time of capture. Additionally, the original map is required to specify the image stitching area, such as... Figure 2 As shown. However, the original map is not required; the image stitching area can also be specified by providing the latitude and longitude coordinates of the top left and bottom right corners of the image stitching area.

[0068] The overall process design of this algorithm is as follows: Figure 3 As shown.

[0069] (1) Create a new map template (new_map.tif) and adopt the “WGS 84” geographic coordinate system. Based on the original map information or the given scan area information (latitude and longitude coordinates of the upper left and lower right corners of the scan area (north facing upwards), obtain the affine transformation parameter information of the new map (north facing upwards): the latitude and longitude coordinates of the upper left corner vertex of the new map and the latitude and longitude span parameter of the new map (different map accuracy levels can be set by adjusting the latitude and longitude span parameter of the map as needed). The affine transformation parameter can realize the mutual conversion between pixel coordinates and latitude and longitude coordinates on the image. At the same time, initialize the R, G, and B band matrices of the map with all black (0).

[0070] The latitude and longitude coordinates are converted into pixel coordinates as shown in Equation (I):

[0071] col = (longitude - transform[0]) / trans[1]

[0072] row=(latitude-transform[3]) / trans[5] Equation (I)

[0073] Where col is the column coordinate of the pixel, row is the row coordinate of the pixel, longitude is the longitude value of the pixel, latitude is the latitude value of the pixel, and transform is the affine transformation parameter (transform[0] represents the longitude value of the upper left corner, transform[1] represents the horizontal resolution of the image, i.e., the longitude span of the pixel, transform[2] represents the rotation angle, which is 0 when facing due north, transform[3] represents the latitude value of the upper left corner, transform[4] represents the rotation angle, which is 0 when facing due north, and transform[5] represents the vertical resolution of the image, i.e., the latitude span of the pixel). In equation (1), the image faces due north, i.e., the values ​​of transform[2] and transform[4] are both 0.

[0074] (2) Monitor the folder where the drone aerial images are stored and update the image stitching sequence in real time to achieve real-time incremental stitching.

[0075] (3) Set the stitching route parameters: flight route angle parameters, adjacent frame movement distance parameters, and stitching image span parameters (stitching with a span of 10%). For the current image to be stitched, combine its flight control data with the previous two frames to determine whether it is currently flying in a straight line at a constant speed along the predetermined route. If so, stitching is performed; otherwise, skip it directly and remove invalid interference images of the UAV before stitching and during turning.

[0076] (4) Image stitching algorithm:

[0077] 1. Image rotation correction, such as Figure 4 As shown. The orientation of the images to be stitched is not always due north, but rather along the current flight path, which can be obtained from the uav_yaw parameter in the flight control data. Since the new map is oriented due north, a rotation matrix needs to be constructed based on the uav_yaw angle and the center point of the image as the rotation center. This matrix will then be used to non-destructively correct the images to be stitched to be oriented due north, and the missing background will be filled with black (0,0,0).

[0078] The rotation matrix is ​​constructed as shown in equation (II):

[0079] Formula (II)

[0080] in, This is the rotation angle (clockwise is positive). It is the center of rotation.

[0081] 2. Geographic Information Measurement. Perform SURF feature point calculation and matching between the current image to be stitched and the previous images (the overlap between the selected images and the image to be stitched should be around 70%). Figure 5 As shown, the KnnMatch matching strategy is adopted. Based on the characteristics of UAV aerial image sequences, the lines connecting correctly matched feature points should be approximately parallel. The RANSAC algorithm is used to remove incorrect matching points. The homography matrix is ​​calculated, and the coordinates of the center point of the previous image are transformed into the current image to be stitched. Based on the pixel coordinates and latitude and longitude coordinates of the transformed points and the center point of the image to be stitched, the latitude and longitude span parameters of the image to be stitched are calculated.

[0082] The latitude and longitude span parameters are calculated as shown in Equation (III):

[0083]

[0084] Formula (3)

[0085] in, The center point of the image to be stitched together Center of the image to be stitched

[0086] The latitude and longitude values ​​of the point To match the center point of the image for features, The latitude and longitude values ​​of the image center point are matched for feature matching.

[0087] 3. Image mosaicking, such as Figure 6As shown. Based on the latitude and longitude span parameters of the image to be stitched, and the pixel coordinates and latitude and longitude coordinates of the image center point, the specific location of the image to be stitched on the new map is calculated, and the R, G, and B band matrix information of the region is updated, while the stitching seams are removed.

[0088] Remove the seam as shown in formula (four):

[0089] Formula (IV)

[0090] If pixel If the pixel is part of the stitching area, update its pixel value; otherwise, leave it unchanged.

[0091] 4. Repeat steps 1 to 3 until all images are stitched together.

[0092] (5) New map generation. After all images are stitched together, the R, G, and B band matrix data are written into the new map file (new_map.tif) to generate the final stitched result, as shown below. Figure 7 As shown, the format is GeoTIFF.

[0093] (6) Center Offset Correction. During image scanning, the UAV should maintain stable flight, meaning the pitch and roll angle deviations should be as small as possible, with a maximum deviation not exceeding 3°. Otherwise, it indicates severe tilting and shaking of the UAV, resulting in significant center offset and edge distortion in the captured images. Compared to the real scene, there will be significant affine distortion, severely affecting the stitching effect. Practical experience shows that when the deviation angle is small, edge distortion is less pronounced and has less impact on the stitching effect. However, center offset (i.e., the latitude and longitude coordinates in the flight control data no longer correspond to the image center point) still affects the stitching effect between strips. Therefore, a unified offset correction is ultimately performed for each strip to further improve the stitching effect.

[0094] This invention addresses the problems existing in current UAV aerial image sequence stitching technology and the characteristics of UAV aerial image sequences by proposing an incremental real-time UAV aerial image sequence stitching algorithm.

[0095] Incremental real-time stitching. Stitching can begin while the drone is performing its scanning task, eliminating the need to input all the images at once, and the stitching process can be displayed in real time.

[0096] The algorithm is universal and robust. It can meet the requirements of stitching tasks in different regions and is not limited by flight time or flight distance.

[0097] Only drone aerial images and their corresponding flight control data are needed; no ground control points or other such data are required.

[0098] The final result is a GeoTIFF format map file, which can be used for other tasks (such as road network extraction, route planning, etc.), has a higher accuracy level than satellite maps, and the map accuracy level can be adjusted as needed.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time incremental stitching of UAV aerial images, characterized in that, include: Create a new map template; Monitor the folder where drone aerial images are stored and update the image stitching sequence in real time; Set the splicing route parameters; Image stitching; New map generated; Center offset correction; The parameters for setting the splicing route include: Set the flight path angle parameters, adjacent frame movement distance parameters, and stitching image span parameters. For the current image to be stitched, combine its flight control data with the previous two frames to determine whether it is currently flying in a straight line at a constant speed along the predetermined route. If so, stitching is performed; otherwise, skip it directly and remove invalid interference images of the drone before stitching and during turning. The center offset correction includes: During the mapping process, the drone should maintain flight stability, that is, the deviation of pitch and roll angle should be as small as possible, and the maximum deviation should not exceed 3°; finally, uniform offset correction should be performed according to the strip.

2. The real-time incremental stitching method for UAV aerial images according to claim 1, characterized in that, The process of creating a new map template includes: Using the WGS 84 geographic coordinate system, based on the original map information or the given scanned area information, the affine transformation parameters of the new map are obtained: the latitude and longitude coordinates of the top left corner vertex of the new map and the latitude and longitude span parameters of the new map. The affine transformation parameters can realize the mutual conversion between pixel coordinates and latitude and longitude coordinates on the image. At the same time, the R, G, and B band matrices of the map are initialized with all black 0. The latitude and longitude coordinates are converted to pixel coordinates as shown in the following formula: col = (longitude - transform[0]) / trans[1] row = (latitude - transform[3]) / trans[5] Where col is the column coordinate of the pixel, row is the row coordinate of the pixel, longitude is the longitude value of the pixel, latitude is the latitude value of the pixel, and transform is the affine transformation parameter. In the above formula, the image faces due north, that is, the values ​​of transform[2] and transform[4] are both 0.

3. The real-time incremental stitching method for UAV aerial images according to claim 1, characterized in that, The image stitching includes: Image rotation correction: The orientation of the images to be stitched is not always due north, but along the current flight path. This can be obtained from the uav_yaw parameter in the flight control data. Based on the uav_yaw angle, a rotation matrix is ​​constructed with the image center point as the rotation center. The images to be stitched are then losslessly corrected to face due north, and the missing background is filled with black (0,0,0). The rotation matrix is ​​constructed as shown in the following equation: ， in, For rotation angle, Center of rotation; Geographic information determination: SURF feature point calculation and matching are performed between the current image to be stitched and the previous image. KnnMatch matching strategy is adopted. According to the characteristics of UAV aerial image sequence, the lines between correctly matched feature points should be approximately parallel. RANSAC algorithm is used to remove incorrect matching points. Homography matrix is ​​calculated. The center point coordinates of the previous image are transformed into the current image to be stitched. Based on the pixel coordinates and latitude and longitude coordinates of the transformed points and the center point of the image to be stitched, the latitude and longitude span parameters of the image to be stitched are calculated. The latitude and longitude span parameters are calculated as follows: ， ， in, The center point of the image to be stitched together , The coordinates of the center point of the images to be stitched together are the latitude and longitude values. To match the center point of the image for features, , To match the latitude and longitude values ​​of the image center point for feature matching; Image mosaicking: Based on the latitude and longitude span parameters of the image to be mosaicked and the pixel coordinates and latitude and longitude coordinates of the image center point, calculate the specific location of the image to be mosaicked on the new map, update the R, G, and B band matrix information of the region, and remove the mosaic seams. Remove the seam as shown in the following formula: ， If pixel If the pixel is part of the stitching area, update the pixel value; otherwise, leave it unchanged. Repeat the above steps until all images are stitched together.

4. The method for real-time incremental stitching of UAV aerial images according to claim 1, characterized in that, The new map generation includes: After all images are stitched together, the R, G, and B band matrix data are written into a new map file to generate the final mosaic result in GeoTIFF format.

Images