Unmanned aerial vehicle image splicing method and device

Abstract

The invention provides an unmanned aerial vehicle image splicing method and device, and relates to the field of image processing. Comprising the steps that multiple original images shot by an unmanned aerial vehicle with the overlapping rate larger than the set overlapping rate are obtained, aerial triangulation reconstruction is conducted through the multiple original images and corresponding initial longitude and latitude information, target pose information of the original images is obtained, and the target pose information comprises target pose coordinates and the confidence coefficient of the target pose coordinates; displaying a control strategy for a user to select on the configuration interface, determining a plurality of to-be-selected control points from the target pose coordinates of the plurality of original images, obtaining the control strategy selected by the user, and determining a plurality of target control points from the plurality of to-be-selected control points based on the control strategy; removing inclined images from the plurality of original images based on the optical axis vectors of the plurality of original images to obtain a plurality of to-be-spliced images; and based on the plurality of target control points, splicing the plurality of to-be-spliced images to obtain a joined image. According to the scheme, the precision and accuracy of the phase-control-free spliced image can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and device for stitching images from unmanned aerial vehicles (UAVs). Background Art

[0002] Aerial photography by drones is often used in fields such as architectural surveying, geographical exploration, and crop monitoring.

[0003] A core requirement of drone aerial photography applications is to stitch aerial images into vector maps. Assuming phase-control-free technology is used for image stitching, this process is susceptible to errors in camera pose estimation, resulting in significant drift. This leads to misalignment of the coordinates in the stitched image, causing seams and distortions in the image content (e.g., misaligned road lines, misaligned or distorted roads), thus affecting the visual quality.

[0004] To ensure natural edge integration of the map and avoid seams and distortions in the large image content, it is necessary to manually lay out a pattern of points with distinct characteristics. During this process, the GPS location information of these points is collected. After the aerial images are stitched into a large image, these points are manually marked, and the geographic coordinates of each pixel in the stitched image are calculated based on these manually marked points for subsequent measurement and calculation. This method is time-consuming and labor-intensive, always requiring manual placement of points on the ground, and the accuracy of subsequent measurement and calculation is affected by camera pose estimation and GPS accuracy.

[0005] Therefore, there is an urgent need for an automated and high-precision method for stitching aerial images to improve the accuracy and visual effect of drone aerial image stitching. Summary of the Invention

[0006] The purpose of this application is to provide a method for stitching up aerial images taken by drones, in order to solve the technical problem of low precision and accuracy in stitching up aerial images taken by drones.

[0007] In a first aspect, this application provides a method for stitching images from unmanned aerial vehicles (UAVs), the method comprising:

[0008] Acquire multiple original images taken by the drone with an overlap rate greater than a set overlap rate, where each original image carries initial latitude and longitude information;

[0009] Aerial triangulation is performed using the multiple original images and their corresponding initial latitude and longitude information to obtain the target pose information of each original image. The target pose information includes the target pose coordinates and the confidence level of the target pose coordinates.

[0010] The configuration interface displays control strategies for users to select, and the control strategies include at least one of the following: a confidence-based control strategy, an inward-based control strategy, and an input command-based control strategy.

[0011] Multiple control points to be selected are determined from the target pose coordinates of the multiple original images, the control strategy selected by the user is obtained, and multiple target control points are determined from the multiple control points to be selected based on the control strategy.

[0012] Obtain the optical axis vectors of the multiple original images, and remove tilted images from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be stitched together;

[0013] Based on the multiple target control points, the multiple images to be stitched together are stitched together to obtain a combined image.

[0014] Optionally, obtaining the optical axis vectors of the multiple original images includes:

[0015] The target pose coordinates of the original image include the pose angle. For each original image, a rotation matrix about the x-axis, a rotation matrix about the y-axis, and a rotation matrix about the z-axis are calculated based on the pose angle of the original image.

[0016] Calculate the composite rotation matrix using the rotation matrices about the x-axis, y-axis, and z-axis.

[0017] Data of the target row is extracted from the synthetic rotation matrix, and the data of the target row is used as the optical axis vector of the original image.

[0018] Optionally, the method further includes:

[0019] The puzzle strategy is displayed on the configuration interface. The puzzle strategy includes at least one of the following: a puzzle strategy that removes tilted images, and a puzzle strategy that retains some tilted images.

[0020] Obtain the display puzzle strategy selected by the user through the configuration interface;

[0021] If the user selects a mosaic strategy that removes tilted images, the process of removing tilted images from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be mosaicked includes: determining a cluster center vector based on the optical axis vector of each original image; and removing tilted images with an angle greater than a set threshold from the multiple original images based on the angle between the optical axis vector of each original image and the cluster center vector to obtain multiple orthophoto images, which are then used as images to be mosaicked.

[0022] If the user chooses a stitching strategy that retains some tilted images, the tilted images are removed from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be stitched. This includes: determining a cluster center vector based on the optical axis vector of each original image; normalizing the optical axis vectors of the multiple original images based on the cluster center vector; defining a plane perpendicular to the cluster center vector as a reference plane; taking the point through which the cluster center vector passes as the center point and the point through which the optical axis vectors of the multiple original images pass as the judgment point in the reference plane; removing the original images corresponding to the judgment points whose distance from the center point is greater than a set distance to obtain multiple images to be stitched.

[0023] Optionally, if the user selects a confidence-based control strategy, multiple target control points are determined from the plurality of candidate control points based on this control strategy, including:

[0024] Multiple initial control points are determined from the plurality of candidate control points. For each initial control point, a candidate control point whose distance from the initial control point satisfies the desired distance is determined from the plurality of candidate control points.

[0025] Search for the control point with the highest confidence among the control points that meet the desired distance, and use it as the target control point corresponding to the initial control point.

[0026] Optionally, multiple initial control points are determined from the plurality of selectable control points, including:

[0027] Obtain a first result view of the aerial triangulation reconstruction, which displays multiple control points to be selected;

[0028] Determine the largest bounding polygon formed by the plurality of control points to be selected in the first result view, and define it as the first polygon;

[0029] Determine a plurality of control points that are closest to a plurality of edges and / or a plurality of corners of the first polygon, and use them as the plurality of initial control points.

[0030] Optionally, if the user selects an inward-based control strategy, multiple target control points are determined from the plurality of selectable control points based on this control strategy, including:

[0031] Obtain the second result view of the aerial triangulation reconstruction, determine the polygon formed by the control points to be selected in the second result view, and define it as the second polygon;

[0032] Multiple first initial control points are determined from the control points to be selected along the first pair of intersection lines of the second polygon, and multiple second initial control points are determined from the control points to be selected along the second pair of intersection lines of the second polygon, wherein the first diagonal direction is different from the second diagonal direction;

[0033] A first target control point is determined by moving the first initial control point inward along the first diagonal towards the center of the second polygon; a second target control point is determined by moving the second initial control point inward along the second diagonal towards the center of the second polygon.

[0034] Optionally, a first target control point is determined by moving the first initial control point inward along the first diagonal towards the center point of the second polygon, including:

[0035] The first initial control point is moved inward along the first diagonal direction towards the center point of the second polygon by a first inward length or a first proportion to obtain the first desired position point; the plurality of candidate control points are traversed, and the candidate control point with the smallest distance from the first desired position point is determined from the plurality of candidate control points as the first target control point corresponding to the first initial control point.

[0036] The second target control point is determined by moving inward from the second initial control point along the second diagonal towards the center of the second polygon, including:

[0037] The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point; the plurality of candidate control points are traversed, and the candidate control point with the smallest distance from the second desired position point is determined from the plurality of candidate control points as the second target control point corresponding to the second initial control point.

[0038] Optionally, the method further includes: the second result view provides an overlap heatmap at the control points to be selected, the overlap heatmap representing the degree of overlap of the control points to be selected;

[0039] The first initial control point is moved inward along the first diagonal direction towards the center point of the second polygon by a first inward length or a first proportion to obtain the first desired position point, including:

[0040] Determine the overlap of each control point to be selected distributed along the first diagonal direction;

[0041] The first ratio is determined based on the overlap of the control points to be selected distributed along the first diagonal direction.

[0042] The first initial control point is moved inward along the first diagonal towards the center point of the second polygon by the first proportion to obtain the first desired position point; wherein, the higher the overlap, the smaller the first proportion; and / or

[0043] The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point, including:

[0044] Determine the overlap of each control point to be selected distributed along the second diagonal direction;

[0045] The second ratio is determined based on the overlap of the selectable control points distributed along the second diagonal direction;

[0046] The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by the second ratio to obtain the second desired position point; wherein, the higher the overlap, the smaller the second ratio.

[0047] Optionally, the confidence level of the target pose coordinates for each original image is determined, including:

[0048] The target pose coordinates include three-dimensional coordinates. Based on the initial three-dimensional coordinates of each original image and the three-dimensional coordinates after aerial triangulation reconstruction, the Euclidean distance between the two coordinates is calculated.

[0049] The confidence level of the target pose coordinates of the original image is determined based on the Euclidean distance between the two coordinates; wherein, the larger the Euclidean distance between the two coordinates, the lower the confidence level of the target pose coordinates of the original image.

[0050] Optionally, if the user selects a control strategy based on input instructions, multiple target control points are determined from the plurality of selectable control points based on the control strategy, including:

[0051] The configuration interface displays multiple control points selected from the clusters of the control points to be selected.

[0052] The control point selected by the user from multiple control points in the recommended cluster is obtained as the target control point.

[0053] Optionally, based on the plurality of target control points, the plurality of images to be stitched together to obtain a joined image includes: determining a transformation matrix based on the plurality of target control points; converting the three-dimensional coordinates of each image to be stitched into pixel coordinates based on the transformation matrix; and stitching the plurality of images to be stitched together based on their pixel coordinates to obtain a joined image; and / or

[0054] Based on the multiple target control points, the multiple images to be stitched together are stitched together to obtain a joined image, including: performing digital differential correction on the multiple images to be stitched based on the multiple target control points, and stitching the digitally differentiated images to be stitched together to obtain a joined image.

[0055] Secondly, this application provides a device for UAV image stitching, the device comprising:

[0056] processor;

[0057] A memory storing computer-readable instructions, which, when executed by the processor, implement the UAV image stitching method described in the first aspect above.

[0058] The monitor is used to display the configuration interface.

[0059] The beneficial effects of the UAV image stitching method and equipment provided in this application include at least the following: obtaining the target pose coordinates and confidence level of multiple original images using aerial triangulation reconstruction; determining multiple selectable control points based on the target pose coordinates of the multiple original images after aerial triangulation reconstruction; and automatically selecting multiple optimal target control points from the multiple selectable control points using at least one of the following control strategies: confidence level control strategy, inward control strategy, and input command control strategy. This automatically avoids control points with low reliability and prioritizes selecting target control points from control points with high reliability, thereby improving the coordinate accuracy of the phase-controlled image stitching and avoiding coordinate misalignment in the stitched image. Utilizing information from all original images during aerial triangulation reconstruction improves the accuracy of target control point selection. Furthermore, during stitching, automatically removing tilted images from the multiple original images based on the optical axis vectors of the original images effectively reduces stretching misalignment and color unevenness in the stitched image, thus significantly improving the accuracy and precision of the phase-controlled image stitching. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of a drone image stitching method provided in an embodiment of this application;

[0062] Figure 2 This is a flowchart of an aerial triangulation reconstruction process provided in an embodiment of this application;

[0063] Figure 3 This is a schematic diagram of a normalized imaging plane provided in an embodiment of this application;

[0064] Figure 4 A schematic diagram illustrating a tilted image recognition method provided in an embodiment of this application;

[0065] Figure 5 A comparative schematic diagram showing the before and after stitching effects of removing tilted images, provided as an embodiment of this application;

[0066] Figure 6 A flowchart illustrating a method for determining target control points based on a confidence-based control strategy, provided in an embodiment of this application;

[0067] Figure 7 This is a schematic diagram of a first result view of aerial triangulation reconstruction provided in an embodiment of this application;

[0068] Figure 8 This is a schematic diagram of the processing procedure of a method for determining target control points based on an inward control strategy provided in an embodiment of this application;

[0069] Figure 9 This is a schematic diagram of an overlapping heatmap provided in an embodiment of this application;

[0070] Figure 10 This is a schematic diagram illustrating a control strategy based on inward retraction for finding a target control point, as provided in an embodiment of this application.

[0071] Figure 11 This is a structural block diagram of a drone image stitching device provided in an embodiment of this application;

[0072] Figure 12 This is a schematic diagram of the structure of a device for UAV image stitching provided in an embodiment of this application. Detailed Implementation

[0073] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0074] like Figure 1 As shown in the figure, this application provides a method for stitching images of a drone, which specifically includes the following steps:

[0075] Step 101: Acquire multiple original images taken by the drone with an overlap rate greater than a set overlap rate. Each original image carries initial latitude and longitude information.

[0076] The drone can fly along a predetermined route, capturing multiple raw images during flight when the lateral and longitudinal overlap rates reach a set threshold. Each raw image carries at least the initial latitude and longitude information at the time of capture. It should be noted that each raw image, in addition to carrying the initial latitude and longitude information, may also carry the time of capture and the initial attitude information at the time of capture. Specifically, the latitude and longitude information includes longitude, latitude, and altitude; the initial attitude information includes roll angle, pitch angle, and yaw angle, which can be represented by Roll, Pitch, and Yaw.

[0077] Step 102: Perform aerial triangulation reconstruction using multiple original images and their corresponding initial latitude and longitude information to obtain the target pose information for each original image. The target pose information includes the target pose coordinates and the confidence level of the target pose coordinates.

[0078] Specifically, aerial triangulation, also known as aerial triangulation, solves for the optimized pose coordinates of each original image by matching feature points from multiple original images and using bundle adjustment (BA).

[0079] The specific steps of aerial triangulation reconstruction are as follows: Figure 2As shown, path A processes multiple input original images to calculate the movement trajectory of feature points. Specifically, path A extracts image features from multiple original images and further matches the features extracted from different original images. Based on the feature matching between multiple original images, the feature point trajectory is obtained, and the feature point trajectory is used as a constraint condition for BA optimization. Path B processes the input initial latitude and longitude information to obtain initial pose coordinates. The initial pose coordinates are used as the initial values ​​for BA optimization, and the optimized target pose coordinates are output after BA optimization. Specifically, path B parses and normalizes the input initial latitude and longitude information to obtain the initial pose coordinates, which can include initial 3D coordinates and initial attitude coordinates. For example, the initial pose coordinates can be represented as (x0, y0, z0, roll0, yaw0, pitch0). Based on the initial pose coordinates, the target pose coordinates after aerial triangulation reconstruction are obtained. The target pose coordinates can include optimized 3D coordinates and attitude coordinates. For example, the target pose coordinates can be represented as (x1, y1, z1, roll1, yaw1, pitch1).

[0080] like Figure 2 As shown, the confidence level of the target pose coordinates can be calculated based on the initial pose coordinates of each original image and the target pose coordinates after aerial triangulation reconstruction. Preferably, based on the initial 3D coordinates of each original image and the 3D coordinates after aerial triangulation reconstruction, the Euclidean distance between the two coordinates is calculated, and the confidence level of the target pose coordinates of the original image is determined based on the Euclidean distance between the two coordinates; wherein, the larger the Euclidean distance between the two coordinates, the lower the confidence level of the target pose coordinates of the original image.

[0081] Furthermore, the specific formula for calculating the confidence level of the target pose coordinates is as follows:

[0082]

[0083] Among them, T i is the confidence score corresponding to the target pose coordinates of the i-th original image; (x0, y0, z0) represents the initial three-dimensional coordinates of the i-th original image, and (x1, y1, z1) represents the three-dimensional coordinates after aerial triangulation reconstruction of the i-th original image.

[0084] Step 103: Display control strategies for user selection on the configuration interface. The control strategies include at least one of the following: confidence-based control strategy, introductory control strategy, and input command-based control strategy.

[0085] Step 104: Determine multiple control points to be selected from the target pose coordinates of multiple original images, obtain the control strategy selected by the user, and determine multiple target control points from the multiple control points to be selected based on the control strategy.

[0086] Among them, the target pose coordinates of multiple original images can all be used as control points to be selected, or the remaining ones after removing some target pose coordinates that do not meet the requirements can be used as control points to be selected, or the remaining ones after removing them according to a certain ratio can be used as control points to be selected.

[0087] Step 105: Obtain the optical axis vectors of multiple original images, and remove tilted images from the multiple original images based on the optical axis vectors of multiple original images to obtain multiple images to be stitched together.

[0088] The optical axis vector refers to the three-dimensional direction vector pointing from the camera's projection center to the target scene. It usually coincides with the optical axis and represents the camera's line of sight. Tilted images can be removed from the original images by using the specific optical axis vector of each original image. For example, if there are 200 original images, and 20 of them have a large tilt angle, these images can be removed, leaving 180 original images for stitching.

[0089] Step 106: Based on multiple target control points, stitch together multiple images to be stitched to obtain a stitched image.

[0090] Method 1: Stitching multiple images to be stitched together based on multiple target control points to obtain a stitched image, including: determining a transformation matrix based on multiple target control points, converting the three-dimensional coordinates of each image to be stitched into pixel coordinates based on the transformation matrix, and stitching multiple images to be stitched together based on their pixel coordinates to obtain a stitched image.

[0091] Specifically, multiple matrix parameters are determined based on multiple target control points. These matrix parameters constitute a transformation matrix. By transforming multiple images to be stitched from three-dimensional coordinates in the three-dimensional coordinate system to pixel coordinates in the camera coordinate system, the stitching accuracy is improved.

[0092] Furthermore, the multiple images to be stitched are converted from three-dimensional coordinates to pixel coordinates using the following formula:

[0093] P = proj(P) C )=(X C f x / Z C +x0,Y C f y / Z C +y0) T Where P represents the pixel coordinates of the target control point, proj represents the transformation matrix from 3D coordinates in the 3D coordinate system to pixel coordinates in the camera coordinate system, and x0, y0, f x and f y Indicates camera intrinsic parameters, X C ,Y C and Z CThis represents the three-dimensional coordinates of the target control point. For example, the three-dimensional coordinates of the target control point can be obtained based on its latitude and longitude information and pose transformation, such as... Figure 3 As shown, taking a 3D coordinate system specific to the camera coordinate system as an example, the specific transformation method is as follows: Assuming the target control point is O, simplify the latitude and longitude information of the target control point O by letting the vector of the target control point be d = [0,0,1]. T (like Figure 3 The yellow arrowed line in the image), vector d can be considered as a spatial point, and the camera optical axis is OA (e.g., the line with the yellow arrow). Figure 3 (The yellow lines in the text) Figure 3 The plane perpendicular to the camera's optical axis, where the red line is located, is the normalized imaging plane, and the pose of the target control point is R. CW The 3D coordinates of d transformed to the camera coordinate system are d' = R. CW *d, because the camera's shooting direction is not vertically downward, therefore d''s Z C The coordinates are not equal to 1; the coordinates of d' (X) C ,Y C Z C Substitute into P = proj(P) C )=(X C f x / Z C +x0,Y C f y / Z C +y0) T The pixel coordinates P of the target control point are obtained.

[0094] Method 2, step 106, involves stitching multiple images to be stitched together based on multiple target control points to obtain a stitched image, including: performing digital differential correction on multiple images to be stitched based on multiple target control points, and stitching the digitally corrected multiple images to be stitched together to obtain a stitched image.

[0095] Method 1 and Method 2 described above can be used individually or in combination to stitch multiple images to be stitched together to obtain a combined image; no limitation is made here.

[0096] For example, digital differential correction can be performed using the OpenCV inverse kinematics method. Specifically, the OpenCV inverse kinematics process is as follows: Assume that pixel P on the image to be stitched after digital differential correction has pixel coordinates (X... p Y pThe X and Y coordinates of point P in the three-dimensional coordinate system are calculated. Based on the calculated X and Y coordinates, the Z coordinates in the corresponding three-dimensional coordinate system are obtained. Based on the X, Y, and Z coordinates, the pixel coordinates of point P in the image to be stitched before correction are determined by the collinearity equation. The gray value is then interpolated using the nearest neighbor method or bilinear interpolation method to obtain the gray value at the pixel coordinate position. This gray value is assigned to the pixel P in the image to be stitched after correction. Thus, the correction of multiple images to be stitched is achieved by correcting the pixels in multiple images to be stitched.

[0097] Due to errors in sensors (such as positioning modules that acquire latitude and longitude information) and registration deviations on the drone, there are local offsets between multiple images to be stitched together. A mapping relationship is established through multiple target control points, and the offset of multiple orthophoto images is corrected based on the mapping relationship to eliminate the influence in order to facilitate accurate alignment and stitching.

[0098] In the aforementioned UAV image stitching method, aerial triangulation is used to obtain the target pose coordinates of multiple original images and a confidence level for measuring the target pose coordinates. Based on the target pose coordinates of the multiple original images after aerial triangulation, multiple candidate control points are determined. At least one control strategy—based on confidence level control, inward control, and input command control—automatically selects multiple optimal target control points from the candidate control points. This automatically avoids control points with low reliability and prioritizes selecting target control points from those with higher reliability, improving the coordinate accuracy of the phase-controlled image stitching and avoiding coordinate misalignment in the stitched image. Since the aerial triangulation process utilizes information from all original images without screening or removing tilted images, it improves the accuracy of target control point selection. During stitching, tilted images are automatically removed from the multiple original images based on the optical axis vectors to eliminate images with excessively large tilt angles, effectively reducing stretching misalignment and color unevenness in the stitched image, thus significantly improving the accuracy and precision of the phase-controlled image stitching.

[0099] In one embodiment, obtaining the optical axis vectors of multiple original images specifically includes:

[0100] The target pose coordinates of the original image include the pose angle. For each original image, a rotation matrix about the x-axis, a rotation matrix about the y-axis, and a rotation matrix about the z-axis are calculated based on the pose angle of the original image. A composite rotation matrix is ​​calculated using the rotation matrices about the x-axis, the y-axis, and the z-axis. Data of the target row is extracted from the composite rotation matrix and used as the optical axis vector of the original image.

[0101] Specifically, the synthesis rotation matrix can be a 3x3 matrix. The first row of the synthesis selection matrix corresponds to the x-axis, the second row to the y-axis, and the third row to the z-axis. Data from the third row can be extracted from the synthesis rotation matrix and used as the optical axis vector of the original image.

[0102] In one embodiment, prior to step 106, the drone image stitching method further includes the following steps:

[0103] Display the jigsaw puzzle strategy on the configuration interface. The jigsaw puzzle strategy includes at least one of the following: a jigsaw puzzle strategy that removes tilted images, a jigsaw puzzle strategy that retains some tilted images, and a jigsaw puzzle strategy that retains all tilted images.

[0104] The display mosaic strategy selected by the user through the configuration interface is obtained. If the user selects the mosaic strategy of removing tilted images, step 105 removes tilted images from multiple original images based on the optical axis vectors of multiple original images to obtain multiple images to be mosaicked. This includes: determining the cluster center vector based on the optical axis vector of each original image; removing tilted images with an angle greater than a set threshold from multiple original images based on the angle between the optical axis vector of each original image and the cluster center vector to obtain multiple orthophoto images, which are then used as images to be mosaicked.

[0105] Specifically, the cluster center vector can be determined based on the optical axis vector of each original image using at least one of the following methods: spherical Gaussian model, geometric median method, and weighted vector average method.

[0106] For example, Vi represents the optical axis vector of the i-th original image, and Vmean represents the cluster center vector. The original image is retained when the cosine angle between the two vectors Vi and Vmean is not greater than the threshold of 15 degrees. If the cosine angle between the two vectors Vi and Vmean is greater than the threshold of 15 degrees, the original image is a tilted image and needs to be removed. Figure 4 This is a schematic diagram for tilted image recognition. Figure 4 The image is identified as tilted based on the optical axis vector of the original image, with red arrows indicating orthophoto and tilted images respectively. Figure 5 This is a comparison diagram showing the stitching effect before and after removing a tilted image. Figure 5 In (a), the stitching effect of the image before removing the tilted image is shown. In (a), the stitched image has severe stretching outside the red line, which causes blank spots, and there is severe color unevenness in the central area. Figure 5 (b) shows the stitching effect of the combined image after removing the tilted image. There are no blank spots in the combined image after removing the tilted image, and there is no serious color unevenness in the central area.

[0107] In one embodiment, if the user chooses a stitching strategy that retains some tilted images, step 105 involves removing the tilted images from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be stitched. This includes: determining a cluster center vector based on the optical axis vector of each original image; normalizing the optical axis vectors of the multiple original images based on the cluster center vector; defining a plane perpendicular to the cluster center vector as a reference plane; using the point through which the cluster center vector passes in the reference plane as the center point and the point through which the optical axis vectors of the multiple original images pass as the judgment point; removing the original images corresponding to the judgment points whose distance from the center point is greater than a set distance to obtain multiple images to be stitched.

[0108] It should be noted that if the images used for stitching are too tilted, it will significantly reduce the stitching effect. Therefore, some of the original images that are too tilted need to be removed. Specifically, the optical axis vectors of the original images can be normalized based on the cluster center vector, so that the optical axis vectors of the original images converge at a point. A plane is intercepted along the direction perpendicular to the cluster center vector as a reference plane, and both the cluster center vector and the optical axis vectors of the original images pass through this reference plane. In the reference plane, the point through which the cluster center vector passes is taken as the center point, and the point through which the optical axis vectors of the original images pass is taken as the judgment point. If the judgment point is too far from the center point, it can be considered that the original image corresponding to the judgment point is too tilted, and it is removed from the original images.

[0109] In one embodiment, if the user chooses a stitching strategy that retains all tilted images, step 105 may involve using all the original images as the images to be stitched together.

[0110] In one embodiment, such as Figure 6 As shown, if the user selects a confidence-based control strategy, step 104 determines multiple target control points from multiple candidate control points based on this control strategy, including the following steps:

[0111] Step 601: Determine multiple initial control points from multiple candidate control points. For each initial control point, determine candidate control points from multiple candidate control points whose distance from the initial control point meets the desired distance.

[0112] Step 602: Search for the control point with the highest confidence among the control points that meet the expected distance, and use it as the target control point corresponding to the initial control point.

[0113] Specifically, for each initial control point, its corresponding target control point is determined using the following formula:

[0114]

[0115] In Formula 1, pi Indicates the control points to be selected; p0 indicates the initial control points; d(,) < D means that the Euclidean distance between the two points p i and p0 is less than the expected distance D; max() means to find the maximum, that is, it means that p i and when the Euclidean distance between p0 and p is less than the expected distance D, select from p i the point with the highest confidence T i as the target control point.

[0116] In one embodiment, determining multiple initial control points from multiple control points to be selected in step 301 specifically includes:

[0117] Obtain the first result view of the aerial triangulation reconstruction, in which multiple control points to be selected are displayed;

[0118] Determine the maximum circumscribed polygon formed by multiple control points to be selected in the first result view, and define it as the first polygon;

[0119] Determine multiple control points to be selected that are closest to multiple sides and / or multiple corner points of the first polygon as multiple initial control points.

[0120] Figure 7 is the first result view of the aerial triangulation reconstruction, as Figure 7 shown, Figure 7 The main figure in it is a top view, and the right side and the lower side are side views respectively. It should be noted that generally, the top view is mainly concerned. In the top view, the abscissa is the x direction, the ordinate is the y direction, and each solid black dot is the target pose coordinate after aerial triangulation reconstruction, that is, the control point to be selected. The blue circles in the top view represent the confidence levels of the control points to be selected. The smaller the circle, the higher the confidence level, and the larger the circle, the lower the confidence level.

[0121] Taking 4 initial control points as an example for detailed description, according to Figure 7 the multiple control points to be selected in it to determine the formed maximum circumscribed polygon, select multiple control points to be selected that are closest to multiple sides and / or multiple corner points of the maximum circumscribed polygon as initial control points. Such as Figure 7 the 1, 2, 3, and 4 initial control points marked in red in it. From Figure 4 it can be seen that the confidence levels of the 1, 2, 3, and 4 control points are relatively low. Therefore, based on these points, adjustments are made, and through the above formula 1, select the four newly marked green control points from the control points to be selected other than the 4 initial control points in Figure 7 as the target control points.

[0122] In this embodiment, by determining the largest bounding polygon formed by the control points to be selected, multiple control points to be selected that are closest to multiple edges and / or multiple corners of the largest bounding polygon are found as multiple initial control points. Based on the multiple initial control points, the corresponding target control points are adjusted to find them. This can effectively avoid points with low confidence, making the control points used for image stitching more reliable.

[0123] In one embodiment, such as Figure 8 As shown, if the user selects an inward-based control strategy, step 104 determines multiple target control points from a pool of selectable control points based on this strategy. Specifically, this includes:

[0124] Step 801: Obtain the second result view of the aerial triangulation reconstruction, determine the polygon formed by the control points to be selected in the second result view, and define it as the second polygon.

[0125] Step 802: Determine multiple first initial control points from the control points to be selected along the first pair of intersecting diagonals of the second polygon, and determine multiple second initial control points from the control points to be selected along the second pair of intersecting diagonals of the second polygon, wherein the first diagonal direction is different from the second diagonal direction.

[0126] Step 803: Determine a first target control point corresponding to the first initial control point by moving the first initial control point inward along the first diagonal towards the center point of the second polygon; determine a second target control point corresponding to the second initial control point by moving the second initial control point inward along the second diagonal towards the center point of the second polygon.

[0127] Specifically, if Figure 9 As shown, the second results view displays an overlap heatmap at the control points to be selected. The overlap heatmap represents the degree of overlap between the control points. It should be noted that the degree of overlap can refer to the number of overlaps between the control points; for example, different colors in the overlap heatmap indicate different numbers of overlaps, with the shade of the color representing different numbers of overlaps (which can range from 1 to 10). Figure 9 In this context, the control points on the outermost edge have lower confidence levels, and the overlap between the four outermost points may be only 1. Using these four points as control points could negatively impact the accuracy of the final geographic coordinate calculation. Alternatively, the overlap of the control points to be selected could refer to their density, etc., which is not limited here.

[0128] Among them, through Figure 10 This illustrates a technical solution for finding target control points based on an inward-facing control strategy, such as... Figure 10 As shown, Figure 10 The gray shape in the image represents the polygon formed by the control points to be selected in the second result view. Figure 10 The four red pentagons represent two initial control points determined along the first pair of intersection lines, and two second initial control points determined along the second pair of intersection lines. By moving these four initial control points inward a certain distance from the center of the polygon, new control points are found and designated as the target control points. Figure 10 The purple four-pointed star in the center is the target control point.

[0129] In one embodiment, step 803, by moving the first initial control point inward along the first diagonal towards the center of the second polygon, determines the first target control point corresponding to the first initial control point. Specifically, this includes: moving the first initial control point inward along the first diagonal towards the center of the second polygon by a first inward length or a first proportion to obtain a first desired position point; traversing multiple candidate control points, and determining the candidate control point with the smallest distance to the first desired position point from among the multiple candidate control points as the first target control point corresponding to the first initial control point.

[0130] In step 803, the second target control point is determined by moving the second initial control point inward along the second diagonal towards the center of the second polygon. Specifically, this includes: moving the second initial control point inward along the second diagonal towards the center of the second polygon by a second inward length or a second proportion to obtain the second desired position point; traversing multiple candidate control points, and determining the candidate control point with the smallest distance to the second desired position point from among the multiple candidate control points, as the second target control point corresponding to the second initial control point.

[0131] Specifically, the formula for calculating the desired location point using the inward strategy is as follows:

[0132] p′1=(1-a1)p1+ap3,

[0133] p′3=(1-a1)p3+ap1,

[0134] p′2=(1-a2)p2+ap4,

[0135] p′4=(1-a2)p4+ap2;

[0136] In this example, p1 and p3 represent two initial control points along the first diagonal, and p2 and p4 represent two initial control points along the second diagonal. It should be noted that reducing the control points by a certain length along the first and second diagonals is one approach, while reducing them by a certain percentage is also feasible. This example uses a percentage reduction as an illustration. In the formula for determining the desired position point, a1 represents the first percentage reduction along the first diagonal, and a2 represents the second percentage reduction along the second diagonal. The first and second percentages can be preset values, and they can be the same or different. For example, if both the first and second percentages are 10%, then p′1 and p′3 represent two desired position points obtained by reducing the two initial control points by the first percentage, and p′2 and p′4 represent two desired position points obtained by reducing the two initial control points by the second percentage.

[0137] Furthermore, based on the first desired location points p′1 and p′3 and the second desired location points p′2 and p′4, the corresponding first target control point and second target control point are found using the following formula:

[0138] min 1≤i≤N,1≤j≤4 d(p i ,p j ′);

[0139] Where N represents the total number of control points to be selected; p i These are the actual locations with GPS data collection points, i.e., the control points to be selected; p j ′ represents the desired location points, specifically p′1, p′2, p′3, and p′4; d(,) calculates the Euclidean distance between these two points; min() finds the nearest neighbor, i.e., from the control point to be selected p i Find the point p at the desired location. j The nearest control point is used as the target control point.

[0140] In one embodiment, the inward movement towards the center point of the second polygon in step 803 can be unequal, and the first and / or second inward proportions can be determined based on the overlap of the control points to be selected. In this embodiment, the first initial control point is moved inward along the first diagonal direction towards the center point of the second polygon by a first inward length or a first proportion to obtain the first desired position point, specifically including:

[0141] Determine the overlap of each control point to be selected distributed along the first diagonal direction;

[0142] The first ratio is determined based on the overlap of the control points to be selected distributed along the first diagonal direction;

[0143] The first initial control point is moved inward along the first diagonal towards the center point of the second polygon by a first proportion to obtain the first desired position point; wherein, the higher the overlap, the smaller the first proportion; and / or

[0144] The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point, including:

[0145] Determine the overlap of each control point to be selected distributed along the second diagonal direction;

[0146] The second ratio is determined based on the overlap of the control points to be selected distributed along the second diagonal direction;

[0147] The second initial control point is moved inward along the second diagonal towards the center of the second polygon by a second ratio to obtain the second desired position point; wherein, the higher the overlap, the smaller the second ratio.

[0148] In one embodiment, if the user selects a control strategy based on input instructions, multiple target control points are determined from multiple selectable control points based on this control strategy, including:

[0149] The configuration interface displays multiple control points selected from the clusters of the control points to be selected.

[0150] The control point selected by the user from multiple control points in the recommended cluster is obtained as the target control point. Based on the same inventive concept, this application also provides a UAV image stitching device for implementing the UAV image stitching method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more UAV image stitching device embodiments provided below can be found in the limitations of the UAV image stitching method above, and will not be repeated here.

[0151] In one embodiment, such as Figure 11 As shown, a UAV image stitching device is provided, including: an acquisition module 111, a reconstruction module 112, a configuration module 113, a filtering module 114, and a stitching module 115, wherein:

[0152] The acquisition module 111 acquires multiple original images taken by the drone with an overlap rate greater than a set overlap rate. Each original image carries initial latitude and longitude information.

[0153] The reconstruction module 112 is used to perform aerial triangulation reconstruction using multiple original images and their corresponding initial latitude and longitude information to obtain the target pose information of each original image. The target pose information includes the target pose coordinates and the confidence level of the target pose coordinates.

[0154] The configuration module 113 is used to display control strategies for user selection on the configuration interface. The control strategies include at least one of the following: a confidence-based control strategy, an inward-based control strategy, and an input command-based control strategy. The configuration module 113 is also used to determine multiple control points to be selected from the target pose coordinates of multiple original images, obtain the control strategy selected by the user, and determine multiple target control points from the multiple control points to be selected based on the control strategy.

[0155] The filtering module 114 is used to obtain the optical axis vectors of multiple original images, and remove tilted images from the multiple original images based on the optical axis vectors of multiple original images to obtain multiple images to be stitched together.

[0156] The stitching module 115 is used to stitch multiple images to be stitched together based on multiple target control points to obtain a stitched image.

[0157] In one embodiment, the filtering module 114 obtains the optical axis vectors of multiple original images, specifically including:

[0158] The target pose coordinates of the original image include the pose angle. For each original image, the rotation matrix around the x-axis, the rotation matrix around the y-axis, and the rotation matrix around the z-axis are calculated based on the pose angle of the original image. The composite rotation matrix is ​​calculated using the rotation matrices around the x-axis, the y-axis, and the z-axis. The target row data is extracted from the composite rotation matrix and used as the optical axis vector of the original image.

[0159] In one embodiment, the configuration module 113 is used to display a jigsaw puzzle strategy on a configuration interface. The jigsaw puzzle strategy includes at least one of the following: a jigsaw puzzle strategy that removes tilted images, and a jigsaw puzzle strategy that retains some tilted images. The configuration module 113 is also used to obtain the display jigsaw puzzle strategy selected by the user through the configuration interface.

[0160] If the user chooses the stitching strategy of removing tilted images, the tilted images are removed from the multiple original images based on the optical axis vectors of multiple original images to obtain multiple images to be stitched. This includes: determining the cluster center vector based on the optical axis vector of each original image; removing tilted images with an angle greater than a set threshold from the multiple original images based on the angle between the optical axis vector of each original image and the cluster center vector to obtain multiple orthophoto images, which are then used as images to be stitched.

[0161] If the user chooses a stitching strategy that retains some tilted images, the tilted images are removed from the original images based on the optical axis vectors of the original images to obtain multiple images to be stitched. This includes: determining a cluster center vector based on the optical axis vector of each original image; normalizing the optical axis vectors of the original images based on the cluster center vector; defining a plane perpendicular to the cluster center vector as a reference plane; using the point through which the cluster center vector passes in the reference plane as the center point and the point through which the optical axis vectors of the original images pass as the judgment point; removing the original images corresponding to the judgment points whose distance from the center point is greater than a set distance to obtain multiple images to be stitched.

[0162] In one embodiment, if the user selects a confidence-based control strategy, the configuration module 113 determines multiple target control points from multiple candidate control points based on the control strategy. Specifically, this includes: determining multiple initial control points from the multiple candidate control points; for each initial control point, determining candidate control points from the multiple candidate control points whose distance from the initial control point meets the expected distance; and searching for the candidate control point with the highest confidence from the candidate control points that meet the expected distance, and using it as the target control point corresponding to the initial control point.

[0163] In one embodiment, the configuration module 113 determines multiple initial control points from multiple selectable control points, specifically including: obtaining a first result view of aerial triangulation reconstruction, in which multiple selectable control points are displayed; determining the largest bounding polygon formed by the multiple selectable control points in the first result view, and defining it as a first polygon; determining multiple selectable control points that are closest to multiple edges and / or multiple corners of the first polygon, as multiple initial control points.

[0164] In one embodiment, if the user selects an inward-based control strategy, the configuration module 113 determines multiple target control points from multiple selectable control points based on the control strategy. Specifically, this includes: acquiring a second result view of aerial triangulation reconstruction; determining the polygon formed by the selectable control points in the second result view and defining it as a second polygon; determining multiple first initial control points from the selectable control points along the first pair of diagonals of the second polygon; determining multiple second initial control points from the selectable control points along the second pair of diagonals of the second polygon, wherein the first diagonal direction is different from the second diagonal direction; determining a first target control point corresponding to the first initial control point by inwardly moving the first initial control point along the first diagonal direction towards the center point of the second polygon; and determining a second target control point corresponding to the second initial control point by inwardly moving the second initial control point along the second diagonal direction towards the center point of the second polygon.

[0165] In one embodiment, the configuration module 113 determines a first target control point corresponding to the first initial control point by moving the first initial control point inward along the first diagonal towards the center point of the second polygon. Specifically, this includes: moving the first initial control point inward along the first diagonal towards the center point of the second polygon by a first inward length or a first proportion to obtain a first desired position point; traversing multiple candidate control points, and determining the candidate control point with the smallest distance to the first desired position point from among the multiple candidate control points as the first target control point corresponding to the first initial control point.

[0166] The configuration module 113 determines the second target control point corresponding to the second initial control point by indenting it along the second diagonal towards the center point of the second polygon. Specifically, this includes: indenting the second initial control point along the second diagonal towards the center point of the second polygon by a second indentation length or a second proportion to obtain the second desired position point; traversing multiple candidate control points, and determining the candidate control point with the smallest distance to the second desired position point from among the multiple candidate control points as the second target control point corresponding to the second initial control point.

[0167] In one embodiment, the second result view includes an overlap heatmap at the control points to be selected, the overlap heatmap representing the degree of overlap of the control points to be selected;

[0168] Configuration module 113 moves the second initial control point inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point, specifically including:

[0169] Determine the overlap of each selectable control point distributed along the first diagonal direction; based on the overlap of the selectable control points distributed along the first diagonal direction, determine a first ratio; move the first initial control point inward along the first diagonal direction towards the center point of the second polygon by the first ratio to obtain the first desired position point; wherein, the higher the overlap, the smaller the first ratio; and / or

[0170] Configuration module 113 moves the second initial control point inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point, specifically including:

[0171] Determine the overlap of each selectable control point distributed along the second diagonal direction; based on the overlap of the selectable control points distributed along the second diagonal direction, determine the second ratio; move the second initial control point inward along the second diagonal direction towards the center point of the second polygon by the second ratio to obtain the second desired position point; wherein, the higher the overlap, the smaller the second ratio.

[0172] In one embodiment, the reconstruction module 112 determines the confidence level of the target pose coordinates of each original image, specifically including: the target pose coordinates include three-dimensional coordinates; based on the initial three-dimensional coordinates of each original image and the three-dimensional coordinates after aerial triangulation reconstruction, the Euclidean distance between the two coordinates is calculated; the confidence level of the target pose coordinates of the original image is determined based on the Euclidean distance between the two coordinates; wherein, the larger the Euclidean distance between the two coordinates, the lower the confidence level of the target pose coordinates of the original image.

[0173] In one embodiment, if the user selects a control strategy based on input instructions, the configuration module 113 determines multiple target control points from multiple selectable control points based on the control strategy. Specifically, this includes: displaying multiple control points selected and clustered from the selectable control points on the configuration interface; and obtaining the control point selected by the user from the multiple selected clustered control points as the target control point.

[0174] In one embodiment, the stitching module 115 stitches multiple images to be stitched together based on multiple target control points to obtain a joined image. Specifically, this includes: determining a transformation matrix based on the multiple target control points; converting the three-dimensional coordinates of each image to be stitched into pixel coordinates based on the transformation matrix; and stitching the multiple images to be stitched together based on their pixel coordinates to obtain the joined image; and / or

[0175] The stitching module 115 stitches multiple images to be stitched together based on multiple target control points to obtain a stitched image. Specifically, it includes: performing digital differential correction on multiple images to be stitched based on multiple target control points, and stitching the digitally corrected multiple images to be stitched together to obtain a stitched image.

[0176] Based on the same inventive concept, this application also provides a system for drone image stitching, the system including a drone and electronic equipment. The drone is configured to acquire multiple original images with an overlap rate greater than a set overlap rate, and the electronic equipment is configured to implement the aforementioned drone image stitching method.

[0177] Based on the same inventive concept, this application also provides a device for image stitching of unmanned aerial vehicles (UAVs). Figure 12 This is a schematic diagram of the structure of the device for UAV image stitching provided in the embodiments of this application, as shown below. Figure 12 As shown, exemplarily, a device 120 for drone image stitching may include a first processor 121.

[0178] For example, the device 120 for drone image stitching may also include a memory 122 and a transceiver 123.

[0179] The first processor 121, memory 122, and transceiver 123 can be connected via a communication bus.

[0180] The following is combined Figure 12 The example illustrates the various components of the device 120 used for drone image stitching:

[0181] The device 120 for UAV image stitching may include the following components: the first processor 121 may be a single processor or a collective term for multiple processing elements. For example, the first processor 121 may be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0182] For example, the first processor 121 can perform various functions of the device 120 for drone image stitching by running or executing software programs stored in memory 122 and calling data stored in memory 122.

[0183] In a specific implementation, as one example, the first processor 121 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 are shown in the diagram.

[0184] As an optional embodiment, the device 120 for drone image stitching may also include multiple processors, such as... Figure 12 The first processor 121 and the second processor 124 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0185] The memory 122 is used to store the software program that executes the present invention and is controlled by the first processor 121. The specific implementation method can be referred to the above-described UAV image stitching method embodiment, which will not be repeated here.

[0186] For example, memory 122 may be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 122 may be integrated with the first processor 121 or may exist independently and be connected via the interface circuit of the device 120 for UAV image stitching. Figure 12 (Not shown in the image) is coupled to the first processor 121, but this embodiment does not specifically limit this.

[0187] Transceiver 123 is used to communicate with network devices or with terminal devices.

[0188] For example, transceiver 123 may include a receiver and a transmitter. Figure 12 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0189] For example, transceiver 123 can be integrated with the first processor 121 or exist independently, and can be connected via the interface circuit of device 120 for drone image stitching. Figure 12 (Not shown in the image) is coupled to the first processor 121, but this embodiment does not specifically limit this.

[0190] For example, the device 120 for drone image stitching may also include a display 125, which may be a liquid crystal display or an e-ink display. This application embodiment does not specifically limit this.

[0191] Furthermore, the technical effects of the device 120 for drone image stitching can be referred to the technical effects of the method for drone image stitching in the above method embodiments, and will not be repeated here.

[0192] Based on the same inventive concept, this application also provides an electronic device, including: at least one memory and at least one processor, wherein the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-described UAV image stitching method.

[0193] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0194] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0195] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via infrared, microwave, or other means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), solid-state drives, etc.

[0196] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0197] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0198] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0199] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0200] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for stitching images from unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire multiple original images taken by the drone with an overlap rate greater than a set overlap rate, where each original image carries initial latitude and longitude information; Aerial triangulation is performed using the multiple original images and their corresponding initial latitude and longitude information to obtain the target pose information of each original image. The target pose information includes the target pose coordinates and the confidence level of the target pose coordinates. The configuration interface displays control strategies for users to select, and the control strategies include at least one of the following: a confidence-based control strategy, an inward-based control strategy, and an input command-based control strategy. Multiple control points to be selected are determined from the target pose coordinates of the multiple original images, the control strategy selected by the user is obtained, and multiple target control points are determined from the multiple control points to be selected based on the control strategy. Obtain the optical axis vectors of the multiple original images, and remove tilted images from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be stitched together; Based on the multiple target control points, the multiple images to be stitched together are stitched together to obtain a combined image.

2. The UAV image stitching method according to claim 1, characterized in that, Obtaining the optical axis vectors of the multiple original images includes: The target pose coordinates of the original image include the pose angle. For each original image, a rotation matrix about the x-axis, a rotation matrix about the y-axis, and a rotation matrix about the z-axis are calculated based on the pose angle of the original image. Calculate the composite rotation matrix using the rotation matrices about the x-axis, y-axis, and z-axis. Data of the target row is extracted from the synthetic rotation matrix, and the data of the target row is used as the optical axis vector of the original image.

3. The UAV image stitching method according to claim 1, characterized in that, The method further includes: The puzzle strategy is displayed on the configuration interface. The puzzle strategy includes at least one of the following: a puzzle strategy that removes tilted images, and a puzzle strategy that retains some tilted images. Obtain the display puzzle strategy selected by the user through the configuration interface; If the user selects a mosaic strategy that removes tilted images, the process of removing tilted images from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be mosaicked includes: determining a cluster center vector based on the optical axis vector of each original image; and removing tilted images with an angle greater than a set threshold from the multiple original images based on the angle between the optical axis vector of each original image and the cluster center vector to obtain multiple orthophoto images, which are then used as images to be mosaicked. If the user chooses a stitching strategy that retains some tilted images, the tilted images are removed from the multiple original images based on the optical axis vectors of the multiple original images to obtain multiple images to be stitched. This includes: determining a cluster center vector based on the optical axis vector of each original image; normalizing the optical axis vectors of the multiple original images based on the cluster center vector; defining a plane perpendicular to the cluster center vector as a reference plane; taking the point through which the cluster center vector passes as the center point and the point through which the optical axis vectors of the multiple original images pass as the judgment point in the reference plane; removing the original images corresponding to the judgment points whose distance from the center point is greater than a set distance to obtain multiple images to be stitched.

4. The UAV image stitching method according to claim 1, characterized in that, If the user selects a confidence-based control strategy, multiple target control points are determined from the plurality of candidate control points based on this control strategy, including: Multiple initial control points are determined from the plurality of candidate control points. For each initial control point, a candidate control point whose distance from the initial control point satisfies the desired distance is determined from the plurality of candidate control points. Search for the control point with the highest confidence among the control points that meet the desired distance, and use it as the target control point corresponding to the initial control point.

5. The UAV image stitching method according to claim 4, characterized in that, Multiple initial control points are determined from the plurality of candidate control points, including: Obtain a first result view of the aerial triangulation reconstruction, which displays multiple control points to be selected; Determine the largest bounding polygon formed by the plurality of control points to be selected in the first result view, and define it as the first polygon; Determine a plurality of control points that are closest to a plurality of edges and / or a plurality of corners of the first polygon, and use them as the plurality of initial control points.

6. The UAV image stitching method according to claim 1, characterized in that, If the user selects an inward-based control strategy, multiple target control points are determined from the plurality of selectable control points based on this control strategy, including: Obtain the second result view of the aerial triangulation reconstruction, determine the polygon formed by the control points to be selected in the second result view, and define it as the second polygon; Multiple first initial control points are determined from the control points to be selected in the direction of the first pair of intersection lines of the second polygon, and multiple second initial control points are determined from the control points to be selected in the direction of the second pair of intersection lines of the second polygon, wherein the direction of the first diagonal line is different from the direction of the second diagonal line; A first target control point is determined by moving the first initial control point inward along the first diagonal towards the center of the second polygon; a second target control point is determined by moving the second initial control point inward along the second diagonal towards the center of the second polygon.

7. The UAV image stitching method according to claim 6, characterized in that, By moving inward from the first initial control point along the first diagonal towards the center of the second polygon, a first target control point corresponding to the first initial control point is determined, including: The first initial control point is moved inward along the first diagonal direction towards the center point of the second polygon by a first inward length or a first proportion to obtain the first desired position point; the plurality of candidate control points are traversed, and the candidate control point with the smallest distance from the first desired position point is determined from the plurality of candidate control points as the first target control point corresponding to the first initial control point. The second target control point is determined by moving inward from the second initial control point along the second diagonal towards the center of the second polygon, including: The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point; the plurality of candidate control points are traversed, and the candidate control point with the smallest distance from the second desired position point is determined from the plurality of candidate control points as the second target control point corresponding to the second initial control point.

8. The UAV image stitching method according to claim 7, characterized in that, The method further includes: the second result view has an overlap heatmap at the control point to be selected, the overlap heatmap representing the degree of overlap of the control point to be selected; The first initial control point is moved inward along the first diagonal direction towards the center point of the second polygon by a first inward length or a first proportion to obtain the first desired position point, including: Determine the overlap of each selectable control point distributed along the first diagonal direction; determine the first ratio based on the overlap of the selectable control points distributed along the first diagonal direction; move the first initial control point inward along the first diagonal direction towards the center point of the second polygon by the first ratio to obtain the first desired position point; wherein, the higher the overlap, the smaller the first ratio; and / or The second initial control point is moved inward along the second diagonal towards the center point of the second polygon by a second inward length or a second proportion to obtain the second desired position point, including: Determine the overlap of each selectable control point distributed along the second diagonal direction; determine the second ratio based on the overlap of the selectable control points distributed along the second diagonal direction; move the second initial control point inward along the second diagonal direction towards the center point of the second polygon by the second ratio to obtain the second desired position point; wherein, the higher the overlap, the smaller the second ratio.

9. The UAV image stitching method according to claim 1, characterized in that, Determine the confidence level of the target pose coordinates for each original image, including: The target pose coordinates include three-dimensional coordinates. Based on the initial three-dimensional coordinates of each original image and the three-dimensional coordinates after aerial triangulation reconstruction, the Euclidean distance between the two coordinates is calculated. The confidence level of the target pose coordinates of the original image is determined based on the Euclidean distance between the two coordinates; wherein, the larger the Euclidean distance between the two coordinates, the lower the confidence level of the target pose coordinates of the original image.

10. The UAV image stitching method according to claim 1, characterized in that, If the user selects a control strategy based on input instructions, multiple target control points are determined from the plurality of selectable control points based on this control strategy, including: The configuration interface displays multiple control points selected from the clusters of the control points to be selected. The control point selected by the user from multiple control points in the recommended cluster is obtained as the target control point.

11. The UAV image stitching method according to claim 1, characterized in that, Based on the multiple target control points, the multiple images to be stitched together are stitched together to obtain a joined image, including: determining a transformation matrix based on the multiple target control points; converting the three-dimensional coordinates of each image to be stitched into pixel coordinates based on the transformation matrix; and stitching the multiple images to be stitched together based on their pixel coordinates to obtain a joined image; and / or Based on the multiple target control points, the multiple images to be stitched together are stitched together to obtain a joined image, including: performing digital differential correction on the multiple images to be stitched based on the multiple target control points, and stitching the digitally differentiated images to be stitched together to obtain a joined image.

12. A device for stitching images from unmanned aerial vehicles (UAVs), characterized in that, The device used for UAV image stitching includes: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the UAV image stitching method as described in any one of claims 1 to 11; A monitor used to display the configuration interface.

Images