Unmanned aerial vehicle real-time image splicing system and splicing method based on navigation assistance and grid homography modeling

Abstract

The invention relates to the technical field of unmanned aerial vehicle airborne calculation image processing, and discloses an unmanned aerial vehicle real-time image splicing system and splicing method based on navigation assistance and grid homography modeling. Comprising a data synchronous acquisition module, a spatial initial value calculation module, an overlapping region determination module, a feature matching module, a local homography matrix calculation module, a directional grid construction module and an image reconstruction and splicing module. And the data synchronous acquisition module is used for synchronously acquiring an airborne image sequence and real-time navigation data. According to the method, the accurate spatial transformation initial value is provided for feature matching through deep fusion of the real-time navigation information of the unmanned aerial vehicle, and the algorithm can convert the traditional global search into target-specific local accurate search in the feature extraction and matching stage, so that the invalid calculation is greatly reduced, the calculation complexity of feature matching is greatly reduced, and the accuracy of feature matching is improved. The real-time performance of the system is obviously improved, and the mismatching rate is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of airborne computational image processing technology for unmanned aerial vehicles (UAVs), specifically to a real-time image stitching system and method for UAVs based on navigation assistance and mesh homography modeling. Background Technology

[0002] UAV-borne real-time image stitching technology mainly relies on image registration and fusion methods from the field of computer vision. Based on its core technology, existing solutions can be summarized into three main categories: algorithms based on classic global feature matching, represented by algorithms such as SIFT and SURF, which extract and match scale- and rotation-invariant feature points across the entire image to solve a global homography matrix to complete image alignment and stitching; algorithms that incorporate external sensor assistance, where some solutions attempt to fuse the UAV's pose information to reduce the computational burden of pure vision; and fast algorithms based on lightweight features, where another category of solutions uses computationally more efficient features to achieve real-time performance. However, under the specific constraints of airborne edge computing platforms, achieving both high precision and high real-time performance in stitching remains a fundamental challenge. Existing technical architectures struggle to balance limited computing power, complex scenarios, and dynamic flight. High-precision algorithms, such as SIFT / SURF, suffer from extremely high computational complexity due to their dense full-image computation and floating-point descriptors, making them unsuitable for real-time processing of high-frame-rate video on embedded platforms like Jetson. Conversely, lightweight features like FAST and ORB, used to pursue real-time performance, lack sufficient discriminative power and robustness in their descriptors. In aerial photography scenarios with wide viewing angles, repetitive textures, and large parallax, they are prone to mismatches, leading to stitching misalignment, ghosting, and other issues, failing to meet the high precision requirements of applications such as surveying and mapping. Therefore, at resource-constrained edge computing environments, existing solutions either sacrifice precision for speed or real-time performance to maintain precision, making it difficult to simultaneously achieve both real-time and high-precision goals. Summary of the Invention

[0003] To address the shortcomings of existing real-time UAV image stitching schemes mentioned in the background art, this invention provides a real-time UAV image stitching method based on navigation assistance and mesh homography modeling, which has the advantages of real-time efficiency and accurate stitching, and solves the technical problems mentioned in the background art.

[0004] This invention provides the following technical solution: a real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling, comprising the following steps: S1. Synchronously acquire the airborne image sequence of the UAV and the corresponding real-time navigation data, wherein the real-time navigation data includes inertial measurement unit (IMU) and / or real-time dynamic carrier phase differential (RTK) data; S2. Based on the real-time navigation data corresponding to two adjacent frames, calculate the initial value of the spatial transformation between the two adjacent frames. The initial value of the spatial transformation is the initial homography matrix. ; S3, Based on the initial homography matrix Determine the overlapping area between the two adjacent image frames; S4. Perform image feature extraction and matching within the overlapping area to obtain matching feature point pairs; S5. Divide the overlapping region into multiple sub-grids, and calculate the local homography matrix corresponding to each sub-grid based on the matching feature point pairs; S6. Based on the heading angle change information in the real-time navigation data, introduce directional constraints for each local homography matrix and construct a directional homography matrix network; S7. Using the directional homography matrix network, the two adjacent frames are mapped to the same coordinate system and fused to output a stitched image.

[0005] Preferably, in step S2, the initial homography matrix is ​​calculated. Specifically, it includes: Based on the real-time navigation data, obtain the UAV attitude change matrix ΔR and displacement vector ΔT at adjacent time points; Based on the camera intrinsic parameter matrix K, the extrinsic parameter transformation relationship between the camera and the UAV body, the transformation matrix ΔR, and the displacement vector ΔT, the initial homography matrix is ​​calculated using a geometric projection model. .

[0006] Preferably, determining the overlapping region in step S3 specifically involves: Using the initial homography matrix The boundary points of the current frame image are projected onto the coordinate system of the previous frame image, and the polygonal region formed by the projection is used as the overlapping region.

[0007] Preferably, step S4 specifically includes: Feature points are extracted within the overlapping area using the FAST corner detection algorithm; The feature points are described using the BRIEF descriptor; Combined with the initial homography matrix The provided spatial constraints are used for feature point matching.

[0008] Preferably, the construction of the directional homography matrix network in step S6 specifically includes: Based on the real-time navigation data, the rate of change of the heading angle of the UAV at the time of acquisition of the two adjacent frames of images is obtained; Using the rate of change of the heading angle as the directional weight parameter θ, an optimization objective function is constructed. The optimization objective function includes at least: a data term that makes the local homography matrix fit the matching feature point pairs in its corresponding subgrid, a smoothing term that keeps the local homography matrices of adjacent subgrids continuous on the boundary, and a global directional consistency term that makes the changing trend of each local homography matrix consistent with the directional weight parameter θ. By solving the objective function, the optimized local homography matrices are obtained, forming the directional homography matrix network.

[0009] Preferably, step S7 specifically includes: Using the optimized local homography matrices in the directional homography matrix network, the image pixels in their corresponding sub-grids are remapped respectively. The remapped image is smoothed using bilinear interpolation in the subgrid boundary region. A multi-resolution pyramid fusion algorithm is used to fuse overlapping images to eliminate stitching gaps.

[0010] The real-time image stitching system for UAVs based on navigation assistance and mesh homography modeling includes a data synchronization acquisition module, a spatial initial value calculation module, an overlapping area determination module, a feature matching module, a local homography matrix calculation module, a directional mesh construction module, and an image reconstruction and stitching module. The data synchronization acquisition module is used to synchronously acquire airborne image sequences and real-time navigation data, and the spatial initial value calculation module is used to calculate the initial homography matrix between adjacent frame images based on the real-time navigation data. The overlapping region determination module is used to determine the overlapping region based on the initial homography matrix. The image overlap region is determined. The feature matching module is used to perform image feature extraction and matching within the overlap region. The local homography matrix calculation module is used to divide the overlap region into grids and calculate the local homography matrix of each sub-grid. The directional grid construction module is used to introduce heading angle change information as a directional constraint to construct a directional homography matrix network. The image reconstruction and stitching module is used to complete image mapping and fusion using the directional homography matrix network and output the stitching result.

[0011] Preferably, the feature matching module specifically includes a feature extraction unit, a feature description unit, and a constraint matching unit; The feature extraction unit is used to extract corner features in the overlapping region using the FAST algorithm; the feature description unit is used to perform binary description of the corner features using the BRIEF descriptor; and the constraint matching unit is used to combine the initial homography matrix. Given the projection position constraints, feature point matching is performed, and the RANSAC algorithm is used to eliminate false matches.

[0012] Preferably, the directional mesh construction module is specifically used for: The rate of change of heading angle is calculated based on real-time navigation data as a directional parameter; Construct an optimization function that includes data items, smoothing items, and a global direction consistency item; Using the local homography matrix of each subgrid as the optimization variable, the optimization function is solved iteratively to obtain a set of optimized local homography matrices that satisfy the directional constraints, thus forming the directional homography matrix network.

[0013] The present invention has the following beneficial effects: 1. This invention provides accurate initial values ​​for spatial transformation for feature matching by deeply fusing real-time navigation information (IMU / RTK) from unmanned aerial vehicles. This enables the algorithm to transform the traditional global search into a targeted local precise search during the feature extraction and matching stages, thereby significantly reducing invalid computation, greatly reducing the computational complexity of feature matching, significantly improving the real-time performance of the system, and effectively reducing the false matching rate.

[0014] 2. This invention, by constructing a directional homography matrix network that integrates the trend of heading angle variation, enables the image stitching model to be upgraded from static geometric alignment to dynamic motion-aware mapping. It can compensate for nonlinear distortions caused by rapid maneuvers such as turning and diving of UAVs, effectively solving the stitching discontinuities and tearing problems that traditional mesh models easily produce in dynamic scenes, thus achieving a smooth and accurate stitching effect even under complex flight attitudes.

[0015] 3. This invention adopts a simple systematic architecture of navigation prediction ROI-local mesh refinement-dynamic network optimization, which enables the entire algorithm to run stably on embedded edge platforms with limited computing resources. It realizes real-time online processing of high-resolution, high-frame-rate aerial video streams, meets the comprehensive and stringent requirements of UAV mapping, inspection and reconnaissance tasks for processing speed, accuracy and power consumption, and has strong engineering implementation capabilities. Attached Figure Description

[0016] Figure 1 This is a diagram showing the overall system framework of the present invention; Figure 2 This is a schematic diagram illustrating the ROI region limitation and dynamic matching based on initial spatial values ​​in this invention. Figure 3 This is a schematic diagram of the directional homography matrix network of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Method Implementation Examples: During the drone's flight, the onboard camera captures image sequences at a fixed frame rate. , , …, Simultaneously, the UAV flight control system outputs a high-frequency navigation data stream in real time via serial port or network interface. This data stream contains at least data from the inertial measurement unit (IMU) and / or the real-time dynamic carrier phase differential (RTK) system, specifically including: three-dimensional acceleration, three-dimensional angular velocity, attitude angle, and three-dimensional position information. To ensure data spatiotemporal alignment, the system uses hardware timestamps to mark each frame of image and each set of navigation data, ensuring that each frame of image... Each can be associated with a precisely corresponding navigation state. .

[0019] Let the current frame be The reference frame is The corresponding navigation states are as follows: and First, according to and Calculate the rotation matrix of the body attitude change from time t-1 to t. Translation vector Next, using the pre-calibrated camera extrinsic parameter matrix M, the body motion is transformed into the camera coordinate system: , Then, assuming that the scene between adjacent frames satisfies the local plane assumption, the initial homography matrix can be solved based on epipolar geometry and the camera intrinsic parameter matrix K. The calculation formula is as follows: ,in, is the transpose of the scene mean plane normal vector, the initial value of which can be estimated from navigation data, and d is the initial distance estimate from the camera to the plane. That is, from arrive The preliminary projection transformation model.

[0020] To minimize unnecessary computation, full-image feature extraction is not performed. The initial homography matrix calculated in the previous step is used instead. , will the current frame The coordinates of the four corner points (0,0), (W,0), (W,H), and (0,H) are projected onto the reference frame. In the coordinate system: ~ Where ~ indicates the equality of homogeneous coordinates, W and H are the image width and height respectively, and the four corner points after projection are connected in the reference frame. This forms a convex quadrilateral region, which is theoretically the maximum overlap region. Further, the circumscribed rectangle of this quadrilateral region can be calculated, and an appropriate expansion of several pixels can be used as a buffer to finally obtain the Region of Interest (ROI) for feature extraction. At the same time, utilizing The inverse matrix can be found in the current frame. Determine the corresponding ROI region above. .

[0021] Employing the highly computationally efficient FAST corner detection algorithm, respectively in and Inner corner detection. To maintain rotation invariance of feature points, the orientation of each detected corner point is calculated. A BRIEF binary descriptor is used to describe each feature point, generating a 256-bit binary string. This process is computationally fast, and the Hamming distance is used for matching, resulting in extremely high efficiency. This utilizes the core elements of navigation priors. For the reference frame ROI... One feature point No longer in the entire current frame ROI Instead of searching for matching points within the search engine, it utilizes... ,Will Projecting onto the current frame yields the predicted position. .Right now Then, only in Within a smaller search window, Hamming distance is used for nearest neighbor search to find the best matching point. This method reduces the search range from tens of thousands of pixels to hundreds of pixels, greatly improving matching speed and accuracy. For the feature point pairs obtained from the initial matching, the RANSAC algorithm is used for iterative filtering with a homography matrix model, ultimately yielding a high-quality set of matching feature point pairs. .

[0022] To handle parallax and local deformation, overlapping regions are meshed. This is based on the ROI region from the previous frame. Based on the baseline, it is uniformly divided into M×N subgrids. For the i-th subgrid... From the matching point pairs obtained in step S4, select point pairs that fall within the grid. Using these local point pairs, calculate a local homography matrix that applies only to this grid using the least squares method or the DLT algorithm. For grids with too few matching points, a global homography matrix can be used temporarily. Alternatively, fill the matrix of adjacent grids.

[0023] From the navigation data, calculate the rate of change ω of the UAV's heading angle within the time interval of adjacent frame acquisitions. This rate of change ω reflects the trend of the aircraft's horizontal turning. Then, combine all the local homography matrices obtained in the previous step { , ,…, As an optimization variable, an energy function E is constructed for global optimization: Requires optimized It can still fit matching point pairs within its mesh very well. It is required that the transformations of adjacent meshes be as continuous as possible at the boundaries, which can be achieved by constraining the consistency of the projected positions of adjacent mesh vertices. Flight trends are incorporated into the visual model. A "major deformation axis direction" is defined for each mesh, and optimization is used to spatially correlate the distribution of all mesh major deformation axis directions with the rate of change of the heading angle ω. For example, when ω is positive (right turn), the deformation degree of the right-hand mesh is constrained to be slightly greater than that of the left-hand mesh to simulate the perspective change trend caused by turning. , , Let be the weight coefficients for each term. Using nonlinear optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method, the energy function E is minimized to obtain an optimal set of local homography matrices that satisfies both local image matching and motion smoothness and directional trends. , , …, This set constitutes the directional homography matrix network.

[0024] Using the constructed directional homography matrix network, the current frame spliced ​​to the reference frame The baseline is a global panoramic view or a local mosaic image. For the current frame. Each subgrid Use its corresponding optimization matrix The algorithm precisely remaps all pixels within the grid onto the output canvas. At grid boundaries, bilinear interpolation is used to ensure smooth pixel value transitions and avoid noticeable grid boundary lines. For overlapping areas between the current frame and existing stitched images, a Laplacian pyramid fusion algorithm is used for multi-resolution fusion. This algorithm performs weighted averaging of the image across different layers of the Gaussian pyramid, maximizing the elimination of lighting differences and stitching gaps for seamless stitching. The fused result is then updated to the panoramic image cache. The current frame... The system uses the above method repeatedly as a reference frame for the next round of processing, thus enabling real-time splicing and output of continuous video streams.

[0025] System Implementation Example: Please see Figure 1 A real-time image stitching system for UAVs based on navigation assistance and mesh homography modeling is typically deployed on an UAV onboard computing platform and includes the following functional modules: The data synchronization acquisition module comprises an image acquisition module and a navigation data acquisition module. The image acquisition module's image acquisition card acquires raw video streams from the camera, while the navigation data acquisition module's navigation data interface reads IMU / RTK data from the flight control computer via serial port or Ethernet. Internally, a time synchronization thread runs, assigning a unified timestamp to each frame of image and each set of navigation data based on the hardware PPS signal or a high-precision system clock, ensuring that the time deviation between the two is within milliseconds, and outputting synchronized data. , Data pairs.

[0026] The preprocessing module includes a spatial initial value calculation module and an overlapping region determination module. The spatial initial value calculation module receives synchronization data from the module. Internally, it stores the calibration values ​​of the camera intrinsic parameter matrix K and extrinsic parameter matrix M, as well as the initial parameters (n, d) of the scene plane. The module determines the overlapping region based on two consecutive pairs of ( , )and( , According to the formula described in the method embodiment, the initial homography matrix is ​​calculated in real time. The calculation process mainly involves matrix multiplication, which is suitable for parallel acceleration on embedded platform GPUs or NPUs; the overlapping region determination module receives the current frame. and the initial homography matrix Its core is a geometric projection calculation unit, which, according to the method described in the implementation example, quickly calculates the corner coordinates and circumscribed rectangle parameters of the overlapping area between the current frame and the reference frame. , The two rectangle parameters are then output as ROI information.

[0027] The feature matching module, also known as the ROI matching module, receives the ROI regions of two images. , )and .

[0028] The feature extraction and matching module performs the FAST corner detection algorithm on the input ROI image patch through the feature extraction unit; the feature description unit calculates the BRIEF descriptor for the detected corners; and the constraint matching unit utilizes... For each feature point in the reference frame, a predicted search region in the current frame is generated, and fast matching is performed within this small region using Hamming distance. After matching is completed, the RANSAC subroutine is called to remove outliers, and finally a refined list of matched point pairs is output.

[0029] The local homography matrix calculation module receives a list of matching point pairs and the ROI region. The module first divides the grid according to a preset number of grids (e.g., 8×8), in... Generate the vertex coordinates of the grid. Then, traverse each grid, collect the matching point pairs falling within it, and use these point pairs to solve for the initial local homography matrix of the grid using linear least squares. .

[0030] The directional mesh building module, also known as the directional homography mesh generation module, receives all initial local homography matrices { } and the rate of change of heading angle ω from navigation data. Internally, the module implements a nonlinear optimizer. The optimizer, based on the energy function E defined in the method embodiment, with { } is the initial value, with ω as... The input terms are used for iterative optimization. After convergence, the final directional homography matrix network is output. }

[0031] The image reconstruction and stitching module receives the current frame image. And directional homography matrix network { }. Includes: Mesh remapping cells apply their corresponding [functions] in parallel to each mesh. Perform pixel coordinate transformation and bilinear interpolation sampling to... Each grid block is mapped to the output canvas; the multi-resolution fusion unit constructs a Laplacian pyramid and performs fusion calculations on the parts of the output canvas that overlap with the existing panoramic image, and finally generates an updated seamless panoramic image, which is then output to the display or storage interface.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling, characterized in that, Includes the following steps: S1. Synchronously acquire the airborne image sequence of the UAV and the corresponding real-time navigation data, wherein the real-time navigation data includes inertial measurement unit (IMU) and / or real-time dynamic carrier phase differential (RTK) data; S2. Based on the real-time navigation data corresponding to two adjacent frames, calculate the initial value of the spatial transformation between the two adjacent frames. The initial value of the spatial transformation is the initial homography matrix. ; S3, Based on the initial homography matrix Determine the overlapping area between the two adjacent image frames; S4. Perform image feature extraction and matching within the overlapping area to obtain matching feature point pairs; S5. Divide the overlapping region into multiple sub-grids, and calculate the local homography matrix corresponding to each sub-grid based on the matching feature point pairs; S6. Based on the heading angle change information in the real-time navigation data, introduce directional constraints for each local homography matrix and construct a directional homography matrix network; S7. Using the directional homography matrix network, the two adjacent frames are mapped to the same coordinate system and fused to output a stitched image.

2. The real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling as described in claim 1, characterized in that: In step S2, the initial homography matrix is ​​calculated. Specifically, it includes: The change matrix of the UAV attitude at adjacent time points is obtained based on the real-time navigation data. and displacement vector ; Based on the camera intrinsic parameter matrix K, the extrinsic parameter transformation relationship between the camera and the UAV body, and the aforementioned transformation matrix and the displacement vector The initial homography matrix is ​​obtained by calculating using a geometric projection model. .

3. The real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling according to claim 1 or 2, characterized in that: In step S3, determining the overlapping region specifically involves: Using the initial homography matrix The boundary points of the current frame image are projected onto the coordinate system of the previous frame image, and the polygonal region formed by the projection is used as the overlapping region.

4. The real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling as described in claim 1, characterized in that: Step S4 specifically includes: Feature points are extracted within the overlapping area using the FAST corner detection algorithm; The feature points are described using the BRIEF descriptor; Combined with the initial homography matrix The provided spatial constraints are used for feature point matching.

5. The UAV real-time image stitching method based on navigation assistance and mesh homography modeling according to claim 1, characterized in that: The construction of the directional homography matrix network in step S6 specifically includes: Based on the real-time navigation data, the rate of change of the heading angle of the UAV at the time of acquisition of the two adjacent frames of images is obtained; Using the rate of change of the heading angle as the directional weight parameter θ, an optimization objective function is constructed. The optimization objective function includes at least: a data term that makes the local homography matrix fit the matching feature point pairs in its corresponding subgrid, a smoothing term that keeps the local homography matrices of adjacent subgrids continuous on the boundary, and a global directional consistency term that makes the changing trend of each local homography matrix consistent with the directional weight parameter θ. By solving the objective function, the optimized local homography matrices are obtained, forming the directional homography matrix network.

6. The real-time image stitching method for UAVs based on navigation assistance and mesh homography modeling according to claim 1, characterized in that: Step S7 specifically includes: Using the optimized local homography matrices in the directional homography matrix network, the image pixels in their corresponding sub-grids are remapped respectively. The remapped image is smoothed using bilinear interpolation in the subgrid boundary region. A multi-resolution pyramid fusion algorithm is used to fuse overlapping images to eliminate stitching gaps.

7. A real-time image stitching system for UAVs based on navigation assistance and mesh homography modeling, used to implement the UAV-borne real-time image stitching method as described in any one of claims 1 to 6, characterized in that: It includes a data synchronization acquisition module, a spatial initial value calculation module, an overlapping region determination module, a feature matching module, a local homography matrix calculation module, a directional grid construction module, and an image reconstruction and stitching module; The data synchronization acquisition module is used to synchronously acquire airborne image sequences and real-time navigation data, and the spatial initial value calculation module is used to calculate the initial homography matrix between adjacent frame images based on the real-time navigation data. The overlapping region determination module is used to determine the overlapping region based on the initial homography matrix. The image overlap region is determined. The feature matching module is used to perform image feature extraction and matching within the overlap region. The local homography matrix calculation module is used to divide the overlap region into grids and calculate the local homography matrix of each sub-grid. The directional grid construction module is used to introduce heading angle change information as a directional constraint to construct a directional homography matrix network. The image reconstruction and stitching module is used to complete image mapping and fusion using the directional homography matrix network and output the stitching result.

8. The UAV real-time image stitching system based on navigation assistance and mesh homography modeling according to claim 7, characterized in that: The feature matching module specifically includes a feature extraction unit, a feature description unit, and a constraint matching unit; The feature extraction unit is used to extract corner features in the overlapping region using the FAST algorithm; the feature description unit is used to perform binary description of the corner features using the BRIEF descriptor; and the constraint matching unit is used to combine the initial homography matrix. Given the projection position constraints, feature point matching is performed, and the RANSAC algorithm is used to eliminate false matches.

9. The UAV real-time image stitching system based on navigation assistance and mesh homography modeling according to claim 7, characterized in that: The directional mesh construction module is specifically used for: The rate of change of heading angle is calculated based on real-time navigation data as a directional parameter; Construct an optimization function that includes data items, smoothing items, and a global direction consistency item; Using the local homography matrix of each subgrid as the optimization variable, the optimization function is solved iteratively to obtain a set of optimized local homography matrices that satisfy the directional constraints, thus forming the directional homography matrix network.

Images