Unmanned aerial vehicle image real-time splicing method and device

Abstract

The invention discloses an unmanned aerial vehicle image real-time splicing method and device, and the method comprises the steps that an unmanned aerial vehicle flies according to a preset route, synchronously collects multi-source data and transmits the multi-source data to an airborne edge calculation unit, and the multi-source data comprises a continuous image sequence, IMU data and GPS data; preprocessing the image of the image sequence to obtain a preprocessed image; performing feature extraction and pose estimation on the preprocessed image by adopting an improved ORB-SLAM3 algorithm, optimizing a global pose in combination with the IMU data and the GPS data, and outputting an optimized pose; performing multi-resolution image registration according to the optimized pose to obtain a registered image; and carrying out adaptive image fusion on the registered image to generate a panorama. According to the method and the device provided by the invention, high-precision, low-delay and high-environment-adaptability unmanned aerial vehicle image real-time splicing can be realized.

Description

Technical Field

[0001] This invention relates to the fields of UAV remote sensing mapping and computer vision technology, and in particular to a method and apparatus for real-time image stitching of UAVs. Background Technology

[0002] Drone aerial photography technology has become an important means of acquiring geographic information due to its advantages such as flexibility, efficiency, and wide coverage. Real-time image stitching is a core technology for rapidly fusing multiple overlapping frames of images continuously acquired by drones into a large-scale panoramic image, which is of great significance for improving data processing efficiency and supporting real-time decision-making (such as disaster assessment and engineering monitoring). However, existing technologies have the following bottlenecks:

[0003] Insufficient real-time performance: Traditional stitching methods rely on offline feature matching (such as SIFT, SURF), which has high computational complexity and is difficult to achieve real-time processing on low-computing-power UAV platforms.

[0004] Limited positioning accuracy: When a drone is in flight, it is affected by airflow and attitude changes. Relying solely on GPS or inertial navigation (INS) is prone to cumulative errors, leading to image registration misalignment.

[0005] Weak environmental adaptability: Complex lighting (such as shadows and highlights) and dynamic occlusion (such as moving vehicles and vegetation) can interfere with feature matching, and traditional algorithms are not robust enough.

[0006] Poor stitching quality: Color difference correction and fusion algorithms in overlapping areas (such as direct averaging) can easily lead to obvious seams and sudden changes in brightness, affecting the visual consistency of panoramic images.

[0007] To address the aforementioned issues, and considering the real-time positioning and mapping capabilities of SLAM technology, there is an urgent need for a high-precision, low-latency, and environmentally adaptable real-time image stitching method and device for UAVs.

[0008] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] This invention provides a method and apparatus for real-time image stitching from unmanned aerial vehicles (UAVs), which can solve at least one of the problems in the background art.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A real-time image stitching method for unmanned aerial vehicles (UAVs) includes: S1: The UAV flies along a preset route, synchronously collects multi-source data and transmits it to an onboard edge computing unit, wherein the multi-source data includes a continuous image sequence, IMU data, and GPS data; S2: The image sequence is preprocessed to obtain a preprocessed image; S3: An improved ORB-SLAM3 algorithm is used to extract features and estimate pose of the preprocessed image, and the global pose is optimized by combining the IMU data and the GPS data to output the optimized pose; S4: Multi-resolution image registration is performed based on the optimized pose to obtain a registered image; S5: Adaptive image fusion is performed on the registered image to generate a panoramic image.

[0012] Preferably, the UAV is equipped with a camera, an IMU module and a GPS module, and flies along the preset route to collect continuous image sequences at a frequency of not less than 30 frames per second, and synchronously records the IMU data and the GPS data. The multi-source data is transmitted to the airborne edge computing unit after being synchronized by timestamp.

[0013] Preferably, the image preprocessing of the image sequence to obtain the preprocessed image includes the following steps: S21: performing distortion correction on the image: based on camera calibration parameters, the Zhang Zhengyou calibration method is used to perform barrel / pincushion distortion correction on the image; S22: performing uniform illumination and color matching on the distortion-corrected image: dividing the distortion-corrected image into pixel sub-blocks, calculating the mean gray value μ and variance σ of each block, adjusting the gray value of the sub-blocks through a Wallis filter to eliminate illumination differences, and obtaining the preprocessed image.

[0014] Preferably, the improved ORB-SLAM3 algorithm is used to perform feature extraction and pose estimation on the preprocessed image, including the following steps: S31: Feature extraction and matching: The ORB algorithm is used to extract feature points of the current frame image, wherein the number of pyramid layers is ≤4, the number of feature points is 800-1200, and the feature points are matched with the previous frame feature points using the BRIEF descriptor, and the RANSAC algorithm is used to remove mismatched points; S32: Initial pose estimation: Combining the IMU data and the matched feature point pairs, the PnP algorithm is used to calculate the initial pose of the camera, including the rotation matrix R and the translation vector t; S33: Sparse point cloud construction: The matched feature points are projected onto three-dimensional space, and points with reprojection errors ≤1 pixel are selected to construct a sparse point cloud map; S34: Multi-sensor fusion optimization: The GPS data is converted into UTM plane coordinates, and fused with the pose estimated by SLAM through Kalman filtering. The state equation and observation equation respectively model the UAV motion and sensor noise, and correct the cumulative error.

[0015] Preferably, the process of obtaining a registered image by multi-resolution image registration based on the optimized pose includes the following steps: S41: Homography matrix calculation: Based on the optimized pose, calculate the homography matrix H of adjacent images and project the image of the next frame onto the coordinate system of the previous frame; S42: Subpixel-level registration: Construct a Laplacian pyramid, optimize the registration parameters layer by layer from low-resolution layer to high-resolution layer, and eliminate subpixel displacement error by maximizing the cross-correlation coefficient.

[0016] Preferably, adaptive Laplacian pyramids are used for adaptive image fusion, and the weights of overlapping regions are dynamically adjusted according to the distance from the pixel to the boundary.

[0017] w(x,y)=d(x,y) / d max

[0018] Where, d max w(x,y) is the maximum width of the overlapping region, w(x,y) is the weight value, and d(x,y) is the distance from the pixel to the boundary. The pyramid is reconstructed layer by layer to generate a seamless panoramic image with seam marks ≤ 0.3 pixels.

[0019] Preferably, the method further includes: encapsulating the panoramic image and tag metadata and outputting them synchronously; the tag metadata includes timestamps and pose information.

[0020] Preferably, the method further includes: synthesizing the panoramic image in real time using the synchronized pose and image data of multiple UAVs.

[0021] This invention also provides a real-time image stitching device for unmanned aerial vehicles (UAVs), implementing any of the methods described above, comprising: a data acquisition module: used to synchronously acquire multi-source data and transmit it to an onboard edge computing unit while the UAV is flying along a preset route, wherein the multi-source data includes continuous image sequences, IMU data, and GPS data; a preprocessing module: used to preprocess the image sequences to obtain preprocessed images; a pose estimation module: used to perform feature extraction and pose estimation on the preprocessed images using an improved ORB-SLAM3 algorithm, optimize the global pose by combining the IMU data and the GPS data, and output the optimized pose; a registration and stitching module: used to perform multi-resolution image registration based on the optimized pose to obtain registered images; and a fusion output module: used to perform adaptive image fusion on the registered images to generate a panoramic image.

[0022] The present invention also provides a non-volatile computer-readable storage medium having a computer program stored thereon, the computer program implementing any of the methods described above.

[0023] The present invention has the following beneficial effects:

[0024] This invention proposes a high-precision, low-latency, and environmentally adaptable real-time image stitching method and device for unmanned aerial vehicles (UAVs). By integrating visual SLAM technology, multi-sensor data, and multi-resolution image processing algorithms, it achieves high real-time performance: real-time stitching of 4K image sequences is completed on an airborne edge computing unit (processing latency ≤200ms); sub-meter-level positioning accuracy: global pose error ≤0.5 meters; strong environmental adaptability: feature matching success rate ≥90% under complex lighting and dynamic occlusion scenarios; seamless fusion: stitching marks ≤0.3 pixels, significantly improving visual consistency. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a real-time image stitching method for unmanned aerial vehicles (UAVs) according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of a multi-source data synchronous acquisition and preprocessing logic block according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of an improved ORB-SLAM3 pose estimation and multi-sensor fusion flowchart according to an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of a Laplace pyramid with multi-resolution registration according to an embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the weight allocation of overlapping regions in a Laplace pyramid fusion according to an embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of a real-time image stitching device for unmanned aerial vehicles (UAVs) according to an embodiment of the present invention. Detailed Implementation

[0031] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0032] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for fixing, coupling, or communication.

[0033] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0035] See Figure 1 An embodiment of the present invention provides a real-time image stitching method for unmanned aerial vehicles (UAVs), characterized in that it includes:

[0036] S1: The UAV flies along a preset route, simultaneously collecting multi-source data and transmitting it to the onboard edge computing unit. The multi-source data includes continuous image sequences, IMU data, and GPS data.

[0037] S2: Image preprocessing of the image sequence yields a preprocessed image;

[0038] S3: The improved ORB-SLAM3 algorithm is used to extract features and estimate pose of the preprocessed image. The global pose is optimized by combining the IMU data and the GPS data, and the optimized pose is output:

[0039] S4: Perform multi-resolution image registration based on the optimized pose to obtain the registered image;

[0040] S5: Perform adaptive image fusion on the registered images to generate a panoramic image.

[0041] This invention is a real-time image stitching method for UAVs based on Simultaneous Localization and Mapping (SLAM) technology. It is applicable to application scenarios that require real-time synthesis of large-scale, high-resolution images, such as UAV aerial surveying and mapping, disaster emergency monitoring, and environmental inspection.

[0042] By integrating visual SLAM technology, multi-sensor data, and multi-resolution image processing algorithms, this invention achieves the following objectives:

[0043] High real-time performance: Real-time stitching of 4K image sequences is completed on the airborne edge computing unit (processing latency ≤200ms);

[0044] Sub-meter level positioning accuracy: global pose error ≤ 0.5 meters;

[0045] Strong environmental adaptability: Feature matching success rate ≥90% in complex lighting and dynamic occlusion scenarios; Seamless fusion: Stitching marks ≤0.3 pixels, significantly improving visual consistency.

[0046] This invention significantly outperforms traditional methods in terms of real-time performance (processing delay ≤200ms), positioning accuracy (global error ≤0.5m), robustness in complex environments (feature matching success rate ≥90%), and stitching quality (seam marks ≤0.3 pixels). It can be widely applied in fields such as UAV aerial photography and mapping, disaster emergency monitoring, and engineering inspection.

[0047] In one embodiment of the present invention, the UAV is equipped with a camera, an IMU module and a GPS module, and flies along the preset route to collect continuous image sequences (resolution ≥ 4K) at a frequency of not less than 30 frames / second, and synchronously records the IMU data (sampling rate ≥ 100Hz) and the GPS data (sampling rate ≥ 5Hz). The multi-source data is transmitted to the airborne edge computing unit after being synchronized by timestamp.

[0048] In one specific embodiment, the IMU data includes angular velocity data and acceleration data; the GPS data includes GPS latitude and longitude information.

[0049] like Figure 2 The diagram shown is a logic block diagram of multi-source data synchronous acquisition and preprocessing according to the present invention.

[0050] In one embodiment of the present invention, image preprocessing of the image sequence to obtain the preprocessed image includes the following steps:

[0051] S21: Perform distortion correction on the image: Based on the camera calibration parameters, use the Zhang Zhengyou calibration method to perform barrel / pincushion distortion correction on the image;

[0052] In one specific embodiment, the camera calibration parameters include the intrinsic parameter matrix K, radial distortion coefficients k1 and k2, and tangential distortion coefficients p1 and p2.

[0053] S22: Perform uniform lighting and color matching on the distortion-corrected image: Divide the distortion-corrected image into pixel sub-blocks, calculate the mean gray value μ and variance σ of each block, adjust the gray value of the sub-blocks through a Wallis filter to eliminate illumination differences, and obtain the preprocessed image.

[0054] In one specific embodiment, the image can be divided into 8×8 pixel sub-blocks. This can be calculated using the following formula:

[0055] I`(x,y)=σσtarget·(I(x,y)-μ)+μtarget

[0056] Where μtarget = 128 and σtarget = 50, bilinear interpolation is used to process the edges of sub-blocks and eliminate lighting differences between images.

[0057] like Figure 3 The diagram shows a flowchart of an improved ORB-SLAM3 pose estimation and multi-sensor fusion process according to the present invention. The diagram illustrates the core sub-steps of SLAM pose estimation, including feature extraction and matching, initial pose calculation, sparse point cloud construction, and multi-sensor (GPS / IMU) fusion optimization. The left side of the diagram represents the core SLAM process (feature extraction → matching → pose calculation → mapping), and the right side represents the multi-sensor fusion branch (GPS / IMU data input → Kalman filtering → pose optimization). Feature extraction and IMU data acquisition are parallel steps (ORB feature extraction and IMU data acquisition are performed simultaneously); sparse point cloud construction and GPS data conversion are parallel steps (GPS coordinate transformation is processed synchronously during mapping); finally, the fusion optimization of SLAM pose and GPS / IMU data through Kalman filtering is a sequential step (requiring waiting for both SLAM pose and sensor data to be ready).

[0058] In one embodiment of the present invention, the feature extraction and pose estimation of the preprocessed image using the improved ORB-SLAM3 algorithm includes the following steps:

[0059] S31: Feature extraction and matching: The ORB algorithm is used to extract feature points of the current frame image, where the number of pyramid layers is ≤4 and the number of feature points is 800-1200. The feature points are matched with the previous frame feature points through the BRIEF descriptor, and the RANSAC algorithm is used to remove mismatched points.

[0060] S32: Initial pose estimation: Combining the IMU data with the matched feature point pairs, the initial pose of the camera is calculated using the PnP algorithm, including the rotation matrix R and the translation vector t;

[0061] S33: Sparse point cloud construction: Project the matching feature points onto a three-dimensional space, and select points with reprojection error ≤ 1 pixel to construct a sparse point cloud map.

[0062] S34: Multi-sensor fusion optimization: The GPS data is converted into UTM plane coordinates and fused with the pose estimated by SLAM through Kalman filtering. The state equation and observation equation respectively model the UAV motion and sensor noise, and correct the cumulative error.

[0063] The state vector includes position, velocity, and attitude angle, while the observation vector consists of GPS and IMU data.

[0064] The improved ORB-SLAM3 pose estimation in this invention is mainly reflected in two aspects: first, adjusting the parameters of ORB feature extraction (number of pyramid layers and number of feature points) to adapt to low-computing-power platforms and improve real-time performance; second, introducing GPS data and fusing it with SLAM and IMU data through Kalman filtering to optimize global pose estimation and improve positioning accuracy. The improved ORB-SLAM3 can still run efficiently on low-computing-power platforms with higher positioning accuracy (global error ≤ 0.5 meters), making it suitable for real-time stitching scenarios using UAVs.

[0065] Specifically, the improved ORB-SLAM3 adjusts the key parameters for ORB feature extraction:

[0066] The number of pyramid layers is limited to ≤4 layers to reduce the computational complexity of feature extraction; the number of feature points is controlled at 800-1200 (1000 in the example) to reduce the amount of computation while ensuring the robustness of feature matching, making it suitable for airborne edge computing platforms with computing power ≤10TOPS.

[0067] like Figure 4 The diagram shows a Laplacian pyramid for multi-resolution registration according to the present invention. The diagram illustrates the hierarchical structure of the Laplacian pyramid and the registration process, including the original image (G0), the downsampled Gaussian pyramids (G1, G2), and the Laplacian pyramids (L0, L1, L2) calculated from the Gaussian pyramids. The top layer is the high-resolution original image (G0), with lower-resolution layers (G1, G2) arranged sequentially downwards. The Laplacian pyramids (L0, L1, L2) correspond one-to-one with the Gaussian pyramids, and the registration direction for each layer is indicated (coarse registration of low-resolution layers → fine registration of high-resolution layers).

[0068] In one embodiment of the present invention, obtaining a registered image by multi-resolution image registration based on the optimized pose includes the following steps:

[0069] S41: Homography matrix calculation: Based on the optimized pose, calculate the homography matrix H of adjacent images, and project the image of the next frame onto the coordinate system of the previous frame;

[0070] S42: Subpixel-level registration: Construct a Laplacian pyramid and optimize the registration parameters layer by layer from low-resolution to high-resolution layers, maximizing the elimination of subpixel displacement errors through cross-correlation coefficients.

[0071] In one specific embodiment, the state vector of the Kalman filter includes the UAV's position (x, y, z) and velocity (v). x ,v γ,v z ) and attitude angles (φ,θ,ψ), the observation vector is the GPS UTM coordinates (X 9ps ,Y 9ps The angular velocity and acceleration of the IMU are fused together, and the global pose error is ≤0.5 meters.

[0072] In one specific embodiment, the findHomography function in the OpenCV library is used to calculate the homography matrix. This function estimates the homography matrix H between two frames of images by matching feature point pairs (effective feature point pairs after extraction by the ORB algorithm, BRIEF descriptor matching, and RANSAC algorithm to remove mismatches).

[0073] In one specific embodiment, a three-layer Laplacian pyramid is constructed, and the registration parameters are optimized layer by layer from the low-resolution layer (1 / 4 size) to the high-resolution layer (original image size); sub-pixel displacement error (≤0.5 pixels) is eliminated by maximizing the cross-correlation coefficient.

[0074] like Figure 5 The diagram shown is a schematic of the overlapping region weight allocation in Laplacian pyramid fusion according to the present invention. It mainly illustrates the pixel weight distribution in the overlapping region. The horizontal axis represents the horizontal position of the image (x-axis), the vertical axis represents the weight value (w), and the shaded area represents the overlapping portion (width D) of the two frames. The weight function is as follows:

[0075] w(x)=d(x) / D

[0076] Where d(x) is the distance from the pixel to the left boundary. The left side of the image is the previous frame (non-overlapping region weight w = 1), the middle is the overlapping region (weight decreases linearly from left to right to w = 0), and the right side is the next frame (non-overlapping region weight w = 0). The weight allocation is a single linear function that covers all pixels in the overlapping region.

[0077] In one embodiment of the present invention, an adaptive Laplacian pyramid is used for adaptive image fusion, and the weights of overlapping regions are dynamically adjusted according to the distance from the pixel to the boundary:

[0078] w(x,y)=d(x,y) / d max

[0079] Where, d max Let w(x,y) be the maximum width of the overlapping region, w(x,y) be the weight value, and d(x,y) be the distance from the pixel to the boundary. It can be understood that the original pixel values ​​are retained in the non-overlapping regions.

[0080] The pyramid is reconstructed layer by layer to generate a seamless panoramic image, with seam marks ≤0.3 pixels.

[0081] As can be seen, the weight function for the overlapping region in Laplacian pyramid fusion is a linear decay function, with the weight value decreasing linearly with the Euclidean distance from the pixel to the boundary.

[0082] The method of the present invention further includes:

[0083] The panoramic image and tag metadata are encapsulated and output synchronously; the tag metadata includes timestamps and pose information.

[0084] In one specific embodiment, the panoramic image and tag metadata (timestamp, pose information) are encapsulated in H.265+JSON format.

[0085] The method of the present invention further includes:

[0086] The panoramic image is synthesized in real time using the synchronized pose and image data of multiple drones.

[0087] This invention supports collaborative stitching of multiple drones, and synchronizes pose and image data through 5G / satellite communication links to achieve real-time synthesis of large-scale scenes.

[0088] This invention uses a high-resolution camera, IMU, and GPS module mounted on a drone to simultaneously collect multi-source data. After distortion correction and uniform lighting and color preprocessing, an improved ORB-SLAM3 algorithm is used for feature extraction and pose estimation. Kalman filtering is combined to fuse SLAM, IMU, and GPS data to optimize the global pose. Subpixel-level image alignment is achieved through multi-resolution Laplacian pyramid registration, and adaptive fusion is performed based on a dynamic weight function to generate a seamless panoramic image.

[0089] like Figure 6 As shown, the present invention also provides a real-time image stitching device for unmanned aerial vehicles (UAVs), which implements any of the methods described above, including:

[0090] Data acquisition module: used to synchronously collect multi-source data and transmit it to the airborne edge computing unit when the UAV flies along a preset route. The multi-source data includes continuous image sequences, IMU data, and GPS data.

[0091] In one embodiment of the present invention, the drone integrates a 4K camera (frame rate ≥30fps), an IMU (ADIS16470, sampling rate ≥100Hz), a GPS module (u-blox M8N, sampling rate ≥5Hz) and a time synchronization unit.

[0092] Preprocessing module: used to preprocess the image sequence to obtain preprocessed images;

[0093] In one embodiment of the present invention, distortion correction and uniform lighting and color are performed using OpenCV's undistort function and Wallis filter for image quality optimization.

[0094] Pose estimation module: Used to perform feature extraction and pose estimation on the preprocessed image using an improved ORB-SLAM3 algorithm, optimize the global pose by combining the IMU data and the GPS data, and output the optimized pose.

[0095] In one embodiment of the present invention, an improved ORB-SLAM3 algorithm is used, integrating a feature extraction and matching unit, a PnP pose calculation unit, and a multi-sensor fusion unit (Kalman filtering).

[0096] The multi-sensor fusion unit is implemented through an extended Kalman filter (EKF), which supports real-time correction of SLAM accumulated errors and has an output frequency ≥30Hz.

[0097] Registration and stitching module: Performs multi-resolution image registration based on the optimized pose to obtain the registered image;

[0098] In one embodiment of the present invention, it includes a homography matrix calculation unit (based on the findHomography function of OpenCV) and a multi-resolution registration unit (Laplacian pyramid).

[0099] Fusion output module: used to perform adaptive image fusion on the registered images to generate panoramic images.

[0100] In one embodiment of the present invention, a seamless panoramic image is generated by Laplacian pyramid fusion, with an output resolution ≥4K.

[0101] In one embodiment of the present invention, the airborne edge computing unit is an NVIDIA Jetson AGX Orin with a computing power of ≥275 TOPS, supporting H.265 video encoding and JSON metadata encapsulation.

[0102] In one specific embodiment, the method and apparatus of the present invention are used to achieve real-time image stitching of unmanned aerial vehicles, as detailed below:

[0103] Step S1: Synchronous Acquisition of Multi-Source Data

[0104] The drone (such as the DJI Matrice 300RTK) is equipped with a Zenmuse P1 camera (45MP, 30fps), an IMU (ADIS16470), and a GPS module (u-blox M8N), flying along a preset route (100 meters altitude, 5 m / s speed). The images captured by the camera (8192×5460 resolution), the angular velocity (ωx, ωy, ωz) and acceleration (ax, ay, az) of the IMU, and the latitude and longitude (lat, lon) of the GPS are synchronized via timestamps (UTC time) and transmitted to the onboard edge computing unit (such as the NVIDIA Jetson AGX Orin, with a computing power of 275 TOPS) via WiFi 6.

[0105] Step S2: Image Preprocessing

[0106] Distortion correction:

[0107] Camera intrinsic parameter matrix With distortion coefficients k1 = -0.01, k2 = 0.002, p1 = 0.001, and p2 = -0.001, the image is corrected using OpenCV's undistort function to eliminate barrel distortion.

[0108] Uniform lighting and color: Divide the image into 8×8 sub-blocks, calculate μ and σ for each block, and adjust the gray value according to the formula. For example, if μ = 150 and σ = 60 for a certain sub-block, then the adjusted gray value I′ = (50 / 60)(I-150) + 128. Use bilinear interpolation to process the edges of the sub-blocks and eliminate lighting differences.

[0109] Step S3: Pose estimation and mapping based on improved ORB-SLAM3

[0110] Feature extraction and matching: The ORB algorithm sets the pyramid to 3 layers and the number of feature points to 1000. It extracts feature point k of the current frame (Frame k) and matches it with the feature points of the previous frame (Frame k-1) using the BRIEF descriptor. The RANSAC algorithm (threshold 2 pixels) removes mismatched points and retains about 800 valid matching pairs.

[0111] Initial pose estimation: Combining the angular velocity (ωx=0.1rad / s) and acceleration (az=9.8m / s2) of the IMU, the initial pose (Rk,tk) is calculated using the PnP algorithm, with a reprojection error of approximately 1.2 pixels;

[0112] Sparse point cloud construction: Project the matching feature points onto the 3D space (camera coordinate system), select points with reprojection error ≤ 1 pixel as key frame map points, and construct a sparse point cloud (approximately 500 map points per key frame).

[0113] Multi-sensor fusion optimization: GPS latitude and longitude (lat=30.123°, lon=120.456°) are converted into UTM coordinates (Xgps=321000m, Ygps=3345000m), which are then fused with the pose estimated by SLAM (Xslam=321002m, Yslam=3345003m) through Kalman filtering to output an optimized pose (Xopt=321001m, Yopt=3345002m) with an error ≤0.5m.

[0114] Step S4: Multi-resolution image registration

[0115] Homography matrix calculation: Based on the optimized pose, the homography matrix HH of adjacent images is calculated using OpenCV's findHomography function, and the image of the next frame is projected onto the coordinate system of the previous frame.

[0116] Subpixel-level registration: Construct a Laplacian pyramid (3 layers), with the low-resolution layer (1 / 4 size) maximizing coarse registration (error ≤ 2 pixels) through cross-correlation coefficients, and the high-resolution layer (original image size) performing fine registration (error ≤ 0.5 pixels) to eliminate subpixel displacement.

[0117] Step S5: Adaptive Image Fusion

[0118] Laplacian pyramid construction: Calculate the Gaussian pyramid for the registered image (G0 is the original image, G1 is 1 / 2 size, G2 is 1 / 4 size), and the Laplacian pyramid Li = Gi-up(Gi+1);

[0119] Weighted fusion: The Laplacian layers L0, L1, and L2 of the overlapping region (200 pixels wide) are fused according to the weight function w(x,y) = d(x,y) / 200 (d(x,y) is the distance from the pixel to the left boundary), while the original pixels are retained in the non-overlapping region;

[0120] Pyramid Reconstruction: The merged Laplace pyramid is upsampled and superimposed layer by layer to generate a 4K panoramic image with stitching marks ≤0.3 pixels.

[0121] The method and apparatus of the present invention have the following advantages:

[0122] Real-time performance improvement: Improved ORB feature extraction parameters (pyramid layers ≤ 4, feature points 800-1200), combined with airborne edge computing unit (computing power 275TOPS), to achieve real-time stitching of 4K image sequences (processing latency ≤ 200ms);

[0123] Enhanced positioning accuracy: By fusing SLAM and GPS / IMU data and correcting accumulated errors through Kalman filtering, the global pose error is ≤0.5 meters, which is 4 times better than the traditional GPS+INS method (error ≥2 meters).

[0124] Environmental adaptability optimization: Wallis uniform lighting process eliminates illumination differences, RANSAC error correction improves robustness in dynamic occlusion scenarios, and feature matching success rate in complex environments (such as shadows and vegetation cover) is ≥90%.

[0125] Improved stitching quality: Multi-resolution registration (subpixel error ≤ 0.5 pixels) and Laplacian pyramid fusion (seam marks ≤ 0.3 pixels) significantly improve visual consistency compared to the traditional direct averaging method (seam error ≥ 2 pixels).

[0126] This invention has been verified in scenarios such as UAV mapping and disaster emergency response, and can support real-time large-scale scene analysis (such as landslide monitoring and building deformation detection), and has broad application value.

[0127] This invention also supports multimodal data fusion: it can be extended to access infrared camera and lidar data to achieve all-weather monitoring; edge-cloud collaboration: the airborne unit completes real-time stitching, and the cloud further performs 3D reconstruction and deep learning analysis; adaptive algorithm updates: SLAM parameters are dynamically optimized through OTA (over-the-air) technology to adapt to different application scenarios.

[0128] According to another aspect of the present invention, an electronic device is also provided. The electronic device includes:

[0129] One or more processors; a storage device for storing one or more programs, which, when executed by one or more processors, enable one or more processors to perform the methods described above.

[0130] According to another aspect of the present invention, a non-volatile computer-readable storage medium is also provided.

[0131] The storage medium contains a computer program that can perform the methods described above.

[0132] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A method for real-time image stitching from unmanned aerial vehicles (UAVs), characterized in that, include: S1: The UAV flies along a preset route, simultaneously collecting multi-source data and transmitting it to the onboard edge computing unit. The multi-source data includes continuous image sequences, IMU data, and GPS data. S2: Image preprocessing of the image sequence yields a preprocessed image; S3: The improved ORB-SLAM3 algorithm is used to extract features and estimate pose of the preprocessed image. The global pose is optimized by combining the IMU data and the GPS data, and the optimized pose is output: S4: Perform multi-resolution image registration based on the optimized pose to obtain the registered image; S5: Perform adaptive image fusion on the registered images to generate a panoramic image.

2. The real-time image stitching method for unmanned aerial vehicles as described in claim 1, characterized in that, The drone is equipped with a camera, an IMU module, and a GPS module. It flies along the preset route and collects continuous image sequences at a frequency of no less than 30 frames per second. It simultaneously records the IMU data and the GPS data. The multi-source data is transmitted to the airborne edge computing unit after being synchronized by timestamps.

3. The real-time image stitching method for unmanned aerial vehicles as described in claim 2, characterized in that, Image preprocessing of the image sequence to obtain the preprocessed image includes the following steps: S21: Perform distortion correction on the image: Based on the camera calibration parameters, use the Zhang Zhengyou calibration method to perform barrel / pincushion distortion correction on the image; S22: Perform uniform lighting and color matching on the distortion-corrected image: Divide the distortion-corrected image into pixel sub-blocks, calculate the mean gray value μ and variance σ of each block, adjust the gray value of the sub-blocks through a Wallis filter to eliminate illumination differences, and obtain the preprocessed image.

4. The real-time image stitching method for unmanned aerial vehicles as described in claim 3, characterized in that, The improved ORB-SLAM3 algorithm is used to perform feature extraction and pose estimation on the preprocessed image, including the following steps: S31: Feature extraction and matching: The ORB algorithm is used to extract feature points of the current frame image, where the number of pyramid layers is ≤4 and the number of feature points is 800-1200. The feature points are matched with the previous frame feature points through the BRIEF descriptor, and the RANSAC algorithm is used to remove mismatched points. S32: Initial pose estimation: Combining the IMU data with the matched feature point pairs, the initial pose of the camera is calculated using the PnP algorithm, including the rotation matrix R and the translation vector *t*. S33: Sparse point cloud construction: Project the matching feature points onto a three-dimensional space, and select points with reprojection error ≤ 1 pixel to construct a sparse point cloud map. S34: Multi-sensor fusion optimization: The GPS data is converted into UTM plane coordinates and fused with the pose estimated by SLAM through Kalman filtering. The state equation and observation equation respectively model the UAV motion and sensor noise, and correct the cumulative error.

5. The real-time image stitching method for unmanned aerial vehicles as described in claim 4, characterized in that, The process of obtaining a registered image by multi-resolution image registration based on the optimized pose includes the following steps: S41: Homography matrix calculation: Based on the optimized pose, calculate the homography matrix H of adjacent images, and project the image of the next frame onto the coordinate system of the previous frame; S42: Subpixel-level registration: Construct a Laplacian pyramid and optimize the registration parameters layer by layer from low-resolution to high-resolution layers, maximizing the elimination of subpixel displacement errors through cross-correlation coefficients.

6. The real-time image stitching method for unmanned aerial vehicles as described in claim 5, characterized in that, Adaptive image fusion is performed using an adaptive Laplacian pyramid, dynamically adjusting the weights of overlapping regions based on the distance from the pixel to the boundary: w(x,y)=d(x,y) / d max Where, d max The maximum width of the overlapping region is w(x,y), the weight value is w(x,y), and the distance from the pixel to the boundary is d(x,y). The pyramid is reconstructed layer by layer to generate a seamless panoramic image with seam marks ≤0.3 pixels.

7. The real-time image stitching method for unmanned aerial vehicles as described in claim 6, characterized in that, Also includes: The panoramic image and tag metadata are encapsulated and output synchronously. The tag metadata includes timestamps and pose information.

8. The real-time image stitching method for unmanned aerial vehicles as described in claim 7, characterized in that, Also includes: The panoramic image is synthesized in real time using the synchronized pose and image data of multiple drones.

9. A real-time image stitching device for unmanned aerial vehicles (UAVs), characterized in that, Implementing the method as described in any one of claims 1-8 includes: Data acquisition module: used to synchronously collect multi-source data and transmit it to the airborne edge computing unit when the UAV flies along a preset route. The multi-source data includes continuous image sequences, IMU data, and GPS data. Preprocessing module: used to preprocess the image sequence to obtain preprocessed images; Pose estimation module: Used to perform feature extraction and pose estimation on the preprocessed image using an improved ORB-SLAM3 algorithm, optimize the global pose by combining the IMU data and the GPS data, and output the optimized pose. Registration and stitching module: Performs multi-resolution image registration based on the optimized pose to obtain the registered image; Fusion output module: used to perform adaptive image fusion on the registered images to generate panoramic images.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program implements the method as described in any one of claims 1-8.

Images