Image stitching method and storage medium applied to drones
By equipping the drone with an inertial measurement unit and a depth camera, combined with a multi-layer perceptron model and queue storage technology, the problem of image blur during high-speed flight of the drone was solved, and efficient image sharpening and stitching were achieved.
Patent Information
- Application Number
- CN202311172218.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-11
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2043-09-11
AI Technical Summary
When drones fly at high speed or shoot high-speed moving objects, images tend to become blurred, making image stitching difficult. Existing technologies make it difficult to achieve effective stitching while maintaining high accuracy and avoiding data redundancy.
The drone is equipped with an inertial measurement unit, an RGB camera, and a depth camera. Through time synchronization and coordinate system alignment, a multi-layer perceptron model is used to perform image sharpening processing, and a first-in-first-out queue is used to store data to achieve RGB-D image fusion and stitching.
It achieves image clarity and quasi-real-time stitching under high-speed flight conditions, improves stitching accuracy and efficiency, and reduces data redundancy.
Smart Images
Figure CN117314807B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of RGB-D image fusion and relates to an image stitching method and a storage medium applied to an aerial camera. Background Art
[0002] When drones and other aerial cameras are reconstructing scenes, they need to use visual methods to stitch the environment together. However, when the drone is moving, the images may become blurry, making stitching difficult. Therefore, the drone is generally required to move slowly to obtain better image quality. However, when the drone is moving, if the captured image contains high-speed moving objects, the image may still be blurry, making it more difficult to stitch together. The usual approach is to reduce the feature importance of the blurred areas in the image. However, this condition is caused by a moving object and typically appears continuously in the image. The characteristics of this object should be taken into consideration during stitching to improve stitching accuracy.
[0003] Current SLAM technology (Simultaneous Localization and Mapping, primarily used to solve the problem of positioning and mapping robots in unknown environments) calculates features for each image. When the image is blurry, the features may be reduced or incorrectly calculated, resulting in tracking errors. In this case, the next image (or the next n images) is used to perform feature matching again, but this can lose or omit potentially useful image data. Another approach involves using a drone to repeatedly capture images from the same angle, performing image restoration, and then performing feature matching on the complete image. This method requires high algorithmic requirements and results in a large amount of data redundancy. Summary of the Invention
[0004] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is to provide an image stitching method and storage medium for drones with high stitching accuracy.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] An image stitching method applied to an aerial camera comprises the following steps:
[0007] S1. Equip the drone with an inertial measurement unit, an RGB camera, and a depth camera, synchronize the time of the inertial measurement unit, the RGB camera, and the depth camera, and align the position and attitude data of the drone's inertial measurement unit, the RGB image of the RGB camera, and the depth image of the depth camera in the same coordinate system;
[0008] S2. Training a multilayer perceptron model using the drone's position and attitude data and RGB image to obtain an image sharpening model; the image sharpening model is configured to sharpen the RGB image based on the RGB image and the corresponding position and attitude data within a time period, and output a sharpened RGB image corresponding to the position based on the input position and attitude data;
[0009] S3, acquiring an RGB image generated by the RGB camera in real time during the flight of the drone, and simultaneously acquiring position and attitude data of the inertial measurement unit and a depth image generated by the depth camera when each frame of the RGB image is acquired;
[0010] S4, using the obtained RGB image and position and posture data as a training set to train the image sharpening processing model, and obtain the RGB image after sharpening processing;
[0011] S5. Fusing the sharpened RGB image with the depth image acquired at the corresponding moment to obtain an RGB-D image, and then stitching the obtained RGB-D image.
[0012] Furthermore, the image sharpening processing model includes a spatial projection model and a pixel blur model; the spatial projection model is used to project rays from the camera to the object in three-dimensional space to a two-dimensional pixel plane, and the pixel blur model is used to generate a clear image through blurred image training prediction.
[0013] Furthermore, the loss function of the spatial projection model is the mean square error Lg of the three-dimensional spatial coordinates and two-dimensional angles between images.
[0014] Furthermore, the pixel blur model uses the illumination of the clear pixel point and the blur convolution kernel h for convolution, and its formula is:
[0015] b p =f(c p '*h)
[0016] Among them, the subscript p represents the pixel number; c p ' represents the scene illumination corresponding to pixel p; f represents the RGB camera response function, which is used to realize the change of mapping from illumination to intensity; b p represents the resulting blurred pixel.
[0017] Furthermore, the loss function Lb of the pixel blur model is
[0018]
[0019] Where R represents the set of all pixels in the image; represents the blurred image predicted by the model; bgt Blurred image representing a real pixel patch.
[0020] Furthermore, the position and attitude data of the inertial measurement unit include position data (x, y, z) and navigation angle data Where x, y, and z represent the coordinates of the drone on the three axes of the pixel coordinate system. θ represents the angle between the drone's flight direction and the horizontal plane and the vertical plane respectively.
[0021] Furthermore, the method of aligning the position and attitude data of the drone's inertial measurement unit, the RGB image of the RGB camera, and the depth image of the depth camera on the same coordinate system includes:
[0022] Align the RGB image with the depth image in the pixel coordinate system so that the RGB image and the depth image are in the same pixel coordinate system;
[0023] Aligning the position and posture data with the depth image in a Cartesian coordinate system so that the position and posture data and the depth image are in the same Cartesian coordinate system;
[0024] Align the position and attitude data of the drone and the inertial measurement unit so that the position and attitude data, RGB images, and depth images are aligned in the same coordinate system.
[0025] Furthermore, the S3 step includes the following sub-steps:
[0026] S301, generating a queue Sigma for storing RGB images and a queue M for storing position and posture data, wherein the queue Sigma and the queue M store the same number of elements;
[0027] S302. Acquire an RGB image generated by the RGB camera in real time during the flight of the drone, and store each frame of the RGB image as an element in a queue Sigma; simultaneously acquire position and attitude data of the inertial measurement unit and a depth image generated by the depth camera each time a frame of the RGB image is acquired; and store each set of position and attitude data of the inertial measurement unit as an element in a queue M;
[0028] The S4 step includes the following sub-steps:
[0029] S401, taking the acquisition time of the earliest RGB image in the queue Sigma as the current splicing time, and using the elements in the queue Sigma and the elements in the queue M as training sets to train the image sharpening processing model;
[0030] S402: Input the earliest element in the queue M into the trained image sharpening processing model, and use the output RGB image as the current spliced RGB image;
[0031] The S5 step includes:
[0032] Extract the depth image corresponding to the current stitching moment from the acquired depth image and fuse it with the current stitched RGB image to obtain the current RGB-D image; if the current RGB-D image is the second frame RGB-D image or an RGB-D image after the second frame, stitch the current RGB-D image with the previous frame RGB-D image; return to execute step S302.
[0033] Furthermore, the aerial camera is a drone.
[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements an image stitching method applied to an aerial camera.
[0035] The present invention provides an image stitching method applicable to high-speed flying drones. A multi-layer perceptron model is used to restore clear RGB images before fusion and stitching. A first-in-first-out queue is used to store position and posture data and RGB images. The multi-layer perceptron model can maintain a fast optimization time after initialization, thereby quickly completing image sharpening processing. The method has the ability to perform feature matching and quasi-real-time stitching even when the camera image is blurred due to the high-speed flight of the drone or the presence of high-speed moving objects within the shooting range. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0037] Figure 1 The figure is a flow chart of an embodiment of the image stitching method applied to an aerial camera according to the present invention.
[0038] Figure 2 Flowchart for training an image sharpening model to obtain RGB images and position and posture data, and then fusing them with depth images for stitching. DETAILED DESCRIPTION
[0039] The following describes the implementation of the present invention through specific examples. The illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention. The following embodiments and features in the embodiments may be combined with each other unless there is any conflict.
[0040] See also Figure 1 , Figure 1This is a flow chart of an embodiment of the image stitching method of the present invention applied to an aerial camera. In this embodiment, a drone is used as the aerial camera. The image stitching method of this embodiment applied to an aerial camera includes the following steps:
[0041] S1. Equip the drone with an IMU, RGB camera, and depth camera. Synchronize the IMU, RGB camera, and depth camera to output data simultaneously through hardware synchronization. Additionally, align the drone's IMU data, the RGB camera's RGB image, and the depth camera's depth image (i.e., depth image) in the same coordinate system.
[0042] The IMU data output by the IMU includes position data (x, y, z) and navigation angle data There are five channels in total; x, y, and z represent the coordinates of the drone on the three coordinate axes of the pixel coordinate system; θ represents the angle between the drone's flight direction and the horizontal and vertical planes, respectively. The RGB image output by the RGB camera includes three channels for the three primary colors. The Depth image output by the depth camera includes one channel for the depth value.
[0043] Methods for aligning the drone IMU data, the RGB image of the RGB camera, and the Depth image of the depth camera on the same coordinate system include:
[0044] Align the RGB image with the depth image in the pixel coordinate system so that they are in the same pixel coordinate system. Align the IMU data with the depth image in the Cartesian coordinate system so that they are in the same Cartesian coordinate system. Align the drone with the IMU data so that the IMU data, RGB image, and depth image are aligned in the same coordinate system.
[0045] S2. Establish a multilayer perceptron model, which may include a spatial projection model and a pixel blur model. The spatial projection model is used to project rays from the camera to the object in three-dimensional space onto a two-dimensional pixel plane. The loss function of the spatial projection model is the mean square error Lg between the three-dimensional spatial coordinates and the two-dimensional angle between the images. The pixel blur model is used to generate a clear image by training and predicting blurred images.
[0046] Previous models typically use convolution with intensity to sharpen blurred areas. However, since it is illuminance, not intensity, that actually affects blurring, the pixel blur model of this embodiment converts pixel intensity to illuminance using the Camera Response Function (CRF). The pixel blur model convolves the illuminance of the clear pixel with the blur convolution kernel h, using the following formula:
[0047] b p =f(c p '*h)
[0048] Among them, the subscript p represents the pixel number; c p ' represents the scene illumination corresponding to the pixel point p; f represents the RGB camera response function CRF, which is used to realize the change of mapping from illumination to intensity; b p represents the resulting blurred pixel.
[0049] The loss function Lb of the pixel blur model is used to supervise the blurred pixels, and its formula is:
[0050]
[0051] Where R represents the set of all pixels in the image; represents the blurred image predicted by the MLP model through deep learning; b gt Represents the blurred image of the real pixel block. When Lb is the smallest, we can get Using pixel blur model, we can get c p The camera response function (CRF) is then used to perform a reverse calculation from pixel illumination to pixel intensity, thus obtaining a clear image.
[0052] The image sharpening model is constructed by training a spatial projection model and a pixel blur model using the position and attitude data from the drone's IMU (Inertial Measurement Unit) and the RGB images generated by the RGB camera. The model sharpens the RGB image based on the RGB image and the corresponding IMU position and attitude data (hereinafter referred to as IMU data) within a time period. The model then outputs the sharpened RGB image at the corresponding position based on the input IMU data.
[0053] S3, during the flight of the UAV, the RGB image generated by the RGB camera is obtained in real time, and each time a frame of RGB image is obtained, the IMU data and the Depth image generated by the depth camera are obtained at the same time. i Represents the i-th RGB image generated by the RGB camera, with X iIndicates the acquired I MU data (i.e., the same as frame i I MU data acquired at the same time), with Depth i Represents the i-th group of Depth images generated by the depth camera (i.e., the same as the frame i Depth image acquired at the same time). See Figure 2 , this step may specifically include the following steps:
[0054] S301: Generate a queue Sigma for storing RGB images and a queue M for storing IMU data. The number of elements stored in the queue Sigma and the queue M is n. The queues Sigma and M both use a FIFO (First Input First Output) data storage method.
[0055] S302, during the flight of the UAV, the RGB image generated by the RGB camera is acquired in real time, and one frame of the RGB image is stored as an element in the queue Sigma. For example, when reading the RGB image, the sliding window is used to read n RGB images each time based on the first-in-first-out principle. The first reading starts from frame 1 and ends at frame 2. n The second reading starts from frame 2 and ends at frame 3. n+1 So far, and it is expressed as Sigma2; and so on.
[0056] When acquiring each RGB image frame, the IMU data and the depth image generated by the depth camera are simultaneously acquired. A set of IMU data is stored as an element in the queue M, and a frame of the depth image is used as an element to form a sequence D.
[0057] S4, using the obtained RGB image and IMU data as a training set to train the image sharpening model, and obtain the RGB image after sharpening. Figure 2 , this step may specifically include the following sub-steps:
[0058] S401, the earliest RGB image acquisition time in the queue Sigma is used as the current splicing time. Assume that the elements in the queue Sigma are frame1, frame2, ..., frame n , that is, queue Sigma1 at this time. Then the elements in queue M are X1, X2, ..., X n ; M1 represents the queue M at this time.
[0059] S402. The acquisition time of the earliest RGB image frame (i.e., frame1) in queue Sigma1 is used as the current splicing time. The image sharpening processing model is trained using the elements in queue Sigma1 and the elements in queue M1 as training sets until the loss function Lg of the spatial projection model and the loss function Lb of the pixel blur model are optimized to the minimum value. MLP1 represents the image sharpening processing model obtained by training the elements in queue Sigma1 and the elements in queue M1. At this time, MLP1 generates n sharpened RGB images, and the n sharpened RGB images correspond one-to-one in time to the n elements in queue M1.
[0060] Input the oldest element in queue M1 (i.e., X1) into the trained image sharpening model MLP1. MLP1 outputs the sharpened RGB image deblur_frame1 corresponding to X1 (i.e., the sharpened RGB image at the same time as X1) as the current stitched RGB image. deblur_frame1 and frame1 have the same position and angle, but deblur_frame1 is sharper than frame1.
[0061] S5. The sharpened RGB image is fused with the depth image acquired at the corresponding moment to obtain an RGB-D image. After scene fusion, a scanned scene with a 3D point cloud and a color image can be obtained. After that, the obtained RGB-D images can be stitched. The specific method is as follows:
[0062] Extract the Depth image corresponding to the current splicing moment from sequence D (this is the first element Depth1 in sequence D) and fuse it with the current spliced RGB image (this is deblur_frame1) to obtain the current RGB-D image (this is the first frame RGB-D image, expressed as RGB-D1). If the current RGB-D image is RGB-D1, no image splicing is performed, and the process returns directly to step S302. If the current RGB-D image is RGB-D2 (i.e., the second frame RGB-D image) or the RGB-D image after the second frame, the current RGB-D image is first spliced with the previous frame RGB-D image; then return to step S302. For example, for the nth frame RGB-D image RGB-D n , and compare it with RGB-D n-1 Perform splicing.
[0063] When step S302 is executed for the second time, the queue M is represented by M2, and the elements included are frame2, frame3, ..., frame n+1Sigma2 represents the queue Sigma at this time, which includes elements X2, X3, ..., X n+1 . The image sharpening processing model is trained using the elements in the queue Sigma2 and the elements in the queue M2 as the training set. The trained image sharpening processing model is represented by MLP2. Input X2 into MLP2, and output the RGB image deblur_frame2 after sharpening corresponding to X2 as the current spliced RGB image. Extract Depth2 from sequence D and fuse it with deblur_frame2 to obtain the RGB-D image RGB-D2. After splicing RGB-D2 with RGB-D1, continue to return to execute step S302, fuse to obtain the next frame of RGB-D image and continue splicing, and loop the above process until the scene scan is completed, thereby realizing quasi-real-time splicing of drone images.
[0064] In this embodiment, an image stitching method applicable to high-speed flying drones and other aerial cameras is provided. The method utilizes a multi-layer perceptron model to restore clear RGB images before performing fusion and stitching. A first-in-first-out queue is used to store position and posture data and RGB images, allowing the multi-layer perceptron model to maintain a fast optimization time after initialization, thereby quickly completing image sharpening processing. This method provides the ability to perform feature matching and quasi-real-time stitching even when the camera image is blurred due to the high-speed flight of the drone or the presence of high-speed moving objects within the shooting range.
[0065] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, part or all of the actual operations of the above-mentioned image stitching method applied to an aerial camera are performed, thereby realizing the functions of any one of the above-mentioned embodiments.
[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.
Claims
1. An image stitching method for an aerial camera, characterized in that: The following steps are involved: S1. Equip the drone with an inertial measurement unit, an RGB camera, and a depth camera, synchronize the time of the inertial measurement unit, the RGB camera, and the depth camera, and align the position and attitude data of the drone's inertial measurement unit, the RGB image of the RGB camera, and the depth image of the depth camera in the same coordinate system; S2. Training a multilayer perceptron model using the drone's position and attitude data and RGB images to obtain an image sharpening model; the image sharpening model is configured to sharpen the RGB image based on the RGB image and the corresponding position and attitude data within a time period, and output a sharpened RGB image of the corresponding position based on the input position and attitude data; the image sharpening model includes a spatial projection model and a pixel blur model; The spatial projection model is used to project the ray from the camera to the object in the three-dimensional space to the two-dimensional pixel plane, and the pixel blur model is used to generate a clear image through blurred image training prediction; S3, acquiring an RGB image generated by the RGB camera in real time during the flight of the drone, and simultaneously acquiring position and attitude data of the inertial measurement unit and a depth image generated by the depth camera when each frame of the RGB image is acquired; S4, using the obtained RGB image and position and posture data as a training set to train the image sharpening processing model, and obtain the RGB image after sharpening processing; S5. Fusing the sharpened RGB image with the depth image acquired at the corresponding moment to obtain an RGB-D image, and then stitching the obtained RGB-D image.
2. The image stitching method for drones according to claim 1, characterized in that: The loss function of the spatial projection model is the mean square error Lg of the three-dimensional spatial coordinates and two-dimensional angles between images.
3. The image stitching method for drones according to claim 2, characterized in that: The pixel blur model uses the illumination of the clear pixel point and the blur convolution kernel h to perform convolution, and its formula is: b p =f(c p '*h) Among them, the subscript p represents the pixel number; c p ' represents the scene illumination corresponding to pixel p; f represents the RGB camera response function, which is used to realize the change of mapping from illumination to intensity; b p represents the resulting blurred pixel.
4. The image stitching method for drones according to claim 3, wherein: The loss function Lb of the pixel blur model is Where R represents the set of all pixels in the image; represents the blurred image predicted by the model; b gt Blurred image representing a real pixel patch.
5. The image stitching method for drones according to claim 1, wherein: The position and attitude data of the inertial measurement unit include position data (x, y, z) and navigation angle data Where x, y, and z represent the coordinates of the drone on the three coordinate axes of the pixel coordinate system. θ represents the angle between the drone's flight direction and the horizontal plane and the vertical plane respectively.
6. The image stitching method for drones according to claim 5, characterized in that: The method for aligning the position and attitude data of the drone's inertial measurement unit, the RGB image of the RGB camera, and the depth image of the depth camera on the same coordinate system includes: Align the RGB image with the depth image in the pixel coordinate system so that the RGB image and the depth image are in the same pixel coordinate system; Aligning the position and posture data with the depth image in a Cartesian coordinate system so that the position and posture data and the depth image are in the same Cartesian coordinate system; Align the position and attitude data of the drone and the inertial measurement unit so that the position and attitude data, RGB images, and depth images are aligned in the same coordinate system.
7. The image stitching method for drones according to any one of claims 1 to 6, characterized in that: The S3 step includes the following sub-steps: S301, generating a queue Sigma for storing RGB images and a queue M for storing position and posture data, wherein the queue Sigma and the queue M store the same number of elements; S302. Acquire an RGB image generated by the RGB camera in real time during the flight of the drone, and store each frame of the RGB image as an element in a queue Sigma; simultaneously acquire position and attitude data of the inertial measurement unit and a depth image generated by the depth camera each time a frame of the RGB image is acquired; and store each set of position and attitude data of the inertial measurement unit as an element in a queue M; The S4 step includes the following sub-steps: S401, taking the acquisition time of the earliest RGB image in the queue Sigma as the current splicing time, and using the elements in the queue Sigma and the elements in the queue M as training sets to train the image sharpening processing model; S402: Input the earliest element in the queue M into the trained image sharpening processing model, and use the output RGB image as the current spliced RGB image; The S5 step includes: Extract the depth image corresponding to the current stitching moment from the acquired depth image and fuse it with the current stitching RGB image to obtain the current RGB-D image; If the current RGB-D image is the second frame RGB-D image or an RGB-D image after the second frame, the current RGB-D image is spliced with the previous frame RGB-D image; and the process returns to step S302.
8. The image stitching method for drones according to claim 7, characterized in that: The aerial camera is a drone.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image stitching method applied to an aerial camera as claimed in any one of claims 1 to 8 is implemented.