A method and device for aligning hyperspectral images of a UAV, a terminal and a storage medium
By acquiring high-resolution satellite images taken at the same time to generate simulated hyperspectral images, and using a fully convolutional network to build a model and align control points, the problem of inaccurate aerial image stitching under a single type of environment is solved, achieving accurate stitching and stability of hyperspectral images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEBEI SAILHERO ENVIRONMENTAL PROTECTION HIGH TECH
- Filing Date
- 2023-10-12
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies often result in inaccurate stitching of aerial images from single-type environments, and conventional methods are cumbersome to operate and have unpredictable results.
By acquiring high-resolution satellite imagery taken at the same time as the first image, a simulated hyperspectral image is generated. A hyperspectral aerial photography simulation model is then constructed using a fully convolutional network to acquire a second image. By aligning the control points of the first and second images, the hyperspectral images are stitched together.
It improves the accuracy and stability of UAV hyperspectral image stitching, reduces image curvature issues, and increases stitching efficiency.
Smart Images
Figure CN117274338B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of UAV hyperspectral imaging technology, and in particular to a method, apparatus, terminal and storage medium for UAV hyperspectral image alignment. Background Technology
[0002] Image stitching, as an important step in UAV aerial photography technology, has been widely used in fields such as aerial surveying and geographic information systems. Its principle is to calculate the relative position and attitude relationship of a large number of aerial photos through feature matching and triangulation techniques, thereby realizing image stitching and three-dimensional reconstruction.
[0003] Currently, when monitoring a single type of environment, the splicing of individual flight strips corresponding to that environment type often results in distortion. For example, when monitoring water bodies, the water often exhibits narrow, straight distributions, severely impacting the accuracy and usability of the splicing results. Conventional solutions include manually adjusting the waypoint positions one by one, and manually aligning the aerial images one by one using a pixel-by-pixel mosaic method. However, both methods are cumbersome and have unpredictable results.
[0004] Therefore, there is an urgent need for a splicing method that enables accurate splicing of flight strips in a single type of environment. Summary of the Invention
[0005] This application provides a method, apparatus, terminal, and storage medium for aligning hyperspectral images of unmanned aerial vehicles (UAVs) to solve the problem of inaccurate stitching of aerial images of a single type of environment in the prior art.
[0006] In a first aspect, this application provides a method for aligning hyperspectral images of a UAV, including:
[0007] A first image is acquired by the UAV while it is flying along a first flight path, capturing an aerial image of a target area of a single type of environment to be tested; the first image includes multiple hyperspectral images of the target aerial area.
[0008] At the same time as the first image is captured, a high-resolution satellite image of the target aerial area that is larger than the first flight strip is acquired, and a second image is acquired based on the high-resolution satellite image. The second image includes a simulated hyperspectral map of a second flight strip located on the first side of the first flight strip.
[0009] The same location points of the overlapping portion between two adjacent hyperspectral images in the first image are used as the first control points; the same location points of the overlapping portion between each hyperspectral image in the first image and the simulated hyperspectral image in the second image are used as the second control points.
[0010] For each hyperspectral image in the first image, the first control point of the hyperspectral image is aligned with the first control point of the second hyperspectral image, and the second control point of the hyperspectral image is aligned with the second control point of the target simulated hyperspectral image; the second hyperspectral image is a hyperspectral image in the first image that overlaps with the hyperspectral image; the target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
[0011] Secondly, this application provides a UAV hyperspectral image alignment device, comprising:
[0012] The first image acquisition module is used to acquire a first image of the target aerial photography area of the single type of environment to be tested when the UAV flies along the first flight path; the first image includes multiple hyperspectral images of the target aerial photography area.
[0013] The second image acquisition module is used to acquire a high-resolution satellite image of the target aerial photography area that is larger than the first flight strip at the same shooting time as the first image, and to acquire a second image based on the high-resolution satellite image. The second image includes a simulated hyperspectral map of a second flight strip located on the first side of the first flight strip.
[0014] The control point acquisition module is used to take the same position points of the overlapping part between two adjacent hyperspectral images in the first image as first control points; and to take the same position points of the overlapping part between each hyperspectral image in the first image and the simulated hyperspectral image in the second image as second control points.
[0015] An image alignment module is used to align a first control point of a hyperspectral image with a first control point of a second hyperspectral image for each hyperspectral image in the first image, and to align a second control point of a hyperspectral image with a second control point of a target simulated hyperspectral image; the second hyperspectral image is a hyperspectral image in the first image that overlaps with the hyperspectral image; the target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
[0016] Thirdly, this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect above.
[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.
[0018] This application provides a method, apparatus, terminal, and storage medium for aligning hyperspectral images of a UAV. By acquiring a second image of a second flight path located on the first side of the first flight path at the same time as the first image was captured, and aligning the second image using the same first control point and the same second control point, the first image can be stitched together. This effectively improves the stitching effect of UAV hyperspectral images, reduces the curvature between UAV hyperspectral images, and solves the problem of inaccurate stitching of aerial images of a single type of environment in the prior art, thereby improving the accuracy and stability of UAV hyperspectral image stitching. Furthermore, by selecting the corresponding second image based on the first image of different fields, UAV hyperspectral image stitching can be achieved for various fields, improving stitching efficiency. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the implementation of the UAV hyperspectral image alignment method provided in this application embodiment;
[0021] Figure 2 This is a schematic diagram of the control point distribution provided in an embodiment of this application;
[0022] Figure 3 This is a schematic diagram of misaligned stitching of the first image provided in an embodiment of this application;
[0023] Figure 4 This is a schematic diagram of the first image alignment splicing provided in the embodiments of this application;
[0024] Figure 5 This is a schematic diagram of the structure of the UAV hyperspectral image alignment device provided in the embodiments of this application;
[0025] Figure 6 This is a schematic diagram of the terminal provided in the embodiments of this application. Detailed Implementation
[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.
[0028] Figure 1 The implementation flowchart of the UAV hyperspectral image alignment method provided in this application embodiment is described in detail below:
[0029] In step 101, a first image of the target aerial photography area of the single type of environment to be tested is acquired when the UAV flies along the first flight path; the first image includes multiple hyperspectral images of the target aerial photography area.
[0030] The first flight path is a straight-line flight path of the UAV in the single type of environment to be tested, which is the flight route of the UAV.
[0031] The target aerial photography area is the area photographed by the drone during its flight over the first flight path.
[0032] In this embodiment of the application, multiple hyperspectral images of a target aerial photography area of a single environmental type to be tested are acquired by an unmanned aerial vehicle (UAV) image transmission device while the UAV is flying along a first flight path, and the multiple hyperspectral images are collectively referred to as the first image.
[0033] In one possible implementation, step 101 may include:
[0034] Set the target aerial photography area for the drone to take aerial photos in the single type of environment to be tested, and set the heading overlap rate and waypoint interval;
[0035] Based on the heading overlap rate and waypoint interval, the positions of multiple first waypoints are determined by the UAV hovering and taking pictures while flying along the first wayline; the positions of the first waypoints include the first longitude and the first latitude.
[0036] Based on the location of the first waypoint, the UAV is controlled to automatically collect the first image and the corresponding positioning information, including longitude, latitude, altitude, heading angle, side angle, and rotation angle.
[0037] Among them, the heading overlap rate refers to the probability that the first picture taken by a drone and the second picture taken when the drone is taking pictures along a flight path will overlap.
[0038] Specifically, in the UAV flight path planning, the area along the direct current of the UAV in the single-type environment to be tested is set as the target aerial photography area. Then, the forward overlap rate, lateral overlap rate, and waypoint interval are set for the UAV. Based on the forward overlap rate and waypoint interval, multiple center points for hovering and photographing by the UAV along the first flight path are determined, defined as the first waypoint position, which includes the first longitude and the first latitude. Based on the set and determined first waypoint position, the UAV automatically acquires the first image and the corresponding positioning information, including longitude, latitude, altitude, forward yaw angle, lateral yaw angle, and rotation angle.
[0039] The lateral overlap rate refers to the probability that photos taken by the drone on the first flight path overlap with photos taken on the second flight path.
[0040] This application embodiment obtains a first image and the corresponding positioning information to provide positioning information for subsequently obtaining the same location points in the overlapping part between two adjacent hyperspectral images in the first image.
[0041] In step 102, a high-resolution satellite image of the target aerial area, which is larger than the first flight strip, is acquired at the same shooting time as the first image. Based on the high-resolution satellite image, a second image is acquired. The second image includes a simulated hyperspectral map of the second flight strip located on the first side of the first flight strip.
[0042] Among them, high-resolution satellite imagery is satellite imagery directly acquired by satellites, and high-resolution satellite imagery includes at least the first flight path of the UAV and the range of the second flight path located on the first side of the first flight path.
[0043] In this embodiment of the application, at the same time as the first image is captured, a high-resolution satellite image of the target aerial area, which is larger than the first flight band, is acquired. Then, from the high-resolution satellite image, a simulated hyperspectral map of the second flight band located on the first side of the first flight band is selected and defined as the second image.
[0044] In one possible implementation, prior to step 102, the method may further include:
[0045] The positioning information of the first image is obtained, including the longitude, latitude, altitude, heading angle, side deviation angle and rotation angle of the first image;
[0046] Accordingly, step 102 may include:
[0047] Based on the longitude, latitude, altitude, heading angle, side deviation angle, and rotation angle of the first image, high-resolution satellite images of the target aerial area that are larger than the first flight strip are acquired at the same time as the first image is captured.
[0048] Specifically, based on the acquired first image and the corresponding positioning information, namely the longitude, latitude, altitude, heading angle, side deviation angle, and rotation angle of the first image, a high-resolution image larger than the first flight strip is acquired within the target aerial photography area at the same shooting time as the first image.
[0049] In one possible implementation, acquiring the second image based on high-resolution satellite imagery may include:
[0050] Based on the forward overlap rate, lateral overlap rate, and waypoint interval of the UAV aerial photography, the location of the second waypoint is simulated when the UAV hovers and takes pictures along the second flight path; the location of the second waypoint includes the second longitude and the second latitude.
[0051] Acquire elevation model data corresponding to high-resolution satellite imagery, including longitude, latitude, and elevation;
[0052] Using 3D graphics software and based on elevation model data, satellite image simulation scenes are created from high-resolution satellite imagery.
[0053] In the satellite imagery simulation scenario, the first waypoint position, the second waypoint position, the flight altitude of the first image, and the image attribute parameters corresponding to the first image are set to generate a simulated aerial three-band color image;
[0054] The simulated aerial three-band color image is input into the hyperspectral aerial photography simulation model, which outputs a second image with 240 bands. The hyperspectral aerial photography simulation model is constructed based on a fully convolutional network.
[0055] Among them, Blender is a free and open-source 3D graphics software that provides a range of solutions for creating animated short films, from modeling, animation, materials, rendering, to audio processing and video editing.
[0056] To simulate images captured by a drone and include positioning information in 3D graphics software, satellite imagery and elevation model data are required. The specific process for creating the simulation scene is as follows:
[0057] 1. Prepare the model: Create a 3D model of the drone. When creating the model, ensure that its size and shape match the actual drone to accurately simulate its flight.
[0058] 2. Import images and elevation models: Create a new project in the 3D graphics software and import the images to be simulated (e.g., high-resolution satellite imagery) and elevation model data. Apply the images as image texture maps to the materials of the drone model and import the elevation model data into the terrain editor of the 3D graphics software.
[0059] 3. Camera Setup: Create a new UV map for the drone model in the 3D graphics software and associate it with the camera image texture. Ensure that the camera's field of view matches the actual shooting range of the drone.
[0060] 4. Create a flight path: Use the particle system in the 3D graphics software to create a path that will follow the desired flight path of the drone, and set appropriate parameters for the particle system, such as speed, direction and duration.
[0061] 5. Set up the animation: In the 3D graphics software, create a new animation and set keyframes to control the drone's flight. Synchronize the keyframes with the particle system parameters to achieve smooth animation transitions.
[0062] 6. Add materials and textures: Add appropriate materials and textures to the drone model to make it more like a real drone.
[0063] 7. Set up the environment: Configure the correct environment in the 3D graphics software, including the sky and lighting. This will make the simulated image closer to the actual shooting effect.
[0064] 8. Rendering: Select a suitable rendering engine (such as Cycles) for rendering. In the rendering settings, ensure that the correct camera and rendering options are enabled.
[0065] 9. Post-processing: After rendering, image editing software (such as GIMP or Photoshop) can be used to post-process the image to improve image quality.
[0066] 10. Export Image and Positioning Data: In the 3D graphics software, select the UAV model and then export its position, attitude, and elevation data. This data can be used for post-processing and analysis.
[0067] To ensure the accuracy of the positioning information obtained from the simulated scenario, it is necessary to ensure that the imported high-resolution satellite imagery and elevation model data have sufficiently high precision.
[0068] The simulated aerial three-band color image is a three-band image. In the embodiments of this application, the output second image is a 240-band image, which has the same number of bands as the first image.
[0069] The image attribute parameters corresponding to the first image include the image size, width, pixel data, and the number of hyperspectral images in the first image. The purpose is to ensure that the simulated second image in the simulated aerial three-band color image is consistent with the first image in shape, pixels, and number, so as to provide basic information for subsequent control point selection and alignment.
[0070] Specifically, based on the forward and lateral overlap rates of the UAV aerial photography, as well as the waypoint intervals, a center point for hovering and shooting while the UAV flies along the second flight path is simulated and defined as the second waypoint location, which includes the second longitude and the second latitude. A satellite imagery simulation scene containing high-resolution satellite imagery is created using the 3D graphics software Blender, and elevation model data is input into the satellite imagery simulation scene to simulate the surface undulations. The satellite imagery simulation scene includes the first and second flight paths. Then, in this satellite imagery simulation scene, the first waypoint location, the second waypoint location, the flight altitude at which the first image was captured, and the corresponding image attribute parameters are set to generate a simulated three-band color image including the first and second flight paths. A hyperspectral aerial photography simulation model based on a fully convolutional network converts the three-band simulated three-band color image into a 240-band second image for output. The simulated three-band color image includes the simulated second image and its corresponding positioning information.
[0071] In one possible implementation, the process of constructing a hyperspectral aerial photography simulation model may include:
[0072] Obtain historical simulated aerial three-band color images;
[0073] Obtain the second historical image of the 240 band corresponding to the historical simulated aerial three-band color image;
[0074] A fully convolutional network model was constructed, and the historical simulated aerial three-band color image was used as input, while the second historical image of 240 bands was used as output. The fully convolutional network model was trained to obtain the hyperspectral aerial simulation model.
[0075] Specifically, historical simulated aerial photography three-band color images and corresponding historical second images of 240 bands are acquired respectively. Based on the constructed fully convolutional network model, the historical simulated aerial photography three-band color images are used as input and the corresponding historical second images of 240 bands are used as output to train the fully convolutional network model and obtain the hyperspectral aerial photography simulation model.
[0076] After obtaining the trained hyperspectral aerial photography simulation model, it is necessary to verify whether the training of the hyperspectral aerial photography simulation model has been completed. Therefore, the main training and verification process is as follows:
[0077] A hyperspectral aerial photography simulation model is designed based on a fully convolutional network model. The input to the hyperspectral aerial photography simulation model consists of a historical first image and historical simulated aerial three-band color images. The output is a corresponding first simulated image and a second simulated image. The first simulated image comprises multiple simulated hyperspectral images of a first flight path, and the second simulated image comprises multiple simulated hyperspectral images of a second flight path. The first and second simulated images are of the same size. The structure of the hyperspectral aerial photography simulation model includes an encoder and a decoder. The encoder repeats convolution and downsampling operations, while the decoder repeats coiling and upsampling operations.
[0078] For the UAV hyperspectral simulation data output by the hyperspectral aerial photography simulation model, the first simulated image is used as the input image, and a historical first image is used as the label image. The UAV hyperspectral simulation data is divided into a training set and a validation set according to a preset ratio, where the intersection of the training set and the validation set is empty. On the training set, the first simulated image is input into the hyperspectral aerial photography simulation model to obtain the first output image. The difference between the first output image and the label image is calculated, and the parameter values of the hyperspectral aerial photography simulation model are adjusted inversely. On the validation set, the second simulated image is input into the hyperspectral aerial photography simulation model to obtain the second output image. The difference between the second output image and the label image is calculated, and the performance of the hyperspectral aerial photography simulation model is estimated. Thus, the trained and validated hyperspectral aerial photography simulation model is obtained.
[0079] In one possible implementation, this application also requires preprocessing the raw data acquired by the satellite, converting the acquired raw data into a format suitable for network processing, such as normalizing the hyperspectral image and color image to scale them to the same resolution.
[0080] In this embodiment, a simulated hyperspectral image of a second flight path located on the first side of the first flight path is obtained through a hyperspectral aerial photography simulation model, and a second image is formed. Since the simulated second image is consistent with the first image in shape, pixels, and quantity, the accuracy of the first image stitching can be improved. Furthermore, the second image is obtained from actual high-resolution satellite imagery, and the second image is consistent with the first image in shape, pixels, and quantity, which can improve the reliability of the first image stitching.
[0081] In step 103, the same location points of the overlapping portion between two adjacent hyperspectral images in the first image are used as first control points; the same location points of the overlapping portion between each hyperspectral image in the first image and the simulated hyperspectral image in the second image are used as second control points.
[0082] In this embodiment of the application, the same location points of the overlapping portion of two adjacent hyperspectral images in the first image obtained in step 101 are used as first control points. For example, refer to Figure 2Hyperspectral image A and hyperspectral image B are any two adjacent hyperspectral images in the first image. The lower x1 of hyperspectral image A and the upper x2 of hyperspectral image B are overlapping parts, where a1 in x1 and x2 is the same location point, which is the first control point.
[0083] The same location points in the overlapping portion of each hyperspectral image in the first image obtained in step 101 and the corresponding simulated hyperspectral image in the second image obtained in step 102 are used as second control points. For example, refer to Figure 2 Hyperspectral image A and simulated hyperspectral image C are two corresponding overlapping hyperspectral images in the first image and the second image, respectively. The right side y1 of hyperspectral image A and the left side y2 of simulated hyperspectral image C are the overlapping parts, where b1 in y1 and y2 is the same location point, which is the second control point.
[0084] In one possible implementation, step 103 may include:
[0085] Obtain the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image, and take the same position points of a first preset percentage among the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image as the first control points; the third hyperspectral image is any hyperspectral image in the first image, and the fourth hyperspectral image is a hyperspectral image that overlaps with the third hyperspectral image;
[0086] Obtain the same position points in the overlapping part between the third hyperspectral image and the first simulated hyperspectral image, and take the same position points in the overlapping part between the third hyperspectral image and the first simulated hyperspectral image as the second control points; the first simulated hyperspectral image is the simulated hyperspectral image in the second image that overlaps with the third hyperspectral image;
[0087] The first preset percentage is greater than the second preset percentage.
[0088] In this embodiment, the first image is stitched together. Therefore, the percentage of the first control point is greater than the percentage of the second control point. That is, the first preset percentage is greater than the second preset percentage. For example, the first preset percentage can be about 60%-80%, and the second preset percentage can be 50%.
[0089] Specifically, the same location points in the overlapping portion between the third hyperspectral image and the fourth hyperspectral image are obtained, and a first preset percentage of these same location points are used as first control points; the same location points in the overlapping portion between the third hyperspectral image and the first simulated hyperspectral image are obtained, and a second preset percentage of these same location points are used as second control points. Here, the third hyperspectral image is any hyperspectral image in the first image, the fourth hyperspectral image is a hyperspectral image that overlaps with the first hyperspectral image, and the first simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the third hyperspectral image.
[0090] In this embodiment, a first control point is obtained to provide positional reference for the initial alignment of the first image; a second control point is obtained to provide positional reference for the subsequent re-alignment of the first image. Based on the two alignments, the stitching alignment of the first image is achieved.
[0091] In step 104, for each hyperspectral image in the first image, the first control point of the hyperspectral image is aligned with the first control point of the second hyperspectral image, and the second control point of the hyperspectral image is aligned with the second control point of the target simulated hyperspectral image; the second hyperspectral image is the hyperspectral image in the first image that overlaps with the hyperspectral image; the target simulated hyperspectral image is the simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
[0092] In this embodiment, the first image is stitched and aligned according to the first and second control points obtained in step 103. Specifically, for each hyperspectral image in the first image, the first control point of the hyperspectral image is aligned with the first control point of the second hyperspectral image, and the second control point of the hyperspectral image is aligned with the second control point of the target simulated hyperspectral image, thereby achieving stitching and alignment of the first image. The second hyperspectral image is a hyperspectral image that overlaps with the first hyperspectral image, and the target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the first hyperspectral image.
[0093] In one possible implementation, after step 104, the method may further include:
[0094] The aligned first image is then subjected to dense point cloud generation, digital elevation model simulation, and digital orthophoto generation to obtain the orthophoto corresponding to the first image.
[0095] Determine whether orthophotos are consistent with high-resolution satellite imagery;
[0096] If the orthophoto matches the high-resolution satellite image, then the first image is considered aligned.
[0097] Dense point clouds refer to point clouds obtained by 3D laser scanners or photogrammetric scanners, which are numerous and dense.
[0098] A Digital Elevation Model (DEM) is a digital simulation of ground topography (i.e., a digital representation of the surface morphology of the terrain) achieved through limited terrain elevation data. It is a physical ground model that represents ground elevation using an ordered array of numerical values.
[0099] Digital orthophoto map (DOM) is image data generated by using a digital elevation model to correct projection differences on each pixel of a scanned digitized aerial photograph, then mosaicking the images and cropping them according to the map sheet area.
[0100] Specifically, the first image after stitching and alignment is subjected to dense point cloud generation, digital elevation model simulation, and digital orthophoto generation to obtain the orthophoto corresponding to the first image after stitching and alignment. The orthophoto is then compared with high-resolution satellite imagery to determine if they are consistent. If they are consistent, the first image is determined to have achieved the stitching effect, and stitching and alignment are realized.
[0101] For example, if the first image is acquired and then stitched together without acquiring the second image, the first flight path will be bent. (Refer to...) Figure 3 The second image is acquired, and by acquiring the first and second control points, the first and second control points are aligned respectively to achieve the stitching alignment of the first image. The alignment effect is as follows: Figure 4 At this point, the first flight strip after the first image is stitched together is a straight flight strip.
[0102] This application provides a method for aligning hyperspectral images of a UAV. By acquiring a second image of a second flight path located on the first side of the first flight path at the same time as the first image, and aligning the second image using the same first control point and the same second control point, the first image can be stitched together. This method can effectively improve the stitching effect of UAV hyperspectral images, reduce the curvature between UAV hyperspectral images, and solve the problem of inaccurate stitching of aerial images of a single type of environment in the prior art, thereby improving the accuracy and stability of UAV hyperspectral image stitching. Furthermore, by selecting different second images according to the first images of different fields, UAV hyperspectral image stitching can be achieved for various fields, improving stitching efficiency.
[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0104] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.
[0105] Figure 5 A schematic diagram of the structure of the UAV hyperspectral image alignment device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown, and are described in detail below:
[0106] like Figure 5 As shown, the UAV hyperspectral image alignment device 5 includes:
[0107] The first image acquisition module 51 is used to acquire a first image of the target aerial photography area of the single type of environment to be tested when the UAV flies along the first flight path; the first image includes multiple hyperspectral images of the target aerial photography area.
[0108] The second image acquisition module 52 is used to acquire high-resolution satellite images of the target aerial area that are larger than the first flight strip at the same shooting time as the first image, and to acquire a second image based on the high-resolution satellite images. The second image includes a simulated hyperspectral map of the second flight strip located on the first side of the first flight strip.
[0109] The control point acquisition module 53 is used to take the same position points of the overlapping part between two adjacent hyperspectral images in the first image as first control points; and to take the same position points of the overlapping part between each hyperspectral image in the first image and the simulated hyperspectral image in the second image as second control points.
[0110] Image alignment module 54 is used to align the first control point of each hyperspectral image in the first image with the first control point of the second hyperspectral image, and to align the second control point of each hyperspectral image with the second control point of the target simulated hyperspectral image; the second hyperspectral image is a hyperspectral image in the first image that overlaps with the hyperspectral image; the target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
[0111] This application provides a UAV hyperspectral image alignment device. By acquiring a second image of a second flight path located on the first side of the first flight path at the same time as the first image capture, and aligning the second image with the same first control point and the same second control point, the device can stitch the first image together. This effectively improves the stitching effect of UAV hyperspectral images, reduces the curvature between UAV hyperspectral images, and solves the problem of inaccurate stitching of aerial images of a single type of environment in the prior art, thereby improving the accuracy and stability of UAV hyperspectral image stitching. Furthermore, by selecting different second images according to the first images of different fields, the device can stitch UAV hyperspectral images of various fields together, improving stitching efficiency.
[0112] In one possible implementation, the first image acquisition module can be used to:
[0113] Set the target aerial photography area for the drone to take aerial photos in the single type of environment to be tested, and set the heading overlap rate and waypoint interval;
[0114] Based on the heading overlap rate and waypoint interval, the positions of multiple first waypoints are determined by the UAV hovering and taking pictures while flying along the first wayline; the positions of the first waypoints include the first longitude and the first latitude.
[0115] Based on the location of the first waypoint, the UAV is controlled to automatically collect the first image and the corresponding positioning information, including longitude, latitude, altitude, heading angle, side angle, and rotation angle.
[0116] In one possible implementation, the device may further include, prior to the second image acquisition module:
[0117] The first positioning information acquisition module is used to acquire the positioning information of the first image, which includes the longitude, latitude, altitude, heading angle, side deviation angle and rotation angle of the first image.
[0118] Accordingly, the second image acquisition module can be used for:
[0119] Based on the longitude, latitude, altitude, heading angle, side deviation angle, and rotation angle of the first image, high-resolution satellite images of the target aerial area that are larger than the first flight strip are acquired at the same time as the first image is captured.
[0120] In one possible implementation, the second image acquisition module can also be used for:
[0121] Based on the forward overlap rate, lateral overlap rate, and waypoint interval of the UAV aerial photography, the location of the second waypoint is simulated when the UAV hovers and takes pictures along the second flight path; the location of the second waypoint includes the second longitude and the second latitude.
[0122] Acquire elevation model data corresponding to high-resolution satellite imagery, including longitude, latitude, and elevation;
[0123] Using 3D graphics software and based on elevation model data, satellite image simulation scenes are created from high-resolution satellite imagery.
[0124] In the satellite imagery simulation scenario, the first waypoint position, the second waypoint position, the flight altitude of the first image, and the image attribute parameters corresponding to the first image are set to generate a simulated aerial three-band color image;
[0125] The simulated aerial three-band color image is input into the hyperspectral aerial photography simulation model, which outputs a second image with 240 bands. The hyperspectral aerial photography simulation model is constructed based on a fully convolutional network.
[0126] In one possible implementation, the process of constructing a hyperspectral aerial photography simulation model may include:
[0127] Obtain historical simulated aerial three-band color images;
[0128] Obtain the second historical image of the 240 band corresponding to the historical simulated aerial three-band color image;
[0129] A fully convolutional network model was constructed, and the historical simulated aerial three-band color image was used as input, while the second historical image of 240 bands was used as output. The fully convolutional network model was trained to obtain the hyperspectral aerial simulation model.
[0130] In one possible implementation, the control point acquisition module can be used for:
[0131] Obtain the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image, and take the same position points of a first preset percentage among the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image as the first control points; the third hyperspectral image is any hyperspectral image in the first image, and the fourth hyperspectral image is a hyperspectral image that overlaps with the third hyperspectral image;
[0132] Obtain the same position points in the overlapping part between the third hyperspectral image and the first simulated hyperspectral image, and take the same position points in the overlapping part between the third hyperspectral image and the first simulated hyperspectral image as the second control points; the first simulated hyperspectral image is the simulated hyperspectral image in the second image that overlaps with the third hyperspectral image;
[0133] The first preset percentage is greater than the second preset percentage.
[0134] In one possible implementation, after the image alignment module, the device can also be used for:
[0135] The aligned first image is then subjected to dense point cloud generation, digital elevation model simulation, and digital orthophoto generation to obtain the orthophoto corresponding to the first image.
[0136] Determine whether orthophotos are consistent with high-resolution satellite imagery;
[0137] If the orthophoto matches the high-resolution satellite image, then the first image is considered aligned.
[0138] Figure 6 This is a schematic diagram of the terminal provided in an embodiment of this application. For example... Figure 6As shown, the terminal 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the various UAV hyperspectral image alignment method embodiments described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module in the above-described device embodiments, for example... Figure 5 The functions of each module are shown.
[0139] For example, the computer program 62 can be divided into one or more modules, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the terminal 6. For example, the computer program 62 can be divided into... Figure 5 The modules shown.
[0140] The terminal 6 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The terminal 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal 6 and does not constitute a limitation on terminal 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal may also include input / output devices, network access devices, buses, etc.
[0141] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0142] The memory 61 can be an internal storage unit of the terminal 6, such as a hard disk or memory of the terminal 6. The memory 61 can also be an external storage device of the terminal 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD card, Flash Card), etc., equipped on the terminal 6. Furthermore, the memory 61 can include both internal storage units and external storage devices of the terminal 6. The memory 61 is used to store the computer program and other programs and data required by the terminal. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0146] In the embodiments provided in this application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0149] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various UAV hyperspectral image alignment method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.
[0150] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for aligning hyperspectral images from unmanned aerial vehicles (UAVs), characterized in that, include: Acquire the first aerial image of the target area of a single type of environment under test when the UAV flies along the first flight path; The first image includes multiple hyperspectral images of the target aerial photograph area; At the same time as the first image is captured, a high-resolution satellite image of the target aerial area that is larger than the first flight strip is acquired, and a second image is acquired based on the high-resolution satellite image. The second image includes a simulated hyperspectral map of a second flight strip located on the first side of the first flight strip. The same location points in the overlapping part between two adjacent hyperspectral images in the first image are used as the first control points; The same location points in the overlapping portion between each hyperspectral image in the first image and the simulated hyperspectral image in the second image are used as the second control points; For each hyperspectral image in the first image, the first control point of the hyperspectral image is aligned with the first control point of the second hyperspectral image, and the second control point of the hyperspectral image is aligned with the second control point of the target simulated hyperspectral image; the second hyperspectral image is the hyperspectral image in the first image that overlaps with the hyperspectral image. The target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
2. The UAV hyperspectral image alignment method according to claim 1, characterized in that, The acquisition of the first image of the target aerial area of the single type of environment to be tested taken by the UAV while flying along the first flight path includes: Set the target aerial photography area for the UAV to take aerial photos in the single type of environment to be tested, and set the heading overlap rate and waypoint interval; Based on the heading overlap rate and waypoint interval, the positions of multiple first waypoints are determined by the UAV hovering and taking pictures while flying along the first flight path; the positions of the first waypoints include first longitude and first latitude. Based on the location of the first waypoint, the UAV is controlled to automatically collect the first image and the corresponding positioning information, which includes longitude, latitude, altitude, heading angle, side angle, and rotation angle.
3. The UAV hyperspectral image alignment method according to claim 1, characterized in that, Before acquiring high-resolution satellite imagery of the target aerial area that is larger than the first flight strip at the same time as acquiring the first image, the method further includes: The positioning information of the first image is obtained, including the longitude, latitude, altitude, heading angle, side deviation angle and rotation angle of the first image; Accordingly, the high-resolution satellite imagery of the target aerial photography area, which is larger than the first flight strip, acquired at the same shooting time as the first image, includes: Based on the longitude, latitude, altitude, heading angle, side deviation angle, and rotation angle of the first image, high-resolution satellite images of the target aerial photography area that are larger than the first flight strip are obtained at the same shooting time of the first image.
4. The UAV hyperspectral image alignment method according to claim 2, characterized in that, The step of acquiring a second image based on the high-resolution satellite imagery includes: Based on the forward overlap rate, lateral overlap rate, and waypoint interval of the UAV aerial photography, the location of the second waypoint is simulated when the UAV hovers and photographs along the second flight path; the location of the second waypoint includes the second longitude and the second latitude. Obtain elevation model data corresponding to the high-resolution satellite imagery, wherein the elevation model data includes longitude, latitude, and elevation; Using 3D graphics software and based on the elevation model data, a satellite image simulation scene is created from the high-resolution satellite imagery. In the satellite image simulation scene, the first waypoint position, the second waypoint position, the flight altitude of the first image, and the image attribute parameters corresponding to the first image are set to generate a simulated aerial three-band color image; The simulated aerial three-band color image is input into the hyperspectral aerial photography simulation model, which outputs a second image with 240 bands. The hyperspectral aerial photography simulation model is constructed based on a fully convolutional network.
5. The UAV hyperspectral image alignment method according to claim 4, characterized in that, The construction process of the hyperspectral aerial photography simulation model includes: Obtain historical simulated aerial three-band color images; Obtain the second historical image of the 240 band corresponding to the historical simulated aerial three-band color image; A fully convolutional network model is constructed, and the historical simulated aerial three-band color image is used as input, while the historical second image of 240 bands is used as output. The fully convolutional network model is trained to obtain the hyperspectral aerial simulation model.
6. The UAV hyperspectral image alignment method according to claim 1, characterized in that, The first control point is the same location point of the overlapping part between two adjacent hyperspectral images in the first image; The same location points in the overlapping portion between each hyperspectral image in the first image and the corresponding simulated hyperspectral image in the second image are used as second control points, including: Obtain the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image, and take the same position points of a first preset percentage among the same position points in the overlapping part between the third hyperspectral image and the fourth hyperspectral image as the first control points; the third hyperspectral image is any hyperspectral image in the first image, and the fourth hyperspectral image is a hyperspectral image that overlaps with the third hyperspectral image; Obtain the same position points of the overlapping portion between the third hyperspectral image and the first simulated hyperspectral image, and take the same position points of a second preset percentage among the same position points of the overlapping portion between the third hyperspectral image and the first simulated hyperspectral image as the second control points; the first simulated hyperspectral image is the simulated hyperspectral image in the second image that overlaps with the third hyperspectral image; The first preset percentage is greater than the second preset percentage.
7. The UAV hyperspectral image alignment method according to claim 1, characterized in that, After aligning the first control point of each hyperspectral image in the first image with the first control point of the second hyperspectral image, and aligning the second control point of each hyperspectral image with the second control point of the target simulated hyperspectral image, the method further includes: The aligned first image is subjected to dense point cloud generation, digital elevation model simulation and digital orthophoto generation respectively to obtain the orthophoto corresponding to the first image. Determine whether the orthophoto is consistent with the high-resolution satellite image; If the orthophoto is consistent with the high-resolution satellite image, then the first image is determined to be aligned.
8. A hyperspectral image alignment device for unmanned aerial vehicles, characterized in that, include: The first image acquisition module is used to acquire a first image of the target aerial photography area of the single type of environment to be tested when the UAV flies along the first flight path; the first image includes multiple hyperspectral images of the target aerial photography area. The second image acquisition module is used to acquire a high-resolution satellite image of the target aerial photography area that is larger than the first flight strip at the same shooting time as the first image, and to acquire a second image based on the high-resolution satellite image. The second image includes a simulated hyperspectral map of a second flight strip located on the first side of the first flight strip. The control point acquisition module is used to take the same position points in the overlapping part between two adjacent hyperspectral images in the first image as the first control points; The same location points in the overlapping portion between each hyperspectral image in the first image and the simulated hyperspectral image in the second image are used as the second control points; An image alignment module is used to align the first control point of each hyperspectral image in the first image with the first control point of a second hyperspectral image, and to align the second control point of each hyperspectral image with the second control point of a target simulated hyperspectral image; the second hyperspectral image is a hyperspectral image in the first image that overlaps with the hyperspectral image. The target simulated hyperspectral image is a simulated hyperspectral image in the second image that overlaps with the hyperspectral image.
9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the UAV hyperspectral image alignment method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the UAV hyperspectral image alignment method as described in any one of claims 1 to 7.