Unmanned aerial vehicle image automatic splicing method and device

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle image automatic splicing method and device, and the method comprises the steps: obtaining a plurality of aerial images of an unmanned aerial vehicle, carrying out the preprocessing of the images, and extracting the feature points of the preprocessed images; obtaining a base map, extracting feature points of the base map, and matching the feature points of the base map with the feature points of the aerial image; and based on a matching result, registering the base map and the aerial images, and splicing the plurality of aerial images after registration. Ground control points are not needed, the feature matching points of the unmanned aerial vehicle shot image and the base image are found by introducing the high-precision base image and combining the feature matching technology, the space coordinates of the feature matching points of the base image are obtained, the unmanned aerial vehicle flies to shoot the image, and the precision and quality of the spliced image are ensured.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image reconstruction, and in particular to a method and device for automatically stitching unmanned aerial vehicle images. Background Art

[0002] At present, UAV remote sensing images have been widely used in environmental monitoring, land use surveys, disaster assessments, etc. However, due to the limitations of the UAV's flight altitude and the field of view of the camera equipment, the images collected in a single flight usually have a limited coverage, and multiple images need to be stitched together to form a large-scale image map. Traditional image stitching methods usually rely on ground control points (GCPs) for geometric correction and registration, which not only increases the workload and cost, but is also restricted by ground conditions and is difficult to implement in complex terrain.

[0003] Therefore, the relevant technology relies on ground control points (image control points) for geometric correction and registration. However, due to factors such as terrain and communication, it is difficult to deploy control points in some areas, and the quality of the produced images is unstable. Summary of the invention

[0004] The main purpose of the present invention is to provide a method and device for automatically stitching drone images to solve the technical problems existing in the related art.

[0005] In order to achieve the above-mentioned purpose, according to the first aspect of the present disclosure, a method for automatic stitching of drone images is provided, comprising acquiring multiple aerial images of a drone and preprocessing the images, extracting feature points of the preprocessed images; acquiring a base map and extracting feature points of the base map, and matching the feature points of the base map with the feature points of the aerial images; based on the matching result, registering the base map and the aerial images, and stitching the multiple registered aerial images.

[0006] Optionally, based on the matching result, registering the base map and the aerial image includes: determining the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; and registering the base map and the aerial image based on the coordinates.

[0007] Optionally, based on the matching results, registering the base map and the aerial image includes: determining a coordinate transformation matrix between the aerial image and the base map based on the matching results and the spatial coordinate system corresponding to the base map; converting the coordinates corresponding to the aerial image into spatial coordinates based on the coordinate transformation matrix; and registering the aerial image converted into spatial coordinates with the base map.

[0008] Optionally, stitching the registered multiple aerial images includes: stitching adjacent images based on an image fusion algorithm.

[0009] Optionally, the feature points of the base map are matched with the feature points of the aerial image: direction values ​​are assigned at the feature point positions: at the feature point positions, different values ​​are assigned in different image gradient directions, wherein the gradient and gradient direction of the image are as follows:

[0010]

[0011] Describe the feature points: calculate the local gradient within the neighborhood of the feature points; match based on the direction assignment and description of the feature points.

[0012] Optionally, the method further includes: automatically adjusting the color and brightness of the spliced ​​area.

[0013] According to a second aspect of the present disclosure, there is provided an automatic stitching device for unmanned aerial vehicle images, comprising a feature extraction unit, configured to obtain a plurality of aerial images of an unmanned aerial vehicle, pre-process the images, and then extract feature points of the pre-processed images; a matching unit, configured to obtain a base map and extract feature points of the base map, and match the feature points of the base map with the feature points of the aerial images; and a stitching unit, configured to register the base map and the aerial images based on the matching result, and stitch the registered plurality of aerial images.

[0014] Optionally, based on the matching result, registering the base map and the aerial image includes: determining the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; and registering the base map and the aerial image based on the coordinates.

[0015] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in any one of the implementation modes of the first aspect.

[0016] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one implementation of the first aspect.

[0017] The method and device for automatically stitching unmanned aerial vehicle images of this embodiment include obtaining multiple unmanned aerial vehicle aerial images and preprocessing the images, extracting feature points of the preprocessed images; obtaining a base map and extracting feature points of the base map, and matching the feature points of the base map with the feature points of the aerial images; based on the matching results, registering the base map and the aerial images, and stitching the multiple registered aerial images. No ground control points are required. By introducing high-precision base map images and combining feature matching technology, feature matching points of the images shot by the unmanned aerial vehicle and the base map images are found, and the spatial coordinates of the feature matching points of the base map images are obtained and assigned to the images shot by the unmanned aerial vehicle in flight, so as to ensure the accuracy and quality of the stitched images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a method for automatically stitching drone images according to an embodiment of the present disclosure;

[0020] Figure 2a is a schematic diagram of multiple aerial images in an embodiment of the present disclosure;

[0021] Figure 2b is a schematic diagram of a splicing result without control points in an embodiment of the present disclosure;

[0022] Figure 2c This is a schematic diagram of the splicing result obtained according to the method of the embodiment of the present disclosure.

[0023] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] refer to Figure 2a It shows multiple drone images. In the related technology, after puzzle without control points, Figure 2b The image shown in the figure has no ground control point puzzle result, and there is an offset phenomenon in the road part in the upper right corner. In order to solve this problem, according to the embodiment of the present disclosure, a method for automatic stitching of drone images is provided, such as Figure 1 As shown, it includes the following steps 101 to 103:

[0028] Step 101: Acquire multiple aerial images from a drone, pre-process the images, and extract feature points of the pre-processed images.

[0029] In this embodiment, multiple images taken by a drone are collected, and pre-processing operations such as radiation correction and color balance are performed on the images. Feature points are extracted from each image using a feature point extraction algorithm such as the SI FT algorithm.

[0030] Step 102: Acquire a base map and extract feature points of the base map, and match the feature points of the base map with the feature points of the aerial image.

[0031] In this embodiment, the UAV remote sensing image with high-precision spatial coordinates input by the user is received as a matching base map and used as a spatial reference benchmark. The feature points of the base map and the UAV image are matched by a matching algorithm.

[0032] Furthermore, when matching feature points, the base map is divided into a rectangular feature point matching range of a preset size, and the aerial image within this range is found based on the base map coordinate range combined with the drone POS data, and then the feature points of the base map and the drone image are matched based on the feature matching algorithm. Specifically, the aerial point position can be converted into a point vector based on the POS data of each aerial image; Figure 1 *Convert the range of 1 km into a surface vector; calculate the spatial position relationship between the point vector and the surface vector; if the point vector is within the range of the surface vector, perform feature point matching.

[0033] Step 103: Based on the matching result, the base map and the aerial image are registered, and the registered multiple aerial images are stitched.

[0034] In this embodiment, reference Figure 2c Based on the jigsaw puzzle result obtained by the method of this embodiment, with the high-precision base map as the registration object, through feature matching and feature selection algorithms, the jigsaw puzzle result can be accurately reconstructed in two dimensions to restore the real geographical location.

[0035] As an optional implementation method of this embodiment, based on the matching result, registering the base map and the aerial image includes: determining the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; and registering the base map and the aerial image based on the coordinates.

[0036] In this optional implementation, the coordinate range of the entire image is calculated based on the coordinates of the base map matching points, and spatial registration is performed.

[0037] As an optional implementation method of this embodiment, the base map and the aerial image are registered based on the matching results, including: determining the transformation matrix between the aerial image and the base map based on the matching results and the spatial coordinate system corresponding to the base map; converting the coordinates corresponding to the aerial image into spatial coordinates based on the transformation matrix; and registering the aerial image converted into spatial coordinates with the base map.

[0038] In this optional implementation, the conversion relationship between the pixel row and column numbers of the drone image feature points and the base map spatial coordinate system is calculated based on the feature point matching results between the base map and the drone image. The coordinates of the drone image are converted into spatial coordinates using a conversion matrix. Based on the converted spatial coordinates, the image is accurately registered. During registration, the RANSAC algorithm is used to remove mismatched points to improve the registration accuracy. Based on the coordinates of the base map matching points, the coordinate range of the entire image is calculated for spatial registration. Combined with the spatial coordinates of the matching points and the drone lens parameters, the resolution information of the aerial image is obtained, the coordinate range of the entire aerial image is calculated, and the current aerial image map projection information is written into the picture.

[0039] Specifically, according to the matching results of the feature points between the base map and the drone image, the conversion relationship between the pixel row and column numbers of the drone image feature points and the base map spatial coordinate system is calculated. According to the map projection parameters of the base map and the POS data of the drone, the deflection angle between the aerial image and the base map is calculated. After the direction of the aerial image is corrected, the conversion matrix is ​​applied to convert the coordinates of the drone image into spatial coordinates. The conversion matrix can correct the direction of the aerial image according to the drone POS data. Calculate the pixel coordinates of the feature points of the aerial image after direction correction. Assign the geographic coordinates of the base map feature points to the pixel coordinates of the aerial image.

[0040] As an optional implementation of this embodiment, stitching the registered multiple aerial images includes: stitching adjacent images based on an image fusion algorithm after the registration.

[0041] As an optional implementation manner of this embodiment, the method further includes: automatically adjusting the color and brightness of the spliced ​​area.

[0042] In this optional implementation, feature points of adjacent images are matched by a matching algorithm, such as the SI FT algorithm. The spatially registered images are seamlessly spliced ​​together using an image fusion algorithm (such as weighted average, pyramid fusion, etc.) based on spatial coordinates. The color and brightness of the spliced ​​area are adjusted to ensure that the spliced ​​image is consistent as a whole. After the splicing is completed, the color and brightness of the spliced ​​area are adjusted to ensure that the spliced ​​image is consistent as a whole.

[0043] Save the stitched images in a specified format, such as GeoTIFF, and generate the digital elevation model (DEM) and orthophoto of the images as needed.

[0044] As an optional implementation of this embodiment, matching the feature points of the base map with the feature points of the aerial image includes: assigning directions at the feature point positions: assigning different values ​​at the feature point positions in different image gradient directions, wherein the gradient and gradient direction of the image are as follows: Describe the feature points: calculate the local gradient within the neighborhood of the feature points; match based on the direction assignment and description of the feature points.

[0046] In this optional implementation, scale space extremum detection: Gaussian Laplacian operator is used to search for interest points that are invariant in images at different scales:

[0047]

[0048] When a pixel is larger or smaller than all its neighbors (different scale domains), the pixel is an interest point (extreme point).

[0049] Feature point positioning: At each feature point position, determine the spatial scale of the feature point.

[0050] Feature direction assignment: At the feature point position, different values ​​are assigned in different image gradient directions. The gradient and gradient direction of the image are as follows:

[0051]

[0052] Feature point description: Calculate the local gradient within the neighborhood of the feature point to improve the ability to resist interference caused by deformation or illumination changes.

[0053] This embodiment uses drones to collect image data and preprocess it. Through feature selection and feature matching algorithms, feature matching points between drone images and high-precision base maps are obtained, pixel spatial coordinates matching the base map feature points are taken out and assigned to drone images, and finally all drone images are spliced ​​together using an image stitching algorithm to achieve large-scale two-dimensional reconstruction of drone images. Compared with existing technical methods, the method of the present invention does not require ground control points. By introducing high-precision base map images and combining SI FT feature matching technology, feature matching points between drone images and base map images are found, and the spatial coordinates of the feature matching points of the base map images are obtained and assigned to drone flight images to ensure the accuracy and quality of the stitched images. This method is applicable to a variety of complex terrains and environmental conditions.

[0054] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] According to an embodiment of the present disclosure, there is also provided an automatic stitching device for drone images, including a feature extraction unit, configured to obtain multiple aerial images of a drone and pre-process the images, and then extract feature points of the pre-processed images; a matching unit, configured to obtain a base map and extract feature points of the base map, and match the feature points of the base map with the feature points of the aerial images; and a stitching unit, configured to align the base map and the aerial images based on the matching result, and stitch the multiple aerial images after the alignment.

[0056] As an optional implementation method of this embodiment, based on the matching result, registering the base map and the aerial image includes: determining the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; and registering the base map and the aerial image based on the coordinates.

[0057] As an optional implementation method of this embodiment, the base map and the aerial image are registered based on the matching results, including: determining the transformation matrix between the aerial image and the base map based on the matching results and the spatial coordinate system corresponding to the base map; converting the coordinates corresponding to the aerial image into spatial coordinates based on the transformation matrix; and registering the aerial image converted into spatial coordinates with the base map.

[0058] As an optional implementation of this embodiment, stitching the registered multiple aerial images includes: stitching adjacent images based on an image fusion algorithm after the registration.

[0059] As an optional implementation of this embodiment, matching the feature points of the base map with the feature points of the aerial image includes: assigning directions at the feature point positions: assigning different values ​​at the feature point positions in different image gradient directions, wherein the gradient and gradient direction of the image are as follows:

[0060]

[0061] Describe the feature points: calculate the local gradient within the neighborhood of the feature points; match based on the direction assignment and description of the feature points.

[0062] As an optional implementation manner of this embodiment, the method further includes: automatically adjusting the color and brightness of the spliced ​​area.

[0063] The present disclosure provides an electronic device, such as Figure 3 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 3 A processor 31 is taken as an example.

[0064] The controller may further include: an input device 33 and an output device 34 .

[0065] The processor 31, the memory 32, the input device 33 and the output device 34 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0066] The processor 31 may be a central processing unit (CPU). The processor 31 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. The general-purpose processor may be a microprocessor or any conventional processor.

[0067] The memory 32 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the control method in the embodiment of the present disclosure. The processor 31 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 32, that is, the enterprise data exchange method based on email in the above method embodiment is implemented.

[0068] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a processing device operated by the server, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0069] The input device 33 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing device of the server. The output device 34 can include display devices such as a display screen.

[0070] One or more modules are stored in the memory 32, and when executed by one or more processors 31, the execution is as follows: Figure 1 The method shown.

[0071] Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method is a program that can be completed by instructing related hardware through a computer program, which can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiment of the above-mentioned method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0072] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for automatically stitching drone images, characterized in that: include: After acquiring a plurality of aerial images from a drone and preprocessing the images, feature points of the preprocessed images are extracted; Acquire a base map and extract feature points of the base map, and match the feature points of the base map with the feature points of the aerial image; Based on the matching results, the base map and the aerial image are registered, and the registered aerial images are stitched together.

2. The automatic stitching method of drone images according to claim 1 is characterized in that: Based on the matching results, the base map and aerial image are registered including: Determine the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; The base map and the aerial image are registered based on the coordinates.

3. The automatic stitching method of drone images according to claim 1 is characterized in that: Based on the matching results, the base map and aerial image are registered including: Determine a conversion matrix between the aerial image and the base map based on the matching result and the spatial coordinate system corresponding to the base map; Converting the coordinates corresponding to the aerial image into spatial coordinates based on the conversion matrix; The aerial image converted into spatial coordinates is registered with the base map.

4. The automatic stitching method of drone images according to claim 1 is characterized in that: Stitching the registered multiple aerial images includes: stitching adjacent images based on an image fusion algorithm after the registration.

5. The automatic stitching method of drone images according to claim 1 is characterized in that: The matching of the feature points of the base map with the feature points of the aerial image comprises: Direction assignment at the feature point position: At the feature point position, different values ​​are assigned in different image gradient directions, where the image gradient and gradient direction are as follows: Describe the feature points: calculate the local gradient within the neighborhood of the feature points; Matching is performed based on the direction assignment and description of feature points.

6. The automatic stitching method of drone images according to claim 4 is characterized in that: The method further comprises: Automatically adjust the color and brightness of the stitched area.

7. An automatic stitching device for drone images, characterized in that: include: A feature extraction unit is configured to obtain a plurality of aerial images of a drone and pre-process the images, and then extract feature points of the pre-processed images; A matching unit is configured to obtain a base map and extract feature points of the base map, and match the feature points of the base map with the feature points of the aerial image; The stitching unit is configured to register the base map and the aerial image based on the matching result, and stitch the registered multiple aerial images.

8. The automatic stitching device for drone images according to claim 7, characterized in that: Based on the matching results, the base map and aerial image are registered including: Determine the coordinates of the aerial image in the spatial coordinate system corresponding to the base map; The base map and the aerial image are registered based on the coordinates.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one of claims 1-6.

Images