An Unmanned Aerial Vehicle Image Geometric Rectification System and Method Based on Multi-Source Image Registration

Abstract

The present invention discloses a UAV image geometric correction system and method based on multi-source image registration, belonging to the technical field of UAV image processing. The method includes the following steps: 1. Convert the reference image and the image to be corrected to the same geographic coordinate system; 2. Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; 3. Slice, register and obtain control points in the format of longitude and latitude coordinates for the two images in the overlapping area; 4. Merge all the obtained homologous feature points to construct a control point set; 5. Implement geometric information correction based on the control points for the image to be corrected in the geographic coordinate system; 6. Perform inverse transformation of the original projection coordinate system. The system includes functional modules for implementing each step. The present invention can achieve rapid geometric correction between images of different sensor types, does not require processing the entire image, and greatly improves the processing efficiency of UAV images. It does not limit the sensor types of UAV images and satellite images.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV image processing, and specifically relates to a UAV image geometric correction system and method based on multi-source image registration. Background Technique

[0002] At present, the application of UAVs in the field of remote sensing shows an increasingly expanding trend. UAVs can carry different high-resolution sensors and can perform imaging with various sensors. However, when the images generated by UAVs are overlaid and displayed with other digital base maps, there will be a phenomenon of inaccurate overlay. Therefore, it is necessary to correct the geometric position of UAV images.

[0003] The classic geometric position correction method is to calculate the coordinate transformation relationship between different images, and then correct the coordinate information of the images in combination with real ground control point data. Due to different imaging sensors and imaging platforms, and the high cost of obtaining real ground control points, it brings difficulties to the geometric correction between UAV images and satellite digital base maps. In addition, due to the large storage space occupied by high-resolution images, it brings great challenges to the efficiency of UAV image geometric correction. Summary of the Invention

[0004] In view of the above problems existing in the prior art, the present invention provides a UAV image geometric correction system and method based on multi-source image registration. By using the technology based on multi-source image registration, rapid geometric correction between images of different sensor types can be achieved without considering the difference problem between different coordinate systems. It is not necessary to process the entire image, which greatly improves the processing efficiency of UAV images. It does not limit the sensor types of UAV images and satellite images, and can better solve the heterogeneity matching problem between the two. The geometric position of the UAV imaging system can be corrected when the UAV starts working, and the generated images can be accurately overlaid with the digital map without processing, further improving the efficiency.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A UAV image geometric correction system based on multi-source image registration, comprising:

[0007] Module 1: Used to convert two images from the projection coordinate system to the same geographic coordinate system;

[0008] Module 2: Used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image;

[0009] Module 3: Used to slice, register and obtain control points in the format of longitude and latitude coordinates for the two images in the overlapping area according to the custom quantity and size;

[0010] Module Four: used to construct a set of control points in the form of [base_lat, base_lon, warp_lat, warp_lon] with the homologous feature points obtained by registration as control points; where base_lat, base_lon, warp_lat, and warp_lon are the longitude coordinates of the corresponding point on the reference image, the latitude coordinates of the corresponding point on the reference image, the longitude coordinates of the corresponding point on the image to be corrected, and the latitude coordinates of the corresponding point on the image to be corrected, respectively;

[0011] Module Five: used to perform geometric information correction based on control points on the image to be corrected in the geographic coordinate system;

[0012] Module Six: used to perform inverse transformation of the original projection coordinate system on the result image of Module Five.

[0013] As a further improvement of the above technical solution:

[0014] Module Three includes a slicing module and a registration module. The slicing module is used to evenly cut out image tiles with geographic information in the overlapping area of the two images according to custom quantity and size. The registration module is used to register the corresponding image tiles and convert the obtained point pixel coordinates into longitude and latitude coordinate formats.

[0015] A method for geometric correction of UAV images based on multi-source image registration, the correction method includes the following steps:

[0016] Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system;

[0017] Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image, aiming to facilitate extracting images at the same position through longitude and latitude;

[0018] Step 3: Slice, register and obtain control points in longitude and latitude coordinate formats for the two images in the overlapping area according to custom quantity and size;

[0019] Step 4: Merge all the obtained homologous feature points to construct a set of control points;

[0020] Step 5: Perform geometric information correction based on control points on the image to be corrected in the geographic coordinate system; <s

[0021] Step 6: Perform inverse transformation of the original projection coordinate system on the result image of Step 5 to achieve geometric position correction of UAV images.

[0022] Step 3 includes the following steps:

[0023] Step 31: Calculate the overlapping area of the two images by computing the four-corner coordinates of the images, and evenly crop custom-numbered and -sized image slices with geographic information within the overlapping area;

[0024] Step 32: Register the corresponding slices and convert the pixel coordinates of the obtained homologous feature points into longitude and latitude coordinate formats.

[0025] Step 32 includes the following steps:

[0026] Step 321: Feature detection to obtain a two-dimensional phase model, and detect Harris corner points on the phase diagram as feature points;

[0027] Step 322: Feature description to obtain the feature descriptors of each feature point, and obtain the homologous feature points between the frame images by matching the descriptors.

[0028] In Step 321, the real and imaginary parts of the image are calculated by combining the Fourier transform in the spatial domain through a Log-Gabor filter. The real and imaginary parts represent the even-symmetric wavelet and odd-symmetric wavelet respectively. The even-symmetric wavelet and odd-symmetric wavelet are respectively used to convolve the image to obtain response components. Considering the analysis results in all directions and all azimuths, and introducing a noise compensation term, a two-dimensional phase model is finally obtained.

[0029] In Step 321, the even-symmetric wavelet is , the odd-symmetric wavelet is , the image is , the response components obtained by convolution are and , , the image The amplitude component and phase component of the image at scale s and direction o can be obtained through . Introduce the noise compensation term as T, and the obtained two-dimensional phase model is , , where is the pixel coordinate, is a weighting function, is the deviation of the phase component, is a number to prevent the denominator from being 0. The image is processed by the phase model to obtain the phase diagram of the image. Harris corner points are detected on this phase diagram, and the detected points are used as the feature points of the image.

[0030] In step 322, given an image, first, calculate the amplitude using the response components obtained in step 321; then, arrange the convolutional layers in sequence to obtain a convolutional sequence, which is a multi-channel convolutional map. For each pixel position in the convolutional map, obtain an ordered array, and then find the maximum value in the array and its corresponding position channel, and set the position channel as the pixel value.

[0031] In step 322, the image is , and the response components obtained in step 321 are and , and the calculated amplitude is the amplitude at scale s and direction o. For direction o, the amplitudes at all scales are summed to obtain the log-Gabor layer , and the obtained multi-channel convolutional map is , where N s is a specific scale value, is the number of directions, with superscript ω = 1, 2, …, and the pixel position is , the ordered array is , and the maximum value in the array is obtained through the formula and its corresponding position channel , and set the value of as the pixel value at pixel position in the feature description map. Here, the feature description map refers to the convolutional map corresponding to pixel position . For each feature point, select a local image patch with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, divide the local area into 6×6 sub-grids, and construct a distribution histogram without bins for each sub-grid. Obtain the feature vector by connecting all histograms. The dimension of the feature vector is 6×6×n; to obtain invariance to illumination changes, finally, normalize the feature vector, which is the feature descriptor of the feature point, where J and n are positive integers.

[0032] ​In step 4, all feature points with the same name are used as control points, the control points are merged, and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into the pixel coordinates (x, y) on the image to be corrected. Finally, a set of control points in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM) and is temporarily set to 0 for the sake of the overall efficiency of the system. (base_lat, base_lon) are the latitude and longitude coordinates of the corresponding point on the reference image, and (warp_lat, warp_lon) are the latitude and longitude coordinates of the corresponding point on the image to be corrected.

[0033] The beneficial effects of the present invention are as follows:

[0034] (1) By using the technology based on multi-source image registration, rapid geometric correction between images of different sensor types can be achieved without considering the differences between different coordinate systems.

[0035] (2) By uniformly extracting feature points with the same name on the image as control points, it is not necessary to process the entire image, which greatly improves the processing efficiency of UAV images.

[0036] (3) It does not limit the sensor types of UAV images and satellite images, can better solve the heterogeneous matching problem between the two, can be used in a variety of application scenarios, and has a small amount of calculation, which can better meet the large-volume image processing requirements of the UAV platform.

[0037] (4) Integrating the system and the digital base map on the airborne platform can correct the geometric position of the UAV imaging system when the UAV starts to work, and the generated image can be accurately registered with the digital map without further processing, further improving the efficiency. Description of the Drawings

[0038] Figure 1 is the flowchart of the method and system for geometric correction of UAV images based on multi-source image registration of the present invention;

[0039] Figure 2 is the schematic diagram of the registration method descriptor structure and matching retrieval of the present invention. Detailed Embodiments

[0040] The following further details the specific embodiments of the present invention with reference to the drawings. It should be understood that the specific embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not intended to limit the present invention.

[0041] For ease of description, spatial relative terms, such as "above", "on top of", "on the upper surface", "over", etc., may be used herein to describe the spatial positional relationship of one device or feature to other devices or features as shown in the figures. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is inverted, a device described as "above" or "on top of" other devices or structures will then be positioned "below" or "beneath" the other devices or structures. Thus, the exemplary term "above" can include both the orientations of "above" and "below". The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and corresponding interpretations of the spatial relative descriptions used herein will be made.

[0042] An unmanned aerial vehicle (UAV) image geometric correction system based on multi-source image registration includes the following modules:

[0043] Module 1: Used to convert two images from a projection coordinate system to the same geographic coordinate system;

[0044] Module 2: Used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image;

[0045] Module 3: Used to slice, register, and obtain control points in the format of longitude and latitude coordinates for the two images in the overlapping area according to a custom quantity and size;

[0046] Module 3 further includes a slicing module and a registration module; wherein:

[0047] The slicing module is used to uniformly cut out tiles with geographic information in the overlapping area of the two images according to a custom quantity and size;

[0048] The registration module is used to register the corresponding tiles and convert the obtained point pixel coordinates into the format of longitude and latitude coordinates.

[0049] Module 4: Used to construct a control point set in the form of [base_lat, base_lon, warp_lat, warp_lon] with the registered homologous feature points as control points;

[0050] Module 5: Used to perform geometric information correction on the image to be corrected in the geographic coordinate system based on the control points;

[0051] Module 6: Used to perform the inverse transformation of the original projection coordinate system on the result image of Module 5.

[0052] It should be noted that the protection scope of the present invention does not involve electrical components and other physical components. The "module" refers to each functional part of the entire system, and the electrical components and other physical components corresponding to each module can adopt components in the prior art, which will not be elaborated here.

[0053] A method for geometric correction of UAV images based on multi-source image registration based on the above system includes the following steps:

[0054] Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system.

[0055] Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image, aiming to facilitate extracting images at the same location through longitude and latitude.

[0056] Step 3: In the overlapping area, slice, register, and obtain control points in the format of longitude and latitude coordinates for the two images according to the custom quantity and size.

[0057] Further, Step 3 includes the following steps:

[0058] Step 31: Calculate the overlapping area of the two images by calculating the four-corner coordinates of the images, and evenly cut out custom quantities and sizes of image slices with geographic information in the overlapping area. The image slices are used for registration. In this embodiment, 500×500 pixel-sized blocks are extracted from the two images in the overlapping area according to the quantity of 6×6, and their geographic information is retained.

[0059] Step 32: Register the corresponding slices, and convert the pixel coordinates of the obtained homologous feature points into the format of longitude and latitude coordinates. The homologous feature points are the corresponding point pairs on the two images obtained in the registration process.

[0060] The registration algorithm used in this solution is a feature transformation algorithm insensitive to radiometric transformation. This method uses the phase consistency principle of the image to eliminate the large radiometric intensity differences between multi-source images, mainly including two parts: feature detection and feature description. That is, the sub-steps further include:

[0061] Step 321: Feature detection. Calculate the real and imaginary parts of the image by combining the Fourier transform with the Log-Gabor filter (LGF) in the spatial domain, which respectively represent the even symmetric wavelet ( ) and the odd symmetric wavelet ( ); Convolve the image with the even symmetric and odd symmetric log-Gabor wavelets respectively to obtain the response components and and , wherein, the amplitude component of the image at the scale s and the direction o Sum-phase component can be obtained through and ; Considering the analysis results in all directions and all azimuths, and introducing the noise compensation term T, the final two-dimensional phase (PC) model is: . Among them, is the pixel coordinate, is a weighting function, is the deviation of the phase component, is a very small number to prevent the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image, and Harris corners are detected on this phase map, and the detected points are used as the feature points of the image.

[0062] Step 322: Feature description. Use the corresponding components and amplitudes obtained in Step 321 for feature description. Specifically, given the image , use the response components and obtained in Step 321, and then calculate the amplitude at the scale s and direction o; for the direction o, all scales (here N s is a specific scale value) of the amplitudes are summed to obtain the log-Gabor layer ; then, the log-Gabor convolution layers are arranged in order to obtain the log-Gabor convolution sequence, which is a multi-channel convolution mapping , where is the number of directions, and the superscript ω = 1, 2, …. Therefore, for each pixel position of the convolution map (the convolution sequence consists of multiple convolution maps in a sequence), we can obtain an ordered array . Then use to find the maximum value in this array and its corresponding position channel . We set the value of as the pixel value at the position in the feature description map. Among them, the feature description map refers to the convolution map corresponding to the pixel position .

[0063] For each feature point, a local image patch with J×J pixels centered on the feature is selected, and a Gaussian function with a standard deviation equal to J / 2 is used to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, the local region is divided into 6×6 sub-grids, and a distribution histogram without bins is constructed for each sub-grid. The feature vector is obtained by concatenating all the histograms. Therefore, the dimension of the feature vector is 6×6×n; to obtain invariance to illumination changes, the feature vector is finally normalized to obtain the feature descriptor of the feature point. By matching the descriptors of the feature points, corresponding homologous feature points between frame images are screened out. Here, J and n are positive integers.

[0064] Figure 2 It is a schematic diagram of the descriptor structure and matching retrieval of the registration method of the present invention. The stacked squares represent the descriptor of the present invention, which is composed of 6 layers of feature maps stacked together. The degree of each layer indicates that the feature map of this layer is generated by step 322 in the direction of this degree. The circular arrow represents the process of matching retrieval, that is, in a circular form, the feature maps generated in different degree directions are placed on the first layer in turn to match the descriptors corresponding to the feature points obtained on two patches. The two feature point pairs with the highest descriptor similarity are the homologous feature point pairs.

[0065] Step 4: Merge all the obtained homologous feature points to construct a control point set.

[0066] In this step, specifically, all the homologous feature points are used as control points, and the control points are merged and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into pixel coordinates (x, y) on the image to be corrected. Finally, a control point set in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM). For the sake of the overall efficiency of the system, z is temporarily set to 0 in the present invention. (base_lat, base_lon) are the longitude and latitude coordinates of the corresponding point on the reference image. (warp_lat, warp_lon) are the longitude and latitude coordinates of the corresponding point on the image to be corrected.

[0067] Step 5: Perform geometric information correction based on control points on the image to be corrected in the geographic coordinate system.

[0068] In this step, the correction method is to traverse (x, y) in the control point file and assign the geographic information of the corresponding pixel position of the image to the new values (base_lat, base_lon), thereby completing the geometric correction of the image.

[0069] Step 6: Perform the inverse transformation of the original projection coordinate system on the result image of Step 5, that is, inverse-transform the corrected image back to its original projection coordinate system to achieve the geometric position correction of the UAV image.

[0070] Finally, it is necessary to state here that the above embodiments are only used to further illustrate the technical solutions of the present invention in detail, and cannot be understood as limiting the protection scope of the present invention. Some non-essential improvements and adjustments made by those skilled in the art based on the above content of the present invention all fall within the protection scope of the present invention.

Claims

1. A method for geometric correction of UAV images based on multi-source image registration, characterized in that The correction method is based on a correction system, and the correction system includes: Module 1: used to convert two images from a projection coordinate system to the same geographic coordinate system; Module 2: used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Module 3: used to slice, register and obtain control points in the format of longitude and latitude coordinates for the two images in the overlapping area according to custom quantity and size; Module 4: used to construct a set of control points in the form of [base_lat, base_lon, warp_lat, warp_lon] with the corresponding feature points obtained by registration as control points; where base_lat, base_lon, warp_lat, and warp_lon are the longitude coordinates of the corresponding points on the reference image, the latitude coordinates of the corresponding points on the reference image, the longitude coordinates of the corresponding points on the image to be corrected, and the latitude coordinates of the corresponding points on the image to be corrected respectively; Module 5: used to perform geometric information correction based on control points on the image to be corrected in the geographic coordinate system; Module 6: used to perform inverse transformation of the original projection coordinate system on the result image of Module 5; The correction method includes the following steps: Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system; Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Step 3: Slice, register and obtain control points in the format of longitude and latitude coordinates for the two images in the overlapping area according to custom quantity and size; Step 4: Merge all the obtained corresponding feature points to construct a set of control points; Step 5: Perform geometric information correction based on control points on the image to be corrected in the geographic coordinate system; Step 6: Perform inverse transformation of the original projection coordinate system on the result image of Step 5 to achieve geometric position correction of the UAV image; Step 3 includes the following steps: Step 31: Calculate the overlapping area of the two images by calculating the four-corner coordinates of the images, and evenly cut out image slices with geographic information of custom quantity and size in the overlapping area; Step 32: Register the corresponding slices and convert the pixel coordinates of the obtained corresponding feature points to the format of longitude and latitude coordinates; Step 32 includes the following steps: Step 321: Feature detection, obtain a two-dimensional phase model, and detect Harris corner points on the phase diagram as feature points; In Step 321, calculate the real and imaginary parts of the image by combining the Fourier transform in the spatial domain with a Log-Gabor filter. The real and imaginary parts represent the even-symmetric wavelet and odd-symmetric wavelet respectively. Convolve the image with the even-symmetric wavelet and odd-symmetric wavelet respectively to obtain response components. Consider the analysis results in all directions and all orientations, and introduce a noise compensation term to finally obtain a two-dimensional phase model; Step 322: Feature description, obtain the feature descriptor of each feature point, and obtain the corresponding feature points between frames of images by matching the descriptors. In step 322, given an image, first, calculate the amplitude using the response components obtained in step 321; then, arrange the convolutional layers in sequence to obtain a convolutional sequence, which is a multi-channel convolutional map. For each pixel position in the convolutional map, obtain an ordered array, and then find the maximum value in the array and its corresponding position channel, and set the position channel as the pixel value.

2. The method for geometric correction of UAV images according to claim 1, characterized in that: Module three includes a slicing module and a registration module. The slicing module is used to uniformly cut out tiles with geographic information in the overlapping area of two images according to a custom quantity and size. The registration module is used to register the corresponding tiles and convert the obtained point pixel coordinates into the format of longitude and latitude coordinates.

3. The method for geometric correction of UAV images according to claim 1, wherein: In step 321, the even symmetric wavelet is , the odd symmetric wavelet is , the image is , the response components obtained by convolution are and , , the image 's amplitude component and phase component at scale s and direction o can be obtained through and . By introducing the noise compensation term as T, the obtained two-dimensional phase model is , , where is the pixel coordinate, is a weighting function, is the deviation of the phase component, is a number to prevent the denominator from being 0. The image is processed by the phase model to obtain the phase diagram of the image. Harris corner detection is performed on this phase diagram, and the detected points are used as the feature points of the image.

4. The method for geometric correction of UAV images according to claim 1, characterized in that: In step 322, the image is , the response components obtained in step 321 are and , the calculated amplitude is the amplitude at scale s and direction o. For direction o, the amplitudes at all scales are summed to obtain the log-Gabor layer , and the resulting multi-channel convolution map is , where N s is a specific scale value, is the number of directions, the superscript ω = 1, 2, …, the pixel position is , the ordered array is , and the maximum value in the array is obtained through the formula and its corresponding position channel . Set the value of to the pixel value at pixel position in the feature description map. Here, the feature description map refers to the convolution map corresponding to pixel position . For each feature point, a local image patch with J×J pixels centered on this feature is selected, and a Gaussian function with a standard deviation equal to J / 2 is used to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without bins is constructed for each sub-grid. The feature vector is obtained by concatenating all histograms. The dimension of the feature vector is 6×6×n. To obtain invariance to illumination changes, the finally normalized feature vector is the feature descriptor of this feature point, where J and n are positive integers.​ 5. The geometric correction method for drone images according to claim 1, wherein: In step 4, all homologous feature points are used as control points, the control points are merged and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into the pixel coordinates (x, y) on the image to be corrected. Finally, a control point set in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM). For the sake of the overall efficiency of the system, it is temporarily set to 0. (base_lat, base_lon) are the longitude and latitude coordinates of the corresponding points on the reference image, and (warp_lat, warp_lon) are the longitude and latitude coordinates of the corresponding points on the image to be corrected.

Images