Unmanned aerial vehicle image geometric correction system and method based on multi-source image registration

Abstract

The invention discloses an unmanned aerial vehicle image geometric correction system and method based on multi-source image registration, and belongs to the technical field of unmanned aerial vehicle image processing, and the method comprises the steps: 1, converting a reference image and a to-be-corrected image into a same geographic coordinate system; 2, resampling the to-be-corrected image in the same geographic coordinate system until the resolution ratio of the to-be-corrected image is consistent with that of the reference image; 3, the two images are sliced and registered in the overlapping area, and longitude and latitude coordinate format control points are obtained; 4, combining all the obtained homonymy feature points, and constructing a control point set; 5, implementing control point-based geometric information correction on the to-be-corrected image of the geographic coordinate system; and 6, inverse transformation of the original projection coordinate system is carried out. The system comprises functional modules for realizing the steps. According to the method, rapid geometric correction between different sensor types of images can be realized, the whole image does not need to be processed, and the processing efficiency of the unmanned aerial vehicle image is greatly improved. And the sensor types of unmanned aerial vehicle images and satellite images are not limited.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle image processing, and in particular relates to a system and method for geometric correction of unmanned aerial vehicle images based on multi-source image registration. Background Art

[0002] At present, the application of drones in the field of remote sensing is showing an increasingly expanding trend. They can be equipped with different high-resolution sensors and can perform multiple sensor imaging. However, the images generated by drones cannot be accurately overlaid when they are overlaid with other digital base maps. Therefore, it is necessary to correct the geometric position of drone images.

[0003] The classic geometric position correction method is to calculate the coordinate transformation relationship between different images, and then correct the coordinate information of the image in combination with the real ground control point data. Due to the differences in imaging sensors and imaging platforms, and the high cost of obtaining real ground control points, it is difficult to correct the geometry between drone images and satellite digital base maps. In addition, since high-resolution images usually occupy a large amount of storage, it brings great challenges to the efficiency of drone image geometric correction. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present invention provides a system and method for geometric correction of drone images based on multi-source image registration. The technology based on multi-source image registration can realize fast geometric correction between images of different sensor types without considering the difference between different coordinate systems. It is not necessary to process the entire image, which greatly improves the processing efficiency of drone images. It does not limit the sensor types of drone images and satellite images, and can better solve the heterogeneous matching problem between the two. The geometric position of the drone imaging system can be corrected when the drone starts working, and the generated image can be accurately matched with the digital map without processing, further improving efficiency.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: A UAV image geometric correction system based on multi-source image registration, comprising: Module 1: used to convert two images from the projected coordinate system to the same geographic coordinate system; Module 2: used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Module 3: Used to slice, align and obtain control points in latitude and longitude coordinate format in overlapping areas according to custom quantity and size; Module 4: Used to construct a control point set in the form of [base_lat, base_lon, warp_lat, warp_lon] using the feature points of the same name obtained by registration as control points; where base_lat, base_lon, warp_lat, and warp_lon are the longitude coordinates of the corresponding point on the reference image, the latitude coordinates of the corresponding point on the reference image, the longitude coordinates of the corresponding point on the image to be corrected, and the latitude coordinates of the corresponding point on the image to be corrected, respectively; Module 5: Used to correct the geometric information of the image to be corrected in the geographic coordinate system based on control points; Module 6: Used to perform inverse transformation of the result image of module 5 to the original projection coordinate system.

[0006] As a further improvement of the above technical solution: Module three includes a slicing module and a registration module. The slicing module is used to evenly crop tiles with geographic information in the overlapping area of ​​two images according to a custom number and size. The registration module is used to register the corresponding tiles and convert the obtained point pixel coordinates into longitude and latitude coordinate format.

[0007] A method for geometric correction of unmanned aerial vehicle images based on multi-source image registration, the correction method comprising the following steps: Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system; Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image, in order to facilitate the extraction of images at the same location through longitude and latitude; Step 3: Slice and register the two images in the overlapping area according to the custom number and size, and obtain the control points in the longitude and latitude coordinate format; Step 4: Merge all the obtained feature points with the same name to construct a control point set; Step 5: Correct the image to be corrected in the geographic coordinate system based on the geometric information of the control points; Step 6: Perform an inverse transformation of the original projection coordinate system on the resulting image in step 5 to achieve geometric position correction of the drone image.

[0008] Step 3 includes the following steps: Step 31: Calculate the overlapping area of ​​the two images by calculating the coordinates of the four corners of the images, and evenly cut out image slices with a custom number and size with geographic information in the overlapping area; Step 32: Align the corresponding slices and convert the obtained pixel coordinates of the feature points with the same name into longitude and latitude coordinate formats.

[0009] Step 32 includes the following steps: Step 321: feature detection, obtaining a two-dimensional phase model, and detecting Harris corner points on the phase map as feature points; Step 322: feature description, obtaining a feature descriptor for each feature point, and obtaining feature points with the same name between frame images by matching the descriptors.

[0010] In step 321, the real and imaginary parts of the image are calculated in the spatial domain by combining the Log-Gabor filter with the Fourier transform. The real and imaginary parts represent the even symmetric wavelet and the odd symmetric wavelet respectively. The image is convolved with the even symmetric wavelet and the odd symmetric wavelet respectively to obtain the response component. The analysis results of all directions and all orientations are considered, and the noise compensation term is introduced to finally obtain the two-dimensional phase model.

[0011] In step 321, the even-symmetric wavelet is , the odd symmetric wavelet is , the image is , the response component obtained by convolution is and , ,image Amplitude component at scale s and direction o and phase component Can be and Obtained, the noise compensation term is introduced as T, and the obtained two-dimensional phase model is , ,in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a number that prevents the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image. Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

[0012] In step 322, given an image, first, the amplitude is calculated using the response component obtained in step 321; then, the convolution layers are arranged in order to obtain a convolution sequence, which is a multi-channel convolution map. For each pixel position of the convolution map, an ordered array is obtained, and then the maximum value in the array and its corresponding position channel are calculated, and the position channel is set to the pixel value.

[0013] In step 322, the image is , the response component obtained in step 321 is and , the calculated amplitude is the amplitude at scale s and direction o. For direction o, all The amplitudes of the scales are summed to obtain the log-Gabor layer , the resulting multi-channel convolution map is ,in, N s is a specific scale value. is the direction number, superscript ω = 1, 2, ..., and the pixel position is , the ordered array is , through the formula Find the maximum value in an array and its corresponding position channel ,Will The value of is set to the pixel position in the feature description map The pixel value of the feature description map refers to the pixel position Corresponding convolution map, for each feature point, select a local image block with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in the feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without face elements is constructed for each sub-grid. The feature vector is obtained by connecting all histograms, and the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the final normalized feature vector is the feature descriptor of the feature point, where J and n are positive integers.

[0014] In step 4, all feature points with the same name are used as control points, and the control points are merged and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into pixel coordinates (x, y) on the image to be corrected, and finally a control point set in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM). In order to take care of the overall efficiency of the system, it is temporarily set to 0, (base_lat, base_lon) is the latitude and longitude coordinates of the corresponding point on the reference image, and (warp_lat, warp_lon) is the latitude and longitude coordinates of the corresponding point on the image to be corrected.

[0015] The beneficial effects of the present invention are: (1) The technology based on multi-source image registration can realize fast geometric correction between images of different sensor types without considering the differences between different coordinate systems.

[0016] (2) By evenly extracting feature points with the same name on the image as control points, there is no need to process the entire image, which greatly improves the processing efficiency of UAV images.

[0017] (3) It does not restrict the sensor types of UAV images and satellite images, and can better solve the heterogeneous matching problem between the two. It can be used in a variety of application scenarios, and has a small amount of calculation, which can better meet the large-scale image processing needs of UAV platforms.

[0018] (4) By integrating the system and the digital base map on an airborne platform, the geometric position of the drone imaging system can be corrected when the drone starts working. The generated image can be accurately matched with the digital map without any processing, further improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a method and system flow chart of UAV image geometric correction based on multi-source image registration of the present invention; Figure 2 It is a schematic diagram of the descriptor structure and matching retrieval of the registration method of the present invention. DETAILED DESCRIPTION

[0020] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0021] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0022] A UAV image geometric correction system based on multi-source image registration includes the following modules: Module 1: used to convert two images from the projected coordinate system to the same geographic coordinate system; Module 2: used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Module 3: Used to slice, align and obtain control points in latitude and longitude coordinate format in overlapping areas according to custom quantity and size; Module three further includes a slicing module and a registration module; wherein: The slicing module is used to evenly crop tiles with geographic information in the overlapping area of ​​two images according to a custom number and size; The registration module is used to register the corresponding image blocks and convert the obtained point pixel coordinates into longitude and latitude coordinate format.

[0023] Module 4: Used to construct a control point set in the form of [base_lat, base_lon, warp_lat, warp_lon] by using the feature points with the same name obtained by registration as control points; Module 5: Used to correct the geometric information of the image to be corrected in the geographic coordinate system based on control points; Module 6: Used to perform inverse transformation of the result image of module 5 to the original projection coordinate system.

[0024] It should be noted that the protection scope of the present invention does not involve electrical components and other physical components. The "module" refers to the various functional parts of the entire system. The electrical components and other physical components corresponding to each module can adopt components in the prior art and will not be repeated here.

[0025] A method for geometric correction of UAV images based on multi-source image registration based on the above system includes the following steps: Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system.

[0026] Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image, in order to facilitate the extraction of images at the same location through longitude and latitude.

[0027] Step 3: Slice and align the two images in the overlapping area according to the custom number and size, and obtain control points in longitude and latitude coordinate format.

[0028] Furthermore, step 3 includes the following steps: Step 31: Calculate the overlap area of ​​the two images by calculating the coordinates of the four corners of the image, and evenly cut out a custom number and size of image slices with geographic information in the overlap area, and the image slices are used for registration. In this embodiment, 6×6 blocks of 500×500 pixels are extracted from the two images in the overlap area, and their geographic information is retained.

[0029] Step 32: Register the corresponding slices and convert the obtained pixel coordinates of the feature points with the same name into the longitude and latitude coordinate format. The feature points with the same name are the corresponding point pairs on the two images obtained in the registration process.

[0030] The registration algorithm used in this scheme is a radiation-insensitive feature transformation algorithm. This method uses the phase consistency principle of images to eliminate the large radiation intensity differences between multi-source images. It mainly includes two parts: feature detection and feature description. That is, the sub-steps further include: Step 321: Feature detection. The real and imaginary parts of the image are calculated by combining the Log-Gabor filter (LGF) in the spatial domain with the Fourier transform to represent the even symmetric wavelet ( ) and odd symmetric wavelet ( ); Use even symmetric and odd symmetric log-Gabor wavelets to image Convolution is performed to obtain the response component and , Among them, the image Amplitude component at scale s and direction o and phase component Can be and Obtain; Considering the analysis results of all directions and all orientations, and introducing the noise compensation term T, the final two-dimensional phase (PC) model is: .in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a very small number to prevent the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image, and Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

[0031] Step 322: Feature description. Use the corresponding components and amplitudes obtained in step 321 to perform feature description. Specifically, given an image , using the response component obtained in step 321 and , then calculate the amplitude at scale s and direction o ; For direction o, all Scale (here N s is a specific scale value) are summed to obtain the log-Gabor layer ; Then, the log-Gabor convolution layers are arranged in order to obtain the log-Gabor convolution sequence, which is a multi-channel convolution mapping ,in is the number of directions, with superscript ω = 1, 2, .... Therefore, for each pixel position of the convolutional image (a convolutional sequence is a sequence of multiple convolutional images), , we can get an ordered array Then use Find the maximum value in this array and its corresponding position channel .we will The value of is set to the position in the feature description map The feature description map refers to the pixel position The corresponding convolution graph.

[0032] For each feature point, a local image block with J×J pixels centered on the feature is selected, and a Gaussian function with a standard deviation equal to J / 2 is used to assign a weight to each pixel. This process avoids sudden changes in the feature description if the window position changes. Then, the local area is divided into 6×6 subgrids, and a distribution histogram without facets is constructed for each subgrid. The feature vector is obtained by connecting all histograms. Therefore, the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the feature vector is finally normalized to obtain the feature descriptor of the feature point. By matching the descriptors of the feature points, the corresponding feature points with the same name between the frame images are screened. Where J and n are positive integers.

[0033] Figure 2 It is a schematic diagram of the descriptor structure and matching retrieval of the registration method of the present invention. The stacked blocks represent the descriptor of the present invention, which is composed of 6 layers of feature maps stacked up. The degree of each layer indicates that the feature map of the layer is generated by step 322 in the direction of the degree. The circular arrow represents the matching retrieval process, that is, in a ring form, the feature maps generated in different degree directions are placed on the first layer in turn to match the descriptors corresponding to the feature points obtained on the two blocks. The two feature point pairs with the highest descriptor similarity are the feature point pairs with the same name.

[0034] Step 4: Merge all feature points with the same name to construct a control point set.

[0035] In this step, specifically, all feature points with the same name are used as control points, the control points are merged and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into pixel coordinates (x, y) on the image to be corrected, and finally a control point set in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM). In order to take care of the overall efficiency of the system, the present invention is temporarily set to 0. (base_lat, base_lon) are the longitude and latitude coordinates of the corresponding point on the reference image. (warp_lat, warp_lon) are the longitude and latitude coordinates of the corresponding point on the image to be corrected.

[0036] Step 5: Correct the image to be corrected in the geographic coordinate system based on the geometric information of the control points.

[0037] In this step, the correction method is to traverse the (x, y) in the control point file and assign new values ​​(base_lat, base_lon) to the geographic information of the corresponding pixel position of the image, thereby completing the geometric correction of the image.

[0038] Step 6: Perform an inverse transformation of the original projection coordinate system on the result image of step 5, that is, transform the corrected image back to its original projection coordinate system to achieve geometric position correction of the drone image.

[0039] Finally, it is necessary to explain here that the above embodiments are only used to further illustrate the technical solution of the present invention in detail and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the above content of the present invention all fall within the scope of protection of the present invention.

Claims

1. A UAV image geometric correction system based on multi-source image registration, characterized in that: include: Module 1: used to convert two images from the projected coordinate system to the same geographic coordinate system; Module 2: used to resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Module 3: Used to slice, align and obtain control points in latitude and longitude coordinate format in overlapping areas according to custom quantity and size; Module 4: Used to construct a control point set in the form of [base_lat, base_lon, warp_lat, warp_lon] using the feature points of the same name obtained by registration as control points; where base_lat, base_lon, warp_lat, and warp_lon are the longitude coordinates of the corresponding point on the reference image, the latitude coordinates of the corresponding point on the reference image, the longitude coordinates of the corresponding point on the image to be corrected, and the latitude coordinates of the corresponding point on the image to be corrected, respectively; Module 5: Used to correct the geometric information of the image to be corrected in the geographic coordinate system based on control points; Module 6: Used to perform inverse transformation of the result image of module 5 to the original projection coordinate system.

2. The UAV image geometric correction system according to claim 1, characterized in that: Module three includes a slicing module and a registration module. The slicing module is used to evenly crop tiles with geographic information in the overlapping area of ​​two images according to a custom number and size. The registration module is used to register the corresponding tiles and convert the obtained point pixel coordinates into longitude and latitude coordinate format.

3. A method for geometric correction of drone images based on multi-source image registration, the correction method being based on the system of claim 1 or 2, characterized in that: The correction method comprises the following steps: Step 1: Convert the reference image and the image to be corrected to the same geographic coordinate system; Step 2: Resample the image to be corrected in the same geographic coordinate system to the same resolution as the reference image; Step 3: Slice and register the two images in the overlapping area according to the custom number and size, and obtain the control points in the longitude and latitude coordinate format; Step 4: Merge all the obtained feature points with the same name to construct a control point set; Step 5: Correct the image to be corrected in the geographic coordinate system based on the geometric information of the control points; Step 6: Perform an inverse transformation of the original projection coordinate system on the resulting image in step 5 to achieve geometric position correction of the drone image.

4. The method for geometric correction of drone images according to claim 3, characterized in that: Step 3 includes the following steps: Step 31: Calculate the overlapping area of ​​the two images by calculating the coordinates of the four corners of the images, and evenly cut out image slices with a custom number and size with geographic information in the overlapping area; Step 32: Align the corresponding slices and convert the obtained pixel coordinates of the feature points with the same name into longitude and latitude coordinate formats.

5. The method for geometric correction of unmanned aerial vehicle images according to claim 4, characterized in that: Step 32 includes the following steps: Step 321: feature detection, obtaining a two-dimensional phase model, and detecting Harris corner points on the phase map as feature points; Step 322: feature description, obtaining a feature descriptor for each feature point, and obtaining feature points with the same name between frame images by matching the descriptors.

6. The method for geometric correction of unmanned aerial vehicle images according to claim 5, characterized in that: In step 321, the real and imaginary parts of the image are calculated in the spatial domain by combining the Log-Gabor filter with the Fourier transform. The real and imaginary parts represent the even symmetric wavelet and the odd symmetric wavelet respectively. The image is convolved with the even symmetric wavelet and the odd symmetric wavelet respectively to obtain the response component. The analysis results of all directions and all orientations are considered, and the noise compensation term is introduced to finally obtain the two-dimensional phase model.

7. The method for geometric correction of unmanned aerial vehicle images according to claim 6, characterized in that: In step 321, the even-symmetric wavelet is , the odd symmetric wavelet is , the image is , the response component obtained by convolution is and , ,image Amplitude component at scale s and direction o and phase component Can be and Obtained, the noise compensation term is introduced as T, and the obtained two-dimensional phase model is , ,in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a number that prevents the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image. Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

8. The method for geometric correction of unmanned aerial vehicle images according to claim 5, characterized in that: In step 322, given an image, first, the amplitude is calculated using the response component obtained in step 321; then, the convolution layers are arranged in order to obtain a convolution sequence, which is a multi-channel convolution map. For each pixel position of the convolution map, an ordered array is obtained, and then the maximum value in the array and its corresponding position channel are calculated, and the position channel is set to the pixel value.

9. The method for geometric correction of unmanned aerial vehicle images according to claim 8, characterized in that: In step 322, the image is , the response component obtained in step 321 is and , the calculated amplitude is the amplitude at scale s and direction o. For direction o, all The amplitudes of the scales are summed to obtain the log-Gabor layer , the resulting multi-channel convolution map is ,in, N s is a specific scale value. is the direction number, superscript ω = 1, 2, ..., and the pixel position is , the ordered array is , through the formula Find the maximum value in an array and its corresponding position channel ,Will The value of is set to the pixel position in the feature description map The pixel value of the feature description map refers to the pixel position Corresponding convolution map, for each feature point, select a local image block with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in the feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without face elements is constructed for each sub-grid. The feature vector is obtained by connecting all histograms, and the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the final normalized feature vector is the feature descriptor of the feature point, where J and n are positive integers.

10. The method for geometric correction of drone images according to claim 3, characterized in that: In step 4, all feature points with the same name are used as control points, the control points are merged and a control point file is constructed. All the obtained control points are merged, and then based on the geographic information of the image to be corrected, (warp_lat, warp_lon) is converted into pixel coordinates (x, y) on the image to be corrected. Finally, a control point set in the form of [x, y, z, base_lat, base_lon] is constructed, where z can be obtained by introducing a digital elevation model (DEM). In order to take care of the overall efficiency of the system, it is temporarily set to 0. (base_lat, base_lon) are the longitude and latitude coordinates of the corresponding points on the reference image, and (warp_lat, warp_lon) are the longitude and latitude coordinates of the corresponding points on the image to be corrected.

Images