A geometric constraint based adaptive ground-to-air image matching method and system

Abstract

The application provides a kind of based on geometric constraint's self-adapting ground-air image matching method and system, comprising: obtaining original ground-air image, original ground-air image includes: ground street view image and unmanned aerial vehicle aerial image;Original ground-air image is corrected by the image correction strategy based on homography matrix, and corrected ground-air image is obtained;Image self-adapting block is carried out to corrected ground-air image, and sub-block image is obtained;Sub-block image is carried out self-adapting channel separation and feature extraction, and sub-block image feature is obtained;Each sub-block image is obtained by sub-block image feature Exhaustive matching obtains first image matching set, and first image matching set is converted to the coordinate plane of original ground-air image, and second image matching set is obtained;By epipolar geometry condition, second image matching set is screened, and final image matching set is obtained.The application can realize the robust matching of ground street view image and unmanned aerial vehicle aerial image, significantly improve matching quantity and precision.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to an adaptive ground-to-air image matching method and system based on geometric constraints. Background Technology

[0002] Image matching refers to using matching algorithms to process images containing the same scene acquired in different ways or from different perspectives, identifying the consistency and similarity of these images in terms of features, structure, and texture, and establishing pixel-level correspondences accordingly. Image matching is one of the fundamental problems in the field of computer vision and a core component of many computer vision algorithms. This technology initially attracted attention in the military field, where it was experimented with and applied. Later, with continuous technological development, it has spread to various industries, such as 3D reconstruction, object detection, image stitching, and geolocation.

[0003] In recent years, with the development of flight control technology and the decrease in manufacturing costs, acquiring aerial imagery from drones has become increasingly easier. Drone aerial imagery is mostly shot from a bird's-eye view, covering a relatively large geographical and spatial area, with comprehensive coverage of buildings and their rooftops. However, in densely built-up urban areas, aerial imagery suffers from problems such as large tilt angles and occlusion due to flight safety and other factors. Meanwhile, ground-based mobile mapping systems have been widely used to collect street view data in recent years, and the democratization of cameras has made ground street view imagery easier to obtain. Unlike aerial imagery, ground street view imagery has a stronger ability to depict the base of buildings and local details, and its imaging platform is not restricted by flight safety, making it more flexible, but it lacks coverage of rooftop information.

[0004] In summary, UAV aerial imagery and ground street view imagery complement each other to some extent. Therefore, researching robust matching methods for ground-to-air imagery is of great significance as it can better serve many matching-based tasks. However, due to changes in imaging platforms, significant differences in image exterior orientation elements, and environmental factors such as lighting, ground-to-air imagery matching faces numerous challenges. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an adaptive ground-to-space image matching method based on geometric constraints, comprising the following steps: S1: Acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery; S2: Correct the original ground-to-air imagery using a homography-based image correction strategy to obtain the corrected ground-to-air imagery; S3: Perform adaptive image segmentation on the corrected ground-to-air image to obtain sub-block images; perform adaptive channel separation and feature extraction on the sub-block images to obtain sub-block image features; S4: Exhaustive matching of each sub-block image is performed using the features of the sub-block image to obtain the first image matching set. The first image matching set is then transformed into the coordinate plane of the original ground-to-air image to obtain the second image matching set. S5: The final image matching set is obtained by filtering the second image matching set through epipolar geometry conditions.

[0006] Preferably, step S2 specifically includes: S21: Adjust the resolution of the ground street view image and the drone aerial image in the original ground-to-air imagery to the preset size to obtain the ground-to-air imagery with adjusted resolution. S22: Perform feature matching on the resolution-adjusted ground-to-air imagery to obtain an image feature matching set; S23: Perform inverse operation on the image feature matching set onto the original resolution ground-to-air image to obtain the matching point set of the original resolution ground-to-air image; S24: Perform a normalization operation on the set of matching points so that the root mean square distance of all matching points to the origin is equal to... , to obtain the set of normalized matching points; S25: Solve for the homography matrix between the original resolution ground-to-air images based on the normalized matching point set; S26: Evaluate the accuracy of the homography matrix and calculate the accuracy value of the homography matrix; S27: Repeat steps S25-S26 until the maximum number of repetitions is reached, and take the homography matrix with the highest homography matrix precision value as the final homography matrix; S28: Perform homography transformation on ground street view images or UAV aerial images using the final homography matrix to reduce the viewpoint difference between ground street view images and UAV aerial images, and obtain corrected ground-to-air images.

[0007] Preferably, step S3 specifically includes: S31: Resolution of the corrected ground-to-air imagery and crop window size Set the number of blocks and the size of the overlapping area for adaptive image segmentation, and perform adaptive image segmentation on the corrected ground-air image to obtain the segmented sub-block image. S32: Extract features from sub-block images and count the number of features using a feature extraction and description network; If the number of features exceeds the set threshold, channel separation will not be performed, and the sub-block image features will be output directly. Otherwise, the color sub-block image is separated by RGB channels, and features are extracted one by one to obtain color features. The grayscale sub-block image is then processed through grayscale channels to obtain grayscale features. The color features and grayscale features are then fused together to output the sub-block image features.

[0008] Preferred: The formula for calculating the number of blocks is as follows:

[0009] in, , These represent the number of blocks in the width and height directions, respectively. Indicates rounding down; The formula for calculating the size of the overlapping region is as follows:

[0010] in, Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

[0011] Preferably, the expression for the transformation relationship between the first image matching set and the coordinate plane of the original ground-to-space image is:

[0012] in, , These represent the indices of the sub-block image n in the first image matching set in the width and height directions of the original ground-to-air image, respectively, starting from 0; and These represent the pixel coordinates of the pixel in the coordinate plane of the sub-block image and the coordinate plane of the original ground-to-air image, respectively. To set the size of the cropping window; Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

[0013] Preferably, step S5 specifically includes: S51: The final fundamental matrix is ​​calculated using the corrected ground-to-air imagery, epipolar geometry, and image correction strategy based on homography matrix. S52: Constrain the second image matching set using the final fundamental matrix, as shown in the following formula:

[0014] After unfolding,

[0015] Where F represents the final fundamental matrix, to For elements in F, Representing images A pair of matching points homogeneous coordinates This indicates a pre-set constraint threshold; S53: Only retain the images in the second image matching set that meet the constraints to obtain the final image matching set.

[0016] An adaptive ground-to-air image matching system based on geometric constraints includes the following modules: The raw ground-to-air imagery acquisition module is used to acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery. The correction module is used to correct the original ground-to-air imagery using a homography matrix-based image correction strategy to obtain the corrected ground-to-air imagery. The sub-block image feature acquisition module is used to adaptively divide the corrected ground-air image into sub-blocks to obtain sub-block images; and to adaptively separate and extract features from the sub-block images to obtain sub-block image features. The image matching module is used to exhaustively match each sub-block image using sub-block image features to obtain a first image matching set, and then transform the first image matching set into the coordinate plane of the original ground-to-air image to obtain a second image matching set. The filtering module is used to filter the second image matching set by epipolar geometry conditions to obtain the final image matching set.

[0017] The present invention has the following beneficial effects: The adaptive ground-to-air image matching method and system based on geometric constraints of the present invention firstly uses a homography matrix-based image correction strategy, taking the ground-to-air image itself as the reference plane, to complete image correction without the need for additional auxiliary information, thereby reducing the difference in viewpoint between ground and air images. Then, it uses an adaptive channel separation-based feature extraction strategy. When the number of features is large, channel separation is not performed and sub-block image features are directly output. When the number of features is small, sub-block image features are output through channel separation and feature fusion. Effective extraction can be performed for different numbers of features. Finally, a matching and filtering strategy based on geometric constraints is used to further filter the image matching set by using epipolar geometry as a strict geometric constraint condition, effectively eliminating low-precision matching pairs. Thus, robust matching of ground street scene images and UAV aerial images can be achieved, significantly improving the number and accuracy of matching. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2This is a schematic diagram of adaptive block segmentation, channel separation, and feature extraction. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Reference Figure 1 This invention provides an adaptive ground-to-space image matching method based on geometric constraints, comprising the following steps: S1: Acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery; S2: Correct the original ground-to-air imagery using a homography-based image correction strategy to obtain the corrected ground-to-air imagery; S3: Perform adaptive image segmentation on the corrected ground-to-air image to obtain sub-block images; perform adaptive channel separation and feature extraction on the sub-block images to obtain sub-block image features; S4: Exhaustive matching of each sub-block image is performed using the features of the sub-block image to obtain the first image matching set. The first image matching set is then transformed into the coordinate plane of the original ground-to-air image to obtain the second image matching set. S5: The final image matching set is obtained by filtering the second image matching set through epipolar geometry conditions.

[0021] Furthermore, step S2 specifically involves: S21: Adjust the resolution of the ground street view image and the drone aerial image in the original ground-to-air imagery to the preset size to obtain the ground-to-air imagery with adjusted resolution. Specifically, the standard for resolution adjustment is that the learning-based feature extraction and description network SuperPoint and the feature matching and optimization network SuperGlue can successfully process the image without exceeding the size of the video memory; preferably, the larger resolution of the ground street view image and the drone aerial image is adjusted to 1000~1600, and the smaller resolution is adjusted accordingly according to the original image resolution ratio. S22: Perform feature matching on the resolution-adjusted ground-to-air imagery to obtain an image feature matching set; Specifically, feature matching is performed on the resolution-adjusted ground-to-air imagery using the SuperPoint and SuperGlue learning networks. S23: Perform inverse operation on the image feature matching set onto the original resolution ground-to-air image to obtain the matching point set of the original resolution ground-to-air image; Specifically, the set of matching points for the original resolution ground-to-air imagery. The calculation formula is as follows:

[0022] in, These are the dimensions of the original image and the dimensions of the corrected image, respectively. The pixels of the original image Corresponding pixels of the corrected image ; S24: Perform a normalization operation on the set of matching points so that the root mean square distance of all matching points to the origin is equal to... , to obtain the set of normalized matching points; Specifically, a normalization operation is performed on the set of matching points to improve the stability and accuracy of the solution. The calculation formula is as follows:

[0023]

[0024]

[0025] in, The number of matching pairs in the matching point set. Represents the coordinates to be normalized. Represents the normalized coordinates; S25: Solve for the homography matrix between the original resolution ground-to-air images based on the normalized matching point set; Specifically, for any point in three-dimensional space It is in two two-dimensional images The above correspond to and ,therefore, and They are a pair of points with the same name, belonging to... and , With homography matrix The following relationship exists:

[0026] Image To Image The homography matrix between them, after expansion:

[0027] and They are and The image has two-dimensional pixel coordinates, therefore the homography matrix can be used to visualize the image. Mapping any point on the image The two points before and after mapping are corresponding points, which serves as the basis for image correction. After organizing the above relationships, we can obtain The homography matrix A has 8 degrees of freedom, and theoretically only four pairs of matchings are needed to solve it. Therefore, four pairs of matchings are randomly selected from the set of matching points, and the homography matrix can be solved by using the least squares method. S26: Evaluate the accuracy of the homography matrix and calculate the accuracy value of the homography matrix; Specifically, since the correctness of randomly selected matching pairs cannot be guaranteed, it is necessary to evaluate the accuracy of the homography matrix. The accuracy is generally measured using the two-way projection error index, calculated as follows:

[0028] For the entire set of matching points, the cumulative score is calculated. Indicates the precision of the homography matrix. The calculation formula is as follows:

[0029] Where t represents the error threshold; S27: Repeat steps S25-S26 until the maximum number of repetitions is reached, and take the homography matrix with the highest homography matrix precision value as the final homography matrix; S28: Homography transformation is performed on ground street view images or UAV aerial images using the final homography matrix to reduce the viewpoint difference between ground street view images and UAV aerial images, and to obtain corrected ground-to-air images. Specifically, if homography is applied to the ground street view image, the corrected ground street view image and the uncorrected UAV aerial image form a ground-to-air image pair; if homography is applied to the UAV aerial image, the corrected UAV aerial image and the uncorrected ground street view image form a ground-to-air image pair. The two pairs of images are then processed separately, and the results are finally merged to enhance the robustness of the algorithm.

[0030] Further reference Figure 2 Step S3 is as follows: S31: Resolution of the corrected ground-to-air imagery and crop window size Set the number of blocks and the size of the overlapping area for adaptive image segmentation, and perform adaptive image segmentation on the corrected ground-air image to obtain the segmented sub-block image. S32: Extract features from sub-block images and count the number of features using a feature extraction and description network; If the number of features exceeds the set threshold, channel separation will not be performed, and the sub-block image features will be output directly. Otherwise, the color sub-block image is separated by RGB channels, and features are extracted one by one to obtain color features. The grayscale sub-block image is then processed through grayscale channels to obtain grayscale features. The color features and grayscale features are then fused together to output the sub-block image features.

[0031] Further: The formula for calculating the number of blocks is as follows:

[0032] in, , These represent the number of blocks in the width and height directions, respectively. Indicates rounding down; Since the number of blocks is rounded up, if the blocks do not overlap, the boundary will inevitably exceed the limit. In order to prevent uneven overlap between blocks from causing large-area repetition of some sub-images, the following formula is designed to calculate the size of the overlapping area. The formula for calculating the size of the overlapping region is as follows:

[0033] in, Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

[0034] Furthermore, the expression for the transformation relationship between the first image matching set and the coordinate plane of the original ground-to-space image is as follows:

[0035] in, , These represent the indices of the sub-block image n in the first image matching set in the width and height directions of the original ground-to-air image, respectively, starting from 0; and These represent the pixel coordinates of the pixel in the coordinate plane of the sub-block image and the coordinate plane of the original ground-to-air image, respectively. To set the size of the cropping window; Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

[0036] Furthermore, step S5 specifically involves: S51: The final fundamental matrix is ​​calculated using the corrected ground-to-air imagery, epipolar geometry, and image correction strategy based on homography matrix. Specifically: S511: Based on the corrected ground-to-air imagery described in S2, its resolution is adjusted so that the SuperPoint and SuperGlue models can process it successfully without exceeding the size of the video memory. S512: Perform matching on the image pairs with adjusted resolution described in S2 using SuperPoint and SuperGlue; S513: Based on the formula described in S23, perform an inverse operation on the matching result described in S512 onto the original resolution image to obtain the set of matching points on the original resolution image. ; S514: Perform a normalization operation on the set of matching points described in S513 according to the formula described in S24; S515: Based on the normalized matching point set described in S514, solve for the fundamental matrix between the original resolution ground-to-space images; the fundamental matrix is ​​the algebraic representation of epipolar geometry, and the fundamental matrix between two image planes is known. and the pixel coordinates of a 3D point on an image plane. Then the pixel coordinates of the 3D point on another image plane can be obtained. The formula is as follows:

[0037] Organize the above relationships as follows The fundamental matrix can be solved using the least squares method in the form of [formula missing]. S516: Evaluate the accuracy of the basic matrix obtained in S515 according to the formula described in S26; S517: Steps S515 and S516 completed a basic matrix estimation and evaluated the accuracy; in order to improve the quality and avoid randomness, the above steps are repeated iteratively until the maximum number of iterations is reached, and the basic matrix with the highest accuracy is used as the final basic matrix. S52: Constrain the second image matching set using the final fundamental matrix, as shown in the following formula:

[0038] After unfolding,

[0039] Where F represents the final fundamental matrix, to For elements in F, Representing images A pair of matching points homogeneous coordinates This represents a pre-set constraint threshold; preferably, the constraint threshold t is set to 1~1.5; S53: Only retain the images in the second image matching set that meet the constraints to obtain the final image matching set.

[0040] This invention provides an adaptive ground-to-air image matching system based on geometric constraints, comprising the following modules: The raw ground-to-air imagery acquisition module is used to acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery. The correction module is used to correct the original ground-to-air imagery using a homography matrix-based image correction strategy to obtain the corrected ground-to-air imagery. The sub-block image feature acquisition module is used to adaptively divide the corrected ground-air image into sub-blocks to obtain sub-block images; and to adaptively separate and extract features from the sub-block images to obtain sub-block image features. The image matching module is used to exhaustively match each sub-block image using sub-block image features to obtain a first image matching set, and then transform the first image matching set into the coordinate plane of the original ground-to-air image to obtain a second image matching set. The filtering module is used to filter the second image matching set by epipolar geometry conditions to obtain the final image matching set.

[0041] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0042] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0043] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An adaptive ground-to-space image matching method based on geometric constraints, characterized in that, Includes the following steps: S1: Acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery; S2: Correct the original ground-to-air imagery using a homography-based image correction strategy to obtain the corrected ground-to-air imagery; S3: Perform adaptive image segmentation on the corrected ground-to-air image to obtain sub-block images; perform adaptive channel separation and feature extraction on the sub-block images to obtain sub-block image features; Specifically: S31: Resolution of the corrected ground-to-air imagery and crop window size Set the number of blocks and the size of the overlapping area for adaptive image segmentation, and perform adaptive image segmentation on the corrected ground-air image to obtain the segmented sub-block image. S32: Extract features from sub-block images and count the number of features using a feature extraction and description network; If the number of features exceeds the set threshold, channel separation will not be performed, and the sub-block image features will be output directly. Otherwise, the color sub-block image is separated according to the RGB channels, and features are extracted one by one to obtain color features. The grayscale sub-block image is obtained through the grayscale channel to obtain grayscale features. The color features and grayscale features are fused together to output the sub-block image features. S4: Exhaustive matching of each sub-block image is performed using the features of the sub-block image to obtain the first image matching set. The first image matching set is then transformed into the coordinate plane of the original ground-to-air image to obtain the second image matching set. S5: The second image matching set is filtered by the epipolar geometry condition to obtain the final image matching set; Specifically: S51: The final fundamental matrix is ​​calculated using the corrected ground-to-air imagery, epipolar geometry, and image correction strategy based on homography matrix. S52: Constrain the second image matching set using the final fundamental matrix, as shown in the following formula: After unfolding, Where F represents the final fundamental matrix, to For elements in F, Representing image I a I b A pair of matching points The homogeneous coordinates of , where t represents the pre-set constraint threshold; S53: Only retain the images in the second image matching set that meet the constraints to obtain the final image matching set.

2. The adaptive ground-to-space image matching method based on geometric constraints according to claim 1, characterized in that, Step S2 is as follows: S21: Adjust the resolution of the ground street view image and the drone aerial image in the original ground-to-air imagery to the preset size to obtain the ground-to-air imagery with adjusted resolution. S22: Perform feature matching on the resolution-adjusted ground-to-air imagery to obtain an image feature matching set; S23: Perform inverse operation on the image feature matching set onto the original resolution ground-to-air image to obtain the matching point set of the original resolution ground-to-air image; S24: Perform a normalization operation on the set of matching points so that the root mean square distance of all matching points to the origin is equal to... , to obtain the set of normalized matching points; S25: Solve for the homography matrix between the original resolution ground-to-air images based on the normalized matching point set; S26: Evaluate the accuracy of the homography matrix and calculate the accuracy value of the homography matrix; S27: Repeat steps S25-S26 until the maximum number of repetitions is reached, and take the homography matrix with the highest homography matrix precision value as the final homography matrix; S28: Perform homography transformation on ground street view images or UAV aerial images using the final homography matrix to reduce the viewpoint difference between ground street view images and UAV aerial images, and obtain corrected ground-to-air images.

3. The adaptive ground-to-space image matching method based on geometric constraints according to claim 1, characterized in that: In step S31, the formula for calculating the number of blocks is as follows: in, , These represent the number of blocks in the width and height directions, respectively. Indicates rounding down; The formula for calculating the size of the overlapping region is as follows: in, Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

4. The adaptive ground-to-space image matching method based on geometric constraints according to claim 1, characterized in that, The expression for the transformation relationship between the first image matching set and the coordinate plane of the original ground-to-air image is: in, , These represent the indices of the sub-block image n in the first image matching set in the width and height directions of the original ground-to-air image, respectively, starting from 0; and These represent the pixel coordinates of the pixel in the coordinate plane of the sub-block image and the coordinate plane of the original ground-to-air image, respectively. To set the crop window size; Represents the size of the overlapping area in the width direction, in pixels; Represents the size of the overlapping area in the height direction, in pixels.

5. An adaptive ground-to-air image matching system based on geometric constraints, characterized in that, The method for performing the geometrically constrained adaptive ground-to-space image matching method according to any one of claims 1-4 includes the following modules: The raw ground-to-air imagery acquisition module is used to acquire raw ground-to-air imagery, which includes: ground street view imagery and drone aerial imagery. The correction module is used to correct the original ground-to-air imagery using a homography matrix-based image correction strategy to obtain the corrected ground-to-air imagery. The sub-block image feature acquisition module is used to adaptively divide the corrected ground-air image into sub-blocks to obtain sub-block images; and to adaptively separate and extract features from the sub-block images to obtain sub-block image features. The image matching module is used to exhaustively match each sub-block image using sub-block image features to obtain a first image matching set, and then transform the first image matching set into the coordinate plane of the original ground-to-air image to obtain a second image matching set. The filtering module is used to filter the second image matching set by epipolar geometry conditions to obtain the final image matching set.

Images