A method for matching drone aerial images with satellite images

By using the camera posture parameters of the drone aerial image and the latitude and longitude information of the satellite image for direction and scale space registration, combined with deep learning feature extraction and QATM matching algorithm, the mesoscale matching of the drone aerial image and satellite image is solved. The low accuracy problem caused by feature differences is achieved, and efficient image matching is achieved.

CN114549871BActive Publication Date: 2024-11-19HARBIN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210162785.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-22
Publication Date
2024-11-19
Estimated Expiration
2042-02-22

AI Technical Summary

Technical Problem

The prior art is difficult to effectively match drone aerial images and satellite images, mainly due to the huge differences in scale and characteristics, resulting in low matching accuracy.

Method used

By obtaining the camera posture parameters of the aerial image of the drone and the latitude and longitude information of the satellite image, the direction and scale space are registered, the corresponding area is intercepted as sub-images, and the feature response score is calculated using deep learning feature extraction and QATM matching algorithm to achieve Image matching.

Benefits of technology

The matching accuracy of drone aerial images and satellite images is improved, the calculation amount and mismatch probability are reduced, and the matching efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549871B_ABST
    Figure CN114549871B_ABST
Patent Text Reader

Abstract

A method for matching drone aerial images and satellite images. The invention relates to a method for matching drone aerial images and satellite images. The purpose of the invention is to solve the problem that the existing method matches drone aerial images with satellite images, and there are great differences in scale and features between images, resulting in low matching accuracy. The process is: 1: directional registration of drone aerial images based on satellite images; 2: obtaining drone aerial images after scale space registration; 3: obtaining sub-images of satellite images; 4: obtaining a trained feature extraction network; 5: extracting feature vectors of aerial images and feature vectors of sub-images of satellite images; calculating feature response scores of feature vectors of aerial images and feature vectors of sub-images of satellite images, and selecting the area with the highest score as the best matching result, thereby achieving matching of drone aerial images with satellite images. The invention is used in the field of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing and is a method for matching drone aerial images with satellite images by utilizing drone aerial camera posture parameters, satellite image latitude and longitude information, deep learning feature extraction, and digital image processing technologies. Background Art

[0002] Image matching is a basic task in computer vision and image processing. Image matching technology can match the same spatial position points in two or more images on each image pixel to achieve registration in the image coordinate space. Therefore, image matching technology is usually used in image stitching to achieve the registration of the coordinate positions of adjacent images to be stitched. In addition, image matching technology is also used in target tracking, computer vision positioning and other fields. Due to the wide application of image matching technology in various fields, researchers have proposed corresponding matching technologies for different matching scene requirements. Existing image matching technologies can be roughly divided into two categories, namely region-based image matching technology and feature-based image matching technology. Among them, region-based image matching technology aims to match images through the pixel intensity of the entire image. For example, the template matching method calculates the pixel grayscale value between the two images by sliding the window, and then selects the area with the smallest sum of squared grayscale difference error as the image matching area. This method can achieve image matching simply and quickly, but it also has obvious defects. When there is rotation and scale change between the matching images, the region matching method obviously cannot match correctly. Feature-based image matching technology extracts image features and then achieves image matching based on the matching relationship of features. The features used for image matching can be global features, such as HOG features, or local features, such as SIFT features. Different matching results can be obtained according to the properties of the features selected for matching. For example, by selecting features with scale invariance and direction invariance to match images, better matching results can be achieved in images to be matched with scale and rotation changes.

[0003] Matching UAV aerial images with satellite images is a relatively new research field. At present, there are relatively few studies in this direction at home and abroad. There are several difficulties in matching UAV aerial images with satellite images. The first is that the scale difference between the two images is huge. Satellite images are large-scale, wide-view images, generally collected by satellites, supplemented by later manual field surveys, mapping, and annotation. The actual ground area covered by a single pixel is relatively large. Relatively speaking, UAV aerial images are small-scale, narrow-view images. Due to the flight altitude limit of the UAV, the actual ground area covered by its aerial images is limited, and the area covered by its single pixel is small. Due to the huge difference between the two images, it is impossible to obtain accurate matching results directly using conventional matching methods. Secondly, the feature difference between the two images is large. Due to the difference in imaging mechanism and image scale, satellite images and UAV aerial images have great differences in color, brightness, contrast, and image clarity. Therefore, the performance of the feature points extracted from the two at the same position is quite different, and it is difficult to match images through these features. Traditional matching methods such as template matching and correlation matching mostly require that the matching image and the reference image be at the same scale to achieve a good match. Therefore, ideal matching results cannot be achieved in image matching scenarios with large scale differences such as drone aerial images and satellite images. Then there is a large difference in the pixel size between drone aerial images and satellite images. Since satellite images have a wide coverage area, the coverage area of ​​drone aerial images is relatively small compared to satellite images. Therefore, the image scale space registration method is used to adjust the two to the same scale. The pixel size of the satellite image will be much larger than that of the drone aerial image, that is, the actual matching area of ​​the drone aerial image is only a part of the sub-image in the satellite image. If the aerial image is matched directly on the satellite image, the computational complexity of feature extraction and feature calculation is very large, and the matching efficiency is low. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing methods of matching drone aerial images with satellite images have great differences in scale and features between images, resulting in low matching accuracy, and propose a method for matching drone aerial images with satellite images.

[0005] A method for matching drone aerial images with satellite images. The specific process is as follows:

[0006] Step 1: Obtain satellite images and drone aerial images, and perform orientation registration of drone aerial images based on satellite images;

[0007] Step 2: Perform scale space registration on the direction-registered UAV aerial image and the satellite image to obtain the scale space-registered UAV aerial image;

[0008] Step 3: Based on the longitude, latitude, and altitude position information of the imaging point recorded during the drone aerial imagery process, determine the area where the drone aerial image is located, determine the area where the aerial image may appear in the satellite image based on the area where the drone aerial image is located and the drone positioning accuracy, and then intercept the area where the aerial image may appear in the satellite image as a sub-image of the satellite image;

[0009] Step 4: Get the trained feature extraction network;

[0010] Step 5: Input the scale-space-registered UAV aerial image obtained in step 2 and the sub-image of the satellite image obtained in step 3 into the feature extraction network trained in step 4 to extract the feature vector of the aerial image and the feature vector of the sub-image of the satellite image;

[0011] The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated, and the area with the highest score is selected as the best matching result, thereby achieving the matching of drone aerial images and satellite images.

[0012] The beneficial effects of the present invention are:

[0013] The purpose of the present invention is to achieve the matching of drone aerial images and satellite images through the camera attitude parameters of drone aerial images and the position parameters of satellite images. Due to the huge gap in scale between aerial images and satellite images, the existing template matching and feature point matching methods are difficult to adapt, so it is necessary to achieve registration of aerial images and satellite images in scale space. Since the aerial image records the pitch angle, azimuth and position of the camera during the image formation during the shooting process, and the satellite image also contains longitude and latitude information, the size of the ground area actually covered by the image and the direction angle between the images can be calculated based on this information, and then the image is rotated, and combined with the pixel size of the image, the image is resampled so that the aerial image and the satellite image are consistent in direction and the ground coverage size represented by the unit pixel, thereby achieving the registration of the aerial image and the satellite image in scale and direction. The registered aerial image and satellite image can be matched by template matching. However, due to the difference in brightness between the two and the fact that the topographic features on the aerial image will change compared to the satellite image, it is obviously inappropriate to use pixel error and other metrics to measure the similarity to achieve matching. Therefore, the present invention extracts the features of the image through deep learning, and then uses feature matching technology to calculate the matching feature response map. Finally, the sliding window matching method is used to find the matching area with the largest score in the feature response map as the best matching result to achieve the matching of the UAV aerial image and the satellite image. The problem that the existing method matches the UAV aerial image with the satellite image, and there are great differences in scale and features between the images, resulting in low matching accuracy, is solved. The present invention improves the matching accuracy of the UAV aerial image and the satellite image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of the method for matching unmanned aerial images and satellite images of the present invention;

[0015] Figure 2 Schematic diagram of satellite image sub-image interception of the present invention, A (lonA, latA) is the latitude and longitude coordinates corresponding to the upper left corner of the satellite image; B (lonB, latB) is the latitude and longitude coordinates corresponding to the lower right corner of the satellite image; (lonT, latT) is the center position of the dotted box in the figure, indicating the latitude and longitude coordinates recorded by the drone aerial image;

[0016] FIG3( a ) is a schematic diagram of a pixel coordinate system of the present invention;

[0017] FIG3( b ) is a schematic diagram of the central coordinate system of the present invention;

[0018] Figure 4 It is the QATM algorithm matching flow chart of the present invention;

[0019] Figure 5This is a schematic diagram of dividing the template image into fragments according to the present invention;

[0020] Figure 6 This is a schematic diagram of an aerial image taken by a UAV of the present invention;

[0021] Figure 7 It is a schematic diagram of satellite image of the present invention;

[0022] Figure 8 A schematic diagram of the image registration of the present invention;

[0023] Fig. 9 This is a schematic diagram of the matching results of the drone aerial image and the satellite image of the present invention. DETAILED DESCRIPTION

[0024] Specific implementation method 1: The specific process of the method for matching drone aerial images with satellite images in this implementation method is as follows:

[0025] The present invention mainly studies the implementation method of matching drone aerial images with satellite images. Since drone aerial images are usually high-precision, small-scale images with a small coverage area taken at low flight altitudes, while satellite images are large-scale images with a wide coverage area formed by satellite scanning, existing image matching technologies cannot adapt to the matching between images with such large scale differences. Therefore, based on the characteristics between drone aerial images and satellite images, the present invention proposes a method of realizing image registration using camera imaging posture parameters, and then matching the registered images using the QATM (Quality-Aware Template Matching For Deep Learning) matching algorithm, thereby realizing a method of matching drone aerial images with satellite images. The specific matching implementation process is as follows: Figure 1 shown.

[0026] Depend on Figure 1 It can be seen that the matching process of the drone aerial image and the satellite image proposed in the present invention is mainly divided into two modules: image registration and image matching.

[0027] In the image registration module, the azimuth angle of the airborne camera at the time of imaging is obtained to determine the directional offset of the drone aerial image from the satellite image, and then the image is rotated in the opposite direction of the azimuth angle to achieve the rotational direction correction registration of the image. The image after the registration direction is then used to calculate the actual ground coverage of the aerial image based on the focal length and imaging height of the airborne camera, and then the pixel scale is adjusted to the size of the satellite image pixel scale through image sampling, thereby achieving the registration of the scale space of the two types of images. In image matching, since the ground coverage area represented by the satellite image is usually relatively wide, while the aerial image coverage area is relatively small, if the aerial image is directly matched with the entire satellite image, the computational complexity of feature extraction and feature matching is very large, and this is also unnecessary matching. Because the latitude, longitude and altitude position information of the imaging point has been recorded during the drone aerial image process, this information can be used to roughly determine the area where the aerial image is located. Therefore, in the matching stage, the area where the aerial image may appear in the satellite image can be determined based on the location information of the aerial image and its positioning accuracy, and then the area is selected as a sub-image of a satellite image to match the aerial image, as shown in the specific diagram. Figure 2 As shown in the figure, the dotted box is the size of the satellite sub-image intercepted from the satellite image according to the latitude and longitude information of the aerial image. This method can effectively reduce the amount of matching calculations between irrelevant satellite image areas and aerial images, and improve the matching rate. In addition, since the irrelevant matching areas are removed, the interference of other areas with similar texture features to the real matching areas on the matching results is avoided, effectively reducing the probability of false matching.

[0028] In the image matching module, the input images are the aerial images that have been registered in the image direction and scale space and the satellite image sub-images that are intercepted from the satellite image according to the aerial image location information. The aerial images and satellite images processed in the image registration stage are respectively extracted through the feature extraction network to extract the feature vectors of the images. Since the focus of image matching is the feature similarity between images, it does not need to extract high-level abstract features of image objects such as image target recognition. Therefore, the present invention uses the ImageNet [1] ([1]https: / / image-net.org) Pre-trained VGG19 on the dataset [2]([2]Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science, 2014.) The network is used to extract the features of the images to be matched and obtain the feature vector matrix. The feature response score is then calculated using the QATM matching algorithm, and the area with the highest score is selected as the best matching result, thereby achieving the matching of drone aerial images and satellite images.

[0029] Step 1: Obtain satellite images and drone aerial images, and perform orientation registration of drone aerial images based on satellite images;

[0030] Step 2: Perform scale space registration on the direction-registered UAV aerial image and the satellite image to obtain the scale space-registered UAV aerial image;

[0031] Step 3: Based on the longitude, latitude, altitude and other location information of the imaging point recorded during the drone aerial imagery process, the area where the drone aerial image is located is roughly determined. Therefore, in the matching stage, the area where the drone aerial image is likely to appear in the satellite image can be determined based on the area where the drone aerial image is located and the drone positioning accuracy, and then the area where the aerial image is likely to appear in the satellite image can be intercepted as a sub-image of the satellite image;

[0032] The satellite image sub-image is matched with the aerial image, as shown in the following figure. Figure 2 As shown in the figure, the dotted box is the size of the satellite sub-image intercepted from the satellite image according to the latitude and longitude information of the aerial image. This method can effectively reduce the amount of matching calculations between irrelevant satellite image areas and aerial images, and improve the matching rate. In addition, since the irrelevant matching areas are removed, the interference of other areas with similar texture features to the real matching areas on the matching results is avoided, effectively reducing the probability of false matching.

[0033] Step 4: Get the trained feature extraction network;

[0034] Step 5: Input the scale-space-registered UAV aerial image obtained in step 2 and the sub-image of the satellite image obtained in step 3 into the feature extraction network trained in step 4 to extract the feature vector of the aerial image and the feature vector of the sub-image of the satellite image;

[0035] The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated through the QATM matching algorithm, and the area with the highest score (the area of ​​the sub-image of the satellite image) is selected as the best matching result, thereby realizing the matching of drone aerial images and satellite images.

[0036] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that in step 1, satellite images and drone aerial images are obtained, and the direction of the drone aerial images is aligned based on the satellite images; the specific process is:

[0037] The pixel coordinate system HOW is established with the upper left corner of the image as the origin O, H is the vertical coordinate of the pixel coordinate system, which is the image height; W is the horizontal coordinate of the pixel coordinate system, which is the image width;

[0038] The coordinates (X′0, Y′0) of a point (X0, Y0) in the image pixel coordinate system after rotating θ degrees clockwise around the center of the image are calculated by the following formula:

[0039]

[0040] The theoretical basis of formula (1) is as follows:

[0041] In the present invention, the orientation registration of the image is achieved by rotating the image. The essence of image rotation is the rotation change of the pixel coordinate position around the center of the image. This change process can be achieved by changing the matrix. However, since the pixel coordinates of the image usually take the upper left corner of the image as the origin, the image rotation coordinate system transformation needs to be achieved before the image transformation. The specific change relationship is shown in Figure 3 (a) and Figure 3 (b).

[0042] The pixel coordinate system HOW is established with the upper left corner of the image as the origin O, H is the vertical coordinate of the pixel coordinate system, which is the image height; W is the horizontal coordinate of the pixel coordinate system, which is the image width;

[0043] The center coordinate system XOY is established with the center of the image as the origin O, where X is the horizontal coordinate of the center coordinate system and Y is the vertical coordinate of the center coordinate system;

[0044] according to Figure 3(a) , 3(b) The relationship between the image pixel coordinate system and the center coordinate system shown in the figure can be obtained. When the image height is assumed to be H and the width is W, the coordinates of a point (X0, Y0) in the pixel coordinate system HOW are transformed to (X1, Y1) in the image center coordinate system XOY. The specific transformation formula is shown in the following formula:

[0045]

[0046] Since the display of the image is based on the pixel coordinate system, after the image is rotated in the image center coordinate system, the pixel coordinates in the center coordinate system need to be inversely transformed to the pixel coordinate system. According to the matrix calculation relationship, the inverse transformation matrix is ​​the inverse matrix of the forward transformation matrix, that is, the transformation formula from the image center coordinate system to the pixel coordinate system is as follows:

[0047]

[0048] The rotation of the image is realized in the image center coordinate system, as shown in Figure 3(b). Assume that the distance from the point (x0, y0) to the origin in the image center coordinate system is R, and the angle with the X-axis is α. The coordinates of the point (x0, y0) after rotating θ degrees clockwise are (x, y). Therefore, the coordinate relationship can be written as follows based on the geometric relationship:

[0049]

[0050]

[0051] There is no limit on the size relationship of θ and α, and there is no limit on the value. A negative value indicates rotation in the opposite direction. Figure 3(b) is a schematic diagram.

[0052] Combining equation (4) and equation (5), combined with the trigonometric function addition theorem, and written in the form of a matrix, the coordinate calculation formula after the coordinate (x0, y0) is rotated by θ degrees is as follows:

[0053]

[0054] The other steps and parameters are the same as those in the first embodiment.

[0055] Specific implementation method three: This implementation method is different from specific implementation method one or two in that in step 2, the scale space registration of the drone aerial image after direction registration and the satellite image is performed to obtain the drone aerial image after scale space registration; the specific process is:

[0056] In the present invention, the scale space registration of the UAV aerial image and the satellite image is achieved by calculating the scale relationship between the aerial image and the satellite image, and then resampling the aerial image and adjusting the image pixel size according to the scale relationship.

[0057] Assume that the ground coverage area of ​​the currently selected satellite image is L aX , width is L aY , the satellite image size is H a ×W a , H a is the number of pixels in the satellite image row, W a The number of pixels in the satellite image column;

[0058] Accordingly, the size of the UAV aerial image is h×w, where h is the number of pixels in the row of the UAV aerial image and w is the number of pixels in the column of the UAV aerial image;

[0059] Therefore, for satellite images, we can calculate the actual ground area occupied by a single pixel point in the X and Y directions of the central coordinate system of the satellite image. The specific calculation method is shown in the following formula:

[0060]

[0061] Similarly, according to the drone aerial image, the ground coverage area is L X and width L Y And the size of the image h×w can be used to obtain the unit pixel size of the aerial image in the X and Y directions of the central coordinate system, respectively du and dv;

[0062]

[0063]

[0064] Here, the scale relationship between the aerial image and the satellite image, that is, the relationship between the actual area size represented by a single pixel point between the aerial image and the satellite image, can be calculated according to the following formulas to calculate the scale size correspondence in the X and Y directions of the central coordinate system;

[0065]

[0066] Among them, σ X Indicates the scale correspondence between the aerial image and the satellite image in the X direction of the central coordinate system, σ Y Indicates the scale correspondence between the aerial image and the satellite image in the Y direction of the central coordinate system;

[0067] At this point, the image scale relationship calculation process between the drone aerial image and the satellite image is completed. Finally, the scale of the drone aerial image can be adjusted by image upsampling to achieve scale registration between the aerial image and the satellite image;

[0068] Based on formula (8), the UAV aerial image is upsampled to obtain the adjusted UAV aerial image, and the scale registration between the aerial image and the satellite image is realized to obtain the UAV aerial image after scale space registration.

[0069] The other steps and parameters are the same as those in the first or second embodiment.

[0070] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that the size of the adjusted drone aerial image obtained by upsampling the drone aerial image based on formula (8) is h'×w', which is calculated according to the following formula:

[0071]

[0072] in, Indicates rounding down the data, h is the number of pixels in the row of the drone aerial image, w is the number of pixels in the column of the drone aerial image, w' is the number of pixels in the column of the adjusted drone aerial image, and h' is the number of pixels in the row of the adjusted drone aerial image.

[0073] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0074] Specific implementation method 5: This implementation method is different from any one of the specific implementation methods 1 to 4 in that the trained feature extraction network is obtained in step 4. The specific process is as follows:

[0075] Use the ImageNet dataset to train the feature extraction network to obtain a trained feature extraction network;

[0076] The feature extraction network is a VGG19 network.

[0077] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0078] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that in step 5, the scale-space-registered drone aerial image obtained in step 2 and the sub-image of the satellite image obtained in step 3 are input into the feature extraction network trained in step 4 to extract the feature vector of the aerial image and the feature vector of the sub-image of the satellite image;

[0079] The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated by the QATM matching algorithm, and the area with the highest score (the area of ​​the sub-image of the satellite image) is selected as the best matching result, thereby achieving the matching of the drone aerial image and the satellite image; the specific process is as follows:

[0080] In the image matching module, the input images are the aerial images that have been registered in the image direction and scale space and the satellite image sub-images that are intercepted from the satellite image according to the aerial image location information. The aerial images and satellite images processed in the image registration stage are respectively extracted through the feature extraction network to extract the feature vectors of the images. Since the focus of image matching is the feature similarity between images, it does not need to extract high-level abstract features of image objects such as image target recognition. Therefore, the present invention uses the ImageNet [1] VGG19 pre-trained on the dataset [2] The network is used to extract the features of the images to be matched to obtain the feature vector matrix, and then the feature response score is calculated through the QATM matching algorithm, and the area with the highest score is selected as the best matching result, thereby realizing the matching of UAV aerial images and satellite images.

[0081] In the present invention, in order to achieve accurate matching between the registered drone aerial image and the satellite image, the present invention uses the VGG19 network to extract the feature vector of the image to be matched, and then matches the image using the QATM matching algorithm. The specific matching process is as follows: Figure 4 As shown in the figure, after the cosine operation is performed on the input feature vector matrix, the feature matching response image is calculated according to the quality perception function calculation method defined by QATM, and then the response score is calculated in the feature response image with the template image size as the sliding window, and the area with the highest score is taken as the final matching result.

[0082] The drone aerial image obtained after scale space registration in step 2 is used as the template image T, and the sub-image of the satellite image obtained in step 3 is defined as the search image S;

[0083] The goal of matching is to find the best matching region of the template image in the search image;

[0084] Divide the scale-space-registered UAV aerial image obtained in step 2 into several square fragments, with some overlap between the fragments;

[0085] Divide the satellite image sub-image obtained in step 3 into a number of square fragments, with a certain overlap between the fragments;

[0086] Input the fragmented drone aerial images and satellite image sub-images into the trained feature extraction network to extract feature vectors of the aerial images and feature vectors of the satellite image sub-images;

[0087] The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated through the QATM matching algorithm, and the area with the highest score (the area of ​​the sub-image of the satellite image) is selected as the best matching result, thereby realizing the matching of drone aerial images and satellite images.

[0088] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0089] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the drone aerial image obtained in step 2 after scale space registration is divided into a plurality of square fragments, and there is a certain overlap between the fragments; the sub-image of the satellite image obtained in step 3 is divided into a plurality of square fragments, and there is a certain overlap between the fragments; the specific process is:

[0090] The block fragments are 4×4 pixels in size, with an offset of 2 pixels between fragments.

[0091] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0092] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the dimension of the feature vector of the extracted aerial image is 320;

[0093] The dimension size of the feature vector of the extracted satellite image sub-image is 320.

[0094] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0095] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated by the QATM matching algorithm, and the area with the highest score (the area of ​​the sub-image of the satellite image) is selected as the best matching result, thereby achieving the matching of the drone aerial image and the satellite image; the specific process is:

[0096] The image matching quality perception function of the QATM algorithm is defined as Q(s, t), where t and s are fragments in the template image T and the search image S, respectively, and the feature vectors corresponding to t and s are expressed as f t and f s ;

[0097] Taking the template image T as an example, the image fragment division is shown as follows: Figure 5 As shown, the input image is divided into several square fragments, and there is a certain overlap between the fragments. In the present invention, the feature vector is extracted by the VGG19 network, the dimension size is 320, the square fragment size is 4×4 pixels, and the offset between the fragments is 2 pixels.

[0098] Given a fragment s in a search image S, the probability that a fragment t in a template image T matches it is defined, that is, the matching similarity score between a fragment s in the search image S and a fragment t in the template image T is P(t|s), and the specific calculation method is shown in the following formula:

[0099]

[0100]

[0101] Where β is a positive weight, ρ(·) represents the cosine operation, and the specific formula is shown in formula (11); t′ is a fragment in the template image T, and the feature vector corresponding to t′ is expressed as f t' ;

[0102] In terms of matching quality, this probability can be interpreted as a ranking of how well the current fragment t matches the fragment s in the search image S relative to all other fragments in the template image T.

[0103] The value of P(t|s) is in the range of [0, 1]. The larger the value of P(t|s), the higher the similarity between the matching of fragments t and s.

[0104] According to the definition, the essence of probability P(t|s) is the matching similarity score between a fragment s of the search image S and a fragment t of the template image T. Similarly, the matching similarity score between a fragment t of the template image T and a fragment s of the search image S can be defined as P(s|t). Therefore, the image matching quality perception function Q(s,t) can be obtained according to formula (12);

[0105] Q(s,t)=P(t|s)×P(s|t) (12)

[0106]

[0107]

[0108] Based on this, according to the principle of overall matching quality optimization, the matching quality response maximum area in the matching quality perception function Q(s, t) can be found as the best matching area. The specific calculation method is shown in the following formula:

[0109]

[0110] Among them, Φ is the best matching area of ​​the template image T found in the search image S.

[0111] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0112] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 9 in that the image matching quality perception function Q(s, t) is divided into five matching situations:

[0113] (1) 1-1 matching: For a fragment t of the template image T, there is a unique fragment s in the search image S that matches the fragment t, and the fragment t is also the only fragment in the template image T that matches the fragment s. In this case, P(s|t)=P(t|s)=1, and the matching quality perception function Q(s,t)=1;

[0114] (2) 1-N matching: For a fragment t of the template image T, there are N fragments in the search image S that match the fragment t. In this case, P(s|t) = 1 / N, P(t|s) = 1, and the matching quality perception function Q(s, t) = 1 / N;

[0115] (3) M-1 matching: For a fragment s of the search image S, there are M fragments in the template image T that match the fragment s. In this case, P(s|t) = 1, P(t|s) = 1 / M, and the matching quality perception function Q(s, t) = 1 / M;

[0116] (4) MN matching: There are N fragments in the search image S that match M fragments in the template image T. In this case, P(s|t) = 1 / N, P(t|s) = 1 / M, and the matching quality perception function Q(s, t) = 1 / NM;

[0117] (5) No match: There are no matching fragments between the template image T and the search image S, and the matching quality perception function Q(s,t) = 0.

[0118] From the above five matching situations, it can be seen that matching situation (1) corresponds to the ideal situation of high matching, in which the image matching effect is the best, and matching situation (5) is the worst matching result.

[0119] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0120] In order to better illustrate this patent, the variables involved in the present invention are summarized as shown in Table 1.

[0121] Table 1 Variable description

[0122]

[0123]

[0124] The following examples are used to verify the beneficial effects of the present invention:

[0125] Embodiment 1:

[0126] In order to verify the feasibility of the method proposed in the present invention, it is necessary to select an experimental scene for drone aerial photography, obtain satellite images of the scene, and match and analyze the aerial images with the satellite images. The venue selected by the present invention is the track and field stadium of Harbin Institute of Technology, and the drone aerial photography height is 100m. Figure 6 The satellite images used in the experiment were downloaded from the Big Map Satellite Image Viewer software. The downloaded satellite images are 18-level images with a scale of 1:1575. Figure 7 The figure shows the actual downloaded satellite image of the Harbin Institute of Technology track and field stadium. The longitude range coordinates of the downloaded image ground coverage area are 126.624550°E-126.632227°E, and the latitude coordinate range is 45.734169°N-45.738873°N.

[0127] The azimuth angle of the airborne camera at the time of aerial image imaging selected in the verification analysis of the present invention is 170.2 degrees. The ground coverage distance of the aerial image calculated according to the flight altitude and the imaging focal length of the airborne camera is 115.10m long and 64.54m wide. Based on this, the aerial image registration result realized by the image registration module is as follows: Figure 8 As shown. Figure 8 It can be seen that the image after registration has achieved the same Figure 7 The direction of the satellite image is consistent, and the image direction is aligned. At the same time, by resampling the image, the image scale space alignment is achieved. Finally, the aligned aerial image is input into the image matching module to achieve the matching result between the drone aerial image and the satellite image. Fig. 9 In order to quantitatively analyze the image matching quality, in the experimental result analysis, the present invention manually selects the feature matching point pairs of the aerial image and the satellite image, and then maps the aerial image to the satellite image according to the matching point pairs as the true value matching position of the image matching, as shown in FIG. Fig. 9 The dotted frame shown is the true value matching position of the experimental aerial image in the satellite image, and the solid line frame shows the matching result of the matching method proposed in the present invention. It can be seen from the matching result that the solid line frame and the dotted line true value frame basically overlap, and the quantitative calculation of the overlap between the two is 82.83%, reaching a relatively high overlap rate, which proves that the UAV aerial image and satellite image matching method proposed in the present invention is feasible.

[0128] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for matching drone aerial images with satellite images, characterized in that: The specific process of the method is: Step 1: Obtain satellite images and drone aerial images, and perform orientation registration of drone aerial images based on satellite images; Step 2: Perform scale space registration on the direction-registered UAV aerial image and the satellite image to obtain the scale space-registered UAV aerial image; Step 3: Based on the longitude, latitude, and altitude position information of the imaging point recorded during the drone aerial imagery process, determine the area where the drone aerial image is located, determine the area where the aerial image may appear in the satellite image based on the area where the drone aerial image is located and the drone positioning accuracy, and then intercept the area where the aerial image may appear in the satellite image as a sub-image of the satellite image; Step 4: Get the trained feature extraction network; Step 5: Input the scale-space-registered UAV aerial image obtained in step 2 and the sub-image of the satellite image obtained in step 3 into the feature extraction network trained in step 4 to extract the feature vector of the aerial image and the feature vector of the sub-image of the satellite image; The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated, and the area with the highest score is selected as the best matching result, thereby achieving the matching of drone aerial images and satellite images.

2. According to claim 1, a method for matching unmanned aerial images with satellite images is characterized by: In step 1, satellite images and drone aerial images are obtained, and the direction of the drone aerial images is registered based on the satellite images. The specific process is as follows: The pixel coordinate system HOW is established with the upper left corner of the image as the origin O, H is the vertical coordinate of the pixel coordinate system, which is the image height; W is the horizontal coordinate of the pixel coordinate system, which is the image width; The coordinates (X′0, Y′0) of a point (X0, Y0) in the image pixel coordinate system after rotating θ degrees clockwise around the center of the image are calculated by the following formula:

3. According to claim 2, a method for matching drone aerial images with satellite images is characterized by: In step 2, the scale space registration of the direction-registered UAV aerial image and the satellite image is performed to obtain the scale space-registered UAV aerial image; the specific process is: Assume that the ground coverage area of ​​the currently selected satellite image is L aX , width is L aY , the satellite image size is H a ×W a , H a is the number of pixels in the satellite image row, W a The number of pixels in the satellite image column; Accordingly, the size of the UAV aerial image is h×w, where h is the number of pixels in the row of the UAV aerial image and w is the number of pixels in the column of the UAV aerial image; Therefore, for satellite images, we can calculate the actual ground area occupied by a single pixel point in the X and Y directions of the central coordinate system of the satellite image. The specific calculation method is shown in the following formula: Similarly, according to the drone aerial image, the ground coverage area is L X and width L Y And the size of the image h×w can be used to obtain the unit pixel size of the aerial image in the X and Y directions of the central coordinate system, respectively du and dv; Here, the scale relationship between the aerial image and the satellite image, that is, the relationship between the actual area size represented by a single pixel point between the aerial image and the satellite image, can be calculated according to the following formulas to calculate the scale size correspondence in the X and Y directions of the central coordinate system; Among them, σ X Indicates the scale correspondence between the aerial image and the satellite image in the X direction of the central coordinate system, σ Y Indicates the scale correspondence between the aerial image and the satellite image in the Y direction of the central coordinate system; Based on formula (8), the UAV aerial image is upsampled to obtain the adjusted UAV aerial image, and the scale registration between the aerial image and the satellite image is realized to obtain the UAV aerial image after scale space registration.

4. According to claim 3, a method for matching unmanned aerial images with satellite images is characterized by: The size of the adjusted drone aerial image obtained by upsampling the drone aerial image based on formula (8) is h'×w', which is calculated according to the following formula: in, Indicates rounding down the data, h is the number of pixels in the row of the drone aerial image, w is the number of pixels in the column of the drone aerial image, w' is the number of pixels in the column of the adjusted drone aerial image, and h' is the number of pixels in the row of the adjusted drone aerial image.

5. The method for matching unmanned aerial images and satellite images according to claim 4, characterized in that: In step 4, the trained feature extraction network is obtained, and the specific process is as follows: Use the ImageNet dataset to train the feature extraction network to obtain a trained feature extraction network; The feature extraction network is a VGG19 network.

6. The method for matching drone aerial images and satellite images according to claim 5, characterized in that: In step 5 Input the scale-space-registered UAV aerial image obtained in step 2 and the sub-image of the satellite image obtained in step 3 into the feature extraction network trained in step 4 to extract the feature vector of the aerial image and the feature vector of the sub-image of the satellite image; Calculate the feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image, and select the area with the highest score as the best matching result, so as to achieve the matching of the drone aerial image and the satellite image; the specific process is: The drone aerial image obtained after scale space registration in step 2 is used as the template image T, and the sub-image of the satellite image obtained in step 3 is defined as the search image S; The goal of matching is to find the best matching region of the template image in the search image; Divide the scale-space-registered UAV aerial image obtained in step 2 into several square fragments; Divide the satellite image sub-image obtained in step 3 into a number of square fragments; Input the fragmented drone aerial images and satellite image sub-images into the trained feature extraction network to extract feature vectors of the aerial images and feature vectors of the satellite image sub-images; The feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image are calculated through the QATM matching algorithm, and the area with the highest score is selected as the best matching result, thereby realizing the matching of UAV aerial images and satellite images.

7. The method for matching unmanned aerial images and satellite images according to claim 6, characterized in that: The scale-space-registered drone aerial image obtained in step 2 is divided into a plurality of square fragments; the satellite image sub-image obtained in step 3 is divided into a plurality of square fragments; the specific process is: The block fragments are 4×4 pixels in size, with an offset of 2 pixels between fragments.

8. The method for matching unmanned aerial images and satellite images according to claim 7, characterized in that: The dimension size of the feature vector of the extracted aerial image is 320; The dimension size of the feature vector of the extracted satellite image sub-image is 320.

9. The method for matching unmanned aerial images and satellite images according to claim 8, characterized in that: The QATM matching algorithm is used to calculate the feature response scores of the feature vector of the aerial image and the feature vector of the sub-image of the satellite image, and the area with the highest score is selected as the best matching result, so as to achieve the matching of the drone aerial image and the satellite image; the specific process is: The image matching quality perception function is defined as Q(s,t), where t and s are fragments in the template image T and the search image S, respectively, and the feature vectors corresponding to t and s are represented as f t and f s ; The matching similarity score between a fragment s of the search image S and a fragment t of the template image T is P(t|s), which is calculated as follows: Where β is a positive weight, ρ(·) represents the cosine operation, and the specific formula is shown in formula (11); t′ is a fragment in the template image T, and the feature vector corresponding to t′ is expressed as f t' ; The value of P(t|s) is in the range of [0, 1]. The larger the value of P(t|s), the higher the similarity between the matching of fragments t and s. Define the matching similarity score between a fragment t of the template image T and a fragment s of the search image S as P(s|t), so the image matching quality perception function Q(s,t) can be obtained according to formula (12); Q(s,t)=P(t|s)×P(s|t) (12) Finding the area with the maximum matching quality response in the matching quality perception function Q(s,t) is the best matching area. The specific calculation method is shown in the formula: Among them, Φ is the best matching area of ​​the template image T found in the search image S.

10. The method for matching unmanned aerial imagery and satellite images according to claim 9, characterized in that: The image matching quality perception function Q(s, t) is divided into five matching situations: (1) 1-1 matching: For a fragment t of the template image T, there is a unique fragment s in the search image S that matches the fragment t, and the fragment t is also the only fragment in the template image T that matches the fragment s. In this case, P(s|t)=P(t|s)=1, and the matching quality perception function Q(s,t)=1; (2) 1-N matching: For a fragment t of the template image T, there are N fragments in the search image S that match the fragment t. In this case, P(s|t) = 1 / N, P(t|s) = 1, and the matching quality perception function Q(s, t) = 1 / N; (3) M-1 matching: For a fragment s of the search image S, there are M fragments in the template image T that match the fragment s. In this case, P(s|t) = 1, P(t|s) = 1 / M, and the matching quality perception function Q(s, t) = 1 / M; (4) MN matching: There are N fragments in the search image S that match M fragments in the template image T. In this case, P(s|t) = 1 / N, P(t|s) = 1 / M, and the matching quality perception function Q(s, t) = 1 / NM; (5) No match: There are no matching fragments between the template image T and the search image S, and the matching quality perception function Q(s,t) = 0.