A remote sensing image registration method based on rotation-invariant SuperGlue matching

Abstract

The present invention relates to the field of remote sensing image processing, and in particular to a remote sensing image registration method based on rotation-invariant SuperGlue matching. In the rotation-invariant SuperGlue, an adaptive median matching confidence criterion (MMC) and geometric constraint adaptive iterative refinement are used to determine the remote sensing image registration control point RCP, and achieve high-precision remote sensing image transformation model parameter estimation. The method adopts MMC adaptive measurement to avoid the shortcomings of SuperGlue's sensitivity to rotational distortion and threshold dependence when determining matching, and adopts a geometric constraint adaptive iterative refinement method to improve the matching point accuracy and achieve robust matching. Experimental results show that the method has good robustness to different types of geometric distortions such as translation, rotation and affine. It can also be applied to SAR, UAV and high-resolution satellite images.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image processing, and in particular to a remote sensing image registration method based on rotationally invariant SuperGlue matching. Background Art

[0002] With the continuous development of information technology and the continuous advancement of Internet technology, remote sensing technology has entered a new era. For example, remote sensing technology plays an important role in agriculture, forestry, hydrology, meteorology, military, natural disasters, etc. Images are an important way to observe things through vision and obtain information. There are obvious geometric and radiometric distortions between remote sensing images obtained at different times, angles, or different sensors. Finding a suitable registration method and establishing the correspondence between space and image features are key steps in the application of remote sensing images.

[0003] In the 1970s, the United States first proposed image registration technology. In the 1980s, image fusion registration technology attracted the attention of many researchers. In 1970, PEAnuta proposed image registration based on cross-correlation image detection calculation using FFT technology to improve the registration rate; Albert Arcese obtained the bipolar reference concept with continuous positive and negative values ​​through convolution calculation; DIBamae et al. proposed the fusion between two structured data sets based on the template sub-image detection algorithm; in 1989, H. Maitre et al. proposed the method of autoregressive model, which achieves correspondence by adjusting the reference of the camera, and is mainly used for image registration with large distortion; in 1992, Flussr proposed adaptive mapping fusion to solve the problem of image deformation. First, the images are segmented according to the image classification requirements. The segmented images have a strong internal similarity, and then the spatial position relationship between the segmented images is used to achieve fusion; in 2004, Professor DC Lowe proposed the SIFT (Scale Invariant Feature Transform) algorithm with good robustness to scale features and noise; in 2006, Herbert Bay et al. proposed the SURF (Speeded Up Robust Features) algorithm. The SIFT algorithm and the SURF algorithm are currently the most widely used automatic image matching algorithms, and both play an important role in the field of remote sensing image registration.

[0004] After years of development, image registration technology has made great progress and improved the data quality of remote sensing satellite basic images. However, due to the particularity of local spectral characteristics of remote sensing images and the influence of external environment, image registration problems still have certain uncertainties and unreliability. Therefore, the realization of "high-precision, high-efficiency, and intelligent" image registration technology is still on the way.

[0005] Image registration is a key issue in remote sensing image processing. Existing methods are usually very sensitive to image distortion. Generally, local features extracted manually or by learning methods are matched to estimate points with the same name. For example: (1) Harris detector is used to extract corner points on the image. It has good repeatability for general translation, rotation and small-scale illumination changes. However, Harris point detector does not have scale and affine deformation invariance, so it can only be used for image pair registration without scale and viewpoint transformation. (2) SIFT algorithm extracts local feature points with scale invariance through DOG algorithm, which has good scale invariance. However, remote sensing images usually have obvious viewpoint changes, resulting in serious affine deformation of the image. At this time, SIFT algorithm is difficult to obtain reliable points with the same name. (3) In order to process images with viewpoint changes, the maximum stable extreme region (MSER) ​​is often used for feature detection. The MSER feature region has good affine invariance and can obtain features with high re-detection rate in wide baseline image pairs. However, the MSER-based method has problems such as inaccurate feature point positioning, a small number of feature points, and even the inability to estimate transformation parameters for some images with poor textures, making it difficult to be widely used in remote sensing image registration algorithms. Summary of the invention

[0006] Image registration is a key issue in remote sensing image processing. Existing methods are usually very sensitive to image distortion. In view of the shortcomings of the existing technology, the present invention proposes a remote sensing image registration method based on rotation-invariant SuperGlue matching with high matching accuracy and good robustness.

[0007] To achieve the above objectives, the present invention adopts the following technical solutions.

[0008] A remote sensing image registration method based on rotation-invariant SuperGlue matching is proposed. In the rotation-invariant SuperGlue, an adaptive median matching confidence criterion (MMC) and adaptive iterative refinement are used to determine the remote sensing image registration transformation model to solve the registration control point (RCP) and achieve high-precision correspondence of image model parameter estimation. Specifically, the method includes the following steps:

[0009] S1. Obtain a set of simulated images at different rotation angles: divide [0,360°] into 30° intervals to determine a set of rotation angles, and transform the image to be registered according to the rotation angles to generate a set of simulated images;

[0010] S2. Simulated image matching: The SuperPoints model is used to extract image features of the reference image and the simulated image, and the reference image and the simulated image are matched using the SuperGlue pre-trained model to obtain the feature point matching confidence matrix of each pair of images; SuperGlue is a neural network that takes local features as input and feature point matching confidence as output, and achieves high precision and recall on indoor and outdoor image pairs with extremely wide baselines;

[0011] S3.MMC adaptive criterion, determine the optimal match based on MMC: take the median of the confidence of the matching points of the simulated image and the reference image as c mmc , comparing the c of different simulated images mmc The value of c mmc The simulated image with the largest value is taken as the best matching image of the reference image;

[0012] S4. Adaptive iterative refinement, under the constraints of epipolar geometry, significant correspondence is obtained through iterative refinement;

[0013] S5. Image registration: The image to be registered is converted into the coordinates of the reference image through the homography model, and the parameters are calculated using the significant correspondence relationship to achieve image registration.

[0014] The specific steps of step S3.MMC self-adaptive criterion are as follows:

[0015] MMC is the median match confidence criterion, which can automatically determine the best matching influence pair and the corresponding simulated image from the sequence image; for the matching of the reference image and the simulated image, the median match confidence calculation method is:

[0016] c mmc =median(C)

[0017] Among them, C represents the matching confidence of feature points between the reference image and the simulated image, and the MMC adaptive criterion value c mmc The larger the value, the better; according to this criterion, determine c mmc The best matching image pair with the highest value, the corresponding simulation image in the best matching image pair is the best simulation image;

[0018] The matching points located on the optimal simulated image The coordinates of the image to be registered are transformed to obtain the optimal matching point set between the reference image and the image to be registered. The calculation formula is as follows:

[0019]

[0020] Among them, A -1 is the inverse of the transformation matrix from the image to be registered to the simulated image, is the position of the i-th matching point on the simulated image, is the position of the i-th matching point on the image to be registered.

[0021] Step S4. The specific steps of adaptive iterative refinement are as follows:

[0022] After the above processing, a large number of matching points on the reference image and the image to be registered have been obtained. However, due to image distortion and matching error images, mismatching is inevitable in the matching point set, which may seriously affect the estimation accuracy of the registration model parameters. Eliminating the mismatching to obtain a high-precision transformation model is a key step in image registration, which specifically includes the following steps:

[0023] 1) Use the RANSAC algorithm to calculate the basic matrix F between the reference image and the image to be registered;

[0024] 2) Calculate the projection error of the matching points under the constraints of the basic matrix F, and remove the external points whose projection error value is greater than 3.0 pixels;

[0025] 3) Determine the number of matching points after removing the outliers and the number of matching points before removing them. If the number of matching points after removing the outliers is less than the number of matching points before, jump back to the first step and recalculate the basic matrix based on the matching point set after removing the outliers; otherwise, terminate the loop and output the final matching point set.

[0026] Step S5. The specific steps of image registration are as follows:

[0027] The global perspective projection model can describe the distortion caused by projecting an object onto the image plane at different distances from the sensor (Zitová and Flusser, 2003). The present invention uses the global perspective projection model to establish a mapping function between the reference image and the image to be registered; the final matching point set obtained in the previous step is used as the registration control point (RCP), and the LSM algorithm is used to calculate the projection transformation H = {h 0 ,h 1 ,...h 7 ,1}; then bilinear interpolation is used to resample I' to achieve the registration of the two images.

[0028] Beneficial effects of the present invention:

[0029] The remote sensing image registration method based on rotation-invariant SuperGlue matching proposed in this invention adopts the MMC adaptive criterion, which avoids the shortcomings of SuperGlue being sensitive to rotation distortion and threshold dependence when determining matching, and adopts an adaptive iterative refinement method to improve the matching accuracy and achieve robust matching. Experimental results show that this method has good robustness to different types of geometric distortions such as translation, rotation and affine. It can also be applied to SAR, UAV and satellite images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flowchart of the remote sensing image registration method based on rotation-invariant SuperGlue matching.

[0031] Figure 2 The present invention is a flowchart of the matching point optimization based on adaptive iterative geometric constraints in the remote sensing image registration method of the present invention.

[0032] Figure 3 (a) is the matching points obtained for the first group of experimental images in the embodiment of the present invention, and (b) is the registration result of the first group of experimental images in the embodiment of the present invention.

[0033] Figure 4 (a) is the matching points obtained for the second group of experimental images in the embodiment of the present invention, and (b) is the registration result of the second group of experimental images in the embodiment of the present invention.

[0034] Figure 5 (a) is the matching points obtained for the third group of experimental images in the embodiment of the present invention, and (b) is the registration result of the third group of experimental images in the embodiment of the present invention.

[0035] Figure 6 (a) is the matching points obtained for the fourth group of experimental images in the embodiment of the present invention, and (b) is the registration result of the fourth group of experimental images in the embodiment of the present invention.

[0036] Figure 7 (a) is the matching points obtained for the fifth group of experimental images in the embodiment of the present invention, and (b) is the registration result of the fifth group of experimental images in the embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0038] The present invention automatically obtains image registration control points through image feature matching based on deep learning to achieve automatic registration of two remote sensing images (reference image and image to be registered). The implementation process is shown in the attached Figure 1 , which is achieved by the following steps:

[0039] Step 1: Rotate the image to be registered at 30° intervals to obtain 12 simulated images. Therefore, there are different degrees of rotation transformation between the simulated image and the image to be registered.

[0040] Step 2: Combine the simulated image and the reference image into image pairs to be matched, and obtain 12 groups of image pairs. Use the SuperPoint pre-trained model to extract feature points from each image, including the reference image and the simulated image, and use the SuperGlue pre-trained model to match the 12 groups of image pairs to obtain the matching confidence of the feature points of each pair of images. The highest matching confidence value of each pair of images is used as the candidate matching point.

[0041] Step 3: Calculate the median matching confidence value c of the candidate matching points in each pair of images mmc = median(C), select c from 12 groups of images mmc The image pair with the largest value is taken as the optimal matching image pair, the matching points in the optimal matching image pair are taken as the optimal matching points, the simulated image in the optimal image matching pair is taken as the optimal simulated image, the rotation angle between the optimal simulated image and the image to be registered is the optimal rotation angle, and the corresponding rotation matrix is ​​the optimal rotation matrix. According to the optimal rotation matrix, the matching points in the optimal matching point pair and the matching points on the optimal simulated image are transformed to the image to be registered, and the optimal matching point set S between the reference image and the image to be registered is obtained. 1 , the number of corresponding matching points is n 1 .

[0042] Step 4: Based on the kernel line constraint, iteratively solve the basic matrix and remove the mismatched points. The process is as follows: Figure 2 As shown, the specific implementation is as follows:

[0043] (1) Based on the optimal matching points between the reference image and the image to be registered obtained in step 3, the basic matrix F between the reference image and the image to be registered is calculated using the RANSAC algorithm.

[0044] (2) Use the F matrix to calculate the epipolar lines l, l' corresponding to each pair of matching points. The calculation formula is as follows:

[0045] l=Fx

[0046] l′=F T x′

[0047] Where x is the matching point on the reference image; x′ is the matching point on the image to be registered; F T is the transpose of F.

[0048] And the distance d from the matching point to the corresponding epipolar line 1 ,d 2 , the calculation formula is as follows:

[0049]

[0050]

[0051] In the formula, xi and x′ i are the coordinate vectors of the matching points on the reference image and the image to be registered respectively; and Fx i The first and second elements.

[0052] Take each pair of d 1 and d 2 The maximum value in is taken as the projection error, and the projection error of the paired matching point is judged whether it is greater than 3.0 pixels. If it is greater than 3.0 pixels, the matching point is considered to be a mismatch and is removed, otherwise it is retained. Therefore, after removing the mismatching points, a new matching point set S can be obtained 2 , the number of corresponding matching points is n 2 .

[0053] (3) Compare and eliminate the new matching point set S obtained after mismatching points 2 The number of matching points n in 2 The number of matching points before removing the wrong matching points n 1 , if n 2 <n 1 Then return to (1) with the new matching point set S 2 Based on, calculate the new basic matrix and execute the loop of steps (1) to (3). Otherwise, output the final matching point set S 3 .

[0054] Step 5: The final matching point set S obtained in step 4 3 As the registration control point, the parameters of the projection transformation model between the reference image and the image to be registered are calculated through the LSM algorithm to obtain the geometric transformation model between the reference image and the image to be registered.

[0055] Step 6: Perform geometric transformation on the image to be registered according to the geometric transformation model between the reference image and the image to be registered, and use bilinear interpolation to resample the image to be registered to obtain the corrected image of the image to be registered. At this point, the image to be registered has been converted to the reference image coordinate system, and the registration of the two images has been achieved. In order to better observe the registration results, the gray value weighted average method is used to fuse the two images and output the final registered image. The calculation formula for the gray value weighted average is as follows:

[0056]

[0057] Where I is the gray value of the reference image; I′ is the gray value of the image to be registered.

[0058] Five groups of different remote sensing images were selected for the experiment. The five groups of images are SAR images with translation transformation, UAV images with perspective and rotation transformation, UAV images with perspective transformation and lack of texture, high-resolution remote sensing images with compression and rotation, and high-resolution remote sensing images with large perspective transformation.

[0059] The matching points and registration results obtained by the method of the present invention are as follows: Figures 3 to 7 As shown. Figures 3 to 7 It can be seen intuitively that the proposed method extracts evenly distributed matching points for all image pairs and achieves image registration. This means that the method is robust to various geometric distortions including translation, rotation, and affine, and can be applied to SAR, UAV, and high-resolution satellite images.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image registration method based on rotation-invariant SuperGlue matching, characterized in that: In the rotation-invariant SuperGlue matching, the adaptive median matching confidence criterion MMC and geometric constraint adaptive iterative refinement are used to obtain high-precision remote sensing image registration control points and realize reliable estimation of the projection transformation model. The specific steps include: S1. Based on the affine transformation model constructed at different rotation angles, the image to be registered is geometrically transformed to obtain a set of simulated images, and a simulated image pair is constructed using a reference image and a simulated image respectively; S2. Use SuperPoints to extract local features of the reference image and the simulated image, then use the SuperGlue pre-trained model to calculate the matching confidence of the feature points in the simulated image pair, and obtain the candidate matching points between the simulated image pairs; S3. Calculate the median matching confidence of the candidate matching points in the simulated image pair, determine the optimal simulated image pair, the optimal simulated image, the optimal rotation angle and the optimal candidate matching point according to the MMC criterion; and transform the optimal candidate matching point to the coordinate system of the image to be registered to obtain the optimal matching point set; S4. Adaptively iteratively refine the optimal matching point set, and obtain the final same-name point pairs as registration control points through iterative refinement under the epipolar geometric constraints; S5. Image registration: The geometric transformation model between the reference image and the image to be registered is calculated based on the registration control points, and based on this, the image to be registered is transformed into the coordinate system of the reference image, the image to be registered is resampled, and superimposed with the reference image to achieve image registration.

2. The remote sensing image registration method based on rotation-invariant SuperGlue matching according to claim 1, characterized in that: In step S3, for the matching between the reference image and the simulated image, the median matching confidence calculation method is: Where C represents the matching confidence of feature points in the reference image and the simulated image, and the adaptive criterion value of MMC is The larger the value, the better; according to this criterion, determine The best matching image pair with the highest value and the corresponding simulated image, converting the coordinates of the matching points on the matched simulated image to the coordinates of the matching points on the image to be registered is as follows: In the formula, is the inverse of the transformation matrix from the image to be registered to the simulated image, For the simulated image The matching point positions, is the first The optimal matching point set between the reference image and the image to be registered is obtained.

3. The remote sensing image registration method based on rotation-invariant SuperGlue matching according to claim 2, characterized in that: The specific steps of the adaptive iterative refinement in step S4 are as follows: 1) According to the optimal matching point set between the reference image and the image to be registered obtained in step S3, the basic matrix F between the reference image and the image to be registered is calculated using the RANSAC algorithm; 2) Calculate the projection error of the matching points under the constraints of the basic matrix F, and remove the external points whose projection error value is greater than 3.0 pixels; 3) Determine the number of matching points after removing the outliers and the number of matching points before removing them. If the number of matching points after removing the outliers is less than the number of matching points before, the loop will jump back to the first step and recalculate the basic matrix based on the matching point set after removing the outliers. Otherwise, terminate the loop and output the final set of matching points.

4. The remote sensing image registration method based on rotation-invariant SuperGlue matching according to claim 3, characterized in that: The specific steps of image registration in step S5 are as follows: The global perspective projection model is used to establish the mapping function between the reference image and the image to be registered; the final matching point set obtained in step S4 is used as the registration control point, and the projection transformation is calculated using the LSM algorithm Parameters; then bilinear interpolation is used to resample the image to be registered to achieve the registration of the two images.

Images