A Heterogeneous Image Registration Method Based on Multi-Scale Image Gradient and Phase Features

Abstract

The present invention discloses a method for registering heterogeneous images based on multi-scale image gradients and phase features, including: obtaining the gradient magnitude map of a reference image by using an improved Sobel operator; performing non-maximum suppression processing on the gradient magnitude map to obtain the feature point information of the reference image; respectively constructing a phase consistency orientation map and a phase consistency structure map for the reference image and the real-time image to be registered; obtaining the phase consistency feature descriptors of the reference image and the real-time image by using the phase consistency orientation map and the structure map; obtaining the rough positions of the feature points of the real-time image according to the feature point information and the projective transformation matrix; obtaining the position errors of the feature points of the real-time image through similarity measurement, and combining with the rough positions of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image. The present invention fully combines and utilizes the gradient and phase consistency information of the image, obtains feature descriptors with a relatively high degree of consistency, and reduces the computational time complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of image registration, and particularly relates to a method for registering heterogeneous images based on multi-scale image gradients and phase features. Background Art

[0002] Image registration technology is an important part of the field of image processing and has been widely applied to military and civilian fields such as aircraft navigation systems, projectile positioning, aerial guidance, computer vision, pattern recognition, urban development monitoring, topographic and mountain mapping, biological diagnosis, and climatology. The registration technology of heterogeneous images can make up for the deficiencies of single-sensor images in image registration technology and achieve the same geometric consistency between the reference image and the image to be registered, which is a research hotspot in the current field of image registration technology. The existing mainstream multi-source remote sensing image registration methods can be divided into traditional registration algorithms and deep learning registration algorithms. The traditional multi-source remote sensing image registration algorithms are mainly represented by image registration methods based on gray information and image registration methods based on feature information. The registration algorithms based on deep learning are divided into supervised learning registration algorithms and unsupervised learning registration algorithms. Due to the non-linear gray difference and radiation intensity difference between multi-modal remote sensing images, the image registration method based on gray information cannot well adapt to the geometric gray changes between heterogeneous remote sensing images, resulting in low registration accuracy and high time complexity. Feature information is a higher-level description of image information and can stably exist in multi-modal remote sensing images. Therefore, the image registration method based on feature information is often used for the matching between multi-modal remote sensing images and has also become the main research direction in this field at home and abroad.

[0003] In order to obtain a robust and accurate heterogeneous image registration result, stable and sufficient feature information needs to be extracted. The most basic feature information in an image is point feature and line feature. Point features have been widely applied in image matching and stereo scene matching. Currently, the commonly used feature extraction algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Radiation-Variation Insensitive Feature Transform (RIFT) algorithms. However, these methods usually use single intensity information, gradient information, or phase information for feature detection and description. The remote sensing images may have a certain degree of radiation distortion, resulting in the failure of these algorithms and making it difficult to obtain a stable registration effect.

[0004] (2) Solutions of the prior art: There are mainly two directions for technical solutions similar to the present invention. The first technical direction is to use a template registration solution, and the second technical direction is to use a single feature registration solution based on intensity, gradient or phase information.

[0005] ① Template registration solution. The method based on template matching started relatively early and has now developed relatively maturely. It is a most commonly used image matching algorithm, which includes two main steps: constructing a regional template and using a similarity measure for similarity measurement. Under the guidance of the similarity measure, the similarity between regional templates is calculated to measure the alignment degree of two images, and then the optimization of the registration process is promoted. In template matching, both expressions used for matching are images. The matching of images can be the matching between the whole image and the whole image, or the matching between sub-images of the images. Since the relative positions of the scenes in different images remain unchanged, the global correspondence relationship can be determined from the local correspondence relationship. If a region is selected as a template in the region where two images may overlap, and then the image region most similar to the template is searched in the other image, that is, a local correspondence relationship is found, and thus the alignment of the overlapping regions of the two images can be completed through the template matching technology. This method compares the similarity of all pixel gray values in the image respectively, and then uses a certain algorithm to search for the transformation model parameter values that minimize or maximize the similarity, so as to determine the similarity between the whole images. Commonly used similarity measure functions in this type of matching algorithm include the sum of squared differences of gray levels between two images, correlation function, covariance function, cross-correlation, and phase correlation, etc. The algorithm based on template matching has a large amount of calculation because it needs to loop through multiple times, and is too sensitive to gray level changes, resulting in poor registration effects.

[0006] ② Single feature registration solution based on intensity, gradient or phase information.

[0007] Feature matching-based methods are a major category of image registration methods. It is common to use intensity or gradient information for feature matching, while using phase information for feature matching is less common. The main commonality of these methods is to first extract key information in the image, and then use the extracted features to generate consistent feature descriptors to complete the matching between the features of two images. An image geometric transformation model is established through the matching relationship of features to complete the registration of images. Among them, the SIFT algorithm is a feature registration algorithm with strong tolerance to changes in the external environment. Later, many improvements have been made on this basis, which to a certain extent meet the requirements of image registration in different scenarios. However, the intensity or gradient information used in such algorithms is sensitive to the non-linear gray difference and radiation intensity difference between heterogeneous images, resulting in a large number of unreliable points in the extracted feature points. The feature matching result has low accuracy, poor reliability, and long time consumption, making it difficult to meet the requirements of high-precision and low-latency multi-modal remote sensing image registration. The RIFT algorithm has good robustness to non-linear radiation distortion (NRD), but the algorithm takes a long time and has a large root mean square error.

[0008] In summary, there are non-linear gray differences and radiation intensity differences between multi-modal remote sensing images. The image registration methods based on gray information are mostly for homologous images, are sensitive to radiation differences and gray changes, are difficult to process multi-modal remote sensing images, have low matching accuracy, seriously reduced reliability, and high time complexity. The image registration methods based on feature information can stably process multi-source remote sensing images. However, due to the possible non-linear radiation distortion in remote sensing images, using intensity information or gradient information for feature detection and description will cause these algorithms to fail and it is difficult to obtain a stable registration effect. Using phase information for feature detection and description has a relatively high matching time complexity and the accuracy of the extracted feature points is relatively low. Summary of the Invention

[0009] In order to solve the above problems existing in the prior art, the present invention provides a method for registering heterogeneous images based on multi-scale image gradients and phase features. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0010] The present invention provides a method for registering heterogeneous images based on multi-scale image gradients and phase features, including:

[0011] S1: Obtain the gradient magnitude map of the reference image by using an improved Sobel operator;

[0012] S2: Perform non-maximum suppression processing on the gradient magnitude map to obtain the feature point information of the reference image;

[0013] S3: Construct the phase consistency orientation map and the phase consistency structure map for the reference image and the real-time image to be registered respectively;

[0014] S4: Obtain the phase consistency feature descriptors of the reference image and the real-time image by using the phase consistency orientation maps and the phase consistency structure maps of the reference image and the real-time image respectively;

[0015] S5: Obtain the rough positions of the feature points of the real-time image according to the feature point information and the projection transformation matrix between the reference image and the real-time image;

[0016] S6: Obtain the position error of the feature points of the real-time image through similarity measurement, and combine it with the rough positions of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image;

[0017] S7: Use the random sample consensus algorithm to remove the mismatched points in the accurate position information of the feature points of the real-time image to obtain the final matching result.

[0018] In an embodiment of the present invention, the S1 includes:

[0019] S1.1: Construct a horizontal template, a vertical template and templates in two diagonal directions:

[0020]

[0021]

[0022] Wherein, h1 and h2 respectively represent the horizontal template and the vertical template, and h3 and h4 respectively represent the templates in two diagonal directions;

[0023] S1.2: Obtain the first-order derivatives of each pixel point in the horizontal direction, vertical direction and two diagonal directions of the reference image, which are D1(x, y), D2(x, y), D3(x, y), D4(x, y) respectively:

[0024] D1(x, y) = h1 * f(x, y)

[0025] D2(x, y) = h2 * f(x, y)

[0026] D3(x, y) = h3 * f(x, y)

[0027] D4(x, y) = h4 * f(x, y)

[0028] Wherein, * represents the convolution operation, and f(x, y) represents the reference image;

[0029] S1.3: Obtain the gradient magnitude map of the reference image:

[0030] F(x, y) = |D1(x, y)| + |D2(x, y)| + |D3(x, y)| + |D4(x, y)|.

[0031] In one embodiment of the present invention, S3 includes:

[0032] S3.1: Obtain the phase consistency information of each pixel position of the reference image and the real-time image at different angles and scales by using a Log-Gabor filter;

[0033] S3.2: Obtain the phase consistency structure diagrams of the reference image and the real-time image respectively by using the phase consistency information;

[0034] S3.3: Calculate the phase consistency direction diagrams integrating multiple scale information of the reference image and the real-time image by using a Log-Gabor filter.

[0035] In one embodiment of the present invention, S3.1 includes:

[0036] S3.11: Obtain a two-dimensional Log-Gabor function:

[0037]

[0038] where ρ and θ represent the polar radius and the polar angle respectively, s and o represent the scale coefficient and the angle coefficient respectively, (ρ s , θ (s,o) ) represents the center frequency, σ ρ and σ θ are bandwidths in different forms;

[0039] S3.12: Perform an inverse Fourier transform on the two-dimensional Log-Gabor function to obtain the spatial domain filter corresponding to the two-dimensional Log-Gabor function:

[0040] L(x, y, s, o) = L even (x, y, s, o) + iL odd (x, y, s, o)

[0041] where L even (x, y, s, o) and L odd (x, y, s, o) are the even-symmetric wavelet expression form and the odd-symmetric wavelet expression form of the spatial domain filter respectively;

[0042] S3.13: Convolve the reference image with the even-symmetric wavelet and the odd-symmetric wavelet respectively to obtain the response components of the reference image at different angles and scales:

[0043] e so(x, y) = f(x, y) * L even (x, y, s, o)

[0044] o so (x, y) = f(x, y) * L odd (x, y, s, o)

[0045] where e so (x, y) is the scale response component of the reference image f(x, y), o so (x, y) is the angular response component of the reference image f(x, y);

[0046] S3.14: Obtain the amplitude component A and phase component φ of the reference image at the scale and direction using the scale response component and the angular response component so (x, y): so (x, y):

[0047]

[0048]

[0049] S3.15: Obtain the phase consistency information of each pixel point in the reference image using the amplitude component and the phase component:

[0050]

[0051] where w0(x, y) is the frequency diffusion weight factor, ξ is the constant factor, T is the noise suppression factor, and Δφ so (x, y) is the phase deviation function;

[0052] S3.16: Obtain the phase consistency information of each pixel point in the real-time image according to S3.11 - S3.15.

[0053] In an embodiment of the present invention, in the S3.15, A so (x, y)Δφ so (x, y) has the following expression:

[0054]

[0055] where E(x, y) represents the local energy of the image,

[0056] In an embodiment of the present invention, the S3.2 includes:

[0057] S3.21: Add the phase consistency information of different directions of the obtained reference image to obtain the phase consistency structure diagram of the reference image;

[0058] S3.22: Add the phase consistency information in different directions of the obtained real-time image to obtain the phase consistency structure diagram of the real-time image.

[0059] In an embodiment of the present invention, the S3.3 includes:

[0060] Calculate the phase consistency direction map integrating multiple scale information according to the angular response component o so (x,y).

[0061]

[0062]

[0063]

[0064] Wherein, represents the phase consistency direction map of the reference image or the real-time image, m represents the energy in the horizontal direction of the phase consistency, n represents the energy in the vertical direction of the phase consistency, o so (θ) represents the angular response component in the direction θ, and arctan(·) represents the arctangent function.

[0065] In an embodiment of the present invention, the S6 includes:

[0066] S6.1: Take a corresponding template window around each feature point of the real-time image, and perform a sliding window search in the region of interest corresponding to the real-time image using the search window to obtain the mean squared error sum value between the template window i and each search window in the region of interest:

[0067]

[0068] Wherein, C1 and C2 respectively represent the phase consistency descriptors of the reference image template window and the real-time image search window, x represents the coordinate of the pixel point in the reference image, T i (·) is a masking function, when x is a pixel point in the template window, T i (x) = 1, otherwise T i (x) = 0; M i (v) represents the mean squared error sum value between the template window i and each search window in the region of interest;

[0069] S6.2: Define the matching function between the template window on the reference image and the search window on the real-time image:

[0070]

[0071] S6.3: Substitute the matching function into the mean squared error formula to obtain:

[0072]

[0073] Among them, the first term is a constant, and the second and third terms are variables;

[0074] S6.4: Perform Fourier transform on the last two terms of the formula in S6.3 to obtain the position error of the feature points in the real-time image:

[0075]

[0076] where v i represents the position error, i represents the template window, F(·), F -1 (·) and F * (·) respectively represent Fourier transform, inverse Fourier transform, and complex conjugate of Fourier transform;

[0077] S6.5: Add the position error to the rough position of the feature points in the real-time image to obtain the accurate position information of the feature points in the real-time image.

[0078] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0079] The present invention proposes a method for registering heterogeneous images based on multi-scale image gradient and phase features. This method uses the gradient information of the image to extract the feature information of the image, which is not affected by the gray change and illumination difference between heterogeneous images, thereby obtaining feature points with strong robustness and stability; uses the phase information of the reference image and the real-time image to generate a phase consistency direction map integrating multiple scale information and a phase consistency structure map integrating multiple direction information, and then constructs a phase consistency feature histogram to overcome the interference of SAR speckle noise and the non-linear radiation difference between images; since the template matching using the sliding window operation is time-consuming, the mean squared error MSD algorithm is used for similarity measurement, and the fast Fourier transform is used to accelerate the MSD algorithm to improve the operation efficiency. Finally, the fast sample consensus FSC algorithm is used to remove the corresponding points with large errors to obtain the final accurate registration result.

[0080] The overall idea of the present invention is different from the traditional method for registering heterogeneous images, which detects and describes features through single intensity, gradient, or phase information of the image. Instead, it fully combines and utilizes the gradient and phase consistency information of the image to extract stable and sufficient feature points, obtains a feature descriptor with a high degree of consistency, and uses the fast Fourier transform to accelerate the MSD algorithm to reduce the time complexity of the algorithm.

[0081] The following will further elaborate on the present invention in conjunction with the accompanying drawings and embodiments. Brief Description of the Drawings

[0082] Figure 1 is a flowchart of a method for registering heterogeneous images based on multi-scale image gradients and phase features provided by an embodiment of the present invention;

[0083] Figure 2 is a schematic diagram of the processing process of a method for registering heterogeneous images based on multi-scale image gradients and phase features provided by an embodiment of the present invention;

[0084] Figure 3 is the registration result of a visible light image and a SAR image obtained by using the method for registering heterogeneous images of an embodiment of the present invention;

[0085] Figure 4 is the registration result of the registration result of a visible light image and an infrared image obtained by using the method for registering heterogeneous images of an embodiment of the present invention;

[0086] Figure 5 is the registration result of a SAR image and a SAR image obtained by using the method for registering heterogeneous images of an embodiment of the present invention. Detailed Embodiment

[0087] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following provides a detailed description of a method for registering heterogeneous images based on multi-scale image gradients and phase features proposed according to the present invention in combination with the accompanying drawings and specific embodiments.

[0088] The foregoing and other technical contents, features, and effects of the present invention can be clearly presented in the following detailed description in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not used to limit the technical solutions of the present invention.

[0089] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant is intended to cover non-exclusive inclusion, so that an article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the article or device including the said element.

[0090] Please refer to Figure 1 andFigure 2 , the method for registering heterogeneous images based on multi-scale image gradients and phase features in this embodiment includes:

[0091] S1: Obtain the gradient magnitude map of the reference image using an improved Sobel operator.

[0092] The template matching algorithm is sensitive to rotation and scaling changes of images. Therefore, the template matching algorithm does not have rotational invariance and scale invariance and can only adapt to translational changes between images. When there are no rotation and scaling changes in the registration scene, registration can be directly performed. If there are, geometric transformation preprocessing needs to be done in advance. Generally, algorithm research solves this problem by pre-selecting control points. The selection of control points is divided into two methods: manual point selection and automatic point selection. The preprocessing control points in this embodiment can be manually selected or obtained by other algorithms, and geometric transformation rough correction is achieved using the preprocessing control points. In engineering practice, real-time images can be corrected through longitude and latitude geographic coordinate information.

[0093] Gradient information is not affected by gray-scale changes and illumination differences between heterogeneous images, and the extracted feature points are more robust. In this embodiment, an improved Sobel operator is used to calculate the gradient map of the reference image. The Sobel operator uses two 3×3 directional templates (horizontal and vertical templates) to perform neighborhood convolution with each point in the image. These two directional templates detect vertical and horizontal edges respectively. The Sobel operator has a smoothing effect on noise and provides relatively accurate edge direction information.

[0094] The traditional Sobel operator uses two templates in the horizontal and vertical directions, with thick edges and low edge positioning accuracy. However, in reality, the gradient direction is unknown, and the traditional operator does not consider the influence of the diagonal direction on the image, resulting in certain errors in the calculated results. Therefore, in this embodiment, two diagonal direction templates at 45° and 135° are added to the original two-direction templates to obtain the gradient image, which can detect edges in multiple directions, so it can solve the problem of azimuth limitation to a certain extent and make the calculation results more accurate.

[0095] S1 in this embodiment includes:

[0096] S1.1: Construct horizontal templates, vertical templates, and two diagonal direction templates:

[0097]

[0098]

[0099] Among them, h1 and h2 respectively represent the horizontal template and the vertical template, and h3 and h4 respectively represent the templates in two diagonal directions. Only the templates in four directions are defined here. Continuing to increase the number of templates can further improve the calculation accuracy, but considering the calculation efficiency and the edge limitations, the number of templates is not increased any further.

[0100] S1.2: Obtain the first-order derivatives in the horizontal direction, vertical direction, and two diagonal directions of each pixel point in the reference image.

[0101] Specifically, let the original reference image be f(x, y). The first-order derivatives of each pixel point in the horizontal direction, vertical direction, and two diagonal directions of the image are D1(x, y), D2(x, y), D3(x, y),

[0102] D4(x, y) respectively. Then, the following can be obtained:

[0103] D1(x, y) = h1 * f(x, y)

[0104] D2(x, y) = h2 * f(x, y)

[0105] D3(x, y) = h3 * f(x, y)

[0106] D4(x, y) = h4 * f(x, y)

[0107] Among them, * represents the convolution operation, and f(x, y) represents the reference image.

[0108] S1.3: Obtain the gradient magnitude map of the reference image

[0109] Specifically, the calculation formula for the gradient magnitude is:

[0110]

[0111] Since the square root operation and the calculation complexity are relatively high, in this embodiment, the absolute value summation is used to calculate the gradient magnitude map, which can improve the calculation efficiency. The expression of this gradient magnitude map is:

[0112] F(x, y) = |D1(x, y)| + |D2(x, y)| + |D3(x, y)| + |D4(x, y)|.

[0113] S2: Perform non-maximum suppression processing on the gradient magnitude map to obtain the feature point information of the reference image.

[0114] S3: Construct a phase consistency direction map and a phase consistency structure map for the reference image and the real-time image to be registered respectively.

[0115] Specifically, S3 of this embodiment includes:

[0116] S3.1: Obtain the phase consistency information at different angles and scales corresponding to the pixel points at each position of the reference image and the real-time image by using the Log-Gabor filter.

[0117] The phase consistency information is not affected by the gray level, radiation, and illumination differences between multi-source images. The image feature description algorithm based on the phase consistency theory is a frequency domain feature description method. The phase consistency information at different angles and scales corresponding to the pixel points at each position of the original reference image and the real-time image is obtained by using the Log-Gabor filter. The two-dimensional Log-Gabor (2Dimensional Log-Gabor Function, 2D-LGF) function is defined as follows:

[0118]

[0119] where ρ and θ represent the polar radius and polar angle respectively, s and o represent the scale coefficient and angle coefficient respectively, which are preset, (ρ s , θ (s,o) ) represents the center frequency, σ ρ and σ θ are bandwidths in different forms, that is, the bandwidths corresponding to the polar radius and polar angle. By performing the inverse Fourier transform on the two-dimensional Log-Gabor function, the spatial domain filter corresponding to the two-dimensional Log-Gabor function is obtained, and its expression form is as follows:

[0120] L(x, y, s, o) = L even (x, y, s, o) + iL odd (x, y, s, o)

[0121] where L even (x, y, s, o) and L odd (x, y, s, o) are the even-symmetric wavelet expression form and odd-symmetric wavelet expression form of the spatial domain filter respectively.

[0122] For an image, the response components at different angles and scales can be obtained by convolving with even and odd symmetric wavelets. Specifically, the reference image is convolved with the dual symmetric wavelet and the odd symmetric wavelet respectively to obtain the response components of the reference image at different angles and scales:

[0123] e so (x, y) = I(x, y) * L even (x, y, s, o)

[0124] o so (x, y) = I(x, y) * L odd (x, y, s, o)

[0125] Among them, e so (x, y) is the scale response component of the reference image f(x, y), and o so (x, y) is the angular response component of the reference image f(x, y). Therefore, the amplitude component A so (x, y) and the phase component φ so (x, y) at the scale and direction of the reference image can be obtained by using the scale response component and the angular response component, and their definitions are as follows:

[0126]

[0127]

[0128] In addition, considering that the image usually contains noise interference, in order to obtain more stable image features, a noise compensation factor is introduced, and the PC (phase congruency) information of each pixel point in the reference image is calculated by the following formula:

[0129]

[0130] where w0(x, y) is the frequency diffusion weight factor, ξ is a very small constant factor to prevent the denominator from being 0, T is the noise suppression factor, and Δφ so (x, y) is the phase deviation function. A so (x, y)Δφ so (x, y) is expressed as follows:

[0131]

[0132] where, E(x, y) represents the local energy of the image,

[0133] Similarly, the phase congruency information of each pixel point in the real-time image is obtained by using the above process.

[0134] S3.2: Obtain the phase congruency structure diagrams of the reference image and the real-time image respectively by using the phase congruency information.

[0135] Specifically, add the phase congruency information of different directions of the obtained reference image to obtain the phase congruency structure diagram of the reference image; add the phase congruency information of different directions of the obtained real-time image to obtain the phase congruency structure diagram of the real-time image.

[0136] S3.3: Calculate the phase congruency direction diagrams integrating multiple scale information of the reference image and the real-time image by using the Log-Gabor filter.

[0137] Specifically, according to the angular response component o so (x, y), a phase consistency direction map integrating multiple scale information is calculated.

[0138]

[0139]

[0140]

[0141] Among them, represents the phase consistency direction map of the reference image or the real-time image, m represents the energy in the horizontal direction of the phase consistency, n represents the energy in the vertical direction of the phase consistency, and o so (θ) represents the angular response component in the direction θ, and arctan(·) represents the arctangent function.

[0142] S4: Obtain the phase consistency feature descriptors of the reference image and the real-time image respectively by using the phase consistency direction maps of the reference image and the real-time image and the phase consistency structure map.

[0143] Specifically, calculate the phase consistency feature histogram by using a generated direction map integrating multiple scale information and a phase consistency structure map integrating multiple direction information, and then normalize the phase consistency feature histogram to obtain the phase consistency feature descriptor, which overcomes the influence of SAR speckle noise interference and the non-linear radiation difference between images and has strong robustness.

[0144] S5: Obtain the approximate position of the feature points of the real-time image according to the feature point information and the projection transformation matrix between the reference image and the real-time image. Specifically, the approximate position of the feature points of the real-time image = the feature point information of the reference image × the projection transformation matrix.

[0145] In this embodiment, the projection transformation matrix from the reference image to the real-time image is calculated by using the preprocessed control points. The preprocessed control points include the coordinate positions of the pixel points of the reference image and the coordinate positions of the pixel points of the real-time image, which can be selected manually or obtained by other registration algorithms.

[0146] S6: Obtain the position error of the feature points of the real-time image through similarity measurement, and combine the approximate position of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image.

[0147] Specifically, step S6 of this embodiment includes:

[0148] First, a corresponding template window is taken around each feature point of the real-time image, and a sliding window search is performed using a search window in the corresponding region of interest in the real-time image to obtain the sum of mean squared errors between the template window i and each search window in the region of interest.

[0149] Specifically, a corresponding template window is taken around each feature point of the real-time image, a sliding window search is performed in the corresponding region of interest in the real-time image, and fast Fourier transform (FFT) is used for acceleration processing to greatly shorten the algorithm time consumption, and finally the best matching point is found. It should be noted that the region of interest can be the entire image or a region circled manually.

[0150] In this embodiment, the sum of mean squared errors (MSD) method is used for similarity measurement, and the specific formula is defined as follows:

[0151]

[0152] Among them, C1 and C2 respectively represent the phase consistency descriptors of the reference image template window and the real-time image search window, x represents the coordinate of the pixel point in the reference image, and T i () is a masking function, when x is the pixel point in the template window, T i (x) = 1, otherwise T i (x) = 0. M i (v) represents the MSD value between the template window i and each search window in the region of interest. Since there are several search windows in a region of interest, M i (v) is in the form of a one-dimensional vector. By minimizing M i (v), the registration window position corresponding to this template window can be found. Therefore, the matching function between the template window on the reference image and the search window on the real-time image is defined as follows:

[0153]

[0154] Substituting the matching function into the sum of mean squared errors MSD formula gives the following formula:

[0155]

[0156] Among them, the first term is a constant, and the second and third terms are variables. Their calculation can be accelerated by FFT for the latter two terms in the above formula to obtain the position error of the real-time image feature point. The specific formula is as follows:

[0157]

[0158] Among them, v i represents the position error, i represents the template window, F(·), F -1(·) and F * (·) represent the Fourier transform, the inverse Fourier transform, and the complex conjugate of the Fourier transform, respectively.

[0159] Subsequently, the position error is added to the rough position of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image.

[0160] S7: Use the Random Sample Consensus (RANSAC) algorithm to eliminate the mismatched points in the accurate position information of the feature points of the real-time image to obtain the final matching result

[0161] Next, the feasibility of the heterologous image registration method of the embodiment of the present invention is verified through a simulation algorithm. The registration results between different types of images are obtained respectively.

[0162] Please refer to Figures 3 to 5 , Figure 3 which are the registration results of visible light images and SAR images obtained by using the heterologous image registration method of the embodiment of the present invention, where Figure 3 (a) is the connection diagram of the registration result, Figure 3 (b) is the fusion diagram, Figure 3 (c) is the checkerboard splicing diagram. Figure 4 are the registration results of visible light images and infrared images obtained by using the heterologous image registration method of the embodiment of the present invention, Figure 4 (a) is the connection diagram of the registration result, Figure 4 (b) is the fusion diagram, Figure 4 (c) is the checkerboard splicing diagram; Figure 5 are the registration results of SAR images and SAR images obtained by using the heterologous image registration method of the embodiment of the present invention, Figure 5 (a) is the connection diagram of the registration result, Figure 5 (b) is the fusion diagram, Figure 5 (c) is the checkerboard splicing diagram. It can be seen from Figures 3 to 5 that whether it is the registration of visible light images and SAR images, the registration of visible light images and infrared images, or the registration of SAR images and SAR images, the method of the embodiment of the present invention can obtain a large number of registration point pairs, the fusion diagram has a clear structure, conforms to the human visual effect, and the checkerboard splicing effect is good.

[0163] Furthermore, the Root Mean Square Error (RMSE) and running time of the above three images are simulated and compared using the RIFT algorithm and the method of the embodiment of the present invention, as shown in Table 1. It can be seen from Table 1 that the method of the embodiment of the present invention obtains a smaller root mean square error, better registration effect, and shorter running time. Thus, the effectiveness of the method proposed in the embodiment of the present invention can be verified.

[0164] Table 1 Comparison results of using the RIFT algorithm and the method of the embodiment of the present invention

[0165] RIFT The method of the present invention RMSE / pixel Time / t RMSE / pixel Time / t The first image 1.8839 13.4244 0.3820 3.4478 The second image 0.8703 12.5505 0.5944 4.7475 The third image 1.9138 10.7921 0.4219 2.8901

[0166] An embodiment of the present invention proposes a method for registering heterogeneous images based on multi-scale image gradients and phase features. First, the gradient information of the images is used to extract the feature information of the images, which is not affected by the gray-scale changes and illumination differences between heterogeneous images, so as to obtain feature points with strong robustness and stability. Secondly, for the reference image and the real-time image, their phase information is used to generate a phase consistency direction map integrating multiple scale information and a phase consistency structure map integrating multiple direction information, and then a phase consistency feature histogram is constructed to overcome the interference of SAR speckle noise and the non-linear radiation differences between images, with strong robustness and more significance for the feature representation of images. Finally, the mean squared error (MSD) algorithm is used for similarity measurement, and the fast Fourier transform is used to accelerate the MSD algorithm to improve the operation efficiency, obtaining a rough registration result, and then the FSC algorithm is used to remove the corresponding points with large errors to obtain the final fine registration result.

[0167] The overall idea of the embodiment of the present invention is different from the traditional method for registering heterogeneous images by detecting and describing features through single intensity, gradient or phase information of the images. Instead, it fully combines and utilizes the gradient and phase consistency information of the images to extract stable and sufficient feature points, obtain a feature descriptor with a high degree of consistency, and use the fast Fourier transform to accelerate the implementation of the MSD algorithm to reduce the time complexity of the algorithm.

[0168] In several embodiments provided by the present invention, it should be understood that the devices and methods disclosed by the present invention can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0169] In addition, each functional module in various embodiments of the present invention can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0170] Another embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the method for registering heterogeneous images based on multi-scale image gradients and phase features described in the above embodiments. Another aspect of the present invention provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the method for registering heterogeneous images based on multi-scale image gradients and phase features described in the above embodiments are implemented. Specifically, the above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes several instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0171] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for registering heterologous images based on multi-scale image gradients and phase features, characterized in that Including: S1: Obtain the gradient magnitude map of the reference image using an improved Sobel operator; S2: Perform non-maximum suppression processing on the gradient magnitude map to obtain the feature point information of the reference image; S3: Construct a phase consistency direction map and a phase consistency structure map for the reference image and the real-time image to be registered respectively; S4: Use the phase consistency direction maps of the reference image and the real-time image and the phase consistency structure map to obtain the phase consistency feature descriptors of the reference image and the real-time image respectively; S5: Obtain the approximate positions of the feature points of the real-time image according to the feature point information and the projection transformation matrix between the reference image and the real-time image; S6: Obtain the position error of the feature points of the real-time image through similarity measurement, and combine it with the approximate positions of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image; S7: Use the Random Sample Consensus (RANSAC) algorithm to eliminate the mismatched points in the accurate position information of the feature points of the real-time image to obtain the final matching result.

2. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 1, wherein The S1 includes: S1.1: Construct horizontal templates, vertical templates, and templates in two diagonal directions: where h1 and h2 represent the horizontal template and the vertical template respectively, and h3 and h4 represent the templates in two diagonal directions respectively; S1.2: Obtain the first-order derivatives in the horizontal direction, vertical direction, and two diagonal directions of each pixel point in the reference image, which are D1(x, y), D2(x, y), D3(x, y), and D4(x, y) respectively: D1(x, y) = h1 * f(x, y) D2(x, y) = h2 * f(x, y) D3(x, y) = h3 * f(x, y) D4(x, y) = h4 * f(x, y) Among them, * represents a convolution operation, and f(x, y) represents a reference image; S1.3: Obtain the gradient magnitude map of the reference image: F(x, y) = |D1(x, y)| + |D2(x, y)| + |D3(x, y)| + |D4(x, y)|.

3. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 1, characterized in that, The S3 includes: S3.1: Use a Log-Gabor filter to obtain the phase consistency information of each pixel point in the reference image and the real-time image at different angles and scales; S3.2: Use the phase consistency information to obtain the phase consistency structure maps of the reference image and the real-time image respectively; S3.3: Use a Log-Gabor filter to calculate the phase consistency direction map that integrates multiple scale information of the reference image and the real-time image.

4. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 3, wherein The S3.1 includes: S3.11: Obtain a two-dimensional Log-Gabor function; where ρ and θ represent the radial distance and the polar angle respectively, s and o represent the scale factor and the angle factor respectively, (ρ s , θ (s,o) ) represents the center frequency, σ ρ and σ θ are bandwidths in different forms; S3.12: Perform an inverse Fourier transform on the two-dimensional Log-Gabor function to obtain the spatial domain filter corresponding to the two-dimensional Log-Gabor function; L(x,y,s,o) = L even (x,y,s,o) + iL odd (x,y,s,o) Among them, L even (x, y, s, o) and L odd (x, y, s, o) are respectively the even-symmetric wavelet expression form and the odd-symmetric wavelet expression form of the spatial domain filter; S3.13: Convolve the reference image with a dual-symmetric wavelet and an odd-symmetric wavelet respectively to obtain the response components of the reference image at different angles and scales; e so (x,y) = f(x,y) * L even (x,y,s,o) o so (x, y) = f(x, y) * L odd (x, y, s, o) Among them, e so (x, y) is the scale response component of the reference image f(x, y), and o so (x, y) is the angle response component of the reference image f(x, y); S3.14: Obtain the amplitude component A<(x,y)> and the phase component φ<(x,y)> of the reference image at the scale and orientation using the scale response component and the angle response component so (x,y) so (x,y): S3.15: Use the amplitude component and the phase component to obtain the phase consistency information of each pixel point in the reference image; where \(w_0(x,y)\) is the frequency diffusion weight factor, \(\xi\) is the constant factor, \(T\) is the noise suppression factor, and \(\Delta\varphi so (x,y)\) is the phase deviation function; S3.16: Obtain the phase consistency information of each pixel point in the real-time image according to S3.11 - S3.

15.

5. The method for registering heterogeneous images based on multi-scale image gradient and phase features according to claim 4, wherein In the said S3.15, A so (x, y)Δφ so (x, y) is expressed as: Among them, E(x, y) represents the local energy of the image, 6. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 3, wherein The S3.2 includes: S3.21: Add the phase consistency information of the reference image in different directions to obtain the phase consistency structure diagram of the reference image; S3.22: Add the phase consistency information of the real-time image in different directions to obtain the phase consistency structure diagram of the real-time image.

7. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 4, characterized in that The S3.3 includes: According to the angular response component o so (x, y), the phase consistency direction map integrating multiple scale information is calculated; Among them, represents the phase consistency orientation map of the reference image or the real-time image, m represents the energy in the horizontal direction of the phase consistency, n represents the energy in the vertical direction of the phase consistency, and o so (θ) represents the angular response component in the direction θ, and arctan(·) represents the arctangent function.

8. The method for registering heterogeneous images based on multi-scale image gradients and phase features according to claim 1, characterized in that The S6 includes: S6.1: Take a corresponding template window around each feature point of the real-time image, perform a sliding window search using the search window in the corresponding region of interest of the real-time image, and obtain the sum of mean squared error values between the template window i and each search window in the region of interest: Among them, C1 and C2 respectively represent the phase consistency descriptors of the reference image template window and the real-time image search window, x represents the coordinates of the pixel points in the reference image, and T i () is a masking function. When x is a pixel point in the template window, T i (x) = 1, otherwise T i (x) = 0; M i (v) represents the mean sum of squared errors between the template window i and each search window in the region of interest; S6.2: Define the matching function between the template window on the reference image and the search window on the real-time image: S6.3: Substitute the matching function into the sum of mean squared error formula to obtain: where the first term is a constant, and the second and third terms are variables; S6.4: Perform Fourier transform on the last two terms of the formula in S6.3 to obtain the position error of the feature points of the real-time image: Among them, v i represents the position error, i represents the template window, F(·), F -1 (·) and F * (·) represent the Fourier transform, the inverse Fourier transform, and the complex conjugate of the Fourier transform, respectively; S6.5: Add the position error to the rough position of the feature points of the real-time image to obtain the accurate position information of the feature points of the real-time image.

Images