Multimodal remote sensing image registration method based on multi-scale 3D frequency domain convolution features

Abstract

The present invention discloses a multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features, including: obtaining a reference image and an image to be registered; using a spatial domain filter based on a two-dimensional Log-Gabor function to measure phase consistency of each image, and obtaining phase consistency response information and convolution response values ​​at different angles and scales; calculating a phase consistency stacking moment image based on the phase consistency response information; accumulating convolution response values ​​at different scales at the same angle to obtain a three-dimensional frequency domain convolution response channel map; performing feature point detection on the phase consistency stacking moment image, and using the feature point position information as the feature point position information of the three-dimensional frequency domain convolution response channel map; and registering the reference image and the image to be registered based on the three-dimensional frequency domain convolution response channel map and its feature point position information. The present invention can achieve high-precision and fast registration of multi-mode remote sensing images.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features. Background Art

[0002] In recent years, sensor imaging technology has become increasingly mature, and a large amount of image data has emerged. Remote sensing images have become one of the effective means for people to observe, describe and analyze surface features due to their convenience and intuitiveness. In this context, facing multi-sensor images with different modes, different spectral bands, different resolutions and different phases, how to integrate and use the information they contain has become a hot topic in the field of image processing.

[0003] Image registration technology is an important link in realizing the integration of heterogeneous image information. It is widely used in military and civilian fields such as missile positioning, aviation guidance, computer vision, pattern recognition, remote sensing technology, medicine, climatology, etc. Among them, the registration technology of heterogeneous images can make up for the shortcomings of single sensor images in image registration technology, and is a research hotspot in the current field of image registration technology. This technology refers to the spatial alignment of images corresponding to the same scene with resolution differences, perspective differences, phase differences and radiation differences. Among them, radiation differences and noise interference are the main factors affecting the performance of heterogeneous image registration, and are also the key and difficult points in this technical field. In the study of heterogeneous image registration, how to extract stable homonymous features between the reference image and the real-time image is the key to whether heterogeneous image registration can be successfully achieved.

[0004] In the prior art, heterogeneous image registration is mainly achieved based on the similarity measurement between the spatial features of heterogeneous images, where the typical spatial features are the gradient features of the image. The point feature extraction operator is used to obtain the respective feature point sets from the heterogeneous images, and then a local descriptor is constructed for each feature point. Based on the descriptors of the feature points, the same-name features with effective similarity mapping relationships are found between the heterogeneous images, thereby achieving heterogeneous image registration.

[0005] However, spatial features such as image gradient features show instability for heterogeneous images with radiation differences, resulting in unsatisfactory registration effects in some registration scenarios (such as multi-modal remote sensing image registration). Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features.

[0007] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] A multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features, comprising:

[0009] Acquire a multi-mode remote sensing image; the multi-mode remote sensing image includes a reference image and an image to be registered;

[0010] A spatial domain filter based on a two-dimensional Log-Gabor function is used to measure the phase consistency of the reference image and the image to be registered, respectively, to obtain phase consistency response information and convolution response values ​​of each image at different angles and scales;

[0011] Based on the phase consistency response information of the image, the minimum moment and the maximum moment of the image are calculated, and the minimum moment and the maximum moment are superimposed to obtain a phase consistency superposition moment image corresponding to the image;

[0012] Among the convolution response values ​​of each image at different angles and scales, the convolution response values ​​at the same angle and scale are accumulated to obtain the three-dimensional frequency domain convolution response channel map corresponding to the image;

[0013] Performing feature point detection on the phase consistency superposition moment image corresponding to the image, and using the detected feature point position information as the feature point position information of the three-dimensional frequency domain convolution response channel map corresponding to the image;

[0014] The reference image and the image to be registered are registered based on the three-dimensional frequency domain convolution response channel map and the feature point position information of the three-dimensional frequency domain convolution response channel map.

[0015] Optionally, the spatial domain filter based on the two-dimensional Log-Gabor function is constructed in the following manner:

[0016] Construct a two-dimensional Log-Gabor frequency filter;

[0017] Performing inverse Fourier transform on the two-dimensional Log-Gabor frequency filter to obtain a spatial domain filter based on the two-dimensional Log-Gabor function.

[0018] Optionally, the adopting of a spatial domain filter based on a two-dimensional Log-Gabor function to perform phase consistency measurement on the reference image and the image to be registered respectively, to obtain phase consistency response information and convolution response values ​​of each image at different angles and scales, includes:

[0019] For each of the reference image and the image to be registered, convolve the image with the even-symmetric Log-Gabor wavelet and the odd-symmetric Log-Gabor wavelet of the spatial domain filter respectively to obtain a first response component corresponding to the even-symmetric Log-Gabor wavelet and a second response component corresponding to the odd-symmetric Log-Gabor wavelet;

[0020] Calculating the amplitude components of the image at different angles and scales using the first response component and the second response component as the convolution response values ​​of the image at different angles and scales;

[0021] Substituting the first response component, the second response component and the amplitude component into a preset two-dimensional phase consistency model to calculate phase consistency response information of the image at different angles and scales;

[0022] The two-dimensional phase consistency model is:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] Where I(x, y) represents the image, PC(x, y) represents the phase consistency response information of the image I(x, y), and A so (x, y) represents the amplitude component of the image I(x, y) at scale s and direction o, e so (x, y) represents the first response component of the image I (x, y) at scale s and direction o, so (x, y) is the second response component of the image I(x, y) at scale s and direction o, w0(x, y) is a preset weighting factor for frequency expansion, T is a preset noise compensation term, ξ is a constant used to prevent the denominator of the fraction from being zero, Indicates A so (x,y)Δφ so (x,y)-T is rounded down.

[0029] Optionally, feature point detection is performed on a phase consistency superposition moment image corresponding to the image, including:

[0030] The block feature detection method is used to detect feature points on the phase consistency superposition moment image corresponding to the image.

[0031] Optionally, a block feature detection method is used to detect feature points on a phase consistency superposition moment image corresponding to the image, including:

[0032] Divide the phase consistency stacking moment image corresponding to the image into a plurality of image blocks of the same size;

[0033] For each of the image blocks, the Harris corner point detection operator is used to calculate the Harris response value of each pixel, and the first n pixels with the largest Harris response values ​​exceeding a preset threshold are taken as feature points; n is a preset positive integer.

[0034] Optionally, based on the three-dimensional frequency domain convolution response channel map and the feature point position information of the three-dimensional frequency domain convolution response channel map, registering the reference image and the image to be registered includes:

[0035] According to the position information of each feature point of the first auxiliary image, a template window is constructed respectively, and each template window is used to search for points with the same name in the second auxiliary image to obtain a set of matching point pairs;

[0036] Among them, the first auxiliary image is a three-dimensional frequency domain convolution response channel map corresponding to the reference image, and the second auxiliary image is a three-dimensional frequency domain convolution response channel map corresponding to the image to be matched.

[0037] Optionally, using each template window to perform a sliding window search in the second auxiliary image for points of the same name includes:

[0038] Using each template window to search for points of the same name in the region of interest of the second auxiliary image;

[0039] The region of interest is obtained by expanding the template window.

[0040] Optionally, the step of using each template window to perform a sliding window search in the second auxiliary image for points of the same name includes:

[0041] For each feature point in the first auxiliary image, the template window constructed for the feature point, the first auxiliary image, and the second auxiliary image are substituted into a preset FFT accelerated matching optimization problem for iterative solution to obtain a point of the same name in the second auxiliary image that matches the feature point;

[0042] The FFT acceleration matching optimization problem is:

[0043]

[0044] Wherein, C1 represents the first auxiliary image, C2 represents the second auxiliary image; F(·) represents Fourier transform, F -1 (·) represents the inverse Fourier transform, F * represents the complex conjugate of Fourier transform; T i (x) represents the template window, x represents the pixel point; v is the temporary point with the same name in the iterative solution process, v i Represents the point with the same name as the i-th feature point obtained by iterative solution.

[0045] Optionally, constructing a prerequisite for the FFT accelerated matching optimization problem includes: using a sum of squared differences (SSD) to measure the similarity between the first auxiliary image and the second auxiliary image.

[0046] Optionally, the multi-mode remote sensing image includes a visible light image and a SAR image, or includes a visible light image and an infrared image.

[0047] In the multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features provided by the present invention, the multi-mode remote sensing image is a heterogeneous image pair. The present invention uses a spatial domain filter based on a two-dimensional Log-Gabor function to measure the phase consistency of the reference image and the image to be registered, respectively, and obtains the phase consistency response information and convolution response value of each image at different angles and scales; since the phase consistency information of the heterogeneous image is not affected by grayscale differences, radiation differences, and illumination differences, it has strong stability and robustness, and is therefore a feature representation that is better than the spatial domain features of the image. In addition, the present invention further uses the phase consistency response information of the image to construct a phase consistency superposition moment image corresponding to the image, thereby facilitating the use of an image feature point detection method to extract strong robust features that are not affected by grayscale differences, radiation differences, and illumination differences, thereby determining the feature point position information. Then, the convolution response values ​​at the same angle and different scales of the image are accumulated to obtain a three-dimensional frequency domain convolution response channel map. The three-dimensional frequency domain convolution response channel map is used to refer to the determined feature point position information to realize the registration of the reference image and the image to be registered. This is more accurate than directly using the spatial domain features detected from heterogeneous images to achieve image registration.

[0048] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features provided by an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of generating a phase consistency superposition moment image in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of performing a sliding window search in an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of the main process of realizing multi-mode remote sensing image registration in an embodiment of the present invention;

[0053] Figure 5 is a set of multimodal remote sensing images to be registered;

[0054] Figure 6(a) to Figure 6(c) The connection diagrams of the same-name points are respectively realized by using the HOPC (Histogram of Orientated Phase Congruency) algorithm, the CFOG (Chanel Feature of Orientated Gradient) algorithm and the method provided in the embodiment of the present invention to realize the registration;

[0055] FIG6( d ) is a chessboard mosaic image registered using the method of an embodiment of the present invention;

[0056] Figure 7 is a set of multimodal remote sensing images to be registered;

[0057] Figure 8(a) to Figure 8(c) The connection diagrams of the same-name points are respectively realized by using the HOPC (Histogram of Orientated Phase Congruency) algorithm, the CFOG (Chanel Feature of Orientated Gradient) algorithm and the method provided in the embodiment of the present invention to realize the registration;

[0058] FIG8( d ) is a chessboard mosaic image registered using the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0060] In order to further improve the registration effect of heterogeneous images, the embodiment of the present invention provides a multi-modal remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features, see Figure 1 As shown, the method comprises the following steps:

[0061] S10: Acquire a multi-mode remote sensing image; the multi-mode remote sensing image includes a reference image and an image to be registered.

[0062] It can be understood that the reference image and the image to be registered are heterogeneous images obtained by photographing the same observation area using different sensors. When they are remote sensing images, they are also called multi-mode remote sensing images.

[0063] In practical applications, multi-mode remote sensing images may include visible light images and SAR images, wherein either one of the visible light images and the SAR images may be used as a reference image, and the other one may be used as an image to be registered. Alternatively, multi-mode remote sensing images may include visible light images and infrared images, wherein either one of the images may be used as a reference image, and the other one may be used as an image to be registered.

[0064] It is understandable that the multi-mode remote sensing images applicable to the embodiments of the present invention are not limited to the examples given here.

[0065] S20: A spatial domain filter based on a two-dimensional Log-Gabor function is used to measure the phase consistency of the reference image and the image to be registered, respectively, to obtain the phase consistency response information and convolution response value of each image at different angles and scales.

[0066] Among them, the spatial domain filter based on the two-dimensional Log-Gabor function can be constructed as follows:

[0067] (1) Construct a two-dimensional Log-Gabor frequency filter;

[0068] Here, the two-dimensional Log-Gabor frequency filter is defined by a two-dimensional Log-Gabor function (2D-LGF). The definition of the 2D-LGF is as follows:

[0069]

[0070] Where (ρ, θ) is the logarithmic polar coordinate; the subscript s represents the scale of the 2D-LGF, and the subscript o represents the direction of the 2D-LGF; the logarithmic polar coordinate (ρ s ,θ (s,o) ) is the center frequency of 2D-LGF; σ ρ and σ θ are the bandwidths denoted by ρ and θ respectively.

[0071] (2) Perform inverse Fourier transform on the two-dimensional Log-Gabor frequency filter to obtain a spatial domain filter based on the two-dimensional Log-Gabor function.

[0072] At this time, the spatial domain filter can be expressed as:

[0073] L(x,y,s,o)=L even (x,y,s,o)+iLodd (x,y,s,o);

[0074] Among them, L even (x, y, s, o) is the even-symmetric Log-Gabor wavelet of the spatial domain filter, L odd (x, y, s, o) is an odd-symmetric Log-Gabor wavelet, and i represents the imaginary unit.

[0075] In step S20, a spatial domain filter based on a two-dimensional Log-Gabor function is used to measure the phase consistency of the reference image and the image to be registered, respectively, to obtain phase consistency response information and convolution response values ​​of each image at different angles and scales, including:

[0076] (1) For each image in the reference image and the image to be registered, the image is convolved with the even-symmetric Log-Gabor wavelet and the odd-symmetric Log-Gabor wavelet of the spatial domain filter respectively to obtain the first response component corresponding to the even-symmetric Log-Gabor wavelet and the second response component corresponding to the odd-symmetric Log-Gabor wavelet.

[0077] Specifically, the operation performed in this step is expressed by the following formula:

[0078] e so (x,y)=I(x,y)*L even (x,y,s,o),

[0079] o so (x,y)=I(x,y)*L odd (x,y,s,o);

[0080] Among them, I(x,y) represents the image targeted in this step, e so (x, y) represents the first response component of the image I(x, y) at scale s and direction o. so (x, y) is the second response component of the image I(x, y) at scale s and direction o.

[0081] (2) The amplitude components of the image at different angles and scales are calculated using the first response component and the second response component as the convolution response values ​​of the image at different angles and scales.

[0082] The specific calculation method is as follows:

[0083]

[0084] Among them, A so(x, y) represents the amplitude component of the image I(x, y) at scale s and direction o, that is, the convolution response value of the image I(x, y) at scale s and direction o. In addition, the phase component φ of the image I(x, y) at scale s and direction o so (x,y) is calculated as follows:

[0085]

[0086] (3) Substituting the first response component, the second response component and the amplitude component into a preset two-dimensional phase consistency model, the phase consistency response information of the image at different angles and scales is calculated.

[0087] The two-dimensional phase consistency model is:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Wherein, PC(x,y) represents the phase consistency response information of the image I(x,y), w0(x,y) is the preset weighting factor for frequency expansion, T is the preset noise compensation term, and ξ is a constant used to prevent the denominator of the fraction from being zero. Δφ so (x, y) is a sensitive phase deviation function, which is not directly involved in the actual calculation, but directly calculates A so (x,y)Δφ so (x,y); Indicates A so (x,y)Δφ so (x,y)-T is rounded down, where the operator The purpose is to prevent A so (x,y)Δφ so (x,y)-T is a negative value, that is, when A so (x,y)Δφ so When (x,y)-T is positive, the result is equal to itself, otherwise it is zero; E(x,y) is a local energy function, and its function value is calculated using the last formula above. The interpretation of the remaining parameters can be found above.

[0094] Although noise behaves differently at different scales and angles, the amplitude of the spatial filter's response to noise is usually directly proportional to its bandwidth. Therefore, a noise compensation term T is introduced into the two-dimensional phase consistency model by comprehensively considering the bandwidth and spatial width of the spatial filter.

[0095] S30: Based on the phase consistency response information of the image, the minimum moment and the maximum moment of the image are calculated, and the minimum moment and the maximum moment are superimposed to obtain a phase consistency superposition moment image corresponding to the image.

[0096] Specifically, the calculation method of the minimum moment and the maximum moment is as follows:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Among them, M Ψ Represents the maximum moment of any pixel (x, y) in the image, m Ψ represents the minimum moment of the pixel (x, y), a, b, c are three intermediate quantities, PC(θ o ) is the image at θ o PC measurement under direction,

[0103] It can be understood that the minimum moment m of the pixel Ψ and the maximum moment M Ψ To perform superposition is to calculate Therefore, the pixel m Ψ The minimum moment diagram composed of the values ​​and the pixel M Ψ The maximum moment image composed of the values ​​is superimposed to obtain a phase consistency superposition moment image. The process principle is shown in Figure 2 shown.

[0104] S40: Accumulating the convolution response values ​​of each image at different angles and different scales and the convolution response values ​​at the same angle and different scales to obtain a three-dimensional frequency domain convolution response channel map corresponding to the image.

[0105] Specifically, the accumulation process is expressed by the following formula:

[0106]

[0107] Among them, N sis the total number of different scales.

[0108] S50: performing feature point detection on the phase consistency superposition moment image corresponding to the image, and using the detected feature point position information as the feature point position information of the three-dimensional frequency domain convolution response channel map corresponding to the image.

[0109] Here, there are multiple specific implementation methods for performing feature point detection on the phase consistency superposition moment image corresponding to the image.

[0110] Exemplarily, in one implementation, feature point detection of the phase consistency superposition moment image corresponding to the image may include: extracting feature points from the phase consistency superposition moment image using a preset feature extraction operator. Among them, the available feature extraction operators may include: Harris corner point detection operator, FAST feature detection operator, SIFT (Scale-invariant feature transform) feature detection operator or SURF (speed uprobust features) feature detection operator, etc.

[0111] In another implementation, in order to improve the detection efficiency, the block feature detection method can be used to detect feature points on the phase consistency superposition moment image corresponding to the image. In this way, feature point detection of multiple image blocks can be implemented in parallel, and the image blocks are used as separate images for feature point detection. Compared with the method of extracting feature points from the entire image, the former method is less likely to miss local feature points.

[0112] Specifically, the block feature detection method is used to detect feature points on the phase consistency superposition moment image corresponding to the image, which may include:

[0113] (1) Divide the phase consistency stacking moment image corresponding to the image into multiple image blocks of the same size;

[0114] (2) For each image block, the Harris corner detection operator is used to calculate the Harris response value of each pixel, and the first n pixels with the largest Harris response values ​​exceeding a preset threshold are taken as feature points; n is a preset positive integer.

[0115] It can be understood that the method of taking the first n pixels whose Harris response values ​​exceed the preset threshold as feature points can obtain more valuable points to be matched.

[0116] In addition, other feature extraction operators may also be used in step (2) to extract feature points from the image block, which is reasonable.

[0117] S60: Based on the three-dimensional frequency domain convolution response channel map and the feature point position information of the three-dimensional frequency domain convolution response channel map, register the reference image and the image to be registered.

[0118] Here, the three-dimensional frequency domain convolution response channel map corresponding to the reference image is used as the first auxiliary image, and the three-dimensional frequency domain convolution response channel map corresponding to the image to be matched is used as the second auxiliary image. There are many ways to achieve the registration of the reference image and the image to be registered based on the first auxiliary image, the second auxiliary image and the feature point position information of both.

[0119] Exemplarily, in one implementation, the similarity of every two feature points between the first auxiliary image and the second auxiliary image may be calculated respectively, using a calculation method such as Euclidean distance or Manhattan distance, etc. Then, each pair of feature pairs with a similarity greater than a preset similarity threshold is taken as a matching point pair, thereby obtaining a set of matching point pairs.

[0120] In another implementation, in order to reduce the amount of calculation, a sliding window search method can be used to achieve registration. Specifically, template windows are constructed according to the position information of each feature point of the first auxiliary image, and each template window is used to search for the same-name points in the second auxiliary image to obtain a set of matching point pairs; then, the reference image and the image to be registered are transformed based on the set of matching point pairs to complete the entire registration process.

[0121] Among them, the template window is constructed according to the position information of each feature point of the first auxiliary image. Specifically, a square neighborhood of a certain size around the feature point is selected as the neighborhood of the template window. It can be understood that the sliding window search process is to first find an initial search window with the same position and size as the first auxiliary image in the second auxiliary image, and then move the search window in the second auxiliary image in steps; wherein, each time a search window is stopped, it will be evaluated whether the image in the search window is similar to the image in the template window of the first auxiliary image, until a similar window is found, at which time the feature points in the window belonging to the second auxiliary image are the searched points of the same name.

[0122] In order to improve the registration efficiency and registration rate at the same time, using each template window to search for the same-name points in the second auxiliary image can include: using each template window to search for the same-name points in the region of interest in the second auxiliary image; wherein the region of interest is obtained by expanding the template window. Figure 3As shown in the figure, assuming that the radius of the template window constructed in the first auxiliary image is 50 pixels, the radius is expanded by 15 pixels at the same position in the second auxiliary image to obtain the region of interest. Then, a sliding window search is performed in the region of interest, and the search step can be set according to the actual image size, for example, the search step can be set to 1 pixel.

[0123] In actual engineering practice, the efficiency and accuracy of heterogeneous image registration algorithms are of equal value. However, existing algorithm research often focuses on registration accuracy and ignores the time complexity of the algorithm. Therefore, in order to overcome the problem that the general image registration algorithm takes too long and the registration accuracy is insufficient, the embodiment of the present invention adopts a method of accelerating matching through FFT. Specifically, in this method, each template window is used to search for the same-name points in the second auxiliary image by sliding the window, including:

[0124] For each feature point in the first auxiliary image, the template window constructed for the feature point, the first auxiliary image and the second auxiliary image are substituted into the preset FFT (Fast Fourier Transform) accelerated matching optimization problem for iterative solution to obtain the same-name point in the second auxiliary image that matches the feature point.

[0125] Among them, constructing the FFT accelerated matching optimization problem requires a prerequisite, including: using the sum of square differences (SSD) to measure the similarity between the first auxiliary image and the second auxiliary image. Specifically, the measurement formula is as follows:

[0126]

[0127] Wherein, C1 represents the first auxiliary image, C2 represents the second auxiliary image; x represents the coordinates of the pixel points of the image, T i (·) is a mask function. When x is a pixel in the template window, T i (x)=1, otherwise T i (x) = 0; S i (v) represents the SSD value calculated when sliding window search is performed in the region of interest using template window i, and v represents the feature point.

[0128] Since there are multiple search windows in a single region of interest, S i (v) is in the form of a one-dimensional vector, and the smallest element in the one-dimensional vector corresponds to the registration window searched from the second auxiliary image. Therefore, the matching function can be constructed as follows:

[0129]

[0130] Transforming the matching function, we get:

[0131]

[0132] It can be seen from the formula that the first term is a constant, and the second and third terms are variables. Therefore, we only need to minimize the last two terms to implement the process of sliding window search for points of the same name by constructing an optimization problem. In addition, FFT is used to accelerate the optimization problem, and the FFT accelerated matching optimization problem is obtained as follows:

[0133]

[0134] Where C1 represents the first auxiliary image, C2 represents the second auxiliary image; F(·) represents Fourier transform, F -1 (·) represents the inverse Fourier transform, F * represents the complex conjugate of Fourier transform; T i (x) represents the template window, x represents the pixel point; v is the temporary point with the same name in the iterative solution process, v i Represents the point with the same name as the i-th feature point obtained by iterative solution.

[0135] In a preferred embodiment, the execution process of the multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features provided by the embodiment of the present invention can be found in Figure 4 shown.

[0136] In addition, in the above-mentioned method of realizing registration through sliding window search, if there is not only a translation transformation but also a scale difference and a rotation difference between the reference image and the image to be registered, the image to be registered or the reference image can be pre-corrected for geometric distortion, thereby reducing the influence of the scale difference and the rotation difference on the subsequent registration of the two. The specific geometric correction method is a common technical means in the field of image processing, and the embodiments of the present invention will not be described in detail.

[0137] In the multi-mode remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features provided by the embodiment of the present invention, the multi-mode remote sensing image is a heterogeneous image pair. The present invention uses a spatial domain filter based on a two-dimensional Log-Gabor function to measure the phase consistency of the reference image and the image to be registered, respectively, and obtains the phase consistency response information and convolution response value of each image at different angles and scales; since the phase consistency information of the heterogeneous image is not affected by grayscale differences, radiation differences, and illumination differences, it has strong stability and robustness, and is therefore a better feature representation than the spatial domain features of the image. In addition, the embodiment of the present invention further uses the phase consistency response information of the image to construct a phase consistency superposition moment image corresponding to the image, so as to facilitate the use of the image feature point detection method to extract strong robust features that are not affected by grayscale differences, radiation differences, and illumination differences, so as to determine the feature point position information. Then, the convolution response values ​​at the same angle and different scales of the image are accumulated to obtain a three-dimensional frequency domain convolution response channel map. The three-dimensional frequency domain convolution response channel map is used to refer to the determined feature point position information to realize the registration of the reference image and the image to be registered. This is more accurate than directly using the spatial domain features detected from heterogeneous images to achieve image registration.

[0138] Furthermore, in the embodiment of the present invention, FFT is used to accelerate the process of sliding window searching for points of the same name, so that the embodiment of the present invention can realize fast registration while realizing high-precision registration of multi-mode remote sensing images.

[0139] The beneficial effects of the embodiments of the present invention are further illustrated by using experimental data below.

[0140] Experiment 1: Select a set of multi-mode remote sensing images obtained by actual shooting in a certain area in China, such as Figure 5 As shown, it includes a SAR image and a visible light image, wherein the visible light image is the reference image and the SAR image is the image to be registered. Figure 5 As can be seen in the figure, the scene contains villages, farmlands, sand and other scenes. The scene structure is relatively complex, and the imaging quality of the captured SAR image is poor.

[0141] Figure 6(a) to Figure 6(c) The following are the connection diagrams of the same-name points for registration using the HOPC algorithm, the CFOG algorithm, and the method provided by the embodiment of the present invention. By comparison, it can be seen that the method provided by the embodiment of the present invention can find more same-name points, thereby achieving more accurate registration. Therefore, compared with the HOPC algorithm and the CFOG algorithm, the method proposed by the embodiment of the present invention has better structural feature expression capabilities.

[0142] Figure 6(d) is a chessboard mosaic image registered by the method of the embodiment of the present invention. It can be observed from the chessboard mosaic image that after the registration by the method of the embodiment of the present invention, the geometric distortion between the reference image and the image to be matched is well corrected, and the edge connection is smooth and complete without any fault phenomenon.

[0143] Experiment 2: A set of multimodal remote sensing images was selected from the open source dataset, such as Figure 7 As shown, it includes infrared images and visible light images, wherein the visible light image is the reference image and the infrared image is the image to be registered. The scene in this group of images is clear and the image quality is good.

[0144] Figure 8(a) to Figure 8(c) The figures are respectively the connection diagrams of the same-name points which are registered by the HOPC algorithm, the CFOG algorithm and the method of the embodiment of the present invention; it can be seen from the comparison that the method of the embodiment of the present invention can find more same-name points, thereby achieving more accurate registration, which once again proves that the method proposed in the embodiment of the present invention has better structural feature expression capability.

[0145] Figure 8(d) is a chessboard mosaic image registered by the method of the embodiment of the present invention. It can also be found from the chessboard mosaic image that after the registration by the method of the embodiment of the present invention, the geometric distortion between the reference image and the image to be matched is well corrected, and the edge connection part is smooth and complete without fault phenomenon.

[0146] In order to make a more objective and accurate measurement of the performance of the above algorithms, the following analysis is conducted by comparing the number of same-name point pairs, RMSE (root mean square error) values, and registration time obtained by registering the same set of image data using different algorithms; the unit of RMSE is pixel, and the unit of registration time is second. The results are shown in Table 1 below:

[0147] Table 1

[0148]

[0149] By comparison, it can be seen that since the embodiment of the present invention adopts the FFT accelerated registration strategy, the time consumption is shorter than the other two methods, especially significantly shorter than the HOPC algorithm. In addition, the registration method provided by the embodiment of the present invention can obtain more homonymous points, and the error is relatively small.

[0150] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an image processing device, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0151] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0152] The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0153] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the present application for which protection is sought, a person skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims.

[0154] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A multi-modal remote sensing image registration method based on multi-scale three-dimensional frequency domain convolution features, characterized in that: include: Acquire multi-mode remote sensing images; The multi-mode remote sensing image includes a reference image and an image to be registered; A spatial domain filter based on a two-dimensional Log-Gabor function is used to measure the phase consistency of the reference image and the image to be registered, respectively, to obtain phase consistency response information and convolution response values ​​of each image at different angles and scales; Based on the phase consistency response information of the image, the minimum moment and the maximum moment of the image are calculated, and the minimum moment and the maximum moment are superimposed to obtain a phase consistency superposition moment image corresponding to the image; Among the convolution response values ​​of each image at different angles and scales, the convolution response values ​​at the same angle and scale are accumulated to obtain the three-dimensional frequency domain convolution response channel map corresponding to the image; Performing feature point detection on the phase consistency superposition moment image corresponding to the image, and using the detected feature point position information as the feature point position information of the three-dimensional frequency domain convolution response channel map corresponding to the image; Based on the three-dimensional frequency domain convolution response channel map and the feature point position information of the three-dimensional frequency domain convolution response channel map, registering the reference image and the image to be registered includes: Template windows are constructed respectively according to the position information of each feature point of the first auxiliary image, and each template window is used to search for the same-name points in the second auxiliary image to obtain a set of matching point pairs; wherein the first auxiliary image is a three-dimensional frequency domain convolution response channel map corresponding to the reference image, and the second auxiliary image is a three-dimensional frequency domain convolution response channel map corresponding to the image to be registered.

2. The method according to claim 1, characterized in that: The spatial domain filter based on the two-dimensional Log-Gabor function is constructed in the following way: Construct a two-dimensional Log-Gabor frequency filter; Performing inverse Fourier transform on the two-dimensional Log-Gabor frequency filter to obtain a spatial domain filter based on the two-dimensional Log-Gabor function.

3. The method according to claim 1, characterized in that The phase consistency measurement is performed on the reference image and the image to be registered respectively using a spatial domain filter based on a two-dimensional Log-Gabor function to obtain phase consistency response information and convolution response values ​​of each image at different angles and scales, including: For each of the reference image and the image to be registered, convolve the image with the even-symmetric Log-Gabor wavelet and the odd-symmetric Log-Gabor wavelet of the spatial domain filter respectively to obtain a first response component corresponding to the even-symmetric Log-Gabor wavelet and a second response component corresponding to the odd-symmetric Log-Gabor wavelet; Calculating the amplitude components of the image at different angles and scales using the first response component and the second response component as the convolution response values ​​of the image at different angles and scales; Substituting the first response component, the second response component and the amplitude component into a preset two-dimensional phase consistency model to calculate phase consistency response information of the image at different angles and scales; The two-dimensional phase consistency model is: Where I(x, y) represents the image, PC(x, y) represents the phase consistency response information of the image I(x, y), and A so (x, y) represents the amplitude component of the image I(x, y) at scale s and direction o, e so (x, y) represents the first response component of the image I (x, y) at scale s and direction o, so (x, y) is the second response component of the image I(x, y) at scale s and direction o, w0(x, y) is a preset weighting factor for frequency expansion, T is a preset noise compensation term, ξ is a constant used to prevent the denominator of the fraction from being zero, Indicates A so (x,y)Δφ so (x,y)-T is rounded down.

4. The method according to claim 1, characterized in that: Perform feature point detection on the phase consistency superposition moment image corresponding to the image, including: The block feature detection method is used to detect feature points on the phase consistency superposition moment image corresponding to the image.

5. The method according to claim 4, characterized in that The block feature detection method is used to detect feature points on the phase consistency superposition moment image corresponding to the image, including: Divide the phase consistency stacking moment image corresponding to the image into a plurality of image blocks of the same size; For each of the image blocks, the Harris corner point detection operator is used to calculate the Harris response value of each pixel, and the first n pixels with the largest Harris response values ​​exceeding a preset threshold are taken as feature points; n is a preset positive integer.

6. The method according to claim 1, characterized in that The method of using each template window to search for points of the same name in the second auxiliary image by sliding a window includes: Using each template window to search for points of the same name in the region of interest of the second auxiliary image; The region of interest is obtained by expanding the template window.

7. The method according to claim 1, characterized in that The step of using each template window to search for points of the same name in the second auxiliary image comprises: For each feature point in the first auxiliary image, the template window constructed for the feature point, the first auxiliary image, and the second auxiliary image are substituted into a preset FFT accelerated matching optimization problem for iterative solution to obtain a point of the same name in the second auxiliary image that matches the feature point; The FFT acceleration matching optimization problem is: Wherein, C1 represents the first auxiliary image, C2 represents the second auxiliary image; F(·) represents Fourier transform, F -1 (·) represents the inverse Fourier transform, F * represents the complex conjugate of Fourier transform; T i (x) represents the template window, x represents the pixel point; v is the temporary point with the same name in the iterative solution process, v i Represents the point with the same name as the i-th feature point obtained by iterative solution.

8. The method according to claim 7, characterized in that The prerequisite for constructing the FFT accelerated matching optimization problem includes: using the sum of squared differences (SSD) to measure the similarity between the first auxiliary image and the second auxiliary image.

9. The method according to any one of claims 1 to 8, characterized in that: The multi-mode remote sensing image includes a visible light image and a SAR image, or includes a visible light image and an infrared image.

Images