Multi-modal Remote Sensing Image Hybrid Matching Method and System

Abstract

The present invention discloses a multi-modal remote sensing image hybrid matching method and system with multi-dimensional directional self-similar features. The present invention mainly includes a hybrid feature rough matching and a multi-dimensional directional self-similar feature fine matching. First, the hybrid feature rough matching is performed: the offset mean filtering method is used to quickly extract the self-similar features of the image, and the direction information of the self-similar features is used to describe the feature points. Through the hybrid feature rough matching, the affine transformation model between multi-modal images is estimated, and the initial affine transformation is performed on the images to be matched. In the second stage, the multi-dimensional directional self-similar feature fine matching is performed: a multi-dimensional directional self-similar template feature is constructed by using the multi-channel self-similarity map obtained in the hybrid feature rough matching stage, and through the constructed template feature thinning strategy, a three-dimensional Gaussian kernel is used to perform convolution enhancement on the directional self-similar features. Finally, a three-dimensional phase consistency measurement strategy is used to identify the high-precision matching homologous points.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing methods, and particularly relates to a multi-modal remote sensing image hybrid matching method and system with multi-dimensional directional self-similar features. Background Art

[0002] Multi-modal remote sensing image (MRSI) matching is a process of identifying corresponding points in images from different sensors and different modalities. MRSI matching can provide technical support for map calibration, precise positioning, feature extraction, target recognition, surface change monitoring, 3D reconstruction, and stereo vision, etc.

[0003] However, multi-modal remote sensing image registration faces problems such as sensor differences, modality differences, and geometric differences, and requires a combination of various technical means to solve. In the past few years, experts and scholars have conducted a large number of studies on MRSI matching and continuously improved algorithms to improve the accuracy and reliability of matching. Although people have made a lot of efforts in enhancing the robustness of multi-modal image matching, there are still challenges in balancing geometric invariance and high-precision registration. Traditional feature-based methods have certain advantages in solving the geometric transformation differences of MRSI, and can overcome geometric transformations such as rotation, scale, and translation of images through flexible feature descriptors. These methods are usually based on local feature points or feature descriptors.

[0004] Due to the large signal-to-noise ratio differences in MRSI, the accuracy of feature matching is often limited. Pixels with a higher signal-to-noise ratio may not be able to correctly extract features, resulting in matching failures or incorrect matches. On the other hand, template matching methods have advantages in the accuracy of identifying corresponding points. The template matching method compares a predefined template with the image to be matched at the pixel level to obtain highly accurate results. This method is particularly suitable for scenarios where accurate positioning of corresponding points is required. However, the matching strategy of calculating pixel by pixel in the template matching method often leads to high computational complexity and poor resistance to geometric transformations of images.

[0005] Therefore, to solve the problems of signal-to-noise ratio interference, non-linear radiation distortion, and geometric transformation differences in MRSI, the present invention proposes a multi-modal remote sensing image hybrid matching method based on multi-dimensional directional self-similar features to obtain high-precision matching. Summary of the Invention

[0006] The present invention proposes a multi-modal remote sensing image hybrid matching method with multi-dimensional directional self-similar features to solve the problem of high-precision matching of multi-modal remote sensing images.

[0007] The technical solution adopted by the present invention is: a multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features. The main core of this algorithm consists of two parts: rough matching of hybrid features (Steps 2-4) and fine matching of multi-dimensional directional self-similar features (Steps 5-8). The steps are as follows:

[0008] Step 1, initialize the calculation parameters for hybrid matching;

[0009] Step 2, use the offset mean filtering method to quickly extract the self-similar features of the reference image, generate a multi-channel self-similar feature map, and perform feature extraction and non-maximum suppression operations on the feature response map to obtain feature points;

[0010] Step 3, describe the feature points using the direction information of the self-similar features, successively complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors, and then perform the nearest neighbor distance ratio matching to output the initial corresponding relationship of the image pair;

[0011] Step 4, perform geometric transformation on the image to be matched using the initial corresponding relationship, and output the result of the transformed image to be matched;

[0012] Step 5, perform directional self-similar feature calculation on the reference image and the newly generated image to be matched, and output a multi-channel self-similar map;

[0013] Step 6, design an angle weighting strategy. For the obtained multi-channel self-similar map, calculate the horizontal gradient and vertical gradient channel by channel, and perform channel thinning to filter out duplicate features, and solve to obtain the multi-dimensional directional self-similar template features;

[0014] Step 7, after generating the multi-dimensional directional self-similarity template features, use a three-dimensional Gaussian convolution kernel to enhance the feature channels;

[0015] Step 8, use a block feature detector to detect the feature points of the reference image, solve the relative positions of the feature points in the newly generated image to be matched using the feature point joint unit matrix, then transform the enhanced template features from the spatial domain to the frequency domain, and use three-dimensional phase correlation as the similarity measure to accelerate the matching of template-like features. Finally, eliminate the gross errors to complete the matching of multimodal remote sensing images.

[0016] Further, the calculation parameters in Step 1 include the neighborhood radius, the thinning interval of multi-dimensional directional self-similar features, and the descriptor neighborhood window.

[0017] Further, in Step 2, for each pixel point on the image, extract its self-similar feature value, and count the n minimum values in its self-similar feature values. The feature response λ of point q is calculated as shown in Equation (1). The feature responses λ of all pixel points form a feature response map, and then local non-maximum suppression is performed on the feature response map to obtain the feature points;

[0018]

[0019] Among them, λ(q) represents the feature response λ at point q, represents the n smallest self-similarity values.

[0020] Furthermore, the specific implementation of step 3 is as follows;

[0021] (1) Calculation of the main direction of feature points: Select a fixed circular neighborhood centered on the feature point, and generate an orientation histogram based on the self-similarity eigenvalues to determine the main direction. The specific steps are as follows: First, evenly divide the histogram into K equal parts. Then, for the feature point P whose main direction is to be determined, uniformly sample K points from the boundary of the feature neighborhood with a radius of r from P. For these K points, count their self-similarity feature sequences, denoted as S1, S2…S K , Next, normalize the self-similarity feature sequence, and select the peak direction with a proportion of more than P% in the histogram as the main direction of the feature point, where both K and P are constants;

[0022] (2) Statistics of descriptor feature vectors: The descriptor neighborhood of the feature point is a circular area extracted from the multi-channel self-similarity map. At each pixel point in the descriptor neighborhood, generate an orientation index map by calculating the index value of the minimum self-similarity direction; finally, perform logarithmic polar coordinate gridding on the orientation index map, and generate a distribution histogram in each grid interval as the feature descriptor vector;

[0023] (3) Calculation of the initial correspondence: Use the nearest neighbor distance ratio matching strategy to determine the initial matching homologous points. At the same time, combine the random sample consensus algorithm to effectively eliminate false matches; where the nearest neighbor distance ratio matching strategy first calculates the Euclidean distance between any two descriptors of the two images; secondly, for each reference image descriptor, calculate the ratio of the minimum distance to the second minimum distance; finally, extract the descriptors with a distance ratio less than a certain threshold d and their nearest descriptors as matching pairs.

[0024] Furthermore, in step 5, first use offset mean filtering to quickly obtain the self-similarity map in the direction of the pixel point q(p,θ) on the image to be matched, including two steps of image cropping and mean filtering, and construct a central sub-image by cropping the original image and an offset sub-image After obtaining the sub-images, the self-similarity map S in the direction of the point q(p,θ) can be calculated by the following formula (2) q :

[0025]

[0026] In the formula, meanFilter(·) represents the mean filtering operation, p represents the pixel distance, θ represents the angle, and the value ranges from [0°, 180°). Pixel offset is performed on the newly generated image to be matched, and then bilinear interpolation is used to obtain the offset sub-image in any direction. The central sub-image is obtained by shrinking one pixel from the four edges of the original input image with point q as the center; the multi-channel self-similarity map can be expressed as C represents the number of channels, and N represents the number of pixels in the feature neighborhood. Since the self-similarity feature has symmetry, performing the offset mean filtering operation C = N / 2 times can obtain the multi-channel self-similarity map of the entire image.

[0027] Furthermore, the specific implementation method of step 6 is as follows;

[0028] First, for the obtained multi-channel self-similarity map calculate the horizontal gradient g x and the vertical gradient g y for each channel using the Sobel operator, as shown in Equation (3):

[0029]

[0030] where g x represents the horizontal gradient, g y represents the vertical gradient, S q represents the input single-channel self-similarity map, represents the convolution operation;

[0031] Then, calculate the gradient direction using Equation (4), and perform the angle weighting strategy on the multi-channel self-similarity map for each channel to construct the single-channel multi-dimensional directional self-similarity feature effectively enhancing the rotational invariance of the self-similarity template feature;

[0032]

[0033] where, represents the single-channel multi-dimensional directional self-similarity feature;

[0034] For the multi-channel self-similarity feature, perform the angle weighting strategy for each channel to construct the multi-channel multi-dimensional directional self-similarity feature In constructing the multi-channel self-similarity map, a circular feature neighborhood of N pixels with a radius of R is adopted, and the number of channels C = N / 2. Therefore, the multi-channel multi-dimensional directional self-similarity feature with a dimension of C is obtained. To improve the matching calculation efficiency of the template feature, a thinning strategy is adopted for the feature channels, and thinning is performed through Equation (6) below to obtain the w-dimensional template feature map

[0035]

[0036] Among them, w represents the dimension of the thinned multi-dimensional oriented self-similarity feature, C represents the dimension of the original multi-dimensional oriented self-similarity feature, and m is the sampling interval. The operation symbol represents rounding up.

[0037] Furthermore, in step 7, the Gaussian convolution kernel is a two-dimensional Gaussian kernel in the horizontal and vertical directions and a kernel d in the Z direction. z = [1, 3, 1] T , as shown in Equation (7):

[0038]

[0039] Among them, represents the feature after two-dimensional Gaussian filtering. represents the Gaussian kernel on the plane. represents the single-channel multi-dimensional oriented self-similarity feature, M σ (x, y) represents the template feature after three-dimensional Gaussian filtering, d z is the Gaussian kernel in the z direction. represents the convolution operation, and σ is the standard deviation of the Gaussian convolution kernel.

[0040] Finally, first perform a normalization operation on the multi-dimensional oriented self-similarity feature, and then combine the features to form a multi-dimensional oriented self-similarity feature map, as shown in Equation (8):

[0041]

[0042] Among them, w represents the dimension of the thinned multi-dimensional oriented self-similarity feature.

[0043] Furthermore, in step 8, the block feature detector adopts the FAST detection model and the strategy of uniform grid division. First, the image is evenly divided into small image blocks with equal areas, and then feature points are extracted in each small image block respectively.

[0044] The random sample consensus algorithm is used to eliminate gross errors.

[0045] Furthermore, it also includes step 9, which uses the root mean square error of homologous points and the number of homologous point pairs to evaluate the matching effect of multi-modal remote sensing images.

[0046] The present invention also provides a multi-modal remote sensing image hybrid matching system for multi-dimensional oriented self-similar features, including the following modules:

[0047] An initialization module, used to initialize the calculation parameters of the hybrid matching.

[0048] The feature point extraction module is used to quickly extract the self-similar features of the reference image by using the offset mean filtering method, generate a multi-channel self-similar feature map, and perform feature extraction and non-maximum suppression operations on the feature response map to obtain feature points;

[0049] The initial correspondence obtaining module is used to describe the feature points by using the direction information of the self-similar features, sequentially complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors, and then perform the nearest neighbor distance ratio matching to output the initial correspondence of the image pair;

[0050] The transformation module is used to perform geometric transformation on the image to be matched by using the initial correspondence and output the result of the transformed image to be matched;

[0051] The oriented self-similar feature calculation module is used to calculate the oriented self-similar features of the reference image and the newly generated image to be matched and output a multi-channel self-similar map;

[0052] The multi-dimensional oriented self-similar template feature obtaining module is used to design an angle weighting strategy, calculate the horizontal gradient and the vertical gradient for each channel of the obtained multi-channel self-similar map, and perform channel thinning to filter out duplicate features, and solve to obtain the multi-dimensional oriented self-similar template features;

[0053] The feature enhancement module is used to enhance the feature channels by using a three-dimensional Gaussian convolution kernel after generating the multi-dimensional oriented self-similarity template features;

[0054] The matching module is used to detect the feature points of the reference image by using a block feature detector, solve the relative positions of the feature points in the newly generated image to be matched by using the feature point joint identity matrix, then transform the enhanced template features from the spatial domain to the frequency domain, and use three-dimensional phase correlation as a similarity measure to accelerate the matching of the template-like features, and finally eliminate the gross errors to complete the multi-modal remote sensing image matching.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] The multi-modal remote sensing image matching method proposed by the present invention mainly consists of two core parts: rough matching of hybrid features and fine matching of multi-dimensional oriented self-similar features. First is the rough matching of hybrid features. In this stage, the oriented self-similarity algorithm is used, and the offset mean filtering method is adopted to quickly extract the self-similar features of the image, and the direction information of the self-similar features is used to describe the feature points. Through the rough matching of hybrid features, the affine transformation model between multi-modal images can be estimated, and the initial affine transformation of the images can be performed to eliminate the geometric differences between them. The second part is the fine matching of multi-dimensional oriented self-similar features. A characterization method based on multi-dimensional oriented self-similar features is proposed. The multi-channel self-similarity map obtained in the rough matching stage of hybrid features is used to construct the multi-dimensional oriented self-similarity template features, and a template feature thinning strategy is designed. The three-dimensional Gaussian kernel is used to convolve the template features to enhance the expression ability of the features. Subsequently, the three-dimensional phase correlation matching strategy is used to establish a more accurate registration relationship, and the random sample consensus algorithm is adopted to eliminate the false matches. Finally, the images are fused to complete the registration of multi-modal remote sensing images. The results show that the method proposed by the present invention can better achieve the matching of multi-modal remote sensing images and is more robust than the traditional methods. Description of the Drawings

[0057] Figure 1 is the flowchart of the method of the present invention;

[0058] Figure 2 is the schematic diagram of the construction process of the multi-dimensional oriented self-similarity template features in the embodiment of the present invention;

[0059] Figure 3 is the multi-modal remote sensing image dataset in the embodiment of the present invention. From left to right, each column is respectively the multi-temporal optical image, the infrared image and the optical image, the night light image and the optical image, the SAR image and the optical image, the navigation map and the optical image, and the optical image and the depth image;

[0060] Figure 4 is the multi-modal remote sensing image matching result in the embodiment of the present invention. Detailed Embodiment

[0061] To facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0062] Please refer to Figure 1 the flowchart. A multi-modal remote sensing image matching method with multi-dimensional oriented self-similar features provided by the present invention includes the following steps:

[0063] Step 1: Initialize parameters such as the neighborhood radius for rough matching of mixed features, the thinning interval of multi-dimensional oriented self-similar features, and the descriptor neighborhood window, and set them to 4, 2, and 96 respectively.

[0064] Step 2: Use the offset mean filtering method to quickly extract the self-similar features of the reference image, generate a multi-channel self-similar feature map, and perform feature extraction and non-maximum suppression operations on the feature response map to obtain feature points.

[0065] In the self-similar feature detector, pixel points that are highly dissimilar to surrounding pixels are regarded as feature points. Therefore, for each pixel point on the image, its self-similar feature value is extracted, and n minimum values in its self-similar feature values are statistically counted. The feature response λ of point q is calculated as shown in Equation (1). The feature responses λ of all pixel points form a feature response map, and then local non-maximum suppression is performed on the feature response map to obtain feature points.

[0066]

[0067] Among them, λ(q) represents the feature response λ of point q, represents n minimum self-similar values (as a preference, the value of n in the present invention is set to 4).

[0068] Step 3: Describe the feature points using the direction information of the self-similar features, and sequentially complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors. Then perform the nearest neighbor distance ratio matching and output the initial correspondence of the image pair.

[0069] (1) Calculation of the main direction of feature points. This step aims to ensure the rotational invariance of the descriptor. The following method is used to calculate the main direction of feature points: Select a fixed circular neighborhood centered on the feature point, and generate an oriented histogram based on the self-similar feature values to determine the main direction. The specific steps are as follows: First, evenly divide the histogram into 36 equal parts, and each part represents an interval of 10°. Then, for the feature point P whose main direction is to be determined, uniformly sample 36 points from the boundary of the feature neighborhood with a radius of r from P. For these 36 points, statistically count their self-similar feature sequences, denoted as S1, S2…S 36 . Next, normalize the self-similar feature sequence, and select the peak direction with a proportion of more than 80% in the histogram as the main direction of the feature point.

[0070] (2) Descriptor feature vector statistics. To improve the feature matching efficiency and reduce the dimension of the feature descriptor, a log-polar coordinate grid is used to statistically analyze the feature vectors. The descriptor neighborhood of the feature points is a circular region extracted from the multi-channel self-similarity graph. At each pixel point within the descriptor neighborhood, an oriented index map is generated by calculating the index value of the minimum self-similarity direction; finally, the oriented index map is log-polar coordinate gridified, and a distribution histogram is generated within each grid interval as the feature descriptor vector.

[0071] (3) Initial correspondence calculation. The nearest neighbor distance ratio matching strategy is adopted to determine the initial matching homologous points. At the same time, the Random Sample Consensus (RANSAC) algorithm is combined to effectively eliminate the false matches. The nearest neighbor distance ratio matching strategy first calculates the Euclidean distance between any two descriptors of the two images; secondly, for each descriptor of the reference image, the ratio of the minimum distance to the second minimum distance is calculated; finally, the descriptors with a distance ratio less than a certain threshold d and their nearest descriptors are extracted as matching pairs.

[0072] Step 4: Geometrically transform the image to be matched using the initial correspondence, and output the result of the transformed image to be matched.

[0073] Step 5: Calculate the oriented self-similarity features for the reference image (i.e., the original input image) and the newly generated image to be matched, and output the multi-channel self-similarity graph. The self-similarity graph in the direction of point q(p,θ) is quickly obtained using offset mean filtering, which includes two steps: image cropping and mean filtering. This point refers to each pixel point of the reference image and the image to be matched, where p represents the pixel distance and θ represents the angle, with a value range of [0°, 180°). The central subgraph is constructed by cropping the newly generated image to be matched in Step 4 and the offset subgraph The newly generated image to be matched is pixel-offset by 2 pixels, and then bilinear interpolation can be used to obtain the offset subgraph in any direction. The central subgraph is a subgraph centered at point q and reduced by one pixel at the four edges of the original image. After obtaining the subgraphs, the self-similarity graph S in the direction of point q(p,θ) can be calculated by the following formula (2) q :

[0074]

[0075] In the formula, meanFilter(·) represents the mean filtering operation. A circular filtering window with a radius of 2 pixels is used, and the circular filtering window can enhance the rotation invariance of the self-similarity features.

[0076] The multi-channel self-similarity graph can be expressed as C represents the number of channels, and N represents the number of pixels within the feature neighborhood. Due to the symmetry of the self-similarity features, the multi-channel self-similarity graph of the entire image can be obtained by performing the offset mean filtering operation C = N / 2 times.

[0077] Step 6: Design an angle weighting strategy. For the obtained multi-channel self-similarity graphs, calculate the horizontal gradient and vertical gradient channel by channel, and perform channel thinning to filter out duplicate features, and solve to obtain multi-dimensional oriented self-similarity template features.

[0078] First, for the obtained multi-channel self-similarity graphs calculate the horizontal gradient g channel by channel using the Sobel operator x and the vertical gradient g y . The Sobel operator template h = [1, 2, 1; 0 0 0; -1, -2, -1], as shown in Equation (3):

[0079]

[0080] where g x represents the horizontal gradient, g y represents the vertical gradient, S q represents the input single-channel self-similarity graph, represents the convolution operation.

[0081] Then, use Equation (4) to calculate the gradient direction, and perform an angle weighting strategy on the multi-channel self-similarity graphs channel by channel to construct single-channel multi-dimensional oriented self-similarity features which effectively enhances the rotational invariance of the self-similarity template features. It should be noted that if the gradient direction is less than 0, π should be added to ensure that the gradient direction value is always greater than 0.

[0082]

[0083] where represents the single-channel multi-dimensional oriented self-similarity feature.

[0084] For the multi-channel self-similarity features, perform an angle weighting strategy channel by channel to construct multi-channel multi-dimensional oriented self-similarity features In constructing the multi-channel self-similarity graphs, a circular feature neighborhood of N pixels with a radius of R is adopted, and the number of channels C = N / 2. Therefore, multi-channel multi-dimensional oriented self-similarity features with a dimension of C are obtained. To improve the matching calculation efficiency of the template features, a thinning strategy is adopted for the feature channels, and thinning is performed through Equation (6) below to obtain a w-dimensional template feature map

[0085]

[0086] where w represents the dimension of the thinned multi-dimensional oriented self-similarity feature, C represents the dimension of the original multi-dimensional oriented self-similarity feature, and m is the sampling interval. The operation symbol represents rounding up, obtaining an integer in the direction of increasing absolute value.

[0087] Step 7: After generating the multi-dimensional directional self-similarity template features, a three-dimensional Gaussian convolution kernel is used to enhance the feature channels to reduce the influence of geometric deformation and non-linear radiation distortion between images on the matching.

[0088] The Gaussian convolution kernel is a two-dimensional Gaussian kernel in the horizontal and vertical directions (preferably, the standard deviation is set to 0.8 in the present invention) and a kernel d in the Z direction z =[1, 3, 1] T . The three-dimensional Gaussian convolution is described as shown in the following formula:

[0089]

[0090] Among them, represents the feature after two-dimensional Gaussian filtering, represents the Gaussian kernel on the xy plane, represents the single-channel multi-dimensional directional self-similarity feature, M σ (x, y) represents the template feature after three-dimensional Gaussian filtering, d z is the Gaussian kernel in the z direction, represents the convolution operation, and σ is the standard deviation of the Gaussian convolution kernel. Then, a normalization operation can scale the feature vector M σ (x, y) to the same scale, thereby enhancing its robustness. It can be expressed by formula (8):

[0091]

[0092] Among them, to avoid the denominator being 0, ε is set to a non-zero extremely small constant value.

[0093] Finally, combine the features to form a multi-dimensional directional self-similarity feature map MOSS(x, y). The mathematical formula is defined as the following formula (9):

[0094]

[0095] Among them, w represents the dimension of the thinned multi-dimensional directional self-similarity feature.

[0096] Step 8: Use a block feature detector (the detector uses FAST detection, and the parameter block strategy is: adopt the strategy of uniform grid division. First, evenly divide the image into small tiles with equal areas, and then extract feature points in each small tile) to detect the feature points of the reference image. Considering that the template feature is three-dimensional, use the Fourier transform to transform the feature template from the spatial domain to the frequency domain, and use three-dimensional phase correlation as the similarity measure to accelerate the matching of similar template features (the matching of the feature points of the reference image with the feature points on the new transformed image calculated by the feature points and the identity matrix), and use the fast sample consensus algorithm to eliminate the false matches. The results are as Figure 4 shown.

[0097] Step 9: Evaluate the matching effect of the multi-modal remote sensing images. The present invention uses 5 groups of multi-modal remote sensing image test methods to test the performance, and the data set is shown in Figure 3 . For each pair of images, use the root-mean-square error (Root-Mean-Square Error, abbreviated as RMSE) of the homologous points and the number of homologous point pairs for quantitative inspection, where the unit of RMSE is pixels. The multi-modal remote sensing image matching method proposed by the present invention is named the MOSS method, and it is compared with several optimal image matching methods (SIFT, PSO-SIFT, RIFT, OSS, and HOWP). The comparison results are shown in Table 1.

[0098] Table 1 Comparison of several image matching methods

[0099]

[0100] As can be seen from Table 1, in the multi-modal remote sensing image data, the MOSS method can obtain more homologous point pairs compared with the SIFT, PSO-SIFT, RIFT, OSS, and HOWP methods. The optimal result can be achieved through the MOSS method proposed by the present invention. Among them, the RMSE of the MOSS method is better than that of the SIFT, PSO-SIFT, RIFT, OSS, and HOWP methods. The average value of the RMSE of the MOSS method proposed by the present invention is within 2 pixels, which further proves that the MOSS method not only greatly increases the number of matched homologous points, but also maintains good matching accuracy. It has better performance in terms of matching scale, rotation, and translation differences, and the proposed MOSS method has scale and rotation invariance.

[0101] The embodiment of the present invention also provides a multi-modal remote sensing image hybrid matching system with multi-dimensional directional self-similar features, including the following modules:

[0102] An initialization module, used to initialize the calculation parameters of the hybrid matching;

[0103] The feature point extraction module is used to quickly extract the self - similar features of the reference image by using the offset mean filtering method, generate a multi - channel self - similar feature map, and perform feature extraction and non - maximum suppression operations on the feature response map to obtain feature points;

[0104] The initial correspondence acquisition module is used to describe the feature points by using the direction information of the self - similar features, sequentially complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors, and then perform the nearest - neighbor distance ratio matching to output the initial correspondence of the image pair;

[0105] The transformation module is used to perform geometric transformation on the image to be matched by using the initial correspondence and output the result of the transformed image to be matched;

[0106] The oriented self - similar feature calculation module is used to calculate the oriented self - similar features of the reference image and the newly generated image to be matched, and output a multi - channel self - similar map;

[0107] The multi - dimensional oriented self - similar template feature acquisition module is used to design an angle - weighted strategy, calculate the horizontal gradient and vertical gradient for each channel of the obtained multi - channel self - similar map, and perform channel thinning to filter out duplicate features, and solve to obtain the multi - dimensional oriented self - similar template features;

[0108] The feature enhancement module is used to enhance the feature channels by using a three - dimensional Gaussian convolution kernel after generating the multi - dimensional oriented self - similar template features;

[0109] The matching module is used to detect the feature points of the reference image by using a block - based feature detector, solve the relative positions of the feature points in the newly generated image to be matched by using the feature points and the identity matrix, then transform the enhanced template features from the spatial domain to the frequency domain, and use three - dimensional phase correlation as the similarity measure to accelerate the matching of the template - like features, and finally eliminate the gross errors to complete the multi - modal remote sensing image matching.

[0110] The specific implementation methods of each module correspond to the respective steps, and are not described in this invention.

[0111] It should be understood that the parts not elaborated in this specification belong to the prior art.

[0112] It should be understood that the above description of the preferred embodiments is relatively detailed, and it should not be considered as a limitation to the protection scope of the invention patent. Under the inspiration of the present invention, those of ordinary skill in the art can also make substitutions or deformations without departing from the scope protected by the claims of the present invention, and all fall within the protection scope of the present invention. The scope of protection requested by the present invention shall be subject to the appended claims.

Claims

1. A multimodal remote sensing image hybrid matching method for multi-dimensional directional self-similar features, characterized in that, It includes the following steps: Step 1, initialize the calculation parameters for hybrid matching; Step 2, use the offset mean filtering method to quickly extract the self-similar features of the reference image, generate a multi-channel self-similar feature map, and perform feature extraction and non-maximum suppression operations on the feature response map to obtain feature points; Step 3, describe the feature points using the direction information of the self-similar features, sequentially complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors, and then perform the nearest neighbor distance ratio matching to output the initial corresponding relationship of the image pair; Step 4, perform geometric transformation on the image to be matched using the initial corresponding relationship, and output the result of the transformed image to be matched; Step 5, perform directional self-similar feature calculation on the reference image and the newly generated image to be matched, and output a multi-channel self-similar map; Step 6, design an angle weighting strategy, for the obtained multi-channel self-similar map, calculate the horizontal gradient and vertical gradient for each channel, and perform channel thinning to filter out duplicate features, and solve to obtain a multi-dimensional directional self-similar template feature; Step 7, after generating the multi-dimensional directional self-similarity template feature, use a three-dimensional Gaussian convolution kernel to enhance the feature channels; Step 8, use a block feature detector to detect the feature points of the reference image, use the feature points to jointly solve the unit matrix to obtain the relative positions of the feature points in the newly generated image to be matched, then transform the enhanced template feature from the spatial domain to the frequency domain, and use three-dimensional phase correlation as a similarity measure to accelerate the matching of the template-like features, and finally eliminate the gross errors to complete the multi-modal remote sensing image matching.

2. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, wherein: The calculation parameters in Step 1 include the neighborhood radius, the thinning interval of the multi-dimensional directional self-similar feature, and the descriptor neighborhood window.

3. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, characterized in that: In Step 2, for each pixel point on the image, extract its self-similar feature value, and count the n minimum values in its self-similar feature values. The feature response λ of point q is calculated as shown in Equation (1). The feature responses λ of all pixel points are composed into a feature response map, and then local non-maximum suppression is performed on the feature response map to obtain the feature points; where λ(q) represents the feature response λ at point q, represents the n smallest self-similarity values.

4. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, characterized in that: The specific implementation method of Step 3 is as follows; (1) Calculation of the main direction of feature points: Select a fixed circular neighborhood centered on the feature point, and generate an orientation histogram based on the self-similarity eigenvalues to determine the main direction. The specific steps are as follows: First, evenly divide the histogram into K equal parts. Then, for the feature point P whose main direction is to be determined, uniformly sample K points from the boundary of the feature neighborhood with a radius of r from P. For these K points, count their self-similarity feature sequences, denoted as S1, S2…S K , Next, normalize the self-similarity feature sequence, and select the peak direction with a proportion of more than P% in the histogram as the main direction of the feature point, where both K and P are constants; (2) Statistics of descriptor feature vectors: The descriptor neighborhood of the feature points is a circular area extracted from the multi-channel self-similar map. On each pixel point in the descriptor neighborhood, by calculating the index value of the minimum self-similar direction, an oriented index map is generated; finally, the oriented index map is logarithmically polar gridded, and a distribution histogram is generated in each grid interval as the feature descriptor vector; (3) Calculation of the initial corresponding relationship: The nearest neighbor distance ratio matching strategy is used to determine the initial matching homologous points. At the same time, the random sample consensus algorithm is combined to effectively eliminate the false matches; Among them, the nearest neighbor distance ratio matching strategy first calculates the Euclidean distance between any two descriptors of the two images; secondly, for each descriptor of the reference image, calculates the ratio of the minimum distance to the second minimum distance; finally, extracts the descriptors with the distance ratio less than a certain threshold d and their nearest descriptors as the matching pairs.

5. The multi-modal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, characterized in that: In step 5, first, a self-similarity map in the direction of pixel point q(p,θ) on the image to be matched is quickly obtained using offset mean filtering, which includes two steps: image cropping and mean filtering. The central sub-map is constructed by cropping the original image and the offset sub-map After obtaining the sub-maps, the self-similarity map S in the direction of point q(p,θ) can be calculated by the following formula (2) q : In the formula, meanFilter(·) represents the mean filtering operation, p represents the pixel distance, θ represents the angle, and the value ranges from [0°, 180°). Pixel offset is performed on the newly generated image to be matched, and then bilinear interpolation is used to obtain the offset sub-image in any direction. The central sub-image is obtained by shrinking one pixel from the four edges of the original input image with point q as the center; the multi-channel self-similarity map can be expressed as C represents the number of channels, and N represents the number of pixels in the feature neighborhood. Since the self-similarity feature has symmetry, performing the offset mean filtering operation C = N / 2 times can obtain the multi-channel self-similarity map of the entire image.

6. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, characterized in that: The specific implementation method of Step 6 is as follows; First, for the obtained multi-channel self-similarity graph calculate the horizontal gradient g and the vertical gradient g for each channel using the Sobel operator x as shown in Equation (3): y as shown in Equation (3): Among them, h represents the Sobel operator template, and g x represents the horizontal gradient, and g y represents the vertical gradient, S q represents the input single-channel self-similarity graph, represents the convolution operation; Then, the gradient direction is calculated using formula (4), and an angle weighting strategy is applied to the multi-channel self-similarity map channel by channel to construct a single-channel multi-dimensional directional self-similarity feature. Effectively enhance the rotational invariance of the self-similar template feature; Among them, represents the single-channel multi-dimensional directional self-similarity feature; For the multi-channel self-similarity features, an angular weighting strategy is applied to each channel to construct multi-channel multi-dimensional directional self-similarity features In constructing the multi-channel self-similarity graph, a circular feature neighborhood of N pixels with a radius of R is adopted, and the number of channels C = N / 2. Thus, multi-channel multi-dimensional directional self-similarity features are obtained The dimension is C. To improve the matching calculation efficiency of the template features, a thinning strategy is adopted for the feature channels, and thinning is performed through the following formula (6) to obtain a w-dimensional template feature map Wherein, w represents the dimension of the downsampled multi-dimensional orientation self-similarity feature, C represents the dimension of the original multi-dimensional orientation self-similarity feature, m is the sampling interval, The operation symbol represents rounding up.

7. The multimodal remote sensing image hybrid matching method for multi-dimensional directional self-similar features according to claim 1, wherein: In step 7, the Gaussian convolution kernel is a two-dimensional Gaussian kernel in the horizontal and vertical directions and a kernel d in the Z direction z =[1, 3, 1] T , as shown in Equation (7): Among them, represents the feature after two-dimensional Gaussian filtering, represents the Gaussian kernel on the plane, represents the single-channel multi-dimensional oriented self-similarity feature, M σ (x, y) represents the template feature after three-dimensional Gaussian filtering, d z is the Gaussian kernel in the z direction, represents the convolution operation, and σ is the standard deviation of the Gaussian convolution kernel; Finally, perform a normalization operation on the multi-dimensional orientation self-similarity features first, and then combine the features to form a multi-dimensional orientation self-similarity feature map, as shown in Equation (8): Among them, w represents the dimension of the thinned multi-dimensional directional self-similarity feature.

8. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, wherein: In step 8, the block feature detector adopts the FAST detection model and the strategy of uniform grid division. First, the image is evenly divided into small tiles with equal areas, and then feature points are extracted from each small tile respectively. The random sample consensus algorithm is used to eliminate gross errors.

9. The multimodal remote sensing image hybrid matching method with multi-dimensional directional self-similar features according to claim 1, wherein: It further includes step 9, which uses the root mean square error of homologous points and the number of homologous point pairs to evaluate the matching effect of multimodal remote sensing images.

10. A multimodal remote sensing image hybrid matching system with multi-dimensional directional self-similar features, characterized in that, It includes the following modules: An initialization module, which is used to initialize the calculation parameters of hybrid matching. A feature point extraction module, which is used to quickly extract the self-similar features of the reference image by using the offset mean filtering method, generate a multi-channel self-similar feature map, and perform feature extraction and non-maximum suppression operations on the feature response map to obtain feature points. An initial correspondence acquisition module, which is used to describe the feature points by using the direction information of the self-similar features, sequentially complete the calculation of the main direction of the feature points and the statistics of the descriptor feature vectors, and then perform the nearest neighbor distance ratio matching to output the initial correspondence of the image pair. A transformation module, which is used to perform geometric transformation on the image to be matched by using the initial correspondence and output the result of the transformed image to be matched. An oriented self-similar feature calculation module, which is used to calculate the oriented self-similar features of the reference image and the newly generated image to be matched and output a multi-channel self-similar map. A multi-dimensional oriented self-similar template feature acquisition module, which is used to design an angle weighting strategy, calculate the horizontal gradient and vertical gradient for each channel of the obtained multi-channel self-similar map, and perform channel thinning to filter out duplicate features, and solve to obtain the multi-dimensional oriented self-similar template feature. A feature enhancement module, which is used to enhance the feature channels by using a three-dimensional Gaussian convolution kernel after generating the multi-dimensional oriented self-similarity template feature. A matching module, which is used to detect the feature points of the reference image by using the block feature detector, solve the relative positions of the feature points in the newly generated image to be matched by using the feature points and the unit matrix, then transform the enhanced template feature from the spatial domain to the frequency domain, and use the three-dimensional phase correlation as the similarity measure to accelerate the matching of the template-like features. Finally, gross errors are eliminated to complete the matching of multimodal remote sensing images.

Images