A robust region-adaptive multi-modal image matching method

Abstract

The application discloses a kind of robust regional adaptive heterogeneous image matching methods of microwave remote sensing technical field, the robust regional adaptive heterogeneous image matching method of the present application includes the following steps: regionally adaptive feature point extraction based on VE-FAST;Similarity graph generation of regional feature matching;Matching point screening based on SK-NMS.Combining image texture features from feature point extraction and matching point screening two links, the robust regional adaptive heterogeneous image matching method can effectively identify weak feature region, realize SAR / optical image in weak feature region fast, robust, high-precision matching.

Description

Technical Field

[0001] This invention relates to the field of microwave remote sensing technology, and more specifically, to a robust regional adaptive heterogeneous image matching method. Background Technology

[0002] Current remote sensing technology can achieve system-level geometric correction of images through platform positioning parameters, effectively eliminating rotation and scale differences between SAR (Synthetic Aperture Radar) images and optical images. Cutting-edge algorithms tend to combine region-based and feature-based methods, developing feature-based template matching algorithms. These algorithms typically utilize similar local structural features between heterogeneous images to extract a set of matching point pairs to construct a geometric transformation model. Research shows that compared to region-based methods, methods combining template regions and features can achieve higher accuracy in SAR / heterogeneous image matching. These methods generally include three main steps: feature point extraction, region feature matching, and matching point pair selection. Depending on the feature descriptors used, common classic algorithms include: directional gradient features proposed by Ye Y et al., and AWOG, which improves the accuracy of gradient calculation; LNIFT using ORB and HOG-like features; SRIF, which has scale invariance based on LNIFT; and LPHOG, which combines line and point features and uses HOG-like descriptors; registration algorithms for SAR and optical images that combine SIFT algorithms from the field of computer vision mainly include PSO-SIFT, OS-SIFT, and RTV-SIFT; and registration algorithms that use phase consistency mainly include HOPC and FED-HOPC.

[0003] When the observed area has abundant ground features and significant structural and textural characteristics, algorithms such as CFOG can effectively achieve high-precision matching of SAR / optical heterogeneous images. However, in practical applications, it is often necessary to process images of weak-feature areas (sparse structural features) such as ports, islands, and other water-land interface areas with large bodies of water, as well as deserts and Gobi deserts. In these cases, traditional matching algorithms generate redundant and meaningless image descriptor points in these weak-feature areas, and even common outlier filtering algorithms may fail to effectively remove these points, resulting in a significant performance degradation of traditional algorithms. Differences in sensor imaging mechanisms, observation perspectives, and ground feature scattering characteristics are key factors limiting the matching performance of SAR / optical heterogeneous images. The block-based Harris algorithm is currently the most commonly used feature point extraction algorithm for heterogeneous image matching. It can extract a large number of feature points evenly. However, this algorithm relies solely on the salience of structural features and ignores the regional differences of weak feature features. It will still extract a certain number of feature points on water surfaces and in deserts and Gobi. These feature points are usually randomly distributed, and even after processing such as regional feature matching and traditional non-maximum matching point filtering to remove most of them, a small number of spurious pairs of identical points often remain in weak feature areas such as water surfaces. Feature point extraction algorithms that do not employ a block-based strategy, such as FAST, although they do not extract feature points in weak feature areas, are prone to clustering, reducing the uniformity of feature point distribution. These factors can easily reduce the accuracy of the geometric transformation model. The robustness of traditional matching algorithms drops significantly in weak feature areas such as ports, islands, and deserts and Gobi with large areas of water. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a robust region-adaptive heterogeneous image matching method. This method solves the technical problems of the traditional matching algorithm's poor robustness in areas with large water areas, such as ports, islands, and deserts, where the proportion of weak feature regions is large and the accuracy of the full-image geometric transformation model is low. By combining image texture features and starting from two stages—feature point extraction and matching point selection—it can effectively identify weak feature regions and achieve fast, robust, and high-precision matching of SAR / optical images in weak feature regions.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] This invention provides a robust region-adaptive heterogeneous image matching method, comprising the following steps:

[0007] S1. Region adaptive feature point extraction based on VE-FAST;

[0008] S2. Similarity map generation based on region feature matching;

[0009] S3. Matching point filtering based on SK-NMS.

[0010] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, S1 includes the following steps:

[0011] S11. Introduce a block-segmentation strategy to divide the image into N×N sub-image blocks, record the FAST feature point scores of each sub-image block, and sort them in descending order;

[0012] S12. According to information theory, based on information entropy H... I The calculation formula states that for each sub-image patch, the information entropy must be greater than a set threshold λ. e Before the selection m u m points are selected as candidate points. If any do not meet the set information entropy threshold, m points are extracted. w One candidate point;

[0013] S13. Determine the template region and search region based on each candidate point, and calculate the variance product Q of the template region and search region corresponding to each candidate point. p ;

[0014] S14. Based on the product of variances Q p The threshold of the product of sum and variance λ v Based on the size relationship, feature points that are adaptively and uniformly distributed in the region are extracted.

[0015] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, wherein H in S12 I The calculation formula is as follows:

[0016]

[0017] Where, p i Let be the probability of the i-th gray level appearing, i∈[1,k].

[0018] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, wherein the variance product Q of the template region and the search region corresponding to each candidate point in S13 is... p The calculation formula is as follows:

[0019]

[0020] Among them, V o The variance of the template region. The minimum variance of all template regions. V represents the maximum variance of the template region. s The variance of the search region, The minimum variance across all search regions. This represents the maximum variance of the search region.

[0021] For a size of M I ×N I The variance V of the image I(x,y) is defined as follows:

[0022]

[0023] Where, μ I The average pixel value of the image is obtained by the following formula (4).

[0024]

[0025] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, S2 includes the following steps:

[0026] S21. Using the template region of the reference optical image and the search region of the SAR image to be matched, feature descriptors are constructed using MOGF respectively;

[0027] S22. Cross-correlation is used as an indicator to evaluate the similarity between feature descriptors in the template region and the search region. This process is transformed to the frequency domain through Fourier transform. Multiplication operations in the frequency domain are used to replace convolution operations in the spatial domain to calculate the normalized cross-correlation value between the template region and the search region in the frequency domain, thereby obtaining the similarity map.

[0028] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, S21 includes the following steps:

[0029] S211. Use filters to obtain the gradient magnitude IG and direction OG of the image in the x and y directions;

[0030] S212. Considering the robustness of the algorithm, the Sobel operator, which takes into account comprehensive factors, is selected to extract gradient information for optical images, and the ROEWA operator is selected to be used for SAR images, considering the influence of multiplicative speckle noise.

[0031] S213. Use the following formula to obtain the gradient vector of each pixel:

[0032]

[0033] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, S3 includes the following steps:

[0034] S31. Using the ratio of primary to secondary peaks R and the skewness S of the similarity plot k The calculation formulas are used to calculate the ratio of the primary and secondary peaks R and the skewness S of the similarity graph, respectively. k And set a threshold λ r and λs ;

[0035] S32. When the ratio R of the primary and secondary peaks of the similarity graph is greater than λ r And skewness S k Greater than λ s At that time, it is considered that only the maximum value of the similarity graph is suitable for calculating the matching point, thus obtaining a reliable matching point pair;

[0036] S33. By processing and filtering the above similarity graph, a set of matching point pairs was obtained;

[0037] S34. Starting from the geometric transformation relationship model of matching point pairs, use the FSC method to eliminate outlier matching points.

[0038] As one aspect of a robust region-adaptive heterogeneous image matching method of the present invention, wherein the ratio of primary to secondary peak values ​​R and the skewness S of the similarity map in S31 are... k The calculation formula is as follows:

[0039]

[0040] Where IoU is the crossover ratio between the main peak window and the secondary peak window, ε is the machine zero introduced to prevent the crossover ratio from being 0, and it is a number close to 0 but greater than 0, P p Main peak intensity, P s This is the second-highest intensity;

[0041]

[0042] in, x is the mean. i Let n be the i-th element and n be the number of samples.

[0043] By adopting the above technical solution, the present invention has the following advantages:

[0044] This invention provides a robust region-adaptive heterogeneous image matching method. It effectively identifies weak-feature regions using a dual threshold criterion of regional information entropy and variance product, appropriately controlling the number of feature points in weak-feature regions to minimize feature point extraction while preserving a small number of valid regions. This invention combines a skewness and non-maximum suppression matching point pair selection strategy. By incorporating the non-maximum suppression method and introducing the concept of skewness in the similarity map, it effectively eliminates false matching points distributed in weak-feature regions, improving the accuracy of the geometric transformation model. The introduction of a region-adaptive processing mechanism and a more comprehensive matching point selection strategy into the heterogeneous image matching method can determine whether a local area is a weak-feature region such as water, desert, or Gobi, thereby controlling the number of feature points and matching point pairs in such areas. Furthermore, since islands, scattered villages, and roads are often scattered within large areas of weak-feature images, and these features are typically effective areas for extracting highly reliable feature points and obtaining matching points, this invention can retain these small effective areas and obtain matching point pairs, thus improving the matching accuracy of the entire image. Attached Figure Description

[0045] Figure 1 The flowchart is a robust region-adaptive heterogeneous image matching method of the present invention;

[0046] Figure 2 This is a flowchart of the VE-FAST feature point extraction algorithm of the present invention;

[0047] Figure 3 This is a flowchart of the matching point filtering based on SK-NMS according to the present invention;

[0048] Figure 4 This is a field measurement of airborne SAR data for a typical region with weak features and a small number of islands scattered in the water.

[0049] Figure 5 This is a comparison of the matching results before and after correction of a typical weak feature image using the method of the present invention. Detailed Implementation

[0050] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0051] Figure 1 The flowchart illustrates a robust region-adaptive heterogeneous image matching method, which mainly includes three parts: (i) region-adaptive feature point extraction; (ii) region feature matching (obtaining a similarity map); and (iii) matching point pair selection, specifically as follows. Figure 1 As shown, for a pair of SAR-optical heterogeneous images, the proposed VE-FAST feature point detection algorithm is first used to extract uniformly distributed key points. Then, the template region and search region are determined with the extracted key points as the center, and a feature descriptor is constructed to obtain a similarity map. Finally, the proposed SK-NMS (Skewness-Non-maximal suppression) homonym filtering algorithm is used to filter out mismatched point pairs and establish the correct geometric transformation model.

[0052] This invention provides a robust region-adaptive heterogeneous image matching method, comprising the following steps:

[0053] S1. Region adaptive feature point extraction based on VE-FAST;

[0054] Figure 2 The flowchart of the feature point extraction algorithm of VE-FAST (Variance and Entropy-Features from Accelerated Segment Test) is shown.

[0055] S1 includes the following steps, as detailed below: Figure 2 As shown:

[0056] S11. Introduce a block-segmentation strategy to divide the image into N×N sub-image blocks, record the FAST feature point scores of each sub-image block, and sort them in descending order;

[0057] S12. According to information theory, based on information entropy H... I The calculation formula shows that the information entropy basically reflects the texture transformation of each sub-image; regions with weak features typically have lower information entropy. For each sub-image patch, if the information entropy is greater than a set threshold λ... e Before the selection m u m points are selected as candidate points. To avoid ignoring small areas of effective feature regions, m points are extracted from sub-image blocks that do not meet the set information entropy threshold. w (m u >m w ) candidate points; in order to improve the information entropy threshold λ e To ensure applicability and avoid bias caused by excessive differences in the numerical values ​​of different features, step S12 calculates the information entropy H of the sub-image block. I First, the image is normalized to a grayscale image within the range [0,1].

[0058]

[0059] Where, p i Let be the probability of the i-th gray level appearing, i∈[1,k].

[0060] S13. Determine the template region and search region based on each candidate point, and calculate the variance product Q of the template region and search region corresponding to each candidate point. p ;

[0061] In S13, the product of the variances of the template region and the search region corresponding to each candidate point is Q. p The calculation formula is as follows:

[0062]

[0063] Among them, V o The variance of the template region. The minimum variance of all template regions. V represents the maximum variance of the template region. s The variance of the search region, The minimum variance across all search regions. This represents the maximum variance of the search region.

[0064] For a size of M I ×N I The variance V of the image I(x,y) is defined as follows:

[0065]

[0066] Where, μ I The average pixel value of the image is obtained by the following formula (4).

[0067]

[0068] S14. Based on the product of variances Q p The threshold of the product of sum and variance λ v The size relationship, when Q p Greater than λ v The region where the candidate point is located is considered to be a feature-rich region, and thus it is identified as a feature point; the variance product Q p Less than the variance product threshold λ v Candidate points are eliminated, and feature points that are adaptive and uniformly distributed in the region are extracted, thereby achieving region adaptation.

[0069] S2. Similarity map generation based on region feature matching;

[0070] S2 includes the following steps:

[0071] S21. After feature point extraction, the template matching method needs to construct feature descriptors for the template region of the reference optical image and the search region of the SAR image to be matched, respectively. Considering that the scale and rotation differences between images have been largely eliminated, MOGF is used to construct feature descriptors for the template region of the reference optical image and the search region of the SAR image to be matched, which has better computational efficiency and matching performance.

[0072] S21 includes the following steps:

[0073] S211. Use filters to obtain the gradient magnitude IG and direction OG of the image in the x and y directions;

[0074] S212. Considering the robustness of the algorithm, the Sobel operator, which takes into account comprehensive factors, is selected to extract gradient information for optical images, and the ROEWA operator is selected to be used for SAR images, considering the influence of multiplicative speckle noise.

[0075] S213. Use the following formula to obtain the gradient vector of each pixel:

[0076]

[0077] S22. Cross-correlation is used as an indicator to evaluate the similarity between feature descriptors in the template region and the search region. This process is transformed to the frequency domain using Fourier transform. Multiplication operations in the frequency domain replace convolution operations in the spatial domain to calculate the normalized cross-correlation value between the template region and the search region, thus obtaining a similarity map. By locating the position of the maximum value in the similarity map, matching point pairs can be determined, and a geometric transformation relationship model can be derived to complete image matching.

[0078] Figure 3 A flowchart of the matching point filtering based on SK-NMS of the present invention is shown.

[0079] S3. Matching point filtering based on SK-NMS.

[0080] S3 includes the following steps, as detailed below: Figure 3 As shown:

[0081] S31. Using the ratio of primary to secondary peaks R and the skewness S of the similarity plot k The calculation formulas are used to calculate the ratio of the primary and secondary peaks R and the skewness S of the similarity graph, respectively. k And set a threshold λ r and λ s ;

[0082] Among them, the ratio of primary to secondary peaks R in S31 and the skewness S of the similarity map k The calculation formula is as follows:

[0083]

[0084] Where IoU is the crossover ratio between the main peak window and the secondary peak window, and ε is the machine epsilon introduced to prevent the crossover ratio from being 0; it is a number close to 0 but greater than 0. P p Main peak intensity, P s This is the second-highest intensity.

[0085]

[0086] in, x is the mean. i Let n be the i-th element and n be the number of samples.

[0087] S32. When the ratio R of the primary and secondary peaks of the similarity graph is greater than λ r And skewness S k Greater than λ s At that time, it is considered that only the maximum value of the similarity graph is suitable for calculating the matching point, thus obtaining a reliable matching point pair;

[0088] S33. By processing and filtering the above similarity graph, a set of matching point pairs was obtained;

[0089] S34. Starting from the geometric transformation relationship model of matching point pairs, use the FSC method to eliminate outlier matching points.

[0090] The measured data used in this invention comes from airborne SAR, specifically as follows: Figure 4 As shown, where Figure 4 (a) shows a typical weak feature region in the airborne SAR measured data 1. Figure 4 (b) is a schematic diagram showing a small number of islands scattered in the water area, based on airborne SAR measured data 2. Figure 4 It can be seen from the measured data that a small number of islands are scattered in a large area of ​​weak feature. Figure 5 To compare the matching results of typical weak feature images before and after correction using the method of this invention, from... Figure 5 It can be seen that the SAR image corrected using this method has an accurate correspondence with the optical image.

[0091] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A robust region-adaptive heterogeneous image matching method, characterized in that, Includes the following steps: S1. Region adaptive feature point extraction based on VE-FAST; S1 includes the following steps: S11. Introduce a block-segmentation strategy to divide the image into N×N sub-image blocks, record the FAST feature point scores of each sub-image block, and sort them in descending order; S12. According to information theory, based on information entropy H... I The calculation formula states that for each sub-image patch, the information entropy must be greater than a set threshold λ. e Before the selection m u m points are selected as candidate points. If any do not meet the set information entropy threshold, m points are extracted. w There are candidate points, and the information entropy H I The calculation formula is as follows: Where, p i Let be the probability of the i-th gray level appearing, where i∈[1,k]; S13. Determine the template region and search region based on each candidate point, and calculate the variance product Q of the template region and search region corresponding to each candidate point. p ; Q, the product of the variances of the template region and the search region corresponding to each candidate point in S13, is... p The calculation formula is as follows: Among them, V o The variance of the template region. The minimum variance of all template regions. V represents the maximum variance of the template region. s The variance of the search region, The minimum variance across all search regions. This represents the maximum variance of the search region. For a size of M I ×N I The variance V of the image I(x,y) is defined as follows: Where, μ I The average pixel value of the image is obtained by the following formula (4). S14. Based on the product of variances Q p The threshold of the product of sum and variance λ v Based on the size relationship, feature points that are adaptively and uniformly distributed in the region are extracted; S2. Similarity map generation based on region feature matching; S3. Matching point filtering based on SK-NMS.

2. The robust region-adaptive heterogeneous image matching method according to claim 1, characterized in that, S2 includes the following steps: S21. Using the template region of the reference optical image and the search region of the SAR image to be matched, feature descriptors are constructed using MOGF respectively; S22. Cross-correlation is used as an indicator to evaluate the similarity between feature descriptors in the template region and the search region. This process is transformed to the frequency domain through Fourier transform. Multiplication operations in the frequency domain are used to replace convolution operations in the spatial domain to calculate the normalized cross-correlation value between the template region and the search region in the frequency domain, thereby obtaining the similarity map.

3. The robust region-adaptive heterogeneous image matching method according to claim 2, characterized in that, S21 includes the following steps: S211. Use filters to obtain the gradient magnitude IG and direction OG of the image in the x and y directions; S212. Considering the robustness of the algorithm, the Sobel operator, which takes into account comprehensive factors, is selected to extract gradient information for optical images, and the ROEWA operator is selected to be used for SAR images, considering the influence of multiplicative speckle noise. S213. Use the following formula to obtain the gradient vector of each pixel:

4. The robust region-adaptive heterogeneous image matching method according to claim 1, characterized in that, S3 includes the following steps: S31. Using the ratio of primary to secondary peaks R and the skewness S of the similarity plot k The calculation formulas are used to calculate the ratio of the primary and secondary peaks R and the skewness S of the similarity graph, respectively. k And set a threshold λ r and λ s ; S32. When the ratio R of the primary and secondary peaks of the similarity graph is greater than λ r And skewness S k Greater than λ s At that time, it is considered that only the maximum value of the similarity graph is suitable for calculating the matching point, thus obtaining a reliable matching point pair; S33. By processing and filtering the above similarity graph, a set of matching point pairs was obtained; S34. Starting from the geometric transformation relationship model of matching point pairs, use the FSC method to eliminate outlier matching points.

5. A robust region-adaptive heterogeneous image matching method according to claim 4, characterized in that, The ratio R of primary and secondary peaks in S31 and the skewness S of the similarity map k The calculation formula is as follows: Where IoU is the crossover ratio between the main peak window and the secondary peak window, ε is the machine zero introduced to prevent the crossover ratio from being 0, and it is a number close to 0 but greater than 0, P p Main peak intensity, P s This is the second-highest intensity; in, x is the mean. i Let n be the i-th element and n be the number of samples.

Images