Fast and scale and rotation invariant multimodal image matching method

Abstract

The application discloses a kind of fast and multi-modal image matching method with scale and rotation invariance, comprising the following steps: step 1, local intensity binary conversion, step 2, FAST feature detection and projection-based scale space construction, step 3, rotate local image block and use SIFT-like descriptor to describe feature points, step 4, with Euclidean distance as the measure, use the nearest neighbor distance ratio-free brute force matching algorithm to match.The application first proposes a kind of local intensity binary conversion model, the similarity between multi-modal images after the conversion is significantly improved, can effectively reduce the nonlinear radiation difference between multi-modal images.Meanwhile the application also proposes a new scale space construction method, i.e.projection-based scale space, not only realizes scale and rotation invariance, but also can realize fast high-precision matching on multi-type large-size multi-modal images, has higher application value and wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of photogrammetry and remote sensing, computer vision and three-dimensional reconstruction, and particularly relates to a fast multi-modal image matching method with scale and rotation invariance. BACKGROUND

[0002] In recent years, photogrammetry and remote sensing technology has developed rapidly, and it has been widely used in rescue and disaster relief, surveying and mapping, infrastructure construction, urban planning and other aspects. Among them, three-dimensional reconstruction, aerial triangulation, target change detection and other technologies provide strong support for smart city, geographic information system database update, major disaster emergency, geographic conditions monitoring and other applications. Image matching is the core basic technology among them, and high-precision and stable and reliable image matching technology is the premise and guarantee for the rapid development of these applications.

[0003] With the rapid development of sensor technology and the increasing application demand, single modal image information is not enough, and comprehensive utilization of multi-modal image data to realize the complementary advantages of information has become necessary, and multi-modal image matching is the premise and basis for realizing the fusion of these image data. Multi-modal image matching refers to the matching process of multi-sensor images or multi-temporal images with different imaging mechanisms. However, multi-modal images usually have serious nonlinear radiometric distortion and complex geometric deformation. The classical feature matching method can usually only handle linear radiometric differences, and basically has no way to deal with nonlinear radiometric differences. At the same time, the existing multi-modal matching method is generally only suitable for matching between a certain type of multi-modal images, and lacks a multi-modal image matching method that is fast, high-precision, scale and rotation invariant, and suitable for multiple types of multi-modal images. SUMMARY

[0004] In view of the above problems, the present application provides a fast multi-modal image feature matching method with scale and rotation invariance and anti-nonlinear radiometric distortion. The method first proposes a local intensity binary transformation model, and the similarity between the multi-modal images after the transformation is significantly improved, which can effectively reduce the nonlinear radiometric difference between the multi-modal images. At the same time, the present application also proposes a new scale space construction method, i.e. a projection-based scale space. The method not only realizes scale and rotation invariance, but also realizes fast and high-precision matching on multi-type large-size multi-modal images.

[0005] The application provides a high-precision multi-modal image feature matching method with scale and rotation invariance. The method first performs local intensity binary transformation (LIBT) on multi-modal image pairs to weaken the nonlinear radiation difference between the multi-modal images. Then, scale space construction based on projection is performed on the target image to achieve the scale invariance of the algorithm, and the rotation invariance of the algorithm is achieved by constructing a gradient direction histogram and local extreme value method. The method can achieve fast and high-precision matching on multi-type multi-modal images.

[0006] To achieve the above purpose, the application designs a multi-modal image feature matching method with scale and rotation invariance and anti-nonlinear radiation distortion, which includes the following steps:

[0007] Step 1), performing local intensity binary transformation on the reference image and the target image;

[0008] Step 2), detecting FAST features on the transformed reference image and target image and constructing a scale space based on projection;

[0009] Step 3), calculating the main direction of the feature points, rotating the local image block and generating a feature vector;

[0010] Step 4), using the Euclidean distance as the metric and adopting a brute-force matching algorithm without nearest neighbor distance ratio to match the feature vectors.

[0011] Further, the specific implementation of step 1) is as follows:

[0012] Let the pixel value of pixel point p in the image be I(p), and Φ p is a circular region centered at pixel point p. All pixel points in Φ p are binary coded, and then the pixel value of the transformed pixel point p is obtained through normalization The mathematical expression is as follows:

[0013]

[0014] In the formula, δ[x] is an indicator function, which is 1 if x is true, and 0 otherwise; q i is the i-th pixel point in Φ p ; |Φ p | is the number of all pixel points in Φ p .

[0015] Further, the specific implementation of step 2) is as follows:

[0016] Step 2.1, first, the FAST algorithm is used to detect the feature points of interest, and the reference image and the target image are detected for FAST corner points respectively;

[0017] Step 2.2, then, the ANMS algorithm is used to obtain a plurality of uniformly distributed feature points;

[0018] Step 2.3, finally, the target image is up-sampled and down-sampled by using the scale space construction method, specifically: the size of the target image is [w, h], assuming that the number of up-sampling or down-sampling is K, and the scale factor is s, then the pyramid contains 2K+1 layers of images in total, the first layer of image is located at the bottom of the pyramid, and the size of the i-th layer of image is [w i ,h i ] is calculated by the following formula:

[0019] [w i ,h i ]=s K+1-i [w,h] (2)

[0020] Wherein, i∈{1,2,…,2K+1}, in order to avoid the distance between feature points being too small, which may cause the local image block to overlap too much during feature description, and affect the subsequent matching, only part of the feature points are randomly selected for projection during down-sampling, and the number of feature points N i of the i-th layer of image is calculated by the following formula:

[0021]

[0022] In the formula, N is the total number of feature points detected in the original target image.

[0023] Further, the specific implementation of the ANMS (adaptive non-maximal suppression) algorithm for suppressing and removing clustered feature points is as follows:

[0024] First, N k >2M feature points are selected from the FAST features according to the Harris response score to form a feature point set C Then, for each feature point k i ∈C, the K-D tree algorithm is used to search for its neighboring feature points within a radius of These neighboring feature points are removed, and finally M uniformly distributed feature points are obtained.

[0025] Further, the specific implementation of step 3 is as follows:

[0026] Step 3.1, for each feature point, a local image block is selected around the feature point in the image after LIBT; ​

[0027] Step 3.2, the gradient of each pixel in the local image block is calculated, the gradient direction ranges from [0°, 360°), which is the same as SIFT, and the gradient direction histogram is formed by statistics, and the maximum direction and the maximum direction greater than P% are taken as the main direction of the feature point;

[0028] Step 3.3, the local image block is rotated according to the main direction;

[0029] Step 3.4, the size of the local image block is adjusted to JxJ pixels;

[0030] Step 3.5, the local image block is divided into N grid x N grid sub-regions, and N bin column histogram statistics are performed on each sub-region;

[0031] Step 3.6, all histogram vectors are sequentially spliced into a feature vector of N grid x N grid x N bin dimension, and normalization processing is further performed to reduce the influence of light changes.

[0032] Further, step 4) uses the Euclidean distance as a measure and adopts a brute force matching algorithm without a nearest neighbor distance ratio to match the feature vectors, first, the feature vectors of the reference image are matched with the feature vectors of each layer of image in the target image pyramid to obtain a matching result, and then the best match is found in the above matching result, which can effectively reduce the matching search space and reduce the probability of false matching and improve the correct matching rate.

[0033] Further, the FAST parameter threshold in step 2.1 is 0.001, and 5000 evenly distributed feature points are obtained by the ANMS algorithm in step 2.2.

[0034] Further, K=3 in step 2.3.

[0035] Further, in step 3.2, the maximum direction and the maximum direction greater than 80% are taken as the main direction of the feature point.

[0036] Further, in step 3.5, N grid =8, N bin =4, and J=96.

[0037] The present application has the following positive effects:

[0038] 1) The application solves the bottleneck problem of severe non-linear radiation distortion in multi-modal image matching. The application proposes a local intensity binary conversion method, and the non-linear radiation difference between the multi-modal images after the conversion can be effectively reduced, thereby effectively improving the accuracy of subsequent matching.

[0039] 2) The application proposes a projection-based scale space construction method, and the matching algorithm can realize scale and rotation invariance at the same time, without relying on any geometric geographic prior information, and has high application flexibility and applicability.

[0040] 3) The application is suitable for multi-type multi-modal image matching, such as optical image and optical image, optical image and infrared image, optical image and SAR image, optical image and laser point cloud depth map, optical image and map data, and optical image and night image matching, and has a wide range of applications.

[0041] 4) The application can also realize fast and high-precision matching on large-size multi-modal images, and has high application value.

[0042] The application can realize feature matching of multi-modal images with severe non-linear radiation distortion, and has the advantages of high precision, large number of matching and fast operation speed. The proposed method can be applied to multi-modal image matching with scale and rotation changes, and has high matching precision for multi-type multi-modal image data. The proposed method has high application value in image stitching, image fusion, aerial triangulation, visual navigation and positioning, target tracking and other related fields, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The flowchart of the application.

[0044] Figure 2 The local intensity binary conversion example in the application.

[0045] Figure 3 The construction of the projection-based scale space in the application.

[0046] Figure 4 The feature descriptor in the application.

[0047] Figure 5 The experimental results of the multi-modal matching method in the application. DETAILED DESCRIPTION

[0048] The technical solutions of the application will be further described below in conjunction with the drawings of the specification.

[0049] As Figure 1As shown, a fast and scale and rotation invariant multi-modal image feature matching method comprises the following steps:

[0050] Step 1), the reference image and the target image are transformed by a local intensity binary transformation formula, specifically: let the pixel value of the pixel point p in the image be I(p), and p for the circular region with the pixel point p as the center, the binary coding is performed on all the pixel points in p , and then the pixel value I(p) of the transformed pixel point p is obtained through normalization, and the mathematical expression is as follows:

[0051]

[0052] In the formula, δ[x] is an indicator function, and the function value is 1 if x is true, otherwise 0; q i is the i-th pixel point in p ; | is the number of all pixel points in p . p .

[0053] The step 1 performs local intensity binary transformation on the reference image and the target image, and first verifies through experiments that the LIBT can effectively reduce the nonlinear radiation difference between the multi-modal images and can improve the similarity between the images. The specific parameters of the LIBT are as follows: the processing radius is 4 pixels, and specifically includes 60 pixel points in the circular neighborhood with the pixel point as the center. The 60 adjacent pixel points are binary coded, and then the pixel value of the center pixel point is obtained through normalization.

[0054] Figure 2 It is an example diagram of the local intensity binary conversion in the application.

[0055] Step 2), the FAST features are detected on the transformed reference image and target image, and a projection-based scale space is constructed. Specifically:

[0056] Step 2.1, first, the FAST algorithm (the FAST parameter threshold is set to 0.001) is used to detect the feature points of interest, and the FAST corner detection is performed on the reference image and the target image respectively.

[0057] Step 2.2, then, the ANMS algorithm is used to obtain 5000 uniformly distributed feature points. Specifically, the neighborhood search range of each feature point is set according to the size of the image, if there are other feature points in the neighborhood range, these adjacent feature points are removed from the feature point set, and finally 5000 uniformly distributed feature points are obtained.

[0058] To make the feature points evenly distributed for matching, the adaptive non-maximal suppression (ANMS) algorithm is used to suppress and remove the clustered feature points. First, N k >2M feature points form a feature point set Then, for each feature point k i ∈C, its neighboring feature points are searched within a radius of using the K-D tree algorithm These neighboring feature points are removed, and finally M evenly distributed feature points are obtained; wherein w and h represent the width and height of the input image, respectively.

[0059] Step 2.3, finally, the scale space construction method proposed in the application is used to up-sample and down-sample the target image, and a feature point pyramid with K=3 and a total of 7 layers is constructed. Figure 3 The K=2 projection-based scale space construction schematic diagram in the application is shown.

[0060] The target image is constructed based on the projection of the scale space, and the construction steps of the traditional pyramid scale space are opposite. The method is to first detect feature points on the original image, and then project the detected feature points to different scale images through up-sampling and down-sampling. Specifically, the target image size is [w, h], and the up-sampling or down-sampling number is K, and the scale factor is s. Then the pyramid contains a total of 2K+1 layers of images, the first layer of image is located at the bottom of the pyramid, and the size of the i-th layer (i∈{1, 2,..., 2K+1}) image is [w i ,h i ] and the calculation formula is:

[0061] [w i ,h i ]=s K+1-i [w,h] (5)

[0062] To avoid the distance between feature points being too small, which may cause the local image block overlap area to be too large during feature description, and affect the subsequent matching, the method randomly selects part of the feature points for projection during down-sampling. The mathematical formula of the number of feature points N i of the i-th layer image is as follows:

[0063]

[0064] In the formula, N is the total number of feature points detected on the original target image.

[0065] Step 3), the main direction of the feature point is calculated, the local image block is rotated, and the feature is described by using the SIFT-like feature descriptor.

[0066] Step 3.1, for each feature point, select a local image block centered at the feature point on the image after LIBT.

[0067] Step 3.2, calculate the gradient of each pixel in the local image block, the gradient direction range is [0°, 360°), same as SIFT, and the maximum value direction and the direction greater than 80% of the maximum value are taken as the main direction of the feature point.

[0068] Step 3.3, rotate the local image block according to the main direction.

[0069] Step 3.4, resize the local image block to JxJ pixels.

[0070] Step 3.5, divide the local image block into N grid x N grid (N grid = 8) sub-regions, and perform N bin (N bin = 4) column histogram statistics on each sub-region.

[0071] Step 3.6, sequentially splice all histogram vectors into an N grid x N grid x N bin dimensional feature vector, and further normalize it to reduce the influence of light changes.

[0072] In this embodiment, first, for the reference image, a local image block of JxJ (J = 96) pixels is selected centered at the feature point, and for the target image, a local image block of sxsxJxJ (s is a scale factor corresponding to the target image) pixels is selected. The gradient of each pixel in the local image block is calculated, and a gradient direction histogram is generated by statistics, and the maximum value and the direction greater than 80% of the maximum value are taken as the main direction of the feature point. The local image block is rotated according to the main direction, and the size of the local image block is resized to JxJ pixels.

[0073] Then, the local image block is divided into 8x8 sub-regions, and a 4-column gradient direction histogram is generated by statistics of the gradient of each pixel in the sub-region. The histogram vectors of each sub-region are sequentially spliced into an 8x8x4 feature vector with a total of 256 dimensions. Figure 4 A schematic diagram of the feature descriptor used in the present application is shown.

[0074] Finally, the feature vector is normalized.

[0075] Step 4), the nearest Euclidean distance between the feature vectors is calculated and the matching is performed by using the brute-force matching algorithm without nearest neighbor distance ratio (NNDR). Different from the general matching algorithm, in this method, the feature vector of the reference image is not matched with all the feature vectors of the target image, but is matched with the feature vectors of each layer of the target image pyramid respectively, and then the best matching is searched from the above matching results. This method can effectively reduce the matching search space and reduce the probability of false matching to improve the correct matching rate.

[0076] Firstly, the feature vector of the reference image is matched with the feature vectors of the target image at different scales respectively by using the brute-force matching algorithm without nearest neighbor distance ratio (NNDR) with the Euclidean distance as the measurement, and 7 matching point sets at different scales are obtained.

[0077] Finally, the best matching point set is searched from the 7 matching point sets at different scales. Figure 5 The experimental result examples of the multi-modal image matching method of the present application are shown.

[0078] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fast, scale- and rotation-invariant multimodal image matching method, characterized in that, Includes the following steps: Step 1), perform local intensity binary transformation on the reference image and the target image; The specific implementation method of step 1) is as follows; Let the pixels in the image be The pixel value , In pixels The central circular area, for All pixels within the array are binary encoded, and then normalized to obtain the transformed pixels. pixel values Its mathematical expression is as follows: (1) In the formula, For indicator functions, if If the result is true, the function value is 1; otherwise, it is 0. for The i-th pixel within; for The number of all pixels in the array; Step 2) Detect FAST features on the transformed reference image and target image and construct a projection-based scale space; Step 3), calculate the principal direction of the feature points, rotate the local image patch and generate the feature vector; The specific implementation method of step 2) is as follows; Step 2.1: First, the FAST algorithm is used to detect interest feature points, and FAST corner detection is performed on the reference image and the target image respectively; Step 2.2: Then, the ANMS algorithm is used to obtain several uniformly distributed feature points; Step 2.3, finally, uses a scale-space construction method to upsample and downsample the target image, specifically: the target image size is... Let the number of upsampling or downsampling operations be . The scaling factor is The pyramids contain a total of Layer image, the first layer is located at the bottom of the pyramid, the second layer... Layer image size The calculation formula is: (2) in, To avoid excessive overlap of local image patches during feature description due to small distances between feature points, which could affect subsequent matching, only a subset of feature points are randomly selected for projection during downsampling. Number of feature points in layer image The mathematical formula is as follows: (3) In the formula, This represents the total number of feature points detected on the original target image; Step 4) Using Euclidean distance as the metric, a brute-force matching algorithm without nearest neighbor distance ratio is used to match the feature vectors.

2. The fast and scale- and rotation-invariant multimodal image matching method according to claim 1, characterized in that: The specific implementation of the ANMS (adaptive non-maximal suppression) algorithm for suppressing and removing clustered feature points is as follows; First, select from FAST features based on Harris response scores. Each feature point forms a feature point set. Then, for each feature point In a radius of Use the KD-tree algorithm to search for its neighboring feature points within the range. Then, these neighboring feature points are removed, and the final result is obtained. A uniformly distributed feature point.

3. The fast and scale- and rotation-invariant multimodal image matching method according to claim 1, characterized in that: The specific implementation method of step 3 is as follows; Step 3.1: For each feature point, select a local image patch centered on the feature point on the LIBT-processed image. Step 3.2: Calculate the gradient of each pixel in the local image patch, with the gradient direction ranging from... Similar to SIFT, gradients are statistically analyzed to form a gradient direction histogram, and the direction of the maximum value and the direction of the maximum value greater than P% are taken as the main direction of the feature point. Step 3.3: Rotate the local image block according to the main direction; Step 3.4, readjust the size of the local image patch. Pixel; Step 3.5: Divide the local image block into... Sub-regions, and perform operations on each sub-region. Column histogram statistics; Step 3.6: Concatenate all histogram vectors sequentially to form... The feature vectors are normalized to reduce the impact of changes in illumination.

4. The fast and scale- and rotation-invariant multimodal image matching method according to claim 1, characterized in that: Step 4) Using Euclidean distance as a metric, a brute-force matching algorithm without nearest neighbor distance ratio is used to match the feature vectors. First, the feature vectors of the reference image are matched with the feature vectors of each layer of the target image pyramid to obtain the matching results. Then, the best match is found among the above matching results. This method can effectively reduce the matching search space and reduce the probability of mismatch and improve the correct matching rate.

5. The fast and scale- and rotation-invariant multimodal image matching method according to claim 1, characterized in that: In step 2.1, the threshold of the FAST parameter is set to 0.

001. In step 2.2, 5000 uniformly distributed feature points are obtained through the ANMS algorithm.

6. The fast and scale- and rotation-invariant multimodal image matching method according to claim 1, characterized in that: In step 2.3 .

7. The fast and scale- and rotation-invariant multimodal image matching method according to claim 3, characterized in that: In step 3.2, the direction of the maximum value and the direction of the maximum value greater than 80% are taken as the main direction of the feature point.

8. The fast and scale- and rotation-invariant multimodal image matching method according to claim 3, characterized in that: In step 3.5 , , .

Images