A two-stage multi-modal image matching method based on point-line feature description

Abstract

The application discloses a two-stage multi-mode image matching method based on point-line feature description, which comprises the following steps: calculating the minimum moment and the maximum moment of phase consistency of each pixel point in a reference image and a to-be-matched image respectively; extracting feature points and line segments of the original input image from the minimum moment graph and the maximum moment graph respectively; for each feature point in the image, constructing a line segment context descriptor of the feature point and clustering; realizing class matching by comparing the similarity of the center description vectors of each feature point class in two images; for each feature point in the matched two groups of feature point classes, constructing a phase consistency maximum response direction histogram descriptor of the feature point; calculating the similarity of the feature descriptors and performing bidirectional feature matching. The application can effectively overcome the problems of multi-mode image imaging gray, large difference in resolution, difficulty in feature point description and low matching efficiency, and the two-stage matching strategy can improve the efficiency and reliability of multi-mode image feature point matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image feature extraction and matching, and in particular relates to a two-stage multimodal image matching method based on point and line feature description. Background Art

[0002] Multimodal image matching is a crucial prerequisite for multimodal image fusion processing. It aims to match two or more images of the same scene, acquired by different imaging devices under different imaging conditions. Due to the varying imaging mechanisms and conditions of multimodal images, significant differences in grayscale, resolution, and even background content can occur between them, posing a significant challenge to multimodal image matching.

[0003] Feature-based matching methods extract image features from two images as matching primitives to determine the correspondence between the reference image and the image to be matched. They have received extensive attention and research due to their high computational efficiency and good robustness to rotation and scale changes between images.

[0004] The most representative feature matching algorithm is the SIFT algorithm proposed by Lowe (D.G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision 60(2)(2004)91-110.). This algorithm has been widely used in natural image matching and provides a basic idea for subsequent feature-based image matching algorithms. However, its performance is poor in multimodal image matching applications. Based on this algorithm, Chen et al. proposed a partial intensity invariant feature descriptor (J.Chen, J.Tian, ​​N.Lee, J.Zheng, R.T.Smith, A.F.Laine, Apartial intensity invariant feature descriptor for multimodal retinal image registration, IEEE Transactions on Biomedical Engineering, 57(7)(2010)1707-1718.) to address the grayscale difference problem between multimodal images. This algorithm has been widely used and improved in multimodal retinal image matching. Considering the problem of difficulty in feature point extraction and description caused by the imaging grayscale differences between multi-mode images, Sui et al. adopted a more stable straight line feature and proposed an iterative feature extraction strategy to optimize the extracted image features (H. Sui, C. Xu, J. Liu, F. Hua, Automatic Optical-to-SAR Image Registration by Iterative Line Extraction and Voronoi Integrated Spectral Point Matching, IEEE Transactions on Geoscience and Remote Sensing, 53(11)(2015)6058-6072.).Wang et al. proposed a straight line segment context descriptor to globally describe the extracted straight line features, and constructed an iterative optimization strategy to overcome the problem of incomplete image straight line extraction (Z.Wang, X.Feng, Y.Wu, G.Xu, M.Qian, An automatic method for matching salientstructures in optical remote sensing images, International Journal of RemoteSensing, 42(21)(2021)8298-8317.).

[0005] Although the above-mentioned existing feature-based matching methods have achieved good matching results in multimodal image matching tasks in specific scenarios, they still have the following defects: First, most of the existing methods only extract feature points of multimodal images as matching primitives, and use local information of feature points to construct feature descriptors. Their matching effects are greatly affected by the imaging grayscale differences between multimodal images; second, the matching methods proposed by researchers in recent years consider using more stable straight line segments extracted from images as matching primitives, but ignore the problem that it is difficult to accurately solve the transformation matrix between images through matching straight line segments; third, there are many interference factors that affect matching between multimodal images. It is difficult to achieve good image matching effects by using image features and their descriptors extracted once. Summary of the Invention

[0006] 1. Technical problem to be solved by the invention

[0007] The purpose of the present invention is to overcome the defects existing in the above-mentioned prior art and provide a two-stage multimodal image matching method based on point and line feature description. The present invention solves the problems of difficulty in describing image feature points and low matching efficiency in multimodal image matching tasks, and improves the efficiency and reliability of multimodal image feature point matching.

[0008] 2. Technical solution

[0009] In order to achieve the above object, the technical solution provided by the present invention is:

[0010] The present invention provides a two-stage multimodal image matching method based on point and line feature description, comprising the following steps:

[0011] Step 1: Calculate the minimum moment and maximum moment of phase consistency at each pixel in the reference image and the image to be matched respectively;

[0012] Step 2: Extract the feature points and straight line segments of the original input image from the minimum moment map and the maximum moment map respectively;

[0013] Step 3: For each feature point in the image, construct its line segment context descriptor and perform clustering to form a feature point class;

[0014] Step 4: Feature point class matching is achieved by comparing the similarity of the central description vectors of each feature point class in the two images;

[0015] Step 5: For each feature point in the two matched feature point classes, construct its phase consistency maximum response direction histogram descriptor;

[0016] Step 6: For each pair of feature points in the two matched feature point classes, calculate the similarity of the feature descriptors and perform bidirectional feature matching to obtain the final image matching result.

[0017] 3. Beneficial effects

[0018] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:

[0019] (1) In view of the fact that most existing feature-based image matching methods only extract feature points of multimodal images as matching primitives and use local information of feature points to construct feature descriptors, which are easily affected by the imaging grayscale differences between multimodal images, leading to the problem of feature mismatching, the present invention simultaneously extracts feature points and straight line segments of the image based on the phase consistency model, effectively overcoming the influence of imaging grayscale differences on feature extraction.

[0020] (2) The present invention provides a two-stage multimodal image matching method based on point and line feature description, which divides feature matching into two stages. First, the position information of the extracted straight line segment relative to the feature point is used to construct a straight line segment context descriptor of the feature point to describe it globally, and a feature point class is formed through a clustering algorithm to achieve coarse matching of the feature point class. Secondly, within the corresponding feature point class, a phase consistency maximum response direction histogram descriptor is constructed to achieve precise matching of the feature points. On the one hand, the constructed descriptor has high robustness to imaging grayscale differences. On the other hand, the two-stage matching strategy helps to improve the efficiency of feature point matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the process of two-stage multimodal image matching of the present invention;

[0022] Figure 2 Schematic diagram of the method for calculating the contextual feature descriptor of a straight line segment of the present invention; specifically, Figure 2 (a) is the feature point c i Relative to the straight line segment l j Schematic diagram of the position relationship; Figure 2 (b) is the process of constructing the straight line segment context descriptor.j Schematic diagram of the weight calculation method. DETAILED DESCRIPTION

[0023] In view of the fact that most existing feature-based image matching methods only extract feature points of multimodal images as matching primitives and use local information of feature points to construct feature descriptors, they are easily affected by the imaging grayscale differences between multimodal images, leading to the problem of feature mismatching. In practical applications, there may also be geometric changes such as translation, rotation and scale between multimodal images, which further affect the efficiency and reliability of image feature point matching. The present invention simultaneously extracts feature points and straight line segments of the image based on the phase consistency model, effectively overcoming the influence of imaging grayscale differences on feature extraction. The constructed straight line segment context descriptor and phase consistency maximum response direction histogram descriptor are both highly robust to the imaging grayscale differences between multimodal images. In addition, the adopted two-stage image matching strategy helps to improve the efficiency of feature point matching.

[0024] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments.

[0025] Example 1

[0026] Combine Figure 1-Figure 2 In this embodiment, a two-stage multimodal image matching method based on point and line feature description includes the following steps:

[0027] Step 1: Calculate the minimum moment and maximum moment of phase consistency at each pixel in the reference image and the image to be matched respectively.

[0028] The specific process of calculating the minimum moment and maximum moment of phase consistency is as follows:

[0029] Step 1-1: Convolve the input image using a Log-Gabor filter and calculate the phase consistency of each pixel in the input image in all directions based on the Kovesi algorithm:

[0030]

[0031] Among them, o and n are the direction mark and scale mark of the Log-Gabor filter respectively, PC(x,y,θ o ) is the phase consistency of the image pixel coordinate (x, y) in the direction o, θ o is the corresponding angle of direction o, W o Represents the weight coefficient of frequency expansion, A no represents the amplitude of the Log-Gabor filter with direction o convolved with the image at scale n, ΔΦ no is a phase difference function, T is the noise threshold, and ε is a very small constant to avoid division by 0;

[0032] Step 1-2: Calculate the following variables at each pixel in the image:

[0033]

[0034] Then, the minimum and maximum moments of phase consistency of each pixel are calculated:

[0035]

[0036] Where m and M represent the minimum moment and maximum moment of phase consistency, respectively.

[0037] Step 2: Use the Harris-Affine algorithm to extract feature points from the minimum moment graph, and use the LSD (Line Segment Detector) algorithm to extract straight line segments from the maximum moment graph.

[0038] Step 3: For each feature point in the image, construct its line segment context descriptor. According to the line segment context descriptor, cluster the feature points to form feature point classes. The specific process of calculating the line segment context descriptor and clustering the feature points is as follows:

[0039] Step 3-1: From step 2, we can get the feature point set and straight line segment set in the original input image, which are respectively denoted as and where n c and n l Indicates the number of extracted feature points and straight line segments. Figure 2 As shown in (a), for any feature point c in the feature point set i , straight line segment l j The position relative to the feature point can be expressed as w ij =(α ij ,β ij ), where α ij is the feature point c i To the straight line segment l j The perpendicular line and the characteristic point c i Angle between the main directions, β ij is the straight line segment l j and feature point c i Angle of the main direction;

[0040] Step 3-2: For all straight line segments l j (j=1,2,…,n l ), calculate the description vector w ij , then the feature point c i The positional relationship with all straight line segments can be described as

[0041] Step 3-3, divide the value range of α [0, 2π) into 8 angle intervals, and the value range of β [0, π) into 4 angle intervals, and calculate W i The histogram descriptor is as follows:

[0042]

[0043] in, is the kth element value in the histogram, bin(k) is the kth angle interval, Γ(a ij )-Γ(b ij ) is the straight line segment l j The weight of Figure 2 As shown in (b), Γ(·) is the distribution function of the standard normal distribution, with the characteristic point c i To the straight line segment l j The foot of the perpendicular is the origin, along the straight line segment l j The coordinate system is established in the direction of a ij and b ij are straight line segments l j The coordinates of the two endpoints of the feature point c can be obtained i The contextual descriptor of the straight line segment is the histogram

[0044] Step 3-4: Calculate the line segment context descriptors for all feature points in both the reference image and the image to be matched, respectively, using the k-means algorithm. Cluster the feature points based on their line segment context descriptors in both images. The initial number of cluster centers, K, in the k-means algorithm can be set between 10 and 20. Experimental results have shown that setting K to 15 yields the best matching results.

[0045] Step 4: Feature point class matching is achieved by comparing the similarity of the central description vectors of each feature point class in the two images. The specific process of feature point class matching is as follows:

[0046] Step 4-1: For each class of feature points in the reference image and the image to be matched, calculate the mean of the line segment context descriptors corresponding to all feature points in the class, which is the class center description vector;

[0047] Step 4-2: For each pair of feature point classes in the two images, calculate the similarity of their class center description vectors, and determine the correspondence between the feature point classes through bidirectional matching: Among them, P f and Q f is the fth pair of matching feature point classes in the two images.

[0048] Step 5: For each feature point in the two matched feature point classes, construct its phase consistency maximum response direction histogram descriptor.

[0049] The specific process of calculating the phase consistency maximum response direction histogram descriptor is as follows:

[0050] Step 5-1: For the feature point c in the image i , while extracting feature points in step 2, the main direction φ of the feature points can be calculated simultaneously i and scale s i , in the corresponding maximum moment diagram, along the feature point c i The main direction of the selected 6As around i ×6As i The rectangular area is divided into 6×6 intervals using a grid. The value of parameter A can be set between 5 and 10. Through actual experiments, it was determined that setting parameter A to 8 can achieve the best matching effect.

[0051] Step 5-2: In each interval, accumulate the phase consistency amplitudes in different directions and record the maximum accumulated value and its corresponding direction. Where t=1,2,…,36, then the feature point c can be obtained i The phase consistency maximum response direction histogram descriptor is as follows:

[0052]

[0053] Step 6: For each pair of feature points in the two matched feature point classes, calculate the similarity of the feature descriptors and perform bidirectional feature matching.

[0054] The specific process of feature point matching is as follows:

[0055] Step 6-1, calculating the phase consistency maximum response direction histogram descriptor of all feature points in the reference image and the image to be matched according to step 5;

[0056] Step 6-2: For each pair of feature points in the corresponding feature point class matched in step 4, calculate their similarity as follows:

[0057]

[0058] Among them, c i and c g are the feature points in the reference image and the image to be matched, respectively. and is the contextual descriptor of the straight line segment between two feature points. The value of parameter λ can be set between 0.8 and 1, and the value of parameter ρ can be set between 3 and 5. Through actual experiments, it was determined that the best matching effect was achieved when the parameters λ and ρ were set to 0.9 and 4 respectively.

[0059] Step 6-3: Calculate the similarity of all feature point pairs in the corresponding feature point classes of the two images according to step 6-2, and use the two-way matching method to screen out the corresponding matching feature point pairs. Based on this, calculate the transformation matrix between the reference image and the image to be matched and perform a consistency check to eliminate point pairs with large matching errors. After eliminating point pairs with large matching errors, the remaining point pairs are the obtained image feature point matching results.

[0060] This embodiment preprocesses the multimodal images using a phase congruency model before extracting features, effectively overcoming the impact of grayscale differences on feature extraction. A two-stage image matching strategy improves feature point matching efficiency. The constructed line segment context descriptor and phase congruency maximum response direction histogram descriptor are both highly robust to grayscale differences between multimodal images, ensuring reliable multimodal image matching.

[0061] The above is a schematic description of the present invention and its embodiments, which is not restrictive. The drawings show only one embodiment of the present invention and are not intended to be limiting. Therefore, if a person skilled in the art is inspired by the above and, without departing from the purpose of the present invention, designs structures and embodiments similar to the technical solution without inventive means, they shall fall within the scope of protection of the present invention.

Claims

1. A two-stage multimodal image matching method based on point and line feature description, characterized in that: The following steps are involved: Step 1: Calculate the minimum moment and maximum moment of phase consistency at each pixel in the reference image and the image to be matched respectively; Step 2: extracting feature points and straight line segments of the original input image from the minimum moment map and the maximum moment map respectively; wherein, the Harris-Affine algorithm is used to extract feature points from the minimum moment map, and the LSD algorithm is used to extract straight line segments from the maximum moment map; Step 3: For each feature point in the image, construct its line segment context descriptor and cluster the feature points to form feature point classes. The specific process of calculating the line segment context descriptor and performing feature point clustering is as follows: Step 3-1: From step 2, we can get the feature point set and straight line segment set in the original input image, which are respectively denoted as and where n c and n l Indicates the number of extracted feature points and straight line segments; for any feature point c in the feature point set i , straight line segment l j The position relative to the feature point is represented as w ij =(α ij ,β ij ), where α ij is the feature point c i To the straight line segment l j The perpendicular line and the characteristic point c i Angle between the main directions, β ij is the straight line segment l j and feature point c i Angle of the main direction; Step 3-2: For all straight line segments l j (j=1,2,…,n l ), calculate the description vector w ij , then the feature point c i The positional relationship with all straight line segments is Step 3-3, divide the value range of α [0, 2π) into 8 angle intervals, and the value range of β [0, π) into 4 angle intervals, and calculate W i The histogram descriptor of , from which the feature point c can be obtained i The line segment context descriptor of ; Step 3-4: Calculate the straight line segment context descriptors of all feature points in the reference image and the image to be matched respectively according to step 3-3, and cluster the feature points in the two images using the k-means algorithm; Step 4: Feature point class matching is achieved by comparing the similarity of the central description vectors of each feature point class in the two images; Step 5: For each feature point in the two matched feature point classes, construct its phase consistency maximum response direction histogram descriptor; Step 6: For each pair of feature points in the two matched feature point classes, calculate the similarity of the feature descriptors and perform bidirectional feature matching to obtain the final image matching result.

2. The two-stage multimodal image matching method based on point and line feature description according to claim 1, characterized in that: In step 1, the specific process of calculating the minimum moment and the maximum moment of phase consistency is as follows: Step 1-1: Convolve the input image using a Log-Gabor filter, and calculate the phase consistency of each pixel of the input image in all directions based on the Kovesi algorithm; Step 1-2: Calculate the variables a, b, and c for each pixel in the image, and then calculate the minimum and maximum moments of phase consistency for each pixel.

3. The two-stage multimodal image matching method based on point and line feature description according to claim 2, characterized in that: The value range of the initial cluster center number K set by the k-means algorithm is 10 to 20.

4. The two-stage multimodal image matching method based on point and line feature description according to claim 3, characterized in that: In step 4, the specific process of feature point class matching is as follows: Step 4-1: For each class of feature points in the reference image and the image to be matched, calculate its class center description vector; Step 4-2: For each pair of feature point classes in the two images, calculate the similarity of their class center description vectors, and determine the correspondence between the feature point classes through bidirectional matching: Among them, P f and Q f is the fth pair of matching feature point classes in the two images.

5. The two-stage multimodal image matching method based on point and line feature description according to claim 4, characterized in that: In step 5, the specific process of calculating the phase consistency maximum response direction histogram descriptor is: Step 5-1: For the feature point c in the image i , its main direction φ can be obtained from step 2 i and scale s i , in the corresponding maximum moment diagram, along the feature point c i The main direction of the selected 6As around i ×6As i A rectangular area is evenly divided into 6×6 intervals using a grid; Step 5-2: In each interval, accumulate the phase consistency amplitudes in different directions and record the maximum accumulated value and its corresponding direction. Where t=1,2,…,36, then the feature point c can be obtained i Phase consistency maximum response direction histogram descriptor.

6. The two-stage multimodal image matching method based on point and line feature description according to claim 5, characterized in that: The value range of parameter A is 5 to 10.

7. The two-stage multimodal image matching method based on point and line feature description according to claim 6, characterized in that: In step 6, the specific process of feature point matching is as follows: Step 6-1, calculating the phase consistency maximum response direction histogram descriptor of all feature points in the reference image and the image to be matched according to step 5; Step 6-2: For each pair of feature points in the corresponding feature point class matched in step 4, calculate their similarity; Step 6-3: Calculate the similarity of all feature point pairs in the corresponding feature point classes of the two images according to step 6-2, and use the two-way matching method to screen out the corresponding matching feature point pairs.

Images