Multi-scale multimode image feature matching method based on Gaussian pyramid

Abstract

The invention discloses a multi-scale multi-mode image feature matching method based on a Gaussian pyramid. The method comprises the following steps: respectively constructing Gaussian pyramid images for an infrared image and a visible light image; performing Harris feature point detection on each layer of the Gaussian pyramid image, amplifying each layer to the same size through an up-sampling method, and performing non-maximum suppression on feature points to remove repeated feature points; curvature calculation is carried out on the feature points after non-maximum suppression processing; distributing a main direction for each feature point, and extracting a PIIFD descriptor; according to the method, the similarity of feature descriptors is calculated, a bidirectional matching method and random sampling consistency check are adopted to obtain initial matching point pairs, a projection matrix is iteratively optimized, high-residual matching points are dynamically replaced, meanwhile, a penalty factor is introduced to stabilize convergence, and finally accurate image matching is achieved. The problem of significant modal difference in infrared and visible light image matching tasks is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image feature extraction and matching, and in particular relates to a multi-scale multi-modal image feature matching method based on a Gaussian pyramid. Background Art

[0002] Electrical equipment plays a vital role in power systems, and its operating status directly impacts their stability and safety. Condition monitoring and maintenance of electrical equipment are crucial for ensuring reliable power system operation. Infrared images can effectively detect equipment overheating faults, while visible light images can help identify cosmetic defects and abnormalities. By matching infrared and visible light images, the advantages of both can be combined to provide more comprehensive status information, enabling real-time monitoring of equipment status and prompt identification and resolution of potential safety hazards.

[0003] Currently, multimodal image matching technologies mainly include region-based matching and feature-based image matching methods. Region-based matching methods, such as normalized cross correlation (NCC) and mean square error (MSE), perform matching by comparing regional similarities within an image. However, these methods are sensitive to local changes in the image and have high computational complexity.

[0004] In the development of feature-based image matching methods, Aguilera et al. introduced the Log-Gabor Histogram Descriptor (LGHD) in 2015, which has strong anti-interference ability and strong robustness to nonlinear intensity changes. In 2020, Jiang et al. proposed a method with the midline of the contour as the main feature direction, called Contour Angle Orientation (CAO). This method improves the matching accuracy of infrared images and visible light images of power equipment under rotation and scale changes to a certain extent. However, this method still has shortcomings in the overlap of matching results, and the repeatability of matching results is not high. In 2020, Li et al. proposed a new method called Radiation-variation Insensitive Feature Transform (RIFT), which uses the phase consistency map to extract key points and constructs a maximum index map for feature description, which has good scale invariance and rotation invariance. In 2022, Chen et al. improved the RIFT algorithm and proposed a new binary pattern string to construct a descriptor. This descriptor is more robust to radiation changes and can represent Log-Gabor convolution sequences, reducing computational complexity. These methods have improved matching accuracy to a certain extent, but still suffer from low reproducibility of matching results, high computational complexity, and insufficient robustness to modal differences. Summary of the Invention

[0005] 1. Technical problem to be solved by the invention

[0006] The purpose of the present invention is to overcome the defects in the prior art and provide a multi-scale multi-modal image feature matching method based on Gaussian pyramid to solve the problem of significant modality differences in infrared and visible light image matching tasks.

[0007] 2. Technical solution

[0008] In order to achieve the above object, the technical solution provided by the present invention is:

[0009] A multi-scale multi-modal image feature matching method based on a Gaussian pyramid of the present invention comprises the following steps:

[0010] Step 1: Construct Gaussian pyramid images for infrared images and visible light images respectively;

[0011] Step 2: Perform Harris feature point detection on each layer of the Gaussian pyramid image, then upsample each layer to the same size, and perform non-maximum suppression on the detected feature points to remove duplicate feature points;

[0012] Step 3: Calculate the curvature of the feature points after non-maximum suppression processing;

[0013] Step 4: Calculate the curvature radius based on the curvature of the feature point, assign the main direction to each feature point, and extract the PIIFD descriptor to describe the feature point;

[0014] Step 5: Calculate the similarity of feature descriptors, use bidirectional matching and random sampling consistency check to obtain initial matching point pairs, iteratively optimize the projection matrix and dynamically replace high residual matching points, while introducing a penalty factor to stabilize the convergence process, and output the final image matching result.

[0015] Furthermore, when constructing the Gaussian pyramid image in step 1, a Gaussian filter is used to perform low-pass filtering on the original infrared image and the visible light image respectively, and a downsampling operation is performed. A multi-scale image iterative decomposition method is used to obtain a sampled image until a preset number of pyramid layers is reached.

[0016] Furthermore, the Harris feature point detection in step 2 includes: calculating the local autocorrelation matrix M of each pixel in the Gaussian pyramid image, and screening the candidate feature points through the Harris response function to obtain a candidate feature point matrix R.

[0017] Furthermore, the step 2 performs non-maximum suppression on the detected feature points, specifically: construct a rectangular area window, retain the candidate feature point with the largest response value in the window through comparison, traverse all candidate feature points in the image, repeat the comparison process, and all retained feature points constitute the final feature point set Among them, n f Indicates the number of feature points extracted.

[0018] Furthermore, the curvature calculation in step 3 specifically includes:

[0019] For the feature point set The autocorrelation matrix M of each feature point is decomposed into eigenvalues to obtain eigenvalues λ1 and λ2; the eigenvalues λ1 and λ2 are then used to calculate the curvature k of the feature point. The curvature k is calculated by the following formula:

[0020]

[0021] Calculate the curvature of all feature points and get k i , i=1,2,3…n f .

[0022] Furthermore, the specific process of step 4 is as follows:

[0023] Step 4-1, assign a main direction φ to each feature point;

[0024] Step 4-2: For feature point P i , first calculate the corresponding curvature radius as follows:

[0025]

[0026] Among them, k i is the curvature at the feature point, r i is the curvature radius of the feature point;

[0027] Select the feature point as the center and the curvature radius as r i The circular area is divided into multiple sub-areas. In each sub-area, the gradient magnitude and gradient direction are calculated, and an 8-bin direction histogram is constructed. The gradient magnitude is normalized, and the 8-bin histograms of multiple sub-areas are spliced into a multidimensional feature vector, namely the PIIFD descriptor.

[0028] Furthermore, the feature descriptor similarity calculation in step 5 satisfies:

[0029]

[0030] Among them, P c1 and P c2 are the feature points in the reference image and the image to be matched, respectively. and is the PIIFD descriptor of two feature points.

[0031] Furthermore, in step 5, the corresponding matching feature point pairs are screened out using a two-way matching method based on the calculated feature descriptor similarity, the transformation matrix between the reference image and the image to be matched is calculated based on the corresponding matching feature point pairs obtained after screening, and a random sampling consistency test is performed to eliminate point pairs with large matching errors. After eliminating the point pairs with large matching errors, the remaining point pairs are the initial matching results after screening and testing.

[0032] Furthermore, the specific process of iteratively optimizing the projection matrix and dynamically replacing high residual matching points in step 5 is as follows:

[0033] For the matching point P obtained after the initial matching c1 and P c2 , calculate its projection matrix M;

[0034] Then, we get the value corresponding to the matching point P. c2 The fitting point P 2fit , calculate the fitting point P 2fit and matching point P c2 The residual ε(i);

[0035] Will Sort in ascending order to find the last 1 / 4 elements that have more residuals than the other 3 / 4, and replace P with the fitted points c2 The points corresponding to the last 1 / 4 elements;

[0036] Repeat the above process and iteratively update P c2 until the sum of the residuals is equal to zero, and the final matching result is obtained.

[0037] Furthermore, in step 5, In the process of sorting in ascending order, a penalty factor β is introduced and the residual is updated as follows:

[0038] ε 总 =ε(k)+(k-1)β

[0039] Where k is the number of iterations and ε(k) is the residual at the kth iteration.

[0040] 3. Beneficial effects

[0041] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:

[0042] (1) This invention proposes a multi-scale, multi-modal image feature matching method based on a Gaussian pyramid. This method leverages the multi-scale properties of the Gaussian pyramid to preserve edge and detail information in the image and enhance the saliency of feature points. In combination with the properties of the Gaussian pyramid, an efficient Harris feature extraction and matching algorithm is designed, enabling the matching task to be completed in a relatively short time.

[0043] (2) The present invention provides a multi-scale and multi-modal image feature matching method based on a Gaussian pyramid. By constructing a Gaussian pyramid, the method realizes feature extraction and matching of images at different scales. It can effectively handle feature matching problems under different scales and rotation transformations, and improves matching robustness.

[0044] (3) The multi-scale multi-modal image feature matching method based on Gaussian pyramid of the present invention can more accurately reflect the geometric characteristics of feature points by calculating the curvature radius, extract straight line segments within a specified range, and capture the significant geometric features around the feature points. The extraction of straight line segments and the construction of descriptors have certain rotation invariance and can maintain the stability of matching when the image is rotated. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of a process of a multi-scale multi-modal image feature matching method based on Gaussian pyramid of the present invention;

[0046] Figure 2It is a schematic diagram of the pyramid structure of the present invention. DETAILED DESCRIPTION

[0047] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments.

[0048] Given that existing infrared and visible light images often have different imaging principles and physical properties, and existing feature matching methods lack sufficient robustness to the modal differences and scale changes between infrared and visible light images, resulting in low image matching accuracy, the present invention provides a multi-scale multi-modal image feature matching method based on a Gaussian pyramid. Gaussian pyramid images are constructed for infrared images and visible light images respectively, Harris feature point detection is performed on each layer of the Gaussian pyramid image, and each layer is enlarged to the same size through an upsampling method. Non-maximum suppression is performed on the feature points to eliminate duplicate feature points, and the curvature of the feature points after non-maximum suppression is calculated. The curvature radius is calculated based on the curvature of the feature points, a main direction is assigned to each feature point to represent the overall direction of the feature point, and a PIIFD descriptor is extracted to describe the feature point. By comparing each pair of feature points in the two images, the similarity of the feature descriptors is calculated, and bidirectional feature matching is performed to obtain a final image matching result. It effectively solves the problem of feature matching under different scales and modal differences, improves matching efficiency, and reduces the possible impact of incorrect matching, greatly improving the robustness of the matching algorithm to scale changes and modal differences between infrared and visible light images.

[0049] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments.

[0050] Example 1

[0051] Combine Figure 1-Figure 2 The present embodiment provides a multi-scale multi-modal image feature matching method based on a Gaussian pyramid, comprising the following steps:

[0052] Step 1: Construct Gaussian pyramid images for the infrared image and the visible light image respectively; specifically including:

[0053] Step 1-1: Use Gaussian filters to perform low-pass filtering on the infrared image and the visible light image to remove high-frequency details;

[0054] Steps 1-2: Downsample the infrared image and visible light image processed by the Gaussian filter respectively, and use the multi-scale image decomposition method with a sampling rate of 2 to obtain the sampled image, which is used as the input image of the next level, and the sampling process is repeated until the iteration ends. It can be expressed as

[0055]

[0056] In formula (1): G k (i, j) is the kth layer of the Gaussian pyramid image; G0 is the original image; G1 is the first layer of the Gaussian pyramid, and so on. w(m, n) = h(m) where h(n) is a low-pass window function, where h is the Gaussian density distribution function.

[0057] Step 2: Harris feature point detection is performed on each layer of the Gaussian pyramid image, and then each layer is enlarged to the same size through upsampling method. Non-maximum suppression is performed on the detected feature points to eliminate duplicate feature points;

[0058] Step 2-1: Perform Harris feature point detection on each layer of the Gaussian pyramid image

[0059] The pixel coordinates in the Gaussian pyramid image are (x, y), and the local autocorrelation matrix M of each pixel in the image is calculated as follows:

[0060]

[0061] Among them, I x and I y are the first-order derivatives of the image in the x and y directions, respectively.

[0062] According to the Harris response function R = det(M) - k·trace 2 (M) is used to determine the candidate feature points and obtain a candidate feature point matrix R.

[0063] Among them, k is 0.04, and each element R(x, y) in the candidate feature point matrix R represents a candidate feature point at position (x, y) in the image.

[0064] Upsampling is performed on the candidate feature points of each layer to unify the candidate feature points detected at each layer of the Gaussian pyramid to the same scale. If the image size of the lth layer is the original image Then the coordinates of the candidate feature points of this layer need to be multiplied by 2 l To map back to the original image, the upsampled image size is consistent with the original image size.

[0065] Step 2-2: For each candidate feature point extracted by the Harris response function, define a 3×3 rectangular window. For each candidate feature point (x, y), compare R(x, y) with the response values of other points in its neighborhood. If R(x, y) is the maximum value in the neighborhood, then retain the point; otherwise, remove it. Traverse all candidate feature points in the image and repeat the above comparison process. All retained feature points constitute the final feature point set. Among them, n fIndicates the number of feature points extracted.

[0066] Step 3: Calculate the curvature of the feature points after non-maximum suppression processing; the specific steps are:

[0067] For the feature point set The autocorrelation matrix M of each feature point is decomposed into eigenvalues to obtain eigenvalues λ1 and λ2. The eigenvalues are calculated using the following formula:

[0068]

[0069] in, is the trace of the matrix M, is the determinant of the matrix M.

[0070] Then use the eigenvalues λ1 and λ2 to calculate the curvature k of a single feature point. The curvature k can be calculated as follows:

[0071]

[0072] Use the above steps to calculate the curvature of all feature points and get k i (i=1,2,3…n f ).

[0073] Step 4: The final feature point set obtained in step 2-2 Each feature point in is assigned a main direction to represent the overall direction of the feature point and extract the PIIFD descriptor;

[0074] Step 4-1, the main direction of the image is

[0075]

[0076] Among them, the main direction of each feature point P(x,y) is defined as φ(x,y), and is the mean square gradient of the image.

[0077] Step 4-2: For feature point P i , first calculate the corresponding curvature radius as follows:

[0078]

[0079] Among them, k i is the curvature at the feature point, r i is the curvature radius of the feature point.

[0080] Select the feature point as the center and the curvature radius as r iThe circular region is divided into 4×4 subregions. Within each subregion, the gradient magnitude and gradient direction are calculated, and an 8-bin direction histogram is constructed. The gradient magnitude is normalized to reduce the influence of illumination variations. The 8-bin histograms of the 4×4 subregions are concatenated into a 128-dimensional feature vector, the PIIFD descriptor.

[0081] It is worth noting that the gradient direction is the original direction and needs to be adjusted according to the main direction obtained in step 4-1 to achieve rotation invariance. Therefore, the main direction assigned to the feature point in step 4-1 is used to align the sub-region in step 4-2. Specifically, it is used to rotate the gradient direction of the sub-region relative to the main direction to ensure that the descriptor is robust to image rotation.

[0082] Step 5: Calculate the similarity of feature descriptors, use bidirectional matching and random sampling consistency check to obtain initial matching point pairs, iteratively optimize the projection matrix and dynamically replace high residual matching points, while introducing a penalty factor to stabilize convergence, and finally achieve accurate image matching.

[0083] Step 5-1: By comparing each pair of feature points in the two images, calculate the similarity of the feature descriptors as follows:

[0084]

[0085] Among them, P c1 and P c2 are the feature points in the reference image and the image to be matched, respectively. and is the PIIFD descriptor of two feature points.

[0086] Step 5-2: Calculate the similarity of all feature point pairs in the corresponding feature point classes of the two images according to step 5-1, and use the two-way matching method to screen out the corresponding matching feature point pairs. Calculate the transformation matrix between the reference image and the image to be matched based on the corresponding matching feature point pairs obtained after screening, and perform a random sampling consistency test to eliminate point pairs with large matching errors. After eliminating point pairs with large matching errors, the remaining point pairs are the initial matching results after screening and testing.

[0087] Step 5-3: Get the correct matching point P after initial matching c1 and P c2 , then the least squares solution of the projection matrix M is:

[0088]

[0089] Then, corresponding to the matching point P c2 The fitting points are expressed as follows:

[0090] P 2fit =P c1 M (10)

[0091] P 2fit and P c2 The residual is expressed as:

[0092] ε(i)=||P 2fit (i)-P c2 (i)||2,i=1,...,Nc (11)

[0093] Where Nc is the number of correct matches.

[0094] Will Sort in ascending order to find the last 1 / 4 elements that have more residuals than the other 3 / 4, and replace P with the fitted points c2 The points corresponding to the last 1 / 4 elements in , that is,

[0095] P c2 ={P c2 (1),...,P c2fit (i),...,P c2fit (j),...,P c2 (n)} (12)

[0096] Among them, P c2fit (i) and P c2fit (j) are those points with large residuals.

[0097] Then update P by repeating (9)-(12) c2 , until the sum of the residuals is equal to zero (considering the floating point storage in the computer, zero is thresholded by ε p =10 -4 replace).

[0098] exist In the process of ascending sorting, the points of the other 3 / 4 elements may become larger due to iteration and become the elements in the last 1 / 4, so a penalty factor β is introduced and the residual is updated as

[0099] ε 总 =ε(k)+(k-1)β (13)

[0100] Where k is the number of iterations and ε(k) is the residual at the kth iteration.

[0101] Update P c2 position to obtain the final matching result.

[0102] This embodiment achieves robust capture of multi-scale features by constructing a Gaussian pyramid when extracting features from infrared and visible light images, effectively overcoming the issue of feature scale sensitivity caused by differences in imaging resolution. Non-maximum suppression eliminates redundant feature points, and curvature calculations are combined to screen key points with high geometric stability. This ensures that the extracted features are highly invariant to nonlinear radiation differences and local deformations, laying a solid foundation for multimodal image matching and ensuring the accuracy and efficiency of the matching process. The penalty factor is progressively weighted to enable the optimization process to converge more smoothly to a globally consistent matching result, further enhancing robustness.

[0103] The above is a schematic description of the present invention and its embodiments, which is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs a structure and embodiment similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A multi-scale multi-modal image feature matching method based on Gaussian pyramid, characterized in that: The following steps are involved: Step 1: Construct Gaussian pyramid images for infrared images and visible light images respectively; Step 2: Perform Harris feature point detection on each layer of the Gaussian pyramid image, then upsample each layer to the same size, and perform non-maximum suppression on the detected feature points to remove duplicate feature points; Step 3: Calculate the curvature of the feature points after non-maximum suppression processing; Step 4: Calculate the curvature radius based on the curvature of the feature point, assign the main direction to each feature point, and extract the PIIFD descriptor to describe the feature point; Step 5: Calculate the similarity of feature descriptors, use bidirectional matching and random sampling consistency check to obtain initial matching point pairs, iteratively optimize the projection matrix and dynamically replace high residual matching points, while introducing a penalty factor to stabilize the convergence process, and output the final image matching result.

2. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 1, characterized in that: When constructing the Gaussian pyramid image in step 1, a Gaussian filter is used to perform low-pass filtering on the original infrared image and the visible light image respectively, and a downsampling operation is performed. A multi-scale image iterative decomposition method is used to obtain a sampled image until a preset number of pyramid layers is reached.

3. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 1 or 2, characterized in that: The Harris feature point detection in step 2 includes: calculating the local autocorrelation matrix M of each pixel in the Gaussian pyramid image, and screening candidate feature points through the Harris response function to obtain a candidate feature point matrix R.

4. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 3, characterized in that: The step 2 performs non-maximum suppression on the detected feature points, specifically: construct a rectangular area window, retain the candidate feature point with the largest response value in the window through comparison, traverse all candidate feature points in the image, repeat the comparison process, and all retained feature points constitute the final feature point set Among them, n f Indicates the number of feature points extracted.

5. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 4, characterized in that: The curvature calculation in step 3 specifically includes: For the feature point set The autocorrelation matrix M of each feature point is decomposed into eigenvalues to obtain eigenvalues λ1 and λ2; the eigenvalues λ1 and λ2 are then used to calculate the curvature k of the feature point. The curvature k is calculated by the following formula: Calculate the curvature of all feature points and get k i , i=1,2,3…n f .

6. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 5, characterized in that: The specific process of step 4 is as follows: Step 4-1, assign a main direction φ to each feature point; Step 4-2: For feature point P i , first calculate the corresponding curvature radius as follows: Among them, k i is the curvature at the feature point, r i is the curvature radius of the feature point; Select the feature point as the center and the curvature radius as r i The circular area is divided into multiple sub-areas. In each sub-area, the gradient magnitude and gradient direction are calculated, and an 8-bin direction histogram is constructed. The gradient magnitude is normalized, and the 8-bin histograms of multiple sub-areas are spliced into a multidimensional feature vector, namely the PIIFD descriptor.

7. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 6, characterized in that: The feature descriptor similarity calculation in step 5 satisfies: Among them, P c1 and P c2 are the feature points in the reference image and the image to be matched, respectively. and is the PIIFD descriptor of two feature points.

8. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 7, characterized in that: In step 5, based on the calculated similarity of the feature descriptors, the corresponding matching feature point pairs are screened out using the two-way matching method, the transformation matrix between the reference image and the image to be matched is calculated based on the corresponding matching feature point pairs obtained after screening, and a random sampling consistency test is performed to eliminate point pairs with large matching errors. After eliminating the point pairs with large matching errors, the remaining point pairs are the initial matching results after screening and testing.

9. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 8, characterized in that: The specific process of step 5 iteratively optimizing the projection matrix and dynamically replacing high residual matching points is as follows: For the matching point P obtained after the initial matching c1 and P c2 , calculate its projection matrix M; Then, we get the value corresponding to the matching point P. c2 The fitting point P 2fit , calculate the fitting point P 2fit and matching point P c2 The residual ε(i); Will Sort in ascending order to find the last 1 / 4 elements that have more residuals than the other 3 / 4, and replace P with the fitted points c2 The points corresponding to the last 1 / 4 elements; Repeat the above process and iteratively update P c2 until the sum of the residuals is equal to zero, and the final matching result is obtained.

10. The multi-scale multi-modal image feature matching method based on Gaussian pyramid according to claim 9, characterized in that: In step 5, In the process of sorting in ascending order, a penalty factor β is introduced and the residual is updated as follows: e 总 =ε(k)+(k-1)β Where k is the number of iterations and ε(k) is the residual at the kth iteration.

Images