An image matching method and device based on improved SIFT
By improving the SIFT algorithm, changing the square region to a circular region, and generating binary descriptors for gradient direction and local intensity, and combining optimized search and correction methods, the problems of high computational complexity and large error of the SIFT algorithm are solved, and efficient image matching is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE 20TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORP
- Filing Date
- 2022-11-22
- Publication Date
- 2026-07-14
AI Technical Summary
Existing SIFT algorithms suffer from high computational complexity and redundancy in image matching, and errors occur during rotation, affecting real-time performance and accuracy.
The square regions are replaced with circular regions to generate binary descriptors of gradient direction and local intensity. These descriptors are then combined with weights to generate keypoint descriptors. Finally, the search is optimized using a kd-tree and mismatch points are corrected using RANSAC.
It improves the real-time performance and accuracy of image matching, while reducing computation time and errors.
Smart Images
Figure CN115761281B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to an image matching method and device based on an improved SIFT. Background Technology
[0002] Image matching is a fundamental and indispensable technology in the field of computer vision, currently widely used in medical imaging, virtual reality, information security, and military applications. Since the introduction of the Scale Invariant Feature Transform (SIFT) algorithm, its excellent performance has driven the development of image matching techniques. However, the local descriptors generated by the SIFT method are computationally complex, have high redundancy, and perform poorly in real-time. Furthermore, because the generated descriptor region is square, rotating the pixels in the region to be registered so that its principal direction is parallel to the principal direction of the reference image results in some pixels not overlapping, causing errors and significantly reducing matching accuracy. The purpose of this invention is to design a new local descriptor generation method to improve the real-time performance and accuracy of the SIFT method in image matching.
[0003] Lowe proposed the SIFT method in 2004 (Lowe DG. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004,60(2): 91-110.). Its principle is to detect extreme points in the Gaussian difference scale space, then fit a three-dimensional quadratic function to remove outliers to obtain a more accurate scale and location of the detected extreme points. Finally, a 128-dimensional scale-invariant descriptor is generated as a matching factor, thus accurately matching images. This algorithm has a certain robustness to rotation, scale, and illumination changes, and its excellent performance has led to its widespread application in many fields. However, the high dimensionality of SIFT results in excessively long computation time. Summary of the Invention
[0004] This application provides an image matching method and device based on an improved SIFT, which uses circular regions to replace square regions to reduce errors, considers the local intensity information of the image itself, generates a binary representation of the intensity, and achieves high accuracy and real-time performance.
[0005] This application provides an image matching method based on an improved SIFT, including:
[0006] Get the image to be matched;
[0007] Based on the image to be matched, the SIFT feature mesh structure containing key points is extracted using the SIFT algorithm;
[0008] The extracted SIFT feature mesh structure is converted into a concentric circle structure, wherein the concentric circle structure includes multiple sub-regions, and each sub-region includes a gradient direction histogram.
[0009] Generate gradient direction binary descriptors and local intensity binary descriptors based on the gradient direction histogram of any sub-region;
[0010] The key point descriptor of the image to be matched is generated based on the gradient direction binary descriptor, the local intensity binary descriptor and the corresponding weights.
[0011] Image matching is performed based on the generated keypoint descriptors and the keypoint descriptors of the reference image.
[0012] Optionally, it also includes numbering the concentric circle structures in a specified order, and numbering the gradient direction histogram of any sub-region.
[0013] Optionally, the gradient direction binary descriptor includes an internal descriptor, a horizontal descriptor, and a vertical descriptor;
[0014] The gradient direction binary descriptor is generated as follows:
[0015] Define a coefficient k to evaluate the threshold, and the internal descriptor satisfies:
[0016]
[0017] in, This represents the internal descriptor with structure number m and gradient direction n. This represents the nth gradient direction in the mth region;
[0018] The horizontal descriptor satisfies:
[0019]
[0020] in, This represents a horizontal descriptor with structure number m and gradient direction n;
[0021] Vertical descriptors satisfy:
[0022]
[0023] in, This represents a horizontal descriptor with structure number m and gradient direction n;
[0024] The gradient direction binary descriptor G satisfies: .
[0025] Optionally, the local intensity binary descriptor includes a center intensity descriptor, a horizontal intensity descriptor, and a vertical intensity descriptor;
[0026] The local intensity binary descriptor is generated as follows:
[0027] The central intensity descriptor satisfies:
[0028]
[0029] in, This represents the average intensity of the sub-region surrounding the key point. Indicates the center intensity descriptor. This represents the average intensity of a key point region with a specified radius.
[0030] The horizontal intensity descriptor satisfies:
[0031]
[0032] in, Indicates the horizontal intensity descriptor;
[0033] The vertical intensity descriptor satisfies:
[0034]
[0035] in, Represents the vertical intensity descriptor;
[0036] The local intensity binary descriptor satisfies .
[0037] Optionally, the keypoint descriptors of the image to be matched are generated based on the gradient direction binary descriptor, the local intensity binary descriptor, and the corresponding weights, satisfying the following:
[0038]
[0039] Where H represents the keypoint descriptor, The weight of the local intensity binary descriptor T.
[0040] Optionally, image matching based on the generated keypoint descriptors and the keypoint descriptors of the reference image includes:
[0041] Based on the key point descriptor of any key point and the key point descriptor of the reference image, an optimized search is performed using a kd-tree to find the approximate nearest neighbor feature point of any key point.
[0042] Based on the distance between any key point and its nearest neighbor feature point, a division operation is performed between the distance between any key point and its second nearest neighbor feature point. If the quotient is within a preset range, it is determined as a matching point to complete the coarse matching of key points.
[0043] Optionally, after coarse matching of key points, the RANSAC method can be used to correct mismatches.
[0044] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned image matching method based on the improved SIFT.
[0045] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned steps of the image matching method based on the improved SIFT.
[0046] This application uses a circular region to replace the square region, reducing the generation of errors. It also considers the local intensity information of the image itself to generate a binary representation of the intensity, achieving high accuracy and real-time performance.
[0047] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0049] Figure 1 This is a basic flowchart illustrating the image matching method based on the improved SIFT in an embodiment of this application;
[0050] Figure 2 This is a flowchart illustrating the image matching method based on the improved SIFT according to an embodiment of this application;
[0051] Figure 3 This is a schematic diagram illustrating the conversion of the SIFT feature mesh structure into a concentric circle structure in the embodiments of this application;
[0052] Figure 4 This is an example of the numbering of the concentric circle structure in the embodiments of this application. Detailed Implementation
[0053] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0054] To improve the accuracy and real-time performance of the SIFT method, this invention modifies the feature descriptor generation region and improves the descriptor generation method, making the calculation method simpler, more convenient, and more efficient, thus achieving improvements in both accuracy and speed. This application provides an image matching method based on an improved SIFT, such as... Figure 1 As shown, it includes the following steps:
[0055] In step S101, the image to be matched is acquired. Due to factors such as lighting, field of view, and shadows, the original image acquired by the camera may contain noise, target occlusion, etc. In this embodiment, the original image can be preprocessed before acquiring the image to be matched, which is beneficial for extracting the required information. The preprocessing process mainly includes histogram equalization and Gaussian smoothing.
[0056] In step S102, based on the image to be matched, a SIFT feature mesh structure containing key points is extracted using the SIFT algorithm. Specifically, based on the image to be matched, the SIFT algorithm is used to extract key points in the image through scale-space extremum detection, and the gradient direction distribution characteristics of the neighboring pixels of the key points are used to assign parameter directions to each key point, thereby making the operator invariant to image rotation. The feature points generated in this way have good invariance to translation, rotation, scaling, brightness changes, occlusion, and noise.
[0057] According to the SIFT algorithm principle, to maintain rotation invariance, SIFT rotates the coordinate axes to align with the principal direction. In some examples, the region where SIFT generates the descriptor is a 4×4 square grid structure. When rotating the pixels in the region to be registered so that its principal direction is parallel to the principal direction of the reference image, some pixels in this region do not overlap, which will produce a large error. In the actual operation of the SIFT algorithm, a larger square region than the reference image is used for rotation, but this results in a larger number of pixels, and the computation time increases accordingly. In step S103, the extracted SIFT feature grid structure is converted into a concentric circle structure, wherein the concentric circle structure includes multiple sub-regions, and each sub-region includes a gradient direction histogram. Figure 3As shown, in this embodiment, the rotation invariance of a circle is used to modify the square region into a circular region, so that the problem of pixel non-overlapping will not occur during rotation, thus simplifying the problem. Specifically, the original 4×4 square neighborhood is transformed into 4 concentric circle regions. The radius of the circle is initially 1 pixel, and the entire circumference is divided into 4 regions in units of 90, thus forming a new 4×4 sub-region.
[0058] In step S104, a gradient direction binary descriptor and a local intensity binary descriptor are generated based on the gradient direction histogram of any sub-region.
[0059] In step S105, the key point descriptor of the image to be matched is generated based on the gradient direction binary descriptor, the local intensity binary descriptor, and the corresponding weights.
[0060] In step S106, image matching is completed based on the generated keypoint descriptors and the keypoint descriptors of the reference image. For example... Figure 2 As shown, the key point descriptors of the reference image can also be determined using the aforementioned process.
[0061] This application uses a circular region to replace the square region, reducing the generation of errors. It also considers the local intensity information of the image itself to generate a binary representation of the intensity, achieving high accuracy and real-time performance.
[0062] Since the original SIFT descriptor is a 128-dimensional floating feature descriptor, it consumes a large amount of memory and has a long processing time. The proposed binary descriptor in this application makes processing more convenient, thereby reducing processing time. Furthermore, the information in the binary descriptor typically determines quality and efficiency. For images, the basic information is their intensity. Using two binary descriptors together for matching significantly improves both accuracy and real-time performance. Specifically, in some embodiments, it also includes numbering the concentric circle structures in a specified order, and numbering the gradient direction histogram of any sub-region. For example… Figure 3 , Figure 4 In the process, the transformed circular descriptor region is numbered from 1 to 16 sub-regions from top to bottom and left to right, and the eight directions of the gradient direction histogram of each sub-region are numbered from smallest to largest.
[0063] In some embodiments, the gradient direction binary descriptor includes generating an internal descriptor, a horizontal descriptor, and a vertical descriptor;
[0064] The gradient direction binary descriptor is generated as follows:
[0065] Define a coefficient k to evaluate the threshold, and the internal descriptor satisfies:
[0066]
[0067] in, The number represents the structure number. m The gradient direction is n The internal descriptor, m =1 to 16, n =1 to 8, Indicates the first m The first region n There are 128 gradient directions. In this example, the internal descriptor represents the difference in gradient magnitude between adjacent directions within the same sub-region. Within each sub-region, the gradient magnitudes between adjacent directions are compared in a clockwise direction, with a threshold related to the gradient magnitudes being compared. In this example, the internal descriptor is 128 bits.
[0068] The horizontal descriptor satisfies:
[0069]
[0070] in, The number represents the structure number. m The gradient direction is n Horizontal descriptors, m =1 to 16, n =1 to 8. The horizontal descriptor represents the relationship between gradient magnitudes in the same direction within adjacent horizontal sub-regions. Note the comparison between the rightmost and leftmost sub-regions. In this example, the horizontal descriptor is 128 bits.
[0071] Vertical descriptors satisfy:
[0072]
[0073] in, The number represents the structure number. m The gradient direction is n The horizontal descriptor. Similar to the horizontal descriptor, the vertical descriptor represents the relationship between the gradient magnitude and the same direction in adjacent vertical sub-regions; the vertical descriptor is 64 bits.
[0074] The gradient direction binary descriptor G satisfies: .
[0075] In some embodiments, the local intensity binary descriptor includes a center intensity descriptor, a horizontal intensity descriptor, and a vertical intensity descriptor;
[0076] The local intensity binary descriptor is generated as follows:
[0077] The local intensity binary descriptor is generated from the local intensity information around the keypoint. In this example, This represents the average intensity of the sub-region surrounding the keypoint. (Definition) Indicates a radius of 3 The average intensity of the key point region is defined. Represents the binary value of the center descriptor. (Comparison) and The value of the center intensity descriptor satisfy:
[0078]
[0079] in, This represents the average intensity of the sub-region surrounding the key point. Indicates the center intensity descriptor. This represents the average intensity of the critical point region with a specified radius.
[0080] By comparing the horizontal intensities of adjacent regions, the horizontal intensity descriptor satisfies:
[0081]
[0082]
[0083] in, This represents the horizontal intensity descriptor.
[0084] By comparing the intensities of adjacent vertical regions, the vertical intensity descriptor satisfies:
[0085]
[0086] in, This represents the vertical intensity descriptor.
[0087] The local intensity binary descriptor satisfies .
[0088] In some embodiments, the key point descriptors of the image to be matched are generated based on the gradient direction binary descriptor, the local intensity binary descriptor, and the corresponding weights, satisfying the following:
[0089]
[0090] Where H represents the keypoint descriptor, The weights of the local intensity binary descriptor T are represented. This represents the weight of the binary SIFT descriptor G.
[0091] This application matches SIFT feature vectors (keypoint descriptors H) based on a similarity metric. While binary representation of the feature descriptors was previously used, this example further employs Hamming distance to match the SIFT feature vectors. Specifically, in some embodiments, image matching is performed based on the generated keypoint descriptors and the keypoint descriptors of the reference image, including:
[0092] Based on the keypoint descriptor of any keypoint and the keypoint descriptor of the reference image, an optimized search using a kd-tree is performed to find the approximate nearest neighbor feature point of any keypoint. That is, after obtaining the SIFT feature vector (keypoint descriptor H), an optimized search using a kd-tree is performed to find the approximate nearest neighbor feature point of each feature point.
[0093] A division operation is performed between the distance from any keypoint to its nearest neighbor feature point and the distance from any keypoint to its second nearest neighbor feature point. If the deviation between the quotients is within a preset range, the keypoint is identified as a matching point, thus completing coarse keypoint matching. Specifically, for example, if the distance between the nearest and second nearest feature points is less than a set percentage threshold, then this pair is accepted as a matching point.
[0094] In some embodiments, after coarse matching of key points, the RANSAC method is also used to correct mismatch points, thereby improving matching accuracy.
[0095] To address the issues of excessive redundancy and computational complexity in the original SIFT descriptors, this application's method employs a binary descriptor for optimization, facilitating computation and improving real-time performance. Since rotating the square region used to generate the SIFT descriptor in the main direction results in some pixel non-overlap, this application's method replaces the square region with a circular region to reduce errors. Furthermore, this application's method redesigns the descriptor generation formula for the new circular region, using a superposition of two binary descriptors in the matching method to achieve both high accuracy and real-time performance.
[0096] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned image matching method based on the improved SIFT.
[0097] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned steps of the image matching method based on the improved SIFT.
[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0099] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0101] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.
Claims
1. An image matching method based on improved SIFT, characterized in that, include: Get the image to be matched; Based on the image to be matched, the SIFT feature mesh structure containing key points is extracted using the SIFT algorithm; The extracted SIFT feature mesh structure is converted into a concentric circle structure, wherein the concentric circle structure includes multiple sub-regions, and each sub-region includes a gradient direction histogram. Generate gradient direction binary descriptors and local intensity binary descriptors based on the gradient direction histogram of any sub-region; The key point descriptor of the image to be matched is generated based on the gradient direction binary descriptor, the local intensity binary descriptor and the corresponding weights. Image matching is completed based on the generated keypoint descriptors and the keypoint descriptors of the reference image; The gradient direction binary descriptor includes an internal descriptor, a horizontal descriptor, and a vertical descriptor; The gradient direction binary descriptor is generated as follows: Define coefficients k Based on the evaluation threshold, the internal descriptor satisfies: in, The number represents the structure number. m The gradient direction is n The internal descriptor, Indicates the first m The first region n One gradient direction; The horizontal descriptor satisfies: in, The number represents the structure number. m The gradient direction is n Horizontal descriptors; Vertical descriptors satisfy: in, The number represents the structure number. m The gradient direction is n Horizontal descriptors; The gradient direction binary descriptor G satisfies: ; The local intensity binary descriptor includes a center intensity descriptor, a horizontal intensity descriptor, and a vertical intensity descriptor; The local intensity binary descriptor is generated as follows: The central intensity descriptor satisfies: in, This represents the average intensity of the sub-region surrounding the key point. Indicates the center intensity descriptor. This represents the average intensity of a key point region with a specified radius. The horizontal intensity descriptor satisfies: in, Indicates the horizontal intensity descriptor; The vertical intensity descriptor satisfies: in, Represents the vertical intensity descriptor; The local intensity binary descriptor satisfies .
2. The image matching method based on improved SIFT as described in claim 1, characterized in that, It also includes numbering the concentric circle structures in a specified order, and numbering the gradient direction histogram of any sub-region.
3. The image matching method based on improved SIFT as described in claim 1, characterized in that, The keypoint descriptor of the image to be matched is generated based on the gradient direction binary descriptor, the local intensity binary descriptor, and the corresponding weights, satisfying the following: in, H Represents the keypoint descriptor. Binary descriptor representing local intensity T The weight.
4. The image matching method based on improved SIFT as described in claim 3, characterized in that, Based on the generated keypoint descriptors and the keypoint descriptors of the reference image, image matching is performed, including: Based on the key point descriptor of any key point and the key point descriptor of the reference image, an optimized search is performed using a kd-tree to find the approximate nearest neighbor feature point of any key point. Based on the distance between any key point and its nearest neighbor feature point, a division operation is performed between the distance between any key point and its second nearest neighbor feature point. If the quotient is within a preset range, it is determined as a matching point to complete the coarse matching of key points.
5. The image matching method based on improved SIFT as described in claim 4, characterized in that, After coarse matching of key points, the RANSAC method is also used to correct mismatches.
6. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the image matching method based on the improved SIFT as described in any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image matching method based on the improved SIFT as described in any one of claims 1 to 5.