Binary feature matching method and system

Abstract

Binary feature matching methods and systems are disclosed. A feature matching method of matching a query binary feature descriptor with reference binary feature descriptors includes assigning reference binary feature descriptors to symbol groups corresponding to leaf nodes of a balanced binary search tree according to bit values of selected bit positions of the reference binary feature descriptors; traversing the binary search tree using the query binary feature descriptor to reach a certain leaf node and determining reference binary feature descriptors assigned to a symbol group corresponding to the leaf node; and when a distance measure between the query binary feature descriptor and a determined reference binary feature descriptor is less than a threshold, the query binary feature descriptor matches to the determined reference binary feature descriptor.

Description

Technical Field

[0001] This disclosure relates to feature matching, and more particularly to real-time or near-real-time matching of binary features. Background Technology

[0002] Feature matching plays a crucial role in many computational tasks. For example, for autonomous robots, feature descriptor matching is a key step in self-localization during navigation. However, real-time matching is often difficult to achieve due to limitations in onboard computing resources. To achieve safe and reliable navigation, autonomous robots must determine their position relative to their environment through localization. In recent years, visual localization relying on camera-fed images has become increasingly popular due to the low cost and rich scene information provided by cameras. The steps of visual localization include: extracting feature descriptors from each image frame; and finding correspondences between such feature descriptors in successive image frames. The process of finding correspondences is called feature descriptor matching. Such correspondences enable pose estimation to track the camera (robot)'s movement in the environment over time. Obviously, feature descriptor matching needs to be performed in real time with high matching accuracy (based on the camera frame rate). However, existing high-precision methods are complex and unsuitable for the resource-constrained computing devices configured in robots.

[0003] Existing technologies have utilized Field-Programmable Gate Arrays (FPGAs) to accelerate feature descriptor matching. Matching in these technologies employs different types of feature descriptors: binary descriptors (such as BRIEF (Binary Robust Independent Basic Features), ORB (Oriented FAST (Accelerated Segmentation Test Features) and Rotated BRIEF)) and non-binary descriptors (such as SIFT (Scale Invariant Feature Transform)). Descriptors are extracted prior to matching through keypoint detection and descriptor computation.

[0004] SIFT descriptors find their counterparts by calculating Euclidean distance, making the matching process more time-consuming. Non-binary descriptors are also large, resulting in high storage requirements. To reduce descriptor complexity, the latest SLAM (Simultaneous Localization and Mapping) visual localization mechanisms (such as ORB-SLAM2 and ORB-SLAM3) all use BRIEF binary descriptors.

[0005] All the hardware architectures described above use the LES (Linear Exhaustive Search) method for descriptor matching. In the LES method, each query descriptor must be compared with all reference descriptors, making the matching process extremely time-consuming. Summary of the Invention

[0006] According to a first aspect of this disclosure, a feature matching method is provided for matching a query binary feature descriptor with a reference binary feature descriptor. The method includes: assigning a reference binary feature descriptor to a symbol group corresponding to a leaf node of a balanced binary search tree based on a bit value at a selected bit position of the reference binary feature descriptor; traversing the binary search tree using the query binary feature descriptor to reach a leaf node, and determining a reference binary feature descriptor assigned to the symbol group corresponding to that leaf node; and matching the query binary feature descriptor to the determined reference binary feature descriptor when a distance metric between the query binary feature descriptor and the determined reference binary feature descriptor is less than a threshold value.

[0007] In one implementation, the method further includes: storing an assigned reference binary feature descriptor for each corresponding symbol group.

[0008] In one implementation, the value of the descriptor corresponding to the selected bit position is removed before storing the allocated reference binary feature descriptor for each corresponding symbol group.

[0009] In one implementation, the balanced binary search tree has a predefined number of levels and a corresponding predefined number of leaf nodes, wherein any reference binary feature descriptors that exceed the size of the leaf node symbol group are discarded.

[0010] In one implementation, the reference binary feature descriptor is assigned to the symbol group by selecting the index number at each level of the balanced binary search tree where the probability of the binary value 1 of the reference binary feature descriptor is closest to 0.5.

[0011] In one implementation, the reference binary feature descriptor and the query binary feature descriptor are image feature descriptors.

[0012] In one implementation, the distance metric is thresholded by performing a ratio test.

[0013] In one implementation, the ratio check includes performing a multiplication operation by 0.5 by performing a single-bit shift operation.

[0014] In one implementation, the distance metric is a Hamming distance metric.

[0015] According to a second aspect of this disclosure, a feature matching system is provided for matching a query binary feature descriptor with a reference binary feature descriptor. The system includes: a binary search tree module, configured to: assign the reference binary feature descriptor to a symbol group corresponding to a leaf node of a balanced binary search tree based on a bit value at a selected bit position of the reference binary feature descriptor; and traverse the binary search tree using the query binary feature descriptor to reach a leaf node and determine a reference binary feature descriptor assigned to the symbol group corresponding to that leaf node; and a matching module, configured to match the query binary feature descriptor to the determined reference binary feature descriptor when a distance metric between the query binary feature descriptor and the determined reference binary feature descriptor is less than a threshold.

[0016] In one implementation, the feature matching system is provided in the form of an integrated circuit.

[0017] In one implementation, the binary search tree module is further configured to store the assigned reference binary feature descriptor for each corresponding symbol group.

[0018] In one implementation, the binary search tree module is further configured to remove the value of the descriptor corresponding to the selected bit position before storing the allocated reference binary feature descriptor for each corresponding symbol group.

[0019] In one implementation, the balanced binary search tree has a predefined number of levels and a corresponding predefined number of leaf nodes, wherein the binary search tree module is further configured to discard any reference binary feature descriptors that exceed the size of the leaf node symbol group.

[0020] In one implementation, the binary search tree module is further configured to assign the reference binary feature descriptor to the symbol group by selecting the index number at each level of the balanced binary search tree whose probability of selecting the binary value 1 of the reference binary feature descriptor is closest to 0.5.

[0021] In one implementation, the reference binary feature descriptor and the query binary feature descriptor are feature descriptors of an image.

[0022] In one implementation, the distance metric is thresholded by performing a ratio test.

[0023] In one implementation, the ratio check includes performing a multiplication operation by 0.5 by performing a single-bit shift operation.

[0024] In one implementation, the distance metric is a Hamming distance metric. Attached Figure Description

[0025] In the following description, embodiments of the invention are illustrated by way of non-limiting example with reference to the accompanying drawings. In the drawings:

[0026] Figure 1 This is a block diagram illustrating an implementation of a feature matching system according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of the feature matching method according to an embodiment of the present invention;

[0028] Figure 3 The diagram shows a three-level binary search tree used in an embodiment of the present invention;

[0029] Figure 4 This is a schematic block diagram of a binary search tree matching module (BSTM-N module) with a depth of N in an embodiment of the present invention;

[0030] Figure 5 for Figure 4 The diagram shows the internal architecture of the BinarySearchTree module within the binary search tree matching module.

[0031] Figure 6 for Figure 5 The diagram shows the internal architecture of the TreeLevelNode unit in the BinarySearchTree module.

[0032] Figure 7A and Figure 7B As shown Figure 4 The data path within the ReferenceDescStore module in the binary search tree matching module is shown below;

[0033] Figure 8 for Figure 4 The diagram shown is an internal architecture diagram of the QueryDescStore module in the binary search tree matching module.

[0034] Figure 9 for Figure 4 The diagram shows the internal architecture of the HammingDistanceCalc module within the binary search tree matching module.

[0035] Figure 10 for Figure 4 The diagram shown is an internal architecture block diagram of the MatchSelector module in the binary search tree matching module.

[0036] Figures 11A to 11C A comparison graph of matching methods. Detailed Implementation

[0037] This disclosure provides a binary search tree matcher for binary feature matching. Binary descriptors are widely used in various applications due to their significant advantages, such as extremely low storage requirements and lower computational costs in the matching process. Among them, BRIEF, ORB, and BRISK binary descriptors are compact, efficient, and therefore popular choices, making them extremely suitable for embedded and real-time systems. In the field of computer vision, there are several commonly used feature descriptor matching techniques, including the following:

[0038] Image feature matching: used for tasks such as pose estimation (using a fundamental matrix), panoramic stitching, augmented reality, and 3D reconstruction;

[0039] Image and map point matching: used for pose estimation (using the Iterative Closest Point (ICP) algorithm) and loop closure detection in visual SLAM (Simultaneous Localization and Mapping);

[0040] Map point matching: Used for multi-map fusion scenarios in single-robot and multi-robot collaborative applications.

[0041] Of the three techniques mentioned above, besides the traditional image feature descriptor matching technique, the latter two techniques focus on other scenarios where the binary search tree matcher of this invention can effectively establish corresponding relationships using binary features. These scenarios often involve massive amounts of data points, thus the ability to quickly and accurately achieve corresponding relationship matching is considered a fundamental element. The binary search tree matcher of this invention, with its ultra-low latency and higher accuracy, provides a promising alternative for such applications. This tool has significant advantages in speed and accuracy, and these advantages are particularly pronounced in resource-constrained environments such as drones and autonomous robots where real-time processing is a critical factor.

[0042] In addition to the feature matching methods described above, the binary search tree matcher of this invention can also be used for the following purposes:

[0043] Network Intrusion Detection: For Network Intrusion Detection Systems (NIDS), the sheer volume and complexity of data make it difficult to detect anomalous or malicious activity from massive amounts of network traffic. Binary signature descriptors can transform packet characteristics into compact binary vectors, which are easy to manage and compare, thus helping to reduce the complexity of the data. By matching these descriptors using a binary search tree, the speed of intrusion detection can be significantly improved. The binary search tree structure enables fast matching processing, which is crucial for real-time threat detection. Binary signature descriptors effectively capture key patterns and anomalous activity, and the binary search tree matcher of this invention can efficiently match these key patterns and anomalous activity with known attack signatures, thereby improving both speed and detection accuracy.

[0044] Document retrieval: Binary feature descriptors, which utilize methods such as MinHash to convert text features like groups of words or phrases (i.e., text fragments composed of a sequentially arranged group of words) into binary vectors, simplify comparisons between different documents and thus play a crucial role in document retrieval. When employing a binary search tree, the system can efficiently organize and match such binary features, significantly accelerating the retrieval process. Because the binary search tree matching mechanism of this invention improves efficiency by reducing the search space of the reference set, it is particularly important for large-scale databases that require real-time comparison of millions of documents to ensure rapid identification of similar or nearly identical documents.

[0045] Fingerprint matching: Fingerprint matching is a fundamental component of biometric identification systems. Representing the entire fingerprint feature using binary feature descriptors allows for the capture of detailed fingerprint features in a compact form, accelerating the comparison process. Therefore, binary feature descriptors are extremely effective for fingerprint applications. Using binary search tree-based matching techniques can facilitate the organization of such binary descriptors, particularly in situations requiring matching individual fingerprints within a large fingerprint database, thus speeding up the matching process.

[0046] Figure 1 This is a block diagram illustrating one implementation of a feature matching system according to an embodiment of the present invention. The feature matching system 150 can be a system based on field-programmable gate array (FPGA) hardware. Figure 1 As shown, the feature matching system 150 inputs features extracted from the reference image 110 and the query image 120. Specifically, the feature extraction tool 130 extracts a reference image feature descriptor 112 from the reference image 110 and a query image feature descriptor 122 from the query image 120. The feature extraction tool 130 may employ a keypoint detection method to extract points of interest. The reference image feature descriptor 112 and the query image feature descriptor 122 are binary feature descriptors.

[0047] The feature matching system 150 includes a binary search tree module 152, a matching module 154, and a memory 156. The binary search tree module 152 is used to assign reference image binary feature descriptors to leaf nodes of a balanced binary search tree and traverse the balanced binary search tree with query image binary feature descriptors, as will be described in further detail below. The matching module 154 is used to determine a distance metric between the identified reference image binary feature descriptors and the query image binary feature descriptors, and to determine a match if the distance metric is less than a threshold.

[0048] Memory 156 stores: reference descriptor storage content 162, which stores a reference image binary feature descriptor; query descriptor storage content 164, which stores a query image binary feature descriptor; and binary search tree storage content 166, which stores index number data for assigning the reference image binary feature descriptors to a balanced binary search tree. Binary search tree storage content 166 can be stored in a configuration register that can be configured at runtime.

[0049] like Figure 1 As shown, the feature matching system 150 outputs a set of correspondences 170 between the reference image feature descriptor 112 and the query image feature descriptor 122.

[0050] As described above, this disclosure relates to an integrated circuit architecture (which may be implemented, for example, in a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)) for real-time matching of binary feature descriptors, wherein key points of the two images are matched accordingly based on feature descriptors extracted from a query image and a reference image.

[0051] The architecture has a matching module and may include the following features:

[0052] 1. An N-level balanced binary search tree module, which divides the feature descriptor into 2... N A set of descriptors, in which any redundant descriptors exceeding the capacity are discarded;

[0053] 2. This module adopts a streaming architecture that uses on-chip memory to build binary trees and perform matching processing;

[0054] 3. Improve the matching accuracy of binary feature descriptor matching by using a selection mechanism based on ratio testing.

[0055] This architecture employs a modular pipeline design that efficiently utilizes resources, making it easy to scale to meet the requirements of different symbol sizes. The FPGA implementation of this architecture demonstrates that it is significantly faster than the best-performing FPGA-based feature matcher in the current technology. The architecture is approximately 10 times faster than the LES (Linear Exhaustive Search) method commonly used in FPGAs.

[0056] Figure 2 This is a flowchart of the feature matching method according to an embodiment of the present invention. Figure 2 The method 200 shown can be derived from Figure 1 The feature matching system 150 shown is implemented.

[0057] In step 202, reference binary feature descriptors are assigned to the respective symbol groups corresponding to the leaf nodes of the balanced binary search tree. The assignment of reference binary feature descriptors is based on the bit value at their selected bit positions.

[0058] In step 204, the balanced binary search tree is traversed using the query binary feature descriptor to reach a leaf node of the balanced binary search tree.

[0059] In step 206, a distance metric is determined between the reference binary feature descriptor corresponding to the leaf node of the balanced binary search tree and the query binary feature descriptor. If the distance metric is less than a threshold, the query binary feature descriptor matches the reference binary feature descriptor corresponding to the leaf node, and a matching indication is generated.

[0060] The feature matching system and method disclosed herein have the following advantages: the binary search tree matching architecture can be implemented as a streaming architecture, which processes the data directly as the input data arrives, without the need for external storage; no external memory needs to be accessed during the binary search tree construction process or the matching process; since the streaming architecture does not need to access external memory during the binary tree construction process and the matching process, it can improve speed and reduce power consumption.

[0061] This binary search tree matching architecture uses a predefined balanced binary search tree (i.e., a fixed number of symbol groups), so there is no need to adjust its branch structure at runtime. For specific application scenarios, users can predetermine a fixed balanced binary search tree suitable for that scenario to prevent the need to use external memory at runtime due to an undetermined search tree size.

[0062] The FPGA-based Binary Search Tree Matching (BSTM) method of this invention solves the problem of high matching complexity in all existing hardware-based feature descriptor matchers that employ the LES method. Specifically, by using a hardware-efficient binary search tree (BST), it avoids exhaustively comparing every descriptor in the query image with all descriptors in the reference image.

[0063] To improve matching accuracy, the above processing also employs a ratio-based outlier removal mechanism. Results show that the ratio-based outlier removal mechanism of the binary search tree matcher in this invention achieves better matching accuracy than the distance threshold-based algorithm commonly used in binary feature descriptor matching.

[0064] The above design can be implemented on the SoC-based Xilinx MPSoC Ultrascale+ platform and evaluated using common datasets. Experimental results show that, compared with existing hardware-based feature descriptor matchers, the binary search tree matcher design of this invention has ultra-low latency and moderate resource usage.

[0065] This invention enables binary descriptor matching by processing binary keypoint descriptors. Given a reference image I and a query image J, descriptor matching aims to find pixels in I and J that correspond to the same physical point in the 3D world. The output is a set of pairwise correspondences, which can be used for projective geometry calculations to determine the camera pose between the two images.

[0066] Before matching, descriptors are extracted for each image through keypoint detection and descriptor computation. Keypoint detection is used to identify points of interest (such as corners) in the image frame. Common keypoint detection methods include Harris corner detection and FAST (accelerated segmentation test features) corner detection. In the descriptor computation step, descriptors are extracted from patches centered on the detected keypoints. Descriptors characterize the local appearance of the keypoints and are represented as vectors of binary or floating-point values. Commonly used binary descriptors are BRIEF (Binary Robust Independent Basic Features), BRISK (Binary Robust Scale Invariant Keypoints), and ORB (Oriented FAST and Rotated BRIEF) descriptors, while commonly used non-binary descriptors include SURF (Accelerated Robust Features) and SIFT (Scale Invariant Feature Transform) descriptors. The underlying design philosophy of these descriptors is to represent regions with locally similar appearances between different images. Matching feature descriptors between two images requires the use of similarity metrics. Specifically, matching non-binary descriptors often uses Euclidean distance, while matching binary descriptors often uses Hamming distance. The latter refers to the number of distinct bit pairs between two descriptors taken from I and J, respectively.

[0067] At a mathematical level, feature matching is the process of finding key points. Most similar key points The operation. In other words:

[0068] ,

[0069] in, Indicate key points Descriptors, This represents a similarity measure between descriptors. For I and The key to matching is eliminating errors. A distance threshold mechanism is often used, discarding matches with a distance less than a specific threshold. In addition to this direct distance thresholding mechanism, other methods may use a threshold based on the ratio between two pairs of shortest distances.

[0070] LES (Linear Exhaustive Search) is the most commonly used feature matching method. This method... With all Compare the distances between them and output the closest one. LES has high matching accuracy, but its complexity is directly proportional to the number of descriptors in J. The number of descriptors in an image is typically between 100 and 10,000, making the computational cost of LES prohibitively high, especially when used on resource-constrained computing platforms commonly used in robotics.

[0071] To address the significant matching latency issue of the LES method, this invention proposes a more hardware-efficient Binary Search Tree (BST) for binary feature descriptor matching. The method evaluates the distances to only a selected number of reference keypoints in I to find the appropriate query path. The corresponding key points. Furthermore, this method also employs a ratio-based outlier elimination mechanism previously used only in non-binary descriptor matching. The method consists of three parts: a binary search tree; the selection of bit index numbers; and the selection of [missing information].

[0072] The method of this invention divides the descriptor space into different descriptor groups based on the bit value of the selected descriptor bit index number. Furthermore, the method relies on a balanced binary search tree with N levels, wherein the descriptor space is divided into 2... N A group of symbols.

[0073] Figure 3 The diagram shows a three-level binary search tree used in an embodiment of the present invention. Figure 3 In the illustrated embodiment, the three-level binary search tree is based on bit index numbers. and mapped to Given query descriptor Accordingly, in each level corresponding to index numbers 2, 0, and 4, the values ​​at these index numbers are discarded. Bits. Descriptor index 0 is the starting index, corresponding to the rightmost bit position.

[0074] Figure 3 The binary search tree shown has N=3, therefore it has 8 symbol groups. Each symbol group stores a set of partial descriptors for the reference image. For example... Figure 3 As shown, each symbol component is equipped with Each descriptor is assigned to a different descriptor group based on the selected descriptor bit index number (bit position). In this embodiment, the selected bit index number... Where K is the descriptor width. The descriptors stored within each descriptor group share a common characteristic. The bit value at that location. For example, All descriptors in their index number Each bit has a value of 1, 0, or 1. To reduce the storage requirements when storing descriptors in a symbol group, the corresponding descriptor bit at the selected bit index is removed (e.g., removing...). Figure 3 middle (bits at the location). For a given query descriptor, the index number The bit value at index number is used to determine the corresponding symbol group containing the reference descriptor to be matched. To facilitate calculating the distance between the query descriptor and the reference descriptor within the corresponding symbol group, the bit value at index number is also discarded. The bits at that location. Thus, compared to the LES method, the upper limit of the number of comparisons for each query descriptor is clearly the number of reference descriptors within the corresponding descriptor group.

[0075] To ensure an even distribution of descriptors across symbol groups, the bit index numbers must be carefully selected. This is crucial because the maximum storage resource allocated to each symbol group is fixed (MAX_DEPTH). If the number of reference descriptors for a given symbol group exceeds MAX_DEPTH, the extra descriptors must be discarded, leading to missed matches. When determining appropriate bit index numbers, the index numbers of each reference descriptor in a set are first calculated. The probability of being "1" at index i is calculated, and then the probability of being "1" at each index i is calculated. ,in, This represents the probability that the value at reference descriptor index i is "1". Finally, select... The N smallest index numbers The index numbers selected in this way ensure that the descriptors within each symbol group have a uniform distribution of "1"s and "0"s. It should be noted that the above process can be performed during runtime by updating a set of counters corresponding to each bit index number whenever a new descriptor is calculated, without consuming significant computational resources. Furthermore, given the minimal difference in image content between successive image frames, the selected descriptor bit index numbers from the previous frame can be used to assign descriptors for the current frame to the appropriate symbol groups.

[0076] To reduce errors, existing hardware implementations for binary feature descriptor matching employ a distance thresholding mechanism, while hardware implementations for non-binary descriptor matching employ a ratio-based selection mechanism.

[0077] In this disclosure, the term "thresholded" refers to the result of applying a thresholding operation to data (generally an image or signal). Thresholding operations include: first setting a threshold; then transforming or modifying the data so that all values ​​above or below the threshold change uniformly. For example, in image processing, thresholding may include: converting a grayscale image to a binary image, wherein all pixel values ​​above a specific threshold are assigned a value (e.g., "white"), while all pixel values ​​below the threshold are assigned another value (e.g., "black"). Therefore, the term "thresholded" refers to data that has been processed in this manner, where the values ​​or features of the data have been modified or segmented according to a preset threshold.

[0078] In this invention, a selection mechanism based on ratio testing and Hamming distance is used for binary feature descriptor matching. Experimental results show that this selection mechanism achieves higher accuracy compared to distance threshold-based methods. The underlying principle of ratio testing is that two descriptors are considered a match only when the corresponding distances between them are sufficiently different. Specifically, two descriptors are considered a match only when they satisfy the following ratio testing conditions:

[0079]

[0080] in, These are the minimum distances between the two pairs of descriptors. The threshold is used. SIFT-based feature matching employs... The ratio test. In this design, the following is adopted. The ratio check is used to replace multiplication operations with single-bit shift operations.

[0081] The hardware architecture of this invention is described below. For a binary search tree implementation of depth N, this architecture is called BSTM-N. This hardware architecture is used for 256-bit ORB binary descriptor matching employed by various well-known visual localization mechanisms.

[0082] The top layer of this design consists of an AXI4LITE interface. The S_AXIS interface and an M_AXIS interface are connected to AXI-DMA to enable streaming input of query descriptors and reference descriptors, and streaming output of index numbers for matching descriptors. The S_AXIS interface is used to feed data to the top-level hardware module and has a 512-bit data bus for carrying 256-bit descriptors and 16-bit descriptor index numbers. Unused bits are filled with "0". In the processing flow, the configuration register in the configuration control module is first configured with the selected bit index numbers calculated according to the above method, and then the set of reference descriptors is fed as an input stream to the BSTM module. Subsequently, query descriptors are fed to the module through the same interface. If a matching descriptor is found, the BSTM module outputs the calculated index numbers for each pair of matching query / reference descriptors. This output is transmitted through the M_AXIS interface of the BSTM module.

[0083] Figure 4 This is a schematic block diagram of a binary search tree matching module (BSTM-N module) with a depth of N in an embodiment of the present invention.

[0084] like Figure 4As shown, the BSTM-N hardware module 400 includes seven sub-modules: a BinarySearchTree module 410; a Switch module 420; a QueryDescStore module 430; a ReferenceDescStore module 440; a HammingDistanceCalc module 450; a MatchSelector module 460; and a Controller module 470. The Controller module 470 performs the function of storing the selected descriptor index number into the configuration register. The BinarySearchTree module 410 is responsible for generating the bin_id of the input descriptor and removing redundant bits. The Switch module 420 switches between the QueryDescStore module 430 and the ReferenceDescStore module 440 depending on whether the descriptor to be stored is a query or a reference descriptor, and transmits the input to the relevant module. The ReferenceDescStore module 440 stores the reference descriptors to be compared with the query descriptor and their index numbers. This module consists of a block random access memory (BRAM) that stores the reference descriptor and its index number in the appropriate location based on the input bin_id. The QueryDescStore module 430 is a FIFO (First-In-First-Out) module that stores the query descriptor and its index number to facilitate pipelined execution in this design. The HammingDistanceCalc module 450 calculates the Hamming distance used for binary descriptor matching. The MatchSelector module 460 selects the correct match based on a ratio-based test method, which is described in further detail below.

[0085] Figure 5 for Figure 4 The diagram shows the internal architecture of the BinarySearchTree module within the binary search tree matching module.

[0086] BinarySearchTree module 410 performs the function of generating bin_id and removing redundant information from the descriptor. The bin_id generated by this module determines the descriptor group required by subsequent modules to accelerate the matching process. The input of BinarySearchTree module 410 is the same as the input of the top-level BSTM module 400. The output of BinarySearchTree module 410 is the reshaped descriptor generated after removing redundant bits from the input descriptor. desc and bin of the input descriptor The BinarySearchTree module 410 includes N TreeLevelNode units, labeled TN0, TN1, TN2, ..., TNN-1.

[0087] Figure 6 for Figure 5 The diagram shows the internal architecture of the TreeLevelNode unit in the BinarySearchTree module.

[0088] Each TreeLevelNode unit 412 represents a level of the binary search tree. The TreeLevelNode unit 412 removes one bit of the input descriptor at each index provided by the Controller module 470 to generate the output reshape. desc (the output descriptor of TreeLevelNode cell 412) and bin_id (the output bin_id of TreeLevelNode cell 412).

[0089] The BSTM module 400 switches between the two descriptor storage hardware modules based on the type of the input descriptor. Specifically, the query descriptor is stored in the QueryDescStore module 430, while the reference descriptor is stored in the ReferenceDescStore module 440. Since the reference descriptor and the query descriptor are stored in the system separately as data blocks in the data stream, the switching process described above occurs when the last signal at the end of each data block reaches the Switch module 420.

[0090] Figure 7A and Figure 7B As shown Figure 4 The data path within the ReferenceDescStore module in the binary search tree matching module is shown. Figure 7A The image shows the DATA_IN_STATE (data input) used by the ReferenceDescStore module to receive and store descriptors. Figure 7B The image shows DATA_OUT_STATE (data output) used to read the descriptor.

[0091] The ReferenceDescStore module 440 is divided into 2 N The BRAM 442 is composed of several partitions. Since the latest FPGA-based localization methods use approximately 700 keypoints per image, the depth of the BRAM 442 is set to 1024 in this implementation. Because this depth is fixed, the maximum depth (MAX_DEPTH) of each BRAM partition is 1024 / 2.N .

[0092] When storing descriptors, if the number of descriptors in a certain descriptor group reaches the MAX_DEPTH limit, the ReferenceDescStore module 440 discards any further descriptors input into that descriptor group. It should be noted that the bit selection mechanism described above should minimize the number of discarded descriptors.

[0093] Figure 8 for Figure 4 The diagram shows the internal architecture of the QueryDescStore module within the binary search tree matching module.

[0094] The QueryDescStore module 430 stores query descriptors during descriptor matching. This module includes a 32-depth FIFO memory 432 and enables the BSTM module to leverage a pipelined architecture in subsequent modules to improve latency performance.

[0095] Figure 9 for Figure 4 The diagram shows the internal architecture of the HammingDistanceCalc module within the binary search tree matching module.

[0096] The HammingDistanceCalc module 450 performs the calculation of the Hamming distance. The QueryDescStore module 430 inputs a query descriptor and its index number to the HammingDistanceCalc module 450, and the ReferenceDescStore module 440 inputs a reference descriptor and its index number to the HammingDistanceCalc module 450. For these two inputs, the Hamming distance can be calculated in parallel by the XOR gate array 452 and its downstream adder tree 454. Since the descriptors received by this submodule are two types of descriptors obtained by modifying the query descriptor and the reference descriptor, these two descriptors need to be padded with N bits before being fed into the adder tree 454.

[0097] In addition, during the Hamming distance calculation, the index number input along with the descriptor is buffered, and the HammingDistanceCalc module 450 outputs the relevant distance calculation results.

[0098] Figure 10 for Figure 4 The diagram shows the internal architecture of the MatchSelector module in the binary search tree matching module.

[0099] The MatchSelector module 460 is used to remove irrelevant components. As shown below, a ratio test mechanism with a threshold of 0.5 is used for selection.

[0100]

[0101] The MatchSelector module 460 takes as input the distance between the query and reference descriptors, along with their index numbers. If any, it outputs the two index numbers of the matched pair of descriptors. This module employs a pipelined design and receives input data in each clock cycle. Figure 10 As shown, the two minimum distance values ​​are stored in registers min_1 (462) and min_2 (464), while the relevant descriptor index numbers are stored in the corresponding registers. When the last_in signal is enabled, if present, the module outputs a pair of index numbers for the matched query and reference descriptors.

[0102] The binary search tree feature matching system of this invention is evaluated below. The hardware of this invention was designed using SystemVerilog and synthesized and manufactured using the Vivado 2021.2 design suite. The hardware design was actually fabricated and verified using the Kria KV260AI starter kit (with K26 SOM) configured with an XCK26-SFVC784-2LV-C Xilinx Zynq Ultrascale+ MP-SoC device. This device includes a quad-core Arm A53 processing system with 4GB of RAM and a clock frequency of 1.33GHz. Furthermore, this hardware implementation was evaluated against a personal computer equipped with an Intel Xeon CPU with a clock frequency of 3.80GHz.

[0103] The following publicly available datasets were used in the evaluation process: leuven, bikes, and boat.

[0104] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, L.V. Gool, “A comparison of affine region detectors”, International Journal of Computer Vision, Vol. 65, pp. 43–72, 2005.

[0105] The aforementioned datasets all exhibit affine covariance characteristics—the leuven dataset shows significant variations in lighting conditions; the bikes dataset exhibits image blurring; and the boat dataset shows substantial rotation and scale variations. Each dataset consists of six images, with varying resolutions across datasets. Specifically, the leuven dataset contains images with a resolution of 921×614, the bikes dataset contains images with a resolution of 1000×700, and the boat dataset contains images with a resolution of 800×640. These challenging datasets allow for the evaluation of the performance and accuracy of the method described in this invention relative to existing implementations.

[0106] Precision is defined as follows:

[0107]

[0108] in, .

[0109] During the evaluation, BSTM-0, which is equivalent to the LES method, was selected as the benchmark. BSTM-0 uses a binary search tree with zero levels, therefore it does not include a BinarySearchTree module, and the BRAM of the ReferenceDescStore module contains only one descriptor group. For comparison purposes, BSTM-0 (LES method) and the design of this invention (BSTM-4) were implemented on the same K26 SOM platform and used the same clock frequency. Furthermore, evaluations were performed on all three datasets, and the same number of descriptors were used in the evaluations.

[0110] The evaluation results are shown in Table I below.

[0111] Table I: Latency Comparison of the BSTM Method and LES Method Implemented in the Programmable Logic of the Same K26 SOM Platform

[0112]

[0113] As can be seen, the execution speed of the method of the present invention is 10 times faster than that of the LES method. Furthermore, the runtime performance of the method of the present invention is evaluated relative to existing descriptor matching techniques, and the results are shown in Table II.

[0114] The prior art used as the comparison object is as follows:

[0115] Chien: C.H. Chien, C.J. Chien, C.C.Hsu, “Hardware-software co-design of an image feature extraction and matching algorithm”, 2nd IEEE International Conference on Intelligent Autonomous Systems (ICoIAS), 2019, pp. 37-41, 2019;

[0116] Fularz: M. Fularz, M. Kraft, A. Schmidt, A. Kasinski, “A high-performance fpga-based image feature detector and matcher based on the fast and brief algorithms”, International Journal of Advanced Robotic Systems, Vol. 12, No. 10, 141 pages, 2015;

[0117] Daoud: L. Daoud, M.K. Latif, H. Jacinto, N. Rafla, “A fully pipelined FPGA accelerator for scale invariant feature transform keypoint descriptor matching”, Microprocessors and Microsystems, Vol. 72, pp. 102,919, 2020.

[0118] Table II: Latency Comparison with Existing Descriptor-Matching Hardware Architectures

[0119]

[0120] All existing methods employ the LES method for descriptor matching. The table further shows the descriptors for the LES method running on a personal computer and a K26 SOM platform processing system. Results show that the method disclosed herein has the highest speed among all implementations listed in Table II. The second-best performing method, Fularz, achieves microsecond-level latency only when using significant resources, as it accelerates the processing with 32 matching cores.

[0121] Furthermore, the FPGA architecture of this invention is also implemented at different binary search tree levels (N=2~6), and resource utilization is evaluated relative to existing architectures. The results are shown in Table III.

[0122] The architecture used for comparison is as follows:

[0123] Wang: J. Wang, S. Zhong, L. Yan, Z. Cao, “An embedded system-on-chip architecture for real-time visual detection and matching”, IEEE transactions on Circuits and Systems for Video Technology, Vol. 24, No. 3, pp. 525-538, 2013;

[0124] Fularz: M. Fularz, M. Kraft, A. Schmidt, A. Kasinski, “A high-performance image feature detector based on FPGA and a matcher based on FAST and BRIEF algorithms”, International Journal of Advanced Robotics Systems, Vol. 12, No. 10, p. 141, 2015.

[0125] Chien: C.H. Chien, C.J. Chien, C.C.Hsu, “Software and Hardware Co-design of Image Feature Extraction and Matching Algorithms”, 2nd IEEE International Conference on Intelligent Autonomous Systems (ICoIAS), 2019, pp. 37-41, 2019;

[0126] Daoud: L. Daoud, M.K. Latif, H. Jacinto, N. Rafla, “A Fully Pipeline FPGA Accelerator for Keypoint Descriptor Matching of Scale-Invariant Feature Transform,” Microprocessors & Microsystems, Vol. 72, pp. 102,919, 2020.

[0127] Belmeshd: N.M. Belmessaoud, Y. Bentoutou, M.C. El-Mezouar, “FPGA implementation of feature detection and matching using ORB”, Microprocessors & Microsystems, Vol. 92, pp. 104,666, 2022.

[0128] Table III: Resource Comparison for Hardware Architectures Matching Existing Descriptors

[0129]

[0130] The results above show that the resources required for this invention are moderate, and increase with the number of binary search tree levels. Experimental results also indicate that the latency of this invention decreases as the number of binary search tree levels increases due to the reduction in the number of descriptors to be compared within each symbol group. For the same set of comparative experiments as in Table II, BSTM-6, with the most levels, achieves a latency in the 0.1 ms range. It should be noted that compared to the Fularz method using 32 cores, BSTM-6 uses smaller lookup tables (LUTs), flip-flops (FFs), and BRAM.

[0131] Finally, the matching accuracy of the implementation of this invention is analyzed, and the comparison results are shown below. Figures 11A to 11C In each image, the three bars correspond to the boat dataset, the Leuven dataset, and the bikes dataset, respectively.

[0132] Figure 11A This is a comparison chart of matching methods with pixel differences not exceeding 5 pixels under homography. The accuracy of each method is measured by determining the percentage of matching feature points whose distance from the corresponding projection point after homography transformation falls within 5 pixels.

[0133] Figure 11B This is a comparison chart of matching methods with a pixel difference of no more than 3 pixels under homography. This comparison allows for a smaller pixel deviation to emphasize accuracy; this parameter reflects each method's ability to achieve high spatial alignment precision. Because the tolerance is only 3 pixels, a closer correspondence between matching points is required. This highlights which methods remain effective under higher precision requirements.

[0134] Figure 11CThis image compares matching methods using a geometric verification method based on RANSAC (Random Sample Consensus Algorithm). The method iteratively fits the geometric model to the features, excluding mismatched outliers, and then calculates the percentage of inliers. Unlike homography-based comparisons, this verification method relies not only on pixel tolerance but also emphasizes geometric consistency. The resulting accuracy metric is the percentage of inliers (i.e., those that match the underlying geometric transformation model), thus evaluating accuracy.

[0135] First, using the experimental method described in the Belmeshd paper and the same dataset, the ratio test-based method was evaluated in BSTM-0 implementation and compared with the matching accuracy of the Belmeshd method. As shown in Figure 11, for each dataset, the ratio test output matching accuracy was 100%, higher than the accuracy described in the Belmeshd paper. Therefore, it is obvious that the ratio test is superior to the distance threshold method in terms of selection. Furthermore, Figure 11 further shows the evaluation results compared with the LES hardware implementation under different accuracy metrics used to select the correct match. Specifically, Figure 11A The evaluation results for the homography matrix of the dataset with a bias within 5 pixels are given. Figure 11B The evaluation results for the homography matrix with a dataset bias within 3 pixels are given. Figure 11C The results are evaluated using the RANSAC-based geometric test with a threshold unit of 1. It can be seen that the accuracy of the method described in this invention is comparable to or better than that of the LES method.

[0136] This disclosure provides an FPGA design for binary descriptor matching utilizing a hardware-efficient binary search tree. This design addresses the high matching latency requirements of existing methods by reducing the number of evaluations between query and reference descriptors. The design employs a ratio-test outlier elimination mechanism. Results show that this design surpasses commonly used distance-threshold-based algorithms in binary feature descriptor matching in terms of matching accuracy. FPGA evaluation results demonstrate that, compared to existing hardware implementations for descriptor matching, this design not only exhibits ultra-low matching latency and moderate resource usage but also achieves comparable accuracy for well-known datasets.

[0137] Although various exemplary embodiments have been described above, those skilled in the art should understand that, within the scope and spirit of this invention, there may be many modifications to the above embodiments.

Claims

1. A feature matching method that matches a query binary feature descriptor with a reference binary feature descriptor, characterized in that, The method includes: Based on the bit value at the selected bit position of the reference binary feature descriptor, the reference binary feature descriptor is assigned to the symbol group corresponding to the leaf node of the balanced binary search tree; The binary search tree is traversed using the query binary feature descriptor to reach a leaf node, and a reference binary feature descriptor is determined for assignment to the symbol group corresponding to the leaf node; and When the distance metric between the query binary feature descriptor and the determined reference binary feature descriptor is less than a threshold, the query binary feature descriptor is matched to the determined reference binary feature descriptor.

2. The method according to claim 1, characterized in that, Further includes: Store the allocated reference binary feature descriptor for each corresponding symbol group.

3. The method according to claim 2, characterized in that, Before storing the assigned reference binary feature descriptor for each corresponding symbol group, remove the value of the descriptor corresponding to the selected bit position.

4. The method according to any one of the preceding claims, characterized in that, The balanced binary search tree has a predefined number of levels and a corresponding predefined number of leaf nodes, wherein any reference binary feature descriptors that exceed the size of the leaf node symbol group are discarded.

5. The method according to any one of the preceding claims, characterized in that, The reference binary feature descriptor is assigned to the symbol group by selecting the index number at each level of the balanced binary search tree where the probability of the binary value 1 of the reference binary feature descriptor is closest to 0.

5.

6. The method according to any one of the preceding claims, characterized in that, The reference binary feature descriptor and the query binary feature descriptor are image feature descriptors.

7. The method according to any one of the preceding claims, characterized in that, The distance metric is thresholded by performing a ratio test.

8. The method according to claim 7, characterized in that, The ratio check includes performing a multiplication operation of 0.5 by performing a single-bit shift operation.

9. The method according to any one of the preceding claims, characterized in that, The distance measurement value is the Hamming distance measurement value.

10. A feature matching system for matching query binary feature descriptors with reference binary feature descriptors, characterized in that, The system includes: A binary search tree module is configured to: assign a reference binary feature descriptor to a symbol group corresponding to a leaf node of a balanced binary search tree based on the bit value of a selected bit position of the reference binary feature descriptor; and traverse the binary search tree using the query binary feature descriptor to reach a leaf node and determine the reference binary feature descriptor assigned to the symbol group corresponding to the leaf node; and The matching module is configured to match the query binary feature descriptor to the determined reference binary feature descriptor when the distance metric between the query binary feature descriptor and the determined reference binary feature descriptor is less than a threshold.

11. The feature matching system according to claim 10, characterized in that, The feature matching system is provided in the form of an integrated circuit.

12. The feature matching system according to claim 10 or 11, characterized in that, The binary search tree module is further used to store the assigned reference binary feature descriptor for each corresponding symbol group.

13. The feature matching system according to claim 12, characterized in that, The binary search tree module is further configured to remove the value corresponding to the selected bit position of the descriptor before storing the corresponding reference binary feature descriptor allocated for each corresponding symbol group.

14. The feature matching system according to any one of claims 10 to 13, characterized in that, The balanced binary search tree has a predefined number of levels and a corresponding predefined number of leaf nodes, wherein the binary search tree module is further used to discard any reference binary feature descriptors that exceed the size of the leaf node symbol group.

15. The feature matching system according to any one of claims 10 to 14, characterized in that, The binary search tree module is further configured to assign the reference binary feature descriptor to the symbol group by selecting the index number at each level of the balanced binary search tree whose probability of selecting the binary value 1 of the reference binary feature descriptor is closest to 0.

5.

16. The feature matching system according to any one of claims 10 to 15, characterized in that, The reference binary feature descriptor and the query binary feature descriptor are image feature descriptors.

17. The feature matching system according to any one of claims 10 to 16, characterized in that, The distance metric is thresholded by performing a ratio test.

18. The feature matching system according to claim 17, characterized in that, The ratio check includes performing a multiplication operation of 0.5 by performing a single-bit shift operation.

19. The feature matching system according to any one of claims 10 to 18, characterized in that, The distance measurement value is the Hamming distance measurement value.

Images