A matching method of a descriptor and a storage medium

Abstract

This invention discloses a descriptor matching method and storage medium, specifically including a model training step, a first classification step, a second classification step, a preliminary matching step, and a second matching step. A descriptor is classified into multiple categories, improving the method's robustness. A single AND operation determines whether two descriptors share the same category label, and distance calculation yields the best matching object for the descriptor to be matched. This invention is simple and practical, improving the matching efficiency, accuracy, and recall rate of descriptor matching methods.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to a descriptor matching method and storage medium. Background Technology

[0002] VR devices are electronic products that can provide an immersive experience in a virtual world. In VR devices, visual SLAM (Simultaneous Localization and Mapping) is the main 6DoF technology they rely on. It refers to the device creating a map based on images captured by a camera in a completely unknown environment when its own position is uncertain, and then using the map to perform autonomous positioning.

[0003] The relocalization schemes in popular visual SLAM mainly include three stages. The first stage is candidate image selection, which requires finding the candidate image that is most similar to the query image in the historical images. The second stage is image feature descriptor matching, which requires matching the query image and the candidate images with image feature descriptors to facilitate the next step of solving the pose. The third stage is geometric verification, which requires using the matching results from the previous step to solve the pose and determine the number of inliers.

[0004] In the image feature descriptor matching stage, the most direct method in the existing technology is brute-force matching, that is, calculating the distance between each descriptor of the query image and all descriptors of the corresponding candidate image, finding the corresponding descriptor that is closest to the current descriptor, and thus obtaining the matching relationship of all descriptors of the two images. This method has the highest accuracy, but it is the least efficient and time-consuming, which is unacceptable in real-time SLAM systems.

[0005] In the image feature descriptor matching stage, another commonly used method in existing technologies is the direct indexing method based on the DBoW algorithm. The direct indexing method refers to the process where, during the generation of BoW vectors, each image feature descriptor needs to find its relevant leaf node in the dictionary tree. In this process, the feature descriptors of both the candidate image and the query image are classified into multiple categories by different nodes in the dictionary tree. The query image descriptor only needs to be matched with the feature descriptors of the candidate image in the same category, reducing the number of calculations and thus speeding up the matching process. This simplifies the entire matching process and accelerates the descriptor matching process. However, the direct indexing method's classification is too absolute, forcibly assigning each descriptor to the nearest classification center without considering the situation where descriptors are close to multiple classification centers. This forced classification will cause actually similar descriptors to be classified into different categories, preventing them from meeting again in subsequent matching, thus reducing the number of correct matches. These problems reduce the accuracy of descriptor matching and significantly impact the recall rate of relocation. Summary of the Invention

[0006] This invention provides a descriptor matching method and storage medium to solve the technical problem that existing image feature descriptor matching methods cannot simultaneously achieve high matching efficiency and high accuracy.

[0007] To solve the above-mentioned technical problems, the present invention discloses the following technical solution:

[0008] This invention provides a descriptor matching method, which specifically includes a model training step, a first classification step, a second classification step, a preliminary matching step, and a second matching step.

[0009] The model training step is used to train a classifier model offline. The classifier model includes a node tree structure composed of multiple classification nodes, each classification node representing the category of a descriptor. The first classification step involves inputting at least one first descriptor into the classifier model, labeling each first descriptor with at least one classification node, and generating at least one first descriptor label. The second classification step involves inputting a second descriptor into the classifier model, labeling the second descriptor with at least one classification node, and generating a second descriptor label. The preliminary matching step involves comparing each first descriptor label with the second descriptor label. When a classification node of a first descriptor is the same as a classification node of a second descriptor, the first descriptor is determined to be initially matched with the second descriptor. The rematch step is used to calculate and compare the distance between each initially matched first descriptor and the second descriptor, and the first descriptor closest to the second descriptor is recorded as the matching object of the second descriptor.

[0010] Furthermore, before the first classification step, a first feature extraction step is included, in which a candidate image is subjected to feature extraction, and at least one image feature descriptor is extracted from the candidate image, denoted as the first descriptor.

[0011] Furthermore, prior to the second classification step, a second feature extraction step is included, in which features are extracted from an image to be retrieved, and at least one image feature descriptor is extracted from the image to be retrieved, denoted as the second descriptor.

[0012] Furthermore, the first classification step specifically includes a first distance calculation step and a first node marking step.

[0013] The first distance calculation step is used to calculate the distance between each first descriptor and each classification node in the classifier model; the first node marking step is to mark all classification nodes whose distance to each first descriptor is less than a preset threshold on the corresponding first descriptor.

[0014] Furthermore, in the first classification step, an initial binary classification label is established for each first descriptor, and each bit of the classification label corresponds to a classification node; when a first descriptor is labeled with at least one classification node, the bits in the classification label of the first descriptor corresponding to those classification nodes are marked as 1, thus obtaining the first descriptor label.

[0015] Furthermore, the second classification step specifically includes a second distance calculation step and a second node labeling step.

[0016] The second distance calculation step is used to calculate the distance between the second descriptor and each classification node in the classifier model; the second node marking step is to mark all classification nodes whose distance from the second descriptor is less than a preset threshold on the second descriptor.

[0017] Furthermore, in the second classification step, an initial binary classification label is established for each second descriptor, and each bit of the classification label corresponds to a classification node; when a second descriptor is labeled with at least one classification node, the bits in the classification label of the second descriptor corresponding to those classification nodes are marked as 1, thus obtaining the second descriptor label.

[0018] Further, the preliminary matching step specifically includes the following steps: performing a bitwise AND operation between the second descriptor tag and each first descriptor tag to obtain at least one operation result; determining whether each operation result is zero; if so, determining that the first descriptor tag and the second descriptor tag have no common nodes; if not, proceeding to the next step; determining that the first descriptor tag and the second descriptor tag are initially matched.

[0019] Furthermore, the re-matching step includes the following steps: calculating the distance between the second descriptor and multiple initially matched first descriptors to obtain at least one descriptor distance; comparing the descriptor distances to find the closest distance and the second closest distance; determining whether the ratio of the closest distance to the second closest distance is less than a preset threshold, and if so, determining that the first descriptor closest to the second descriptor is the matching object of the second descriptor.

[0020] The present invention also provides a storage medium, a memory for storing executable program code, and a processor for reading the executable program code to run a computer program corresponding to the executable program code to perform at least one step in the descriptor matching method.

[0021] Compared with the prior art, the present invention has at least the following technical effects:

[0022] This invention provides a descriptor matching method. Each descriptor is categorized and assigned a category label. Through a bitwise AND operation of the category labels, descriptors with the same category label as a specific descriptor are marked as initial matches. Other descriptors without a matching relationship to the specific descriptor are quickly filtered out, retaining only a small number of descriptors that initially match the specific descriptor. Then, distance calculation is used to find the best match for the descriptor to be matched. This process effectively reduces the processor's computational load, improving descriptor matching efficiency and speed while effectively ensuring matching accuracy. This invention has better robustness and can improve the recall rate of relocation. Attached Figure Description

[0023] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 This is a flowchart of the descriptor matching method according to an embodiment of the present invention;

[0025] Figure 2 This is a structural diagram of the classifier model described in an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of the method for the first classification step according to an embodiment of the present invention;

[0027] Figure 4 This is a flowchart of the classification steps for the first descriptor D0 according to an embodiment of the present invention;

[0028] Figure 5 This is a flowchart of the second classification step method according to an embodiment of the present invention;

[0029] Figure 6This is a flowchart of the classification steps for the second descriptor D1 according to an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the storage medium structure according to an embodiment of the present invention.

[0031] The components in the diagram are labeled as follows:

[0032] 80 VR glasses, 81 memory, 82 processor. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, based on a VR device, the present invention provides a descriptor matching method, which is implemented in software and specifically includes steps S1) model training, S2) first feature extraction, S3) first classification, S4) second feature extraction, S5) second classification, S6) preliminary matching, and S7) re-matching.

[0035] like Figure 2 As shown, step S1) is a model training step used to train a classifier model offline. The classifier model includes a node tree structure composed of multiple classification nodes. In this embodiment, the classifier model includes 64 classification nodes, namely Node0 to Node63. Each classification node represents a category of a descriptor. Preferably, the data type of the node tree is consistent with the data type of the descriptor.

[0036] Step S2) First feature extraction step: Perform feature extraction on a candidate image and extract at least one image feature descriptor from the candidate image, denoted as the first descriptor. When there are multiple candidate images, at least one image feature descriptor must be extracted for each candidate image. Only after all descriptors of a candidate image have been matched can all descriptors of the next candidate image be matched. This method is applicable to all image descriptors, whether it is a commonly used binary descriptor such as BRIEF and BAD, or a floating-point descriptor such as SIFT and SURF.

[0037] Step S3) First classification step: Input at least one first descriptor into the classifier model, label at least one classification node for each first descriptor, and generate at least one first descriptor label. In this embodiment, the first descriptor D0 is taken as an example.

[0038] like Figure 3 As shown, step S3) the first classification step specifically includes step S31) label establishment step, step S32) first distance calculation step and step S33) first node marking step.

[0039] Step S31): First tag creation step, such as Figure 4 As shown, an initial binary category label Label D0 is created for the first descriptor D0. The initial binary category label Label D0 is divided into 64 sub-labels, which correspond to 64 category nodes Node0~Node63. The number of category labels is represented by a 64-bit unsigned integer uint64_t. Each bit of the category label corresponds to a category node, and each category node is initially set to 0. When a first descriptor is labeled with at least one category node, the bit in the category label of the first descriptor corresponding to that category node is set to 1, thus obtaining the first descriptor label Label D0.

[0040] Step S32) First distance calculation step: Taking the first descriptor D0 as an example, calculate the distance between the first descriptor D0 and each classification node in the classifier model. Calculate the distance between the first descriptor D0 and each of the 64 classification nodes Node0~Node63 in the classifier model, and record the distance between the first descriptor D0 and the node closest to the first descriptor D0 as S1.

[0041] Step S33) First node marking step: All classification nodes whose distance to the first descriptor D0 is less than a preset threshold S1*n are marked on the corresponding first descriptor. In this embodiment, n=1.2. Taking the first descriptor D0 as an example, if the distance between node Node1 and node Node2 and the first descriptor D0 is less than S1*1.2, these two nodes are marked on the first descriptor D0. At this time, the bit position corresponding to node Node1 and node Node2 in the descriptor label D0 is marked as 1, which indicates that the first descriptor D0 is divided into two categories, Node1 and Node2.

[0042] Step S4) Second feature extraction step: extract features from an image to be retrieved, and extract at least one image feature descriptor from the image to be retrieved, denoted as the second descriptor. In this embodiment, the second descriptor D1 is taken as an example. This method is applicable to all image descriptors, whether it is a commonly used binary descriptor such as BRIEF and BAD, or a floating-point descriptor such as SIFT and SURF.

[0043] Step S5) The second classification step is to input the second descriptor D1 into the classifier model, label the second descriptor with at least one classification node, and generate the second descriptor label Label D1.

[0044] like Figure 5 As shown, step S5) the second classification step specifically includes step S51) label establishment step, step S52) second distance calculation step and step S53) second node marking step.

[0045] Step S51) Second tag creation step, such as Figure 6 As shown, an initial binary category label Label D1 is created for the second descriptor D1. The initial binary category label Label D1 is divided into 64 sub-labels, which correspond to 64 category nodes Node0 to Node63. Each bit of the category label corresponds to a category node. When a second descriptor is labeled with at least one category node, the bits in the category label of the second descriptor corresponding to those category nodes are marked as 1, thus obtaining the second descriptor label Label D1.

[0046] Step S52) Second distance calculation step: Taking the second descriptor D1 as an example, calculate the distance between the second descriptor D1 and each classification node in the classifier model. Calculate the distance between the second descriptor D1 and each of the 64 classification nodes Node0 to Node63 in the classifier model, and record the distance between the second descriptor D1 and the node closest to the second descriptor D1 as S2.

[0047] Step S53) Second node marking step: All classification nodes whose distance to the second descriptor D1 is less than a preset threshold S2*n are marked on the corresponding second descriptor D1. In this embodiment, n=1.2. Node0 and Node2 are less than S2*1.2 from the second descriptor D1, so these two nodes are marked on the second descriptor D1. At this time, the bit position corresponding to Node0 and Node2 in the descriptor label D1 is marked as 1, indicating that the second descriptor D1 is divided into two categories: Node0 and Node2.

[0048] Preferably, this scheme compares a second descriptor of an image to be retrieved with multiple first descriptors of a historical image to find a matching object; during the image retrieval process, it is necessary to match each second descriptor of the image to be retrieved using this method to find historical images that match the image to be retrieved.

[0049] Step S6) The preliminary matching step is to compare each first descriptor label with the second descriptor label. When a category node of a first descriptor is the same as a category node of a second descriptor, it is determined that the first descriptor and the second descriptor are initially matched.

[0050] In this embodiment, step S6) preliminary matching step specifically includes the following steps: comparing the label of each first descriptor with the label of the second descriptor D1. During the matching process, performing a bitwise AND operation between the classification label of each first descriptor and the classification label of the second descriptor D1 to obtain at least one operation result. If the result is 0, it means that the first descriptor and the second descriptor D1 have no common nodes, and the current first descriptor can be skipped directly. If the result is not 0, it means that the first descriptor and the second descriptor D1 have common nodes, and it is determined that the first descriptor and the second descriptor are initially matched, obtaining at least one initially matched first descriptor, and proceeding to the next step S7).

[0051] When a binary second descriptor label is ANDed with a binary first descriptor label, if any bit of the operation results in a 1, it indicates that the corresponding classification node is a common classification node for both the second and first descriptors, allowing for a preliminary match. If all bits of the operation result in a 0, it indicates that the second and first descriptors do not share any classification nodes, and therefore, they do not match. For example, performing an AND operation between a second descriptor label and five first descriptor labels may result in all mismatches or one to five matches.

[0052] The purpose of performing a bitwise AND operation between the second descriptor label and the first descriptor label is to filter out first descriptors that do not match the second descriptor during this process, and only retain the first descriptors that initially match the second descriptor for subsequent distance judgment. This can skip a large number of irrelevant descriptors, accelerate the descriptor matching process, effectively reduce the amount of computation on the processor, and achieve a faster response.

[0053] like Figure 7As shown, a bitwise AND operation is performed between the first descriptor label D0 and the second descriptor label D1. Since the Node2 operation result of the first descriptor label D0 and the second descriptor label D1 is 1, the first descriptor D0 is an initial matching object of the second descriptor D1.

[0054] Step S7) Matching step again: Calculate and compare the distance between each initially matched first descriptor and the second descriptor, and record the first descriptor that is closest to the second descriptor as the matching object of the second descriptor. In this embodiment, the first descriptor D0 is the matching object of the second descriptor D1.

[0055] In this embodiment, the re-matching step S7) includes the following steps: calculating the distance between the second descriptor D1 and multiple initially matched first descriptors to obtain at least one descriptor distance; comparing the descriptor distances to find the closest distance and the second closest distance, i.e., obtaining the first descriptor D0 with the closest distance and the first descriptor D2 with the second closest distance; determining whether the ratio of the closest distance to the second closest distance is less than a preset threshold, and if so, determining that the first descriptor D0 with the closest distance to the second descriptor is the matching object of the second descriptor.

[0056] The purpose of the rematch step is to verify the obtained matching results. The rematch result between the second descriptor and multiple initially matched first descriptors is that the first descriptor closest to the second descriptor is its matching object. To verify the accuracy of this matching result, it is also necessary to find the second closest first descriptor and calculate the nearest and second closest distances. If the ratio of the nearest distance to the second closest distance is less than a preset threshold, the matching result is considered correct. If the second descriptor closest to the first descriptor is an anomalous feature descriptor, its distance from other descriptors will be relatively large, and the ratio of the nearest distance to the second closest distance will definitely be greater than the preset threshold.

[0057] In visual SLAM relocalization schemes, the first stage is candidate image selection, which involves finding the candidate image most similar to the query image in historical images. Then, the second stage proceeds to descriptor matching.

[0058] For example, in the first stage, three candidate images have been found, which need to be compared and matched separately. First, extract multiple descriptors from one of the candidate images, for example, five. These five descriptors are then used as the first descriptors and input into the classifier model to generate five first descriptor labels N1, N2, N3, N4, and N5.

[0059] Similarly, five image feature descriptors are extracted from the image to be retrieved, denoted as the second descriptors, and input into the classifier model to generate five second descriptor labels M1, M2, M3, M4 and M5.

[0060] The five second descriptor tags M1, M2, M3, M4, and M5 need to be ANDed with the five first descriptor tags N1, N2, N3, N4, and N5 respectively. Taking the second descriptor tag M1 as an example, after one AND operation, the first descriptor tags N1, N2, and N4 with non-zero results are obtained. N1, N2, and N4 are the initial matching objects of the second descriptor tag M1. Then, the distance is calculated to obtain N1, which is closest to the second descriptor tag M1, and N4, which is second closest to the second descriptor tag M1. The nearest distance and the second closest distance are obtained. By calculating whether the ratio of the nearest distance and the second closest distance is less than a preset threshold, if so, the matching result is considered correct, and N1 is the best matching object of the second descriptor tag M1.

[0061] The matching process for the second descriptor tags M2, M3, M4, and M5 is the same as that for M1, and will not be repeated here.

[0062] After the matching of image feature descriptors is completed, the geometric verification stage begins. The descriptor matching results in this invention are used to solve the pose and determine the number of inliers.

[0063] For example, the image features of the image to be retrieved are M1, M2, and M3, and the image features of the candidate images are N1, N2, and N3. Descriptor matching yields one-to-one matching relationships: M1-N1, M2-N2, and M3-N3. This matching relationship allows us to calculate the change in camera pose T between the two images. The pixel coordinates of the image features M1, M2, and M3 of the image to be retrieved, after this change T, become M1', M2', and M3'. Ideally, if this change T is calculated accurately, the coordinates of N1 and M1', N2 and M2', and N3 and M3' should be identical. However, due to noise and other interference, if the distance between their coordinates is less than a threshold, they can be considered inliers. If the number of inliers exceeds a preset threshold, the calculated change in camera pose T is considered correct.

[0064] like Figure 7As shown, the present invention also provides a VR device 80, which includes a memory 81 and a processor 82. The storage medium in the memory 81 is used to store executable program code. The processor 82 reads the executable program code to run a computer program corresponding to the executable program code to execute at least one step from step S1) to step S7) in the descriptor matching method.

[0065] The advantage of this invention lies in classifying a descriptor into multiple categories, which makes the method more robust. Each descriptor is classified and assigned a category label. By performing an AND operation on the category labels, descriptors with the same category label as a specific descriptor are marked as preliminary matches. Other descriptors that are not matched with a specific descriptor are quickly filtered out, leaving only a small number of descriptors that are initially matched with a specific descriptor. Then, by calculating the distance, the best matching object for the descriptor to be matched is found. This process effectively reduces the amount of computation of the processor, improving the matching efficiency and speed of descriptors while effectively ensuring the matching accuracy of descriptors.

[0066] The descriptor matching method provided in the embodiments of the present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of the present invention. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of matching descriptors, characterized by, Specifically comprising the following steps: a model training step of training a classifier model offline, the classifier model comprising a node tree structure composed of multiple classification nodes, each classification node representing a category of a descriptor; a first feature extraction step of performing feature extraction on a candidate image, extracting at least one image feature descriptor from the candidate image, denoted as a first descriptor; a first classification step of inputting the at least one first descriptor into the classifier model, marking at least one classification node for each first descriptor, and generating at least one first descriptor label; the first classification step specifically comprises the following steps: a first distance calculation step of calculating the distance between each first descriptor and each classification node in the classifier model; and a first node marking step of marking all classification nodes with a distance less than a preset threshold to each first descriptor on the corresponding first descriptor; in the first classification step, an initialized binary classification label is established for each first descriptor, each digit of the classification label corresponding to a classification node; when a first descriptor is marked with at least one classification node, the bits corresponding to the classification nodes in the classification label of the first descriptor are marked as 1, obtaining a first descriptor label; a second classification step of inputting a second descriptor into the classifier model, marking at least one classification node for the second descriptor, and generating a second descriptor label; the second classification step specifically comprises the following steps: a second distance calculation step of calculating the distance between the second descriptor and each classification node in the classifier model; and a second node marking step of marking all classification nodes with a distance less than a preset threshold to the second descriptor on the second descriptor; in the second classification step, an initialized binary classification label is established for each second descriptor, each digit of the classification label corresponding to a classification node; when a second descriptor is marked with at least one classification node, the bits corresponding to the classification nodes in the classification label of the second descriptor are marked as 1, obtaining a second descriptor label; a preliminary matching step of comparing each first descriptor label with the second descriptor label respectively, and determining that a first descriptor and the second descriptor are preliminarily matched when a classification node of the first descriptor is the same as a classification node of the second descriptor; and a re-matching step of calculating and comparing the distance between each preliminarily matched first descriptor and the second descriptor, and marking the first descriptor with the closest distance to the second descriptor as the matching object of the second descriptor.

2. The descriptor matching method of claim 1, wherein before the second classification step, further comprising a second feature extraction step of performing feature extraction on a to-be-searched image, extracting at least one image feature descriptor from the to-be-searched image, denoted as a second descriptor.

3. The descriptor matching method of claim 1, wherein the preliminary matching step specifically comprises the following steps: ​ performing an AND operation between the second descriptor tag and each of the first descriptor tags to obtain at least one operation result; determining whether each of the operation results is zero, if yes, determining that the first descriptor tag and the second descriptor tag have no common node, if no, performing the next step; and determining that the first descriptor and the second descriptor are preliminarily matched.

4. The descriptor matching method of claim 1, wherein the re-matching step comprises the following steps: calculating distances between the second descriptor and the first descriptors preliminarily matched to obtain at least one descriptor distance; comparing the descriptor distances to find a nearest distance and a second nearest distance; and determining whether a ratio of the nearest distance to the second nearest distance is less than a preset threshold, if yes, determining that the first descriptor closest to the second descriptor is a matching object of the second descriptor.

5. A storage medium for storing executable program codes, wherein a processor reads the executable program codes to run a computer program corresponding to the executable program codes to perform at least one step of the descriptor matching method of any one of claims 1-4. ​ ​

Images