Binocular feature matching method and device based on quadtree and optical flow estimation, and medium

Abstract

The invention relates to a quadtree and optical flow estimation-based binocular feature matching method and device, and a medium, and the method comprises the following steps: respectively reading a left eye image and a right eye image, and respectively extracting AGAST features from the left eye image and the right eye image; establishing a left quadtree and a right quadtree, dividing the features into quadtree nodes, and screening left feature points and right feature points respectively; using a BEBLID algorithm to respectively describe the screened left eye feature points and right eye feature points, and respectively generating a left eye binary descriptor set and a right eye binary descriptor set; estimating coordinates of the screened left eye feature points in the right eye image by using an optical flow algorithm; traversing the screened left eye feature points, and querying right quadtree nodes according to the estimated coordinates to obtain an adjacent point set; and performing a 2-nearest neighbor algorithm on the neighbor point set to obtain an accurate matching pair of the left-eye feature points and the right-eye feature points. Compared with the prior art, the method can accelerate the feature matching process and improve the matching correctness.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature extraction and matching, and in particular to a binocular feature matching method, device and medium based on quadtree and optical flow estimation. Background Art

[0002] After extracting features from continuous images using computer vision technology, multi-view geometric modeling can restore the camera's motion and external environment map by establishing associations between features. This technology is called "visual simultaneous localization and mapping", or visual SLAM for short. Fast and accurate feature matching can improve the real-time and robustness of visual SLAM. For feature matching from left to right in binocular visual SLAM, the main technical solutions are:

[0003] (1) Calculate the descriptors of the left and right eyes, and match them by comparing their vector distances, i.e., brute force matching.

[0004] (2) Divide the feature points of the left and right eyes into a uniform grid, traverse the feature points of the left eye, find the row in the right eye grid according to its coordinates, take the feature points of the right eye in the same row as candidate points, calculate the Hamming distance between binary descriptors, and use a K-nearest neighbor algorithm such as KNN to obtain the matching result.

[0005] The logic of solution (1) is simple and easy to implement, but the calculation time is huge and the matching results are not accurate enough. Solution (2) can significantly shorten the feature matching time and obtain more accurate feature matching results, but there is no reasonable screening process for a large number of features, and there is still room for improvement in real-time performance.

[0006] Therefore, a fast and accurate image feature matching method is urgently needed. Summary of the invention

[0007] The purpose of the present invention is to overcome the problems existing in the prior art, provide a binocular feature matching method, device and medium based on quadtree and optical flow estimation, use a quadtree to divide the left eye and right eye feature sets respectively, facilitate the query of neighboring points, give the position estimate of the left eye feature in the right eye through optical flow tracking, query the right eye quadtree to obtain neighboring points and use 2 nearest neighbors to complete the matching, which can accelerate the feature matching process and improve the matching accuracy.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A first object of the present invention is to provide a binocular feature matching method based on quadtree and optical flow estimation, the method comprising the following steps:

[0010] Read the left image and the right image respectively, and extract AGAST features from the left image and the right image respectively;

[0011] Establishing a quadtree and dividing the features into quadtree nodes, respectively screening left feature points and right feature points, wherein the quadtree includes a left quadtree and a right quadtree;

[0012] Use the BEBLID algorithm to describe the filtered left feature points and right feature points respectively, and generate a binary descriptor set for the left target and a binary descriptor set for the right target respectively;

[0013] Based on the binary descriptor set of the left eye and the binary descriptor set of the right eye, the optical flow algorithm is used to estimate the coordinates of the filtered left eye feature points in the right eye image;

[0014] Traverse the filtered left feature points, query the right quadtree node according to the estimated coordinates to obtain the neighboring point set;

[0015] The 2 nearest neighbor algorithm is used on the neighbor point set to obtain an exact matching pair of left eye feature points and right eye feature points.

[0016] Furthermore, the steps of reading the left image and the right image respectively and extracting AGAST features from the left image and the right image respectively include:

[0017] When a frame image of the left eye and a frame image of the right eye are received, the AGAST algorithm is used to extract features of the left eye and the right eye to obtain a left eye feature point set and a right eye feature point set.

[0018] Furthermore, the steps of establishing a left quadtree and a right quadtree, dividing the features into quadtree nodes, and respectively screening left feature points and right feature points include:

[0019] A quadtree data structure is established for the left feature point set and the right feature point set for division. A node capacity threshold N is set for the entire quadtree to record the maximum number of feature points that all nodes can store. The subnodes in the node are designed as 4 pointer arrays, which point to the upper left, lower left, upper right or lower right sub-areas respectively.

[0020] Divide the feature points into nodes, determine whether the current node is a leaf node, if it is a leaf node, determine whether the capacity of the current node is full, if it is full, split the current node and re-divide it, if it is not full, save the feature points directly, if it is not a leaf node, map it to one of the four sub-regions of upper left, lower left, upper right or lower right according to its horizontal and vertical coordinates, and divide it into its corresponding sub-node;

[0021] Recursively divide all feature points into leaf nodes;

[0022] Maintain a quadtree for the left and right eyes respectively. Sort the AGAST features of the left eye from high to low according to the response value and divide them into the left quadtree. Sort the AGAST features of the right eye from high to low according to the response value and divide them into the right quadtree. Each node automatically selects the M feature points with the highest local response value.

[0023] Furthermore, in the step of estimating the coordinates of the filtered left-eye feature points in the right-eye image using an optical flow algorithm based on the left-eye binary descriptor set and the right-eye binary descriptor set, the optical flow algorithm is an LK optical flow algorithm.

[0024] Furthermore, the step of estimating the coordinates of the filtered left eye feature points in the right eye image using an optical flow algorithm based on the left eye binary descriptor set and the right eye binary descriptor set includes:

[0025] Based on the binary descriptor set of the left eye and the binary descriptor set of the right eye, the LK optical flow algorithm is used to track the position of the filtered left eye feature points in the right eye image, the window size of the LK optical flow algorithm is set, the number of pyramid levels is set, the two-dimensional coordinate point data of the binary descriptor set of the left eye is passed in, and the tracked two-dimensional coordinate point data in the right eye is obtained. The tracked two-dimensional coordinates in the right eye indicate the position of the filtered left eye feature points in the right eye image.

[0026] Furthermore, the step of traversing the filtered left feature points and querying the right quadtree node according to the estimated coordinates to obtain a neighboring point set includes:

[0027] Traverse the set of feature points successfully tracked by the left eye, use the tracked right eye coordinate points to query the right quadtree, and obtain the set of neighboring points.

[0028] Furthermore, the step of using a 2-nearest-neighbor algorithm on the neighbor point set to obtain an accurate matching pair of left-eye feature points and right-eye feature points includes:

[0029] Use the 2 nearest neighbor algorithm on the subset of neighbor point sets to calculate the two feature points with the closest Hamming distance of the descriptor. When the distance of the closest feature point is less than 0.7 of the distance of the second closest feature point, save this matching pair, which is the feature pair matched according to the descriptor;

[0030] Traverse to get the exact matching pair of left eye features and right eye features.

[0031] Furthermore, the node type of the quadtree node stores data including starting point coordinates, width, height, leaf node flag, feature point array, feature point index array and node depth.

[0032] Another object of the present invention is to provide an electronic device, comprising a memory and a processor, wherein the processor is used to execute a program in the memory, so as to implement a binocular feature matching method based on quadtree and optical flow estimation as described in any one of claims 1 to 8.

[0033] Another object of the present invention is to provide a storage medium containing computer executable instructions, which, when executed by a computer processor, is used to execute the above-mentioned binocular feature matching method based on quadtree and optical flow estimation.

[0034] Compared with the prior art, the present invention has achieved the following beneficial effects: 1) The present technical solution provides a binocular feature matching method based on quadtree and optical flow estimation, which uses a quadtree to divide the left and right eye feature sets, facilitates the query of neighboring points, gives the position estimate of the left eye feature in the right eye through optical flow tracking, queries the right eye quadtree to obtain neighboring points and uses 2 nearest neighbors to complete the matching.

[0035] 2) The present technical solution provides a binocular feature matching method based on quadtree and optical flow estimation. By using the initial position estimation given by the optical flow and the convenience of querying neighboring points by quadtree, the number of neighboring point traversals for feature matching is greatly reduced compared to the grid method, thereby accelerating the feature matching process and improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The accompanying drawings are only provided for reference and explanation and are not intended to limit the present invention.

[0037] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In the following description of the present invention, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "inside" and "outside" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description, and do not mean that the device must have a specific direction.

[0039] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0041] S1. When a new frame of left and right eye images is received, the AGAST algorithm is used to extract features of the left and right eyes to obtain a left eye feature point set and a right eye feature point set.

[0042] S2. Establish a quadtree data structure for each feature point set for partitioning.

[0043] The node type of the quadtree stores the following data:

[0044] (1) Starting point coordinates. Save the horizontal and vertical coordinates representing the upper left corner of the image area.

[0045] (2) Width. Saves the width of the image area.

[0046] (3) Height. Saves the height of the image area.

[0047] (4) Leaf node flag: A flag that stores whether the current node is a leaf node.

[0048] (5) Feature point array. Save the feature points in the current node.

[0049] (6) Feature point index array. Save the index number corresponding to the feature point in the current node.

[0050] (7) Node depth. Save the depth of the current node.

[0051] The entire quadtree will set a node capacity threshold N to record the maximum number of feature points that can be saved by all nodes. The subnodes in the node are designed as 4 pointer arrays. The feature points divided into the node first determine whether the current node is a leaf node. If it is a leaf node, determine whether the capacity is full. If it is full, split the current node and re-divide it. If it is not full, save the feature points directly. If it is not a leaf node, map it to one of the four sub-regions of upper left, lower left, upper right or lower right according to its horizontal and vertical coordinates (using spatial hashing technology, according to the ratio of the horizontal coordinate to the image width, the ratio of the vertical coordinate to the image height, round down, generate an index mapped to 0 to 3, representing the upper left, upper right, lower left, and lower right regions respectively), and divide it into its own child nodes. In this way, all feature points are recursively divided into the leaf nodes where they are located.

[0052] The above quadtree is maintained for both the left and right eyes. The AGAST features of the left eye are sorted from high to low according to the response value and then divided into the left quadtree. The same operation is performed on the right eye. Due to the capacity limit of the node, each node will automatically filter out the M feature points with the highest local response value, thereby achieving overall feature screening and realizing a relatively uniform distribution of features.

[0053] S3. Use the BEBLID algorithm to describe the filtered binocular features and generate a binary descriptor set for the left eye and the right eye.

[0054] S4, use LK optical flow algorithm to track the position of left eye feature points in the right eye image, the algorithm window setting size is 50*50 pixels, the pyramid level number is set to 2, the algorithm is passed to the two-dimensional coordinate point data of the left eye feature point set, and the tracked two-dimensional coordinate point data in the right eye will be obtained. The tracked two-dimensional coordinates indicate the position of the left eye feature in the right eye image. Then traverse the left eye tracking successful feature point set, use the tracked right eye coordinates to query the right quadtree, and obtain the neighbor point set, which is the nearby right eye feature. Use the 2 nearest neighbor algorithm for this subset, calculate the two features with the closest descriptor Hamming distance, and save this matching pair when the distance of the nearest feature is less than 0.7 of the second closest distance. After the traversal is completed, a more accurate matching pair (accurate matching pair) of the left eye feature and the right eye feature will be obtained.

[0055] Based on the above method, this embodiment also proposes an electronic device for a binocular feature matching method based on quadtree and optical flow estimation, the electronic device includes a processor and a memory, the processor and the memory are coupled, the memory stores program instructions, and the above method is implemented when the program instructions stored in the memory are executed by the processor. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components; the memory may include a random access memory (RAM) and may also include a non-volatile memory (Non-VolatileMemory), such as at least one disk storage. The memory can be an internal memory of the random access memory (RAM) type, and the processor and the memory can be integrated into one or more independent circuits or hardware, such as: an application-specific integrated circuit (ASIC). It should be noted that the computer program in the above memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention.

[0056] This embodiment also proposes a storage medium of computer executable instructions, wherein the storage medium stores computer instructions, and when the storage medium of the computer executable instructions is executed by a computer processor, it is used to execute the above-mentioned binocular feature matching method based on quadtree and optical flow estimation. The storage medium can be an electronic medium, a magnetic medium, an optical medium, an electromagnetic medium, an infrared medium or a semiconductor system or a propagation medium. The storage medium can also include semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk and an optical disk. The optical disk can include a compact disk-read only memory (CD-ROM), a compact disk-read / write (CD-RW) and a DVD.

[0057] The above is only a preferred embodiment of the present invention, which shows and describes the basic principles and main features of the present invention and the advantages of the present invention, but does not limit the scope of patent protection of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. In addition to the above embodiments, the present invention may have other implementation modes without departing from the spirit and scope of the present invention. The present invention may also have various changes and improvements, and all technical solutions formed by equivalent substitution or equivalent transformation fall within the scope of protection required by the present invention. The scope of protection required by the present invention is defined by the attached claims and their equivalents. The technical features not described in the present invention can be realized by or using existing technologies, which will not be repeated here.

Claims

1. A binocular feature matching method based on quadtree and optical flow estimation, characterized in that: The method comprises the following steps: Read the left image and the right image respectively, and extract AGAST features from the left image and the right image respectively; Establishing a quadtree and dividing the features into quadtree nodes, respectively screening left feature points and right feature points, wherein the quadtree includes a left quadtree and a right quadtree; Use the BEBLID algorithm to describe the filtered left feature points and right feature points respectively, and generate a binary descriptor set for the left target and a binary descriptor set for the right target respectively; Based on the binary descriptor set of the left eye and the binary descriptor set of the right eye, the optical flow algorithm is used to estimate the coordinates of the filtered left eye feature points in the right eye image; Traverse the filtered left feature points, query the right quadtree node according to the estimated coordinates to obtain the neighboring point set; The 2 nearest neighbor algorithm is used on the neighbor point set to obtain an exact matching pair of left eye feature points and right eye feature points.

2. According to claim 1, a binocular feature matching method based on quadtree and optical flow estimation is characterized in that: The steps of reading the left image and the right image respectively and extracting AGAST features from the left image and the right image respectively include: When a frame image of the left eye and a frame image of the right eye are received, the AGAST algorithm is used to extract features of the left eye and the right eye to obtain a left eye feature point set and a right eye feature point set.

3. The binocular feature matching method based on quadtree and optical flow estimation according to claim 2, characterized in that: The steps of establishing a left quadtree and a right quadtree, dividing the features into quadtree nodes, and respectively screening left feature points and right feature points include: A quadtree data structure is established for the left feature point set and the right feature point set for division. A node capacity threshold N is set for the entire quadtree to record the maximum number of feature points that all nodes can store. The subnodes in the node are designed as 4 pointer arrays, which point to the upper left, lower left, upper right or lower right sub-areas respectively. Divide the feature points into nodes, determine whether the current node is a leaf node, if it is a leaf node, determine whether the capacity of the current node is full, if it is full, split the current node and re-divide it, if it is not full, save the feature points directly, if it is not a leaf node, map it to one of the four sub-regions of upper left, lower left, upper right or lower right according to its horizontal and vertical coordinates, and divide it into its corresponding sub-node; Recursively divide all feature points into leaf nodes; Maintain a quadtree for the left and right eyes respectively. Sort the AGAST features of the left eye from high to low according to the response value and divide them into the left quadtree. Sort the AGAST features of the right eye from high to low according to the response value and divide them into the right quadtree. Each node automatically selects the M feature points with the highest local response value.

4. The binocular feature matching method based on quadtree and optical flow estimation according to claim 3, characterized in that: In the step of estimating the coordinates of the filtered left-eye feature points in the right-eye image using an optical flow algorithm based on the left-eye binary descriptor set and the right-eye binary descriptor set, the optical flow algorithm is an LK optical flow algorithm.

5. The binocular feature matching method based on quadtree and optical flow estimation according to claim 4, characterized in that: The step of estimating the coordinates of the filtered left eye feature points in the right eye image using an optical flow algorithm based on the left eye binary descriptor set and the right eye binary descriptor set comprises: Based on the binary descriptor set of the left eye and the binary descriptor set of the right eye, the LK optical flow algorithm is used to track the position of the filtered left eye feature points in the right eye image, the window size of the LK optical flow algorithm is set, the number of pyramid levels is set, the two-dimensional coordinate point data of the binary descriptor set of the left eye is passed in, and the tracked two-dimensional coordinate point data in the right eye is obtained. The tracked two-dimensional coordinates in the right eye indicate the position of the filtered left eye feature points in the right eye image.

6. The binocular feature matching method based on quadtree and optical flow estimation according to claim 5, characterized in that: The step of traversing the filtered left feature points and querying the right quadtree node according to the estimated coordinates to obtain a neighboring point set comprises: Traverse the set of feature points successfully tracked by the left eye, use the tracked right eye coordinate points to query the right quadtree, and obtain the set of neighboring points.

7. The binocular feature matching method based on quadtree and optical flow estimation according to claim 5, characterized in that: The step of using the 2 nearest neighbor algorithm on the neighbor point set to obtain an accurate matching pair of the left eye feature point and the right eye feature point comprises: Use the 2 nearest neighbor algorithm on the subset of neighbor point sets to calculate the two feature points with the closest Hamming distance of the descriptor. When the distance of the closest feature point is less than 0.7 of the distance of the second closest feature point, save this matching pair, which is the feature pair matched according to the descriptor; Traverse to get the exact matching pair of left eye features and right eye features.

8. The binocular feature matching method based on quadtree and optical flow estimation according to claim 1, characterized in that: The node type of the quadtree node stores data including starting point coordinates, width, height, leaf node flag, feature point array, feature point index array and node depth.

9. An electronic device, comprising a memory and a processor, characterized in that: The processor is used to execute the program in the memory to implement the binocular feature matching method based on quadtree and optical flow estimation as described in any one of claims 1 to 8.

10. A storage medium containing computer executable instructions, characterized in that: When the storage medium of the computer executable instructions is executed by a computer processor, it is used to execute the binocular feature matching method based on quadtree and optical flow estimation as described in any one of claims 1 to 8.

Images