Binocular vision stereo matching method based on local features
By combining SURF and CLBP descriptors and using Bayesian inference and NNDR matching strategies to optimize the disparity map, the problem of insufficient real-time and matching accuracy in binocular stereoscopic stereo matching is solved, and the generation of high-quality disparity maps is achieved.
Patent Information
- Application Number
- CN202311133219.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-04
- Publication Date
- 2025-03-04
AI Technical Summary
The existing binocular stereoscopic visual stereo matching methods have shortcomings in real-time and matching accuracy, especially in local feature point matching, it is difficult to obtain high-quality disparity maps.
The feature point extraction method combined with SURF and CLBP descriptors was adopted to obtain the initial matching points through the NNDR matching strategy, and the matching points with high confidence were screened using Bayesian inference and Soble filtering, and the disparity map was further optimized through neighborhood disparity estimation algorithm, median filtering and left-right consistency test.
The accuracy and real-time nature of feature point matching are improved, and high-quality disparity maps can be generated while maintaining the matching speed, especially the matching effect in weak texture areas and depth discontinuities is significantly improved.
Smart Images
Figure CN119559225A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a binocular vision stereo matching method based on local features, and belongs to the field of image processing. Background Art
[0002] The main process of binocular stereo matching includes a series of steps that directly affect the quality and real-time performance of 3D reconstruction and depth estimation, such as feature extraction, matching cost calculation, cost aggregation, matching point selection, and disparity calculation. Disparity optimization is to obtain a more accurate and continuous depth map through the steps of cost aggregation, disparity initialization, disparity smoothing, and disparity refinement of the initial disparity map, thereby improving the accuracy and stability of applications such as 3D reconstruction and target recognition.
[0003] At present, SIFT, SURF and ORB are commonly used. Among them, SIFT is a floating-point scale-invariant feature matching algorithm with scale invariance and descriptor robustness. This results in high computational complexity and a long matching process. Based on the SIFT algorithm, the SURF algorithm uses a faster scale space extreme point detection method, which can reduce the descriptor from 128 dimensions to 64 dimensions to improve matching efficiency. However, despite these improvements, real-time performance is still a problem, and the matching accuracy is not high. ORB is a feature point extraction algorithm with rotation invariance. It uses binary description vectors and Hamming distance to achieve fast image matching, but its distinguishability is generally weak, resulting in a decrease in the accuracy of feature point matching.
[0004] For local feature point stereo matching, only sparse matching point pairs can be obtained, and effective disparity maps cannot be directly generated. It is impossible to obtain effective image information. The currently commonly used region growing densification method is too complex to calculate and takes too long. The densified initial disparity map requires further disparity optimization to obtain a more accurate disparity map. Summary of the invention
[0005] The purpose of the present invention is to disclose a binocular vision stereo matching method based on local features, which improves the matching effect of weak texture areas and depth discontinuities by enhancing the robustness of feature points and combining with Bayesian reasoning.
[0006] In an embodiment of the present disclosure, the local feature-based binocular vision stereo matching method comprises the following steps:
[0007] Step 1: Construct SURF descriptor and CLBP descriptor, and combine SURF descriptor and CLBP descriptor into a new feature point vector;
[0008] Step 2: Match the feature point vectors obtained in step 1 according to NNDR to obtain the initial matching points;
[0009] Step 3: Filter the initial matching points in step 2 through Soble filtering, select the matching points with higher confidence, and solve the posterior probability density of the left disparity map given the initial feature point set and the right view (i.e., the observation) according to the Bayesian reasoning method, and then estimate the disparity through the MAP method to obtain the initial disparity map;
[0010] Step 4: Optimize the disparity map formed in step 3 using the neighborhood disparity estimation algorithm, median filtering, and left-right consistency check
[0011] In a further technical solution, step 1 specifically comprises the following steps:
[0012] Step 1.1: Decompose the image block into multiple Haar wavelet response maps, and obtain the local feature information of each wavelet response map. Then merge these response maps to generate a SURF descriptor. Then calculate the size and direction of the feature point through the Hessian matrix in the SURF algorithm. The formula of the SURF descriptor is as follows:
[0013]
[0014] Among them, L xx (x0, s) is the Gaussian second-order differential Convolution with image M at point x0. Other parameters have the same meaning as L xx (x0, s) is similar.
[0015] Step 1.2: The CLBP descriptor uses the statistical information of local binary patterns to describe the texture structure of the image and generates a binary code. By comparing each pixel of the image with its surrounding neighboring pixels, the ability to capture image texture features can be enhanced, thereby improving the robustness and expressiveness of image features. The SURF feature point K obtained in 1.1 is taken as the center of the circle, and P pixels are sampled at equal intervals on the circumference of the circle with a radius of R. The grayscale values of the P sampled points are compared with the grayscale value of the center point K, and the encoding is performed based on the comparison result. The formula of the CLBP descriptor is as follows:
[0016]
[0017] The SURF descriptor is the center point, R is the radius, and the gray value of the center point is g. c , the grayscale values of the P pixels on its neighborhood circle are g p , s is the sign function, which is used to binarize the difference result of pixel values.
[0018] Step 1.3: Combine the SURF feature points and the CLBP descriptor, and use the new feature vector to describe the feature points. The formula for combining the SURF descriptor and the CLBP descriptor is as follows:
[0019] D{SURF-CLBP}(x,y,sa)=[CLBP(x,y),G(x,y,sa)]
[0020] Among them, (x, y) represents the position of pixel P, sa represents the standard deviation of the Gaussian kernel, and G(x, y, sa) represents the gradient histogram descriptor obtained after the image at (x, y) is filtered by Gaussian at scale sa.
[0021] A further technical solution is that step 2 evaluates the reliability of the nearest neighbor point by comparing the distance between the nearest neighbor point and the next nearest neighbor point, which takes into account the distance relationship between the nearest neighbor point and the next nearest neighbor point. This improves the accuracy of feature point matching. Step 2 matches the feature point vector obtained in step 1 according to NNDR to obtain the initial matching point. The NNDR formula is as follows:
[0022]
[0023] Among them, NN(P) represents the nearest neighbor distance of P, and NND(P) represents the next nearest neighbor distance of P.
[0024] In a further technical solution, step 3 specifically comprises the following steps:
[0025] Step 3.1: Use Sobel filtering to filter out the accurate sparse points after feature matching and select the matching points with higher confidence as the initial dense points. Take the left picture as the reference picture. The disparity in the picture is the random variable to be solved. The probability generation model formula is as follows:
[0026]
[0027] Where S = (s1, s2, ..., s m ) is the initial feature point set, each initial point is s m =(u m , v m , d m ), where d m is m , v m ) points, and the left image is used as the reference image, O = (o1, o2, ..., o n ) represents the set of observation points, each observation point is represented by o n =(u n , v n , f n ),make and Indicates the corresponding pixels in the left and right images.
[0028] Step 3.2: According to the probability generation model, the random disparity value of the left image pixel is obtained by the given initial point set of step 3.1 and the pixel information of the left image. Then, the pixel information of the right image can be obtained according to the disparity value and the pixel information of the left image. The posterior probability density of the left disparity image is solved by Bayesian reasoning given the initial point set and the right view - that is, the observation quantity. Then, the disparity formula is estimated by the MAP method as follows:
[0029]
[0030] In a further technical solution, step 4 specifically comprises the following steps:
[0031] Step 4.1: The neighborhood disparity estimation algorithm searches for pixels in the four neighborhood directions and finds the nearest pixel with a non-zero value to fill the occluded and hole areas. Specifically, the algorithm records the minimum value in such pixels and uses it as the pixel value of the central occluded point.
[0032] Step 4.2: Left-right consistency check is to find possible errors and inconsistencies by comparing the differences between the left and right disparity maps. Based on these findings, the quality and accuracy of the disparity map can be further improved by correcting inconsistent pixels.
[0033] Step 4.3: The processed disparity map is subjected to connected domain noise removal, median filtering and adaptive median filtering techniques to further reduce noise.
[0034] The present invention provides a local feature based binocular vision stereo matching method, which has the following beneficial effects:
[0035] 1) The present invention is a local feature point stereo matching algorithm for a surface. The box filter and integral image of the SURF algorithm are first used to achieve fast feature point extraction and descriptor calculation to ensure matching speed. The CLBP descriptor is introduced as a method for extracting texture features. The CLBP descriptor has rotation invariance and scale invariance, and can extract similar texture features at different positions and scales of the image. Its characteristics enable the descriptor to enhance the robustness of feature points.
[0036] 2) The present invention adopts the NNDR matching strategy to match the constructed feature point vectors to obtain the initial matching points. NNDR is used to evaluate the similarity between feature points and to determine whether to match by comparing the distance between the nearest neighbor and the next nearest neighbor. Since the construction and matching process of the feature point vector is relatively efficient, this method has a high matching accuracy while maintaining the matching speed.
[0037] 3) The present invention is a method for a local stereo matching algorithm to output a valid disparity map. Compared with the traditional local stereo matching algorithm which can only obtain sparse matching point pairs, the present invention introduces probability density estimation to construct a probability generation model for stereo matching. This method can perform effective depth estimation between different views, improve the accuracy of the depth map, and thus perform dense stereo matching and generate a valid disparity map.
[0038] 4) The present invention comprehensively considers the quality of the disparity map, especially the weak texture area of the image and the local area of depth discontinuity. In order to obtain a more accurate disparity map, the neighborhood disparity estimation algorithm, median filtering and left-right consistency check are used to optimize the initial disparity map. Neighborhood disparity estimation and median filtering help smooth the disparity map and reduce the impact of noise. The left-right consistency check is used to detect inconsistent areas in the disparity map, further improving the stability and accuracy of the disparity map. A more comprehensive and accurate solution is provided for the quality of the disparity map. Description of the drawings:
[0039] Figure 1 It is a flow chart of a binocular vision stereo matching method based on local features involved in the present invention; DETAILED DESCRIPTION
[0040] The present invention proposes a binocular vision stereo matching method based on local features, which uses local feature enhancement to achieve image optimization disparity map results. In order to more clearly explain the purpose, technical solution and advantages of the present invention, the technical solution of the present invention will be described in detail with reference to the accompanying drawings. It should be pointed out that the specific embodiments described below are only used to explain the principles of the present invention and are not used to limit the scope of implementation of the present invention.
[0041] The present invention first provides a binocular vision stereo matching method based on local features, specifically referring to Figure 1 , including the following steps:
[0042] Step 1: Construct SURF descriptor and CLBP descriptor, and combine SURF descriptor and CLBP descriptor into a new feature point vector;
[0043] Step 2: Match the feature point vectors obtained in step 1 according to NNDR to obtain the initial matching points;
[0044] Step 3: Filter the initial matching points in step 2 through Soble filtering, select the matching points with higher confidence, and solve the posterior probability density of the left disparity map given the initial feature point set and the right view (i.e., the observation) according to the Bayesian reasoning method, and then estimate the disparity through the MAP method to obtain the initial disparity map;
[0045] Step 4: Optimize the disparity map formed in step 3 using the neighborhood disparity estimation algorithm, median filtering, and left-right consistency check
[0046] In a further technical solution, step 1 specifically comprises the following steps:
[0047] Step 1.1: Decompose the image block into multiple Haar wavelet response maps, and obtain the local feature information of each wavelet response map. Then merge these response maps to generate a SURF descriptor. Then calculate the size and direction of the feature point through the Hessian matrix in the SURF algorithm. The formula of the SURF descriptor is as follows:
[0048]
[0049] Among them, L xx (x0, s) is the Gaussian second-order differential Convolution with image M at point x0. Other parameters have the same meaning as L xx (x0, s) is similar.
[0050] Step 1.2: The CLBP descriptor uses the statistical information of local binary patterns to describe the texture structure of the image and generates a binary code. By comparing each pixel of the image with its surrounding neighboring pixels, the ability to capture image texture features can be enhanced, thereby improving the robustness and expressiveness of image features. The SURF feature point K obtained in 1.1 is taken as the center of the circle, and P pixels are sampled at equal intervals on the circumference of the circle with a radius of R. The grayscale values of the P sampled points are compared with the grayscale value of the center point K, and the encoding is performed based on the comparison result. The formula of the CLBP descriptor is as follows:
[0051]
[0052] The SURF descriptor is the center point, R is the radius, and the gray value of the center point is g. c , the grayscale values of the P pixels on its neighborhood circle are g p , s is the sign function, which is used to binarize the difference result of pixel values.
[0053] Step 1.3: Combine the SURF feature points and the CLBP descriptor, and use the new feature vector to describe the feature points. The formula for combining the SURF descriptor and the CLBP descriptor is as follows:
[0054] D{SURF-CLBP}(x,y,sa)=[CLBP(x,y),G(x,y,sa)]
[0055] Among them, (x, y) represents the position of pixel P, sa represents the standard deviation of the Gaussian kernel, and G(x, y, sa) represents the gradient histogram descriptor obtained after the image at (x, y) is filtered by Gaussian at scale sa.
[0056] A further technical solution is that step 2 evaluates the reliability of the nearest neighbor point by comparing the distance between the nearest neighbor point and the next nearest neighbor point, which takes into account the distance relationship between the nearest neighbor point and the next nearest neighbor point. This improves the accuracy of feature point matching. Step 2 matches the feature point vector obtained in step 1 according to NNDR to obtain the initial matching point. The NNDR formula is as follows:
[0057]
[0058] Among them, NN(P) represents the nearest neighbor distance of P, and NND(P) represents the next nearest neighbor distance of P.
[0059] In a further technical solution, step 3 specifically comprises the following steps:
[0060] Step 3.1: Use Sobel filtering to filter out the accurate sparse points after feature matching and select the matching points with higher confidence as the initial dense points. Take the left picture as the reference picture. The disparity in the picture is the random variable to be solved. The probability generation model formula is as follows:
[0061]
[0062] Where S = (s1, s2, ..., s m ) is the initial feature point set, each initial point is s m =(u m , v m , d m ), where d m is m , v m ) points, and the left image is used as the reference image, O = (o1, o2, ..., o n ) represents the set of observation points, each observation point is represented by o n =(u n , v n , f n ),make and Indicates the corresponding pixels in the left and right images.
[0063] Step 3.2: According to the probability generation model, the random disparity value of the left image pixel is obtained by the given initial point set of step 3.1 and the pixel information of the left image. Then, the pixel information of the right image can be obtained according to the disparity value and the pixel information of the left image. The posterior probability density of the left disparity image is solved by Bayesian reasoning given the initial point set and the right view - that is, the observation quantity. Then, the disparity formula is estimated by the MAP method as follows:
[0064]
[0065] In a further technical solution, step 4 specifically comprises the following steps:
[0066] Step 4.1: The neighborhood disparity estimation algorithm searches for pixels in the four neighborhood directions and finds the nearest pixel with a non-zero value to fill the occluded and hole areas. Specifically, the algorithm records the minimum value in such pixels and uses it as the pixel value of the central occluded point.
[0067] Step 4.2: Left-right consistency check is to find possible errors and inconsistencies by comparing the differences between the left and right disparity maps. Based on these findings, the quality and accuracy of the disparity map can be further improved by correcting inconsistent pixels.
[0068] Step 4.3: The processed disparity map is subjected to connected domain noise removal, median filtering and adaptive median filtering techniques to further reduce noise.
[0069] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.
[0070] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.
Claims
1. A binocular vision stereo matching method based on local features, the method comprising the following steps: Step 1: Construct SURF descriptor and CLBP descriptor, and combine SURF descriptor and CLBP descriptor into a new feature point vector; Step 2: Match the feature point vectors obtained in step 1 according to NNDR to obtain the initial matching points; Step 3: Filter the initial matching points in step 2 through Soble filtering, select the matching points with higher confidence, and solve the posterior probability density of the left disparity map given the initial feature point set and the right view (i.e., the observation) according to the Bayesian reasoning method, and then estimate the disparity through the MAP method to obtain the initial disparity map; Step 4: Optimize the initial disparity map obtained in step 3 using the neighborhood disparity estimation algorithm, median filtering, and left-right consistency check.
2. The local feature based binocular vision stereo matching method according to claim 1, characterized in that: The step 1 specifically comprises the following steps: Step 1.1: Decompose the image block into multiple Haar wavelet response maps, and obtain the local feature information of each wavelet response map. Then merge these response maps to generate a SURF descriptor. Then calculate the size and direction of the feature point through the Hessian matrix in the SURF algorithm. The formula of the SURF descriptor is as follows: Among them, L xx (x0, s) is the Gaussian second-order differential Convolution with image M at point x0. Other parameters have the same meaning as L xx (x0, s) is similar. Step 1.2: The CLBP descriptor uses the statistical information of local binary patterns to describe the texture structure of the image and generates a binary code. By comparing each pixel of the image with its surrounding neighboring pixels, the ability to capture image texture features can be enhanced, thereby improving the robustness and expressiveness of image features. The SURF feature point K obtained in 1.1 is taken as the center of the circle, and P pixels are sampled at equal intervals on the circumference of the circle with a radius of R. The grayscale values of the P sampled points are compared with the grayscale value of the center point K, and the encoding is performed based on the comparison result. The formula of the CLBP descriptor is as follows: The SURF descriptor is the center point, R is the radius, and the gray value of the center point is g c , the grayscale values of the P pixels on its neighborhood circle are g p , s is the sign function, which is used to binarize the difference result of pixel values. Step 1.3: Combine the SURF feature points and the CLBP descriptor and use the new feature vector to describe the feature points. The formula for combining the SURF descriptor and the CLBP descriptor is as follows: D{SURF-CLBP}(x,y,sa)=[CLBP(x,y),G(x,y,sa)] Among them, (x, y) represents the position of pixel P, sa represents the standard deviation of the Gaussian kernel, and G(x, y, sa) represents the gradient histogram descriptor obtained after the image at (x, y) is filtered by Gaussian at scale sa.
3. The local feature based binocular vision stereo matching method according to claim 1, characterized in that: The reliability of the nearest neighbor point is evaluated by comparing the distance between the nearest neighbor point and the next nearest neighbor point, which takes into account the distance relationship between the nearest neighbor point and the next nearest neighbor point. This improves the accuracy of feature point matching. The process of matching the feature point vector according to NNDR in step 2 to obtain the initial matching point is the matching of the initial feature point. The NNDR formula is as follows: Among them, NN(P) represents the nearest neighbor distance of P, and NND(P) represents the next nearest neighbor distance of P.
4. The local feature based binocular vision stereo matching method according to claim 1, characterized in that: The step 3 specifically comprises the following steps: Step 3.1: Use Sobel filtering to filter out the accurate sparse points after feature matching and select the matching points with higher confidence as the initial dense points. Take the left picture as the reference picture. The parallax is d n The probability generation model formula of the random variable to be solved is as follows: Where S = (s1, s2, ..., s m ) is the initial feature point set, each initial point is s m =(u m , v m , d m ), where d m is m , v m ) points, and the left image is used as the reference image, O = (o1, o2..., o n ) represents the set of observation points, each observation point is represented by o n =(u n , v n , f n ),make and Indicates the corresponding pixels in the left and right images. Step 3.2: According to the probability generation model, the random disparity value of the left image pixel is obtained by the given initial point set of step 3.1 and the pixel information of the left image. Then, the pixel information of the right image can be obtained based on the disparity value and the pixel information of the left image. The posterior probability density of the left disparity image is solved by Bayesian reasoning given the initial point set and the right view - that is, the observation quantity. Then, the disparity formula is estimated by the MAP method as follows:
5. The local feature based binocular vision stereo matching method according to claim 1, characterized in that: The step 4 specifically comprises the following steps: Step 4.1: The neighborhood disparity estimation algorithm searches for pixels in the four neighborhood directions and finds the nearest pixel with a non-zero value to fill the occluded and hole areas. Specifically, the algorithm records the minimum value in such pixels and uses it as the pixel value of the central occluded point. Step 4.2: Left-right consistency check is to find possible errors and inconsistencies by comparing the differences between the left and right disparity maps. Based on these findings, the quality and accuracy of the disparity map can be further improved by correcting inconsistent pixels. Step 4.3: The processed disparity map is subjected to connected domain noise removal, median filtering and adaptive median filtering techniques to further reduce noise.