Local stereo matching method based on fusion cost computation and weighted guided filtering

Abstract

The application discloses a local stereo matching method based on fusion cost calculation function and weighted guided filtering, and comprises the following steps: S1, a reference image and a matching image are obtained; S2, improved three-state Census transformation cost, enhanced gradient cost and AD-Census cost of each pixel in the reference image and the matching image are calculated, and the cost values are fused by adopting an exponential fusion algorithm; S3, an aggregation region is constructed, then cost aggregation is carried out on the aggregation region, and a cost space after aggregation is generated; S4, a winner-takes-all algorithm is adopted in the cost space after aggregation, and an initial disparity map is obtained; S5, different interpolation strategies are adopted to fill disparity for different types of matching error points, and finally, sub-pixel refinement and median filtering are carried out on the disparity map after disparity filling, so that a final disparity map is obtained. Compared with the prior art, the application overcomes the shortcomings of single cost, can generate a better disparity map, and the precision is improved by 4.9% compared with the traditional Census transformation.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a local stereo matching algorithm for binocular stereo vision. Background Technology

[0002] Binocular stereo vision-based 3D reconstruction technology boasts advantages such as high accuracy, simple equipment, and strong real-time performance, and is widely used in fields such as virtual reality, industrial measurement, and robot navigation. In recent years, it has become a research hotspot in the field of machine vision. Stereo matching algorithms, as the core technology in binocular stereo vision, play a crucial role in the effectiveness of 3D reconstruction due to their matching accuracy.

[0003] The purpose of stereo matching algorithms is to find matching point pairs between a reference image and a matching image, calculate disparity, and generate a disparity map. These algorithms are categorized into global stereo matching algorithms, semi-global stereo matching algorithms, and local stereo matching algorithms. Local stereo matching algorithms, also known as sliding window-based matching algorithms, offer advantages such as low computational cost, high accuracy, and ease of hardware implementation, making them a current research hotspot in matching algorithms. Existing technologies have optimized stereo matching algorithms into four steps: cost calculation, cost aggregation, disparity calculation, and disparity optimization.

[0004] Currently, the most commonly used similarity metric function in cost calculation is the Census transform, which is robust to noise but does not fully utilize the pixel value itself and the cross-correlation between pixels, and is highly dependent on the center pixel. Summary of the Invention

[0005] This invention aims to propose a local stereo matching method based on fusion cost calculation and weighted guided filtering. It obtains the final disparity map by constructing a fusion cost as the overall cost calculation function and applying weighted guided filtering to the disparity map.

[0006] This invention is achieved using the following technical solution:

[0007] A local stereo matching method based on a fusion cost calculation function and weighted guided filtering, the method comprising the following steps:

[0008] Step S1: Use two cameras to build a binocular stereo vision system and obtain the color images captured by the left and right cameras as the reference image and the matching image, respectively;

[0009] Step S2: Calculate the improved three-state Census transform cost, enhanced gradient cost, and AD-Census cost for each pixel in the reference image and the matching image, and use the exponential fusion algorithm to fuse the above three costs according to their weights.

[0010] Step S3: Construct an aggregation region for each target pixel based on the cross-shaped adaptive generation domain method, and then perform cost aggregation on the aggregation region to generate the aggregated cost space;

[0011] Step S4: In the aggregated cost space, the winner-takes-all algorithm is used to initially determine the minimum pixel cost. The disparity value corresponding to each minimum pixel cost is the optimal disparity, thus obtaining the initial disparity map.

[0012] Step S5: Classify the mismatched points in the initial disparity map using the left-right consistency principle. Then, use different interpolation strategies to fill the disparity for different types of mismatched points. Finally, perform sub-pixel refinement and median filtering on the disparity map after disparity filling to obtain the final disparity map.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1) It overcomes the shortcomings of traditional Census transform that over-relies on the center pixel, enhances the utilization of edge information, and overcomes the inadequacy of single cost, thus generating better disparity maps.

[0015] 2) Tests were conducted on the Middleburry datasets Teddy, Tsukuba, Venus, and Cones. The results showed good robustness against various images and improved accuracy by 4.9% compared to the traditional Census transform. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of a local stereo matching method based on a fusion cost calculation function and weighted guided filtering according to the present invention.

[0017] Figure 2 This is a diagram illustrating an example of a binocular stereo vision system.

[0018] Figure 3 This is a disparity comparison diagram of the improved three-state Census transform and the traditional Census transform in the weakly textured region of the Middleburry dataset Motorcycle.

[0019] Figure 4 This is the true disparity map in the standard dataset described in the preferred embodiment of the present invention.

[0020] Figure 5 The disparity map is obtained by the algorithm in this example case, which is a preferred embodiment of the present invention.

[0021] Figure 6 This is a comparison chart of algorithm errors in a preferred embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of 8 directions. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] This invention discloses a local stereo matching method based on a fused cost calculation function and weighted guided filtering. In cost calculation, it proposes an improved three-state Census transform and an enhanced gradient cost calculation method, fusing AD costs to form the cost calculation function of this algorithm. In cost aggregation, it proposes an improved weighted guided filtering algorithm. Then, a winner-takes-all algorithm is used to select the disparity corresponding to the minimum cost value as the optimal disparity value, and the initial disparity map is refined in multiple steps to further improve the matching accuracy of the algorithm. The method specifically includes the following steps:

[0025] Step S1: Use two cameras to build a binocular stereo vision system and obtain the color images captured by the left and right cameras as the reference image and the matching image, respectively;

[0026] Step S2: Calculate the improved three-state Census transform cost, enhanced gradient cost (x-direction and y-direction), and AD cost for each pixel in the reference image and the matching image. Use the exponential fusion algorithm to fuse the above cost values ​​according to the weights to generate a three-dimensional initial cost space with the image width as the x-axis, the image height as the y-axis, and the disparity range as the z-axis.

[0027] Step S3: Construct an aggregation region for each target pixel based on the cross-shaped adaptive generation domain method, and then perform cost aggregation on the aggregation region to generate the aggregated cost space;

[0028] Step S4: In the aggregated cost space, the winner-takes-all algorithm is used to initially determine the minimum pixel cost. The disparity value corresponding to each minimum pixel cost is the optimal disparity, thus obtaining the initial disparity map.

[0029] Step S5: Classify the mismatched points in the initial disparity map using the left-right consistency principle. Then, use different interpolation strategies to fill the disparity for different types of mismatched points. Finally, perform sub-pixel refinement and median filtering on the disparity map after disparity filling to obtain the final disparity map.

[0030] Furthermore, step S2 specifically includes the following steps:

[0031] Step S21: Convert the reference image and the matching image into grayscale images. Then, establish a 9x7 neighborhood window centered on the target pixel p on the grayscale image of the reference image. Use this neighborhood window as the Census transform window. Perform an improved three-state Census transform within the Census transform window. The improved three-state Census transform process is defined as follows:

[0032]

[0033] In the formula, Let I(q) be the average pixel value of the target pixel p, I(q) be the pixel value of the neighboring pixel q, ξ[·] be the 01 sequence, which is the basis for constructing the improved three-state Census transform code, and α be the local texture contrast threshold. The expression for α is as follows:

[0034]

[0035] In the formula, sum1 is the sum of gray values ​​of all pixels in the Census transform window that are greater than the mean pixel value, sum2 is the sum of gray values ​​of all pixels in the Census transform window that are less than the mean pixel value, n1 and n2 are the number of all pixels in the Census transform window that are greater than the mean pixel value and the number of all pixels in the Census transform window that are less than the mean pixel value, respectively, and β is the normalization parameter, which is set to 32.

[0036] By implementing the Census transform code concatenation of the Census transform window, the improved three-state Census transform code is obtained as shown in the following equation:

[0037]

[0038] In the formula, W(p) is the neighborhood window centered on the target pixel p, I(p) and I(q) are the gray values ​​of pixels p and q, respectively, and c(p) is the Census transform code of the target pixel p.

[0039] Suppose that the target pixel p in the reference image has a disparity d, and that pixel p in the matching image... d Forming a matching point pair;

[0040] The improved three-state Census cost is obtained using Hamming distance, as shown in the following equation:

[0041] C Census (p, d) = Hamming[c1(p), c r (p d )]

[0042] Step S22: Perform contrast-limited adaptive histogram equalization on the grayscale images of the reference image and the matching image, and then calculate the enhancement gradient costs in the x and y directions on the processed image, as shown in the following formula:

[0043]

[0044] In the formula, I′ L and I′ R These are the pixel values ​​of the reference image and the matched image after contrast-limited adaptive histogram equalization, respectively. and These are the gradients of the target pixel p in the x and y directions, respectively;

[0045] Step S23: Calculate the AD-Census cost for each pixel in the reference image and the matching image, as shown in the following formula:

[0046]

[0047] In the formula, d represents the parallax. This represents the pixel value in the i-th channel of the target pixel p in the reference image;

[0048] Step S24: Combine the above three costs to construct the overall cost calculation function:

[0049] C(p, d) = ρ(C Census (p, d), γ Census )+ρ(C Grad (p, d), γ Grad )+ρ(C AD (p, d), γ AD )

[0050] In the formula, C Census (p, d), C Grad (p, d) and C AD (p, d) represent the costs of the improved three-state Census transformation, the enhanced gradient cost (x and y directions), and the AD-Census cost, respectively. γ is an adjustable parameter; adjusting the value of γ can adjust the proportion of the three costs, avoiding excessive dependence of the final generation value on any one cost. C is a variable representing the three matching costs substituted into the above equation. ρ(c, γ) is the control function with respect to c, as shown in the following equation:

[0051]

[0052] Furthermore, step 3 specifically includes the following steps:

[0053] Step S31: Construct the skeleton of the adaptive generation domain: First, construct a cross with the target pixel p as the perpendicular point as the initial skeleton of the adaptive region. The cross consists of a horizontal arm H(p) and a vertical arm V(p), with an arm length of {h}. p + h p - v p + v p -}. (The length of the left horizontal arm) For example, the arm length extension criterion is as follows:

[0054]

[0055] In the formula, D s (p, q) = |pq| represents the distance difference, and τ and L are the preset color threshold and distance threshold, respectively. The color difference between the target pixel p and the pixel q in the aggregation region is defined, and growth stops when the arm length does not meet the above arm length expansion criteria.

[0056] Step S32: Construct an adaptive generation domain based on the initial skeleton, and use the arm length {h} of the target pixel point p obtained in S31. p + h p - v p + v p -}, thus obtaining the orthogonal cross-shaped regions H(p) and V(p), as shown in the following equation:

[0057]

[0058] Finally, along the vertical direction, a horizontal support region H(p) is constructed for the pixels q within the aggregation region of V(p) according to the arm length expansion criterion in step S31. The adaptive region for any target pixel p in the image is obtained by combining all horizontal support regions H(p), as shown in the following equation:

[0059]

[0060] Step S33: Obtain the aggregated cost value, as shown in the following formula:

[0061]

[0062] In the formula, N is the number of pixels in the adaptive region U(p).

[0063] Furthermore, step 5 specifically includes the following steps:

[0064] Step S51: Perform left-right consistency detection on the initial disparity map obtained in step S4 according to the following formula, where the left-right consistency criterion is as follows:

[0065] |d L (x, y)-d R [(xd L (x, y)), y]|<1

[0066] In the formula, d L (x, y) represents the disparity value of the target pixel coordinates p(x, y) in the reference image, and d R (xd L (x, y), y) represents the matching point (xd) of the target pixel point coordinate p(x, y) under disparity d. L The disparity values ​​of (x, y), y);

[0067] If the pixels in the initial disparity map satisfy the left-right consistency criterion, they are considered correctly matched points; otherwise, they are considered mismatched points. Further, based on the limit constraint criterion, mismatched points are categorized into incorrectly matched points and occluded points.

[0068] Furthermore, according to the limit constraint criterion, matching points are divided into mismatched points and occluded points; the epipolar constraint criterion specifies that matching point pairs should be on the same horizontal line. That is, if a matching point pair can be found on the same horizontal line, then the point is an mismatched point, otherwise it is an occluded point.

[0069] Step S52: Interpolate the mismatched points and occluded points using different interpolation strategies: ① Use a region iterative interpolation strategy to interpolate the occluded points: Within the cross-shaped window of the occluded points, perform histogram voting on points with reasonable disparity (points with reasonable disparity are the correctly matched points), with a total of N votes. sum The disparity value that receives the most votes is denoted as d. H The number of votes is denoted as N. vote (d H If equation N is satisfied sum >τ N and Given two conditions, the disparity value of the occluded point is updated to d. H , τ N and τ R This is the threshold, a custom value, taken as 50 and 0.5 respectively. Five iterations of region interpolation are performed following the above process, updating the disparity values ​​of all occluded points, and visual filling is completed for mismatched points.

[0070] ② For mismatched points, select the disparity value from the reasonable points in the 8 directions that is closest in color to the target pixel p in the original reference image to update the original disparity value, and complete the visual filling for the mismatched points.

[0071] The reference image is the left image, and the matching image is the right image.

[0072] Step S53: Perform sub-pixel refinement on the disparity map processed in step S52, and perform median filtering based on a 3×3 filter kernel to obtain the final disparity map.

[0073] like Figure 2 The diagram shows an embodiment of the constructed binocular stereo vision system. This system includes a left camera 1, a right camera 2, a mounting plate 3, and a screw structure 4 for fixing the cameras and the mounting plate. The monocular camera is an S50PT1080 model, using a USB interface, supporting the UVC communication protocol, and featuring a built-in CMOS (OV2710) image sensor chip. It has a field of view of 110 degrees and a resolution of 1920x1080. Other key parameters of the camera are shown in Table 1.

[0074] Table 1

[0075]

[0076]

[0077] like Figure 3 (a) is the original reference image of "Motorcycle". (b) and (c) are disparity maps generated by the traditional Census transform and the improved three-state Census transform, respectively, without noise. A comparison of (b) and (c) shows that, without noise, the improved three-state Census slightly improves the matching effect of the traditional Census. Adding Gaussian noise to the original image, the disparity maps generated by the two Census transforms are (d) and (e), respectively. As shown in the figure, the improvement effect of the improved three-state Census transform is very significant, and the generated disparity map is not much different from the disparity map without noise, indicating that the improved three-state Census proposed in this invention has good anti-interference ability against noise.

[0078] The above description is merely a preferred embodiment to illustrate the technical ideas and features of the present invention. Its purpose is to enable those skilled in the art to understand the invention and implement it accordingly. It is not intended to limit the present invention. Modifications, substitutions, equivalent changes, etc., made within the spirit and principles of the present invention should all be within the protection scope of the present invention.

Claims

1. A local stereo matching method based on a fusion cost calculation function and weighted guided filtering, characterized in that, The method includes the following steps: Step S1: Use two cameras to build a binocular stereo vision system and obtain the color images captured by the left and right cameras as the reference image and the matching image, respectively; Step S2: Calculate the improved three-state Census transform cost, enhanced gradient cost, and AD cost for each pixel in the reference image and the matching image, and fuse these three costs according to their weights using an exponential fusion algorithm; specifically, this includes the following steps: Step S21: Convert the reference image and the matching image to grayscale images, and then use the target pixels on the grayscale image of the reference image. A 9x7 neighborhood window is established around the center, and this neighborhood window is used as the Census transformation window. An improved three-state Census transformation is performed within the Census transformation window, and the improved three-state Census transformation process is defined as follows: ； In the formula, For target pixel The average pixel value, For pixels in the neighborhood pixel values, The 0-1 sequence forms the basis for the improved three-state census transform code. This is the local texture contrast threshold. The expression is as follows: ； In the formula, This is the sum of the gray values ​​of all pixels in the Census transform window that are greater than the mean pixel value. This is the sum of the gray values ​​of all pixels in the Census transform window that are less than the mean pixel value. and These represent the number of all pixels in the Census transform window that are greater than the average pixel value and the number of all pixels in the Census transform window that are less than the average pixel value, respectively. As a normalization parameter, set it to 32; By implementing the Census transform code concatenation of the Census transform window, the improved three-state Census transform code is obtained as shown in the following equation: ； In the formula, To target pixels The center's neighborhood window, and Pixels and pixels grayscale value, For target pixel Census transform code; Let the target pixel in the reference image be... In parallax At that time, with pixels in the matching image Forming a matching point pair; The improved three-state Census cost is obtained using Hamming distance, as shown in the following equation: ； Step S22: Perform contrast-limited adaptive histogram equalization on the grayscale images of the reference image and the matching image, and then calculate the enhancement gradient costs in the x and y directions on the processed image, as shown in the following formula: ； In the formula, and These are the pixel values ​​of the reference image and the matched image after contrast-limited adaptive histogram equalization, respectively. and Target pixels Gradients in the x and y directions; Step S23: Calculate the AD cost for each pixel in the reference image and the matching image, as shown in the following formula: ； In the formula, For parallax, Represents the target pixel in the reference image No. Pixel values ​​in the channel; Step S24: Combine the above three costs to construct the overall cost calculation function: ； In the formula, , and These represent the costs of the improved three-state Census transform, the enhanced gradient, and the AD transformation, respectively, where γ is an adjustable parameter and C is a variable. For about The control function is shown in the following equation: ； Step S3: Construct an aggregation region for each target pixel based on the cross-shaped adaptive generation domain method, and then perform cost aggregation on the aggregation region to generate the aggregated cost space; Step S4: In the aggregated cost space, the winner-takes-all algorithm is used to initially determine the minimum pixel cost. The disparity value corresponding to each minimum pixel cost is the optimal disparity, thus obtaining the initial disparity map. Step S5: Classify the mismatched points in the initial disparity map using the left-right consistency principle. Then, use different interpolation strategies to fill the disparity for different types of mismatched points. Finally, perform sub-pixel refinement and median filtering on the disparity map after disparity filling to obtain the final disparity map.

2. The local stereo matching method based on a fusion cost calculation function and weighted guided filtering as described in claim 1, characterized in that, Furthermore, step S3 specifically includes the following steps: Step S31: Construct the skeleton of the adaptive generation domain: First, construct the skeleton of the target pixel. The cross, with its perpendicular point, serves as the initial skeleton of the adaptive region. The cross is composed of horizontal arms. and vertical arm Composition, arm length is Growth stops when the arm length does not meet the above arm length expansion criteria. Step S32: Construct an adaptive generation domain based on the initial skeleton, and utilize the target pixel points obtained in S31. arm length This leads to the orthogonal cross-shaped regions. and As shown in the following formula: ； Finally, along the vertical direction Pixels in the aggregation area Construct the horizontal support region according to the arm length extension criterion in step S31. Unite all horizontal support areas Get any target pixel in the image The adaptive region is shown in the following equation: ； Step S33: Obtain the aggregated cost value, as shown in the following formula: ； In the formula, For adaptive regions The number of pixels within.

3. The local stereo matching method based on a fusion cost calculation function and weighted guided filtering as described in claim 1, characterized in that, Furthermore, step S5 specifically includes the following steps: Step S51: Perform left-right consistency detection on the initial disparity map obtained in step S4 according to the following formula, where the left-right consistency criterion is as follows: ； In the formula, The coordinates of the target pixel in the reference image The disparity value, Coordinates of the target pixel In parallax matching point under disparity value; If the pixels in the initial disparity map satisfy the left-right consistency criterion, they are considered correctly matched; otherwise, they are considered mismatched. Furthermore, according to the limit constraint criterion, the matching points are divided into mismatched points and occluded points. If a matching point pair can be found on the same horizontal line, the point is an mismatched point; otherwise, it is an occluded point. Step S52: Interpolate mismatched points and occluded points using different interpolation strategies: ① Use a region iterative interpolation strategy for occluded points: Perform histogram voting on reasonable disparity points within the cross-shaped window of the occluded points, with a total number of votes. The disparity value that receives the most votes is denoted as The number of votes is recorded as If the equation is satisfied and Given two conditions, the disparity value of the occluded point is updated as follows: , and The threshold is used; five iterations of region interpolation are performed according to the above process to update the disparity values ​​of all occluded points, and visual filling is completed for points with mismatches. ② For incorrect matching points, select the target pixel from the reasonable points in 8 directions. The original disparity value is updated using the disparity value that is closest in color to the original reference image, and visual filling is performed for points that are mismatched. Step S53: Perform sub-pixel refinement on the disparity map processed in step S52, and perform median filtering based on a 3×3 filter kernel to obtain the final disparity map.

Images