Stereo Matching Method Based on Adaptive Dynamic Cost Calculation and Aggregation for Texture Regions

Abstract

The present invention relates to a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation. Specifically: (1) Input binocular images; (2) Cost calculation: The image is divided into strong, weak, and textureless regions. In the strong texture region, cost calculation combines AD and Census, and in the weak and textureless regions, cost calculation combines AD and gradient; (3) Cost aggregation: During the construction of the cross arm, the constraint condition of the difference in pixel values is weakened, and the constraint condition of the difference in gradient magnitudes is increased, making the construction arm length longer in the weak and textureless regions of the image; (4) Use the WTA strategy to select the minimum cost value and calculate the disparity; (5) Post-processing for disparity optimization; (6) Output the final disparity map. The method of the present invention has good robustness, can better adapt to different scenarios, and can also be applied in the development trends of different fields such as 3D map reconstruction, target detection, driverless, and virtual reality, with broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of binocular stereo vision, and particularly relates to a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation. Background Art

[0002] In recent years, with the wide application of binocular stereo vision in fields such as 3D map reconstruction, object detection, unmanned driving, and virtual reality, binocular stereo vision has become a research hotspot in the field of computer vision. Stereo matching is to find pixel matching points of the same scene in two or more images taken from different perspectives at the same time point, then calculate the matching cost for each pixel point, find the optimal matching pixel point, calculate the disparity, and further obtain the three-dimensional depth information of the real space.

[0003] Traditional binocular stereo vision adopts a cost calculation method that combines a single or several cost functions, and also adopts a unified support domain calculation constraint condition in the cost aggregation stage. This will make its robustness poor and unable to adapt to different occasions, such as different lighting, different exposures, weak texture regions, and textureless regions. Therefore, this will greatly limit the application occasions of traditional methods, and at the same time lead to inaccurate extracted disparity values, which cannot be used in fields with high-precision requirements. Summary of the Invention

[0004] The purpose of the present invention is to provide a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation, which realizes the adaptation to different occasions and dynamically adjusts the texture region threshold.

[0005] The technical solution to achieve the purpose of the present invention is: a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation, which is characterized by including the following steps:

[0006] Step (1): Input the rectified binocular images;

[0007] Step (2): Cost calculation: Divide the image into strong texture regions, weak texture regions, and textureless regions. In the strong texture regions, combine AD and Census for cost calculation, and in the weak texture regions and textureless regions, combine AD and gradient for cost calculation;

[0008] Step (3): Cost aggregation: Weaken the constraint condition of the difference in pixel point values and increase the constraint condition of the difference in gradient magnitudes during the construction of the cross arm, so that the construction arm lengths in the weak texture regions and textureless regions of the image become longer;

[0009] Step (4): Use the WTA strategy to screen out the minimum cost value and calculate the corresponding disparity value;

[0010] Step (5): Post-processing of parallax optimization: Perform a four-step optimization strategy on the parallax values obtained in step (4) to obtain an optimized parallax map;

[0011] Step (6): Output the final parallax map.

[0012] Furthermore, the specific method of "dividing the image into strong texture regions, weak texture regions, and textureless regions" in step (2) is as follows:

[0013] The strong texture region, weak texture region, and textureless region are respectively denoted as Us, Uw, and Uq. Add the values of n pixels around any pixel point and take the average value, and set a judgment threshold τ. The calculation formula is as follows:

[0014]

[0015] Equation (1) is used to compare and judge the average gray value of surrounding pixel points with the threshold τ, and equation (2) is used to compare and judge the average value of the three-channel color values of surrounding pixel points with the threshold τ. flag is a judgment flag. If it is 1, this pixel point belongs to Us; otherwise, it belongs to Uw and Uq.

[0016] Furthermore, the specific formula for cost calculation in step (2) is as follows:

[0017]

[0018] In the above formula, different cost functions are normalized, and different cost calculation functions are used according to the divided different regions. The value range of cost is [0, 2], λ AD 、λ Census 、λ Grad are respectively the conversion coefficients for normalizing the three cost functions, C AD is the cost value calculated by the AD function, C Census is the cost value calculated by the Census function, C Grad is the cost value calculated by the gradient function.

[0019] Furthermore, step (3) is specifically as follows:

[0020] Set two constraint conditions of the difference in color values and the difference in spatial arm lengths in the strong texture region; set two constraint conditions of the gradient and the difference in spatial arm lengths in the weak texture and textureless regions; the specific expressions are as follows:

[0021]

[0022] Among them, D G (p,q) = |D G (p) - D G (q)| represents the absolute value of the difference in gradient values between two pixel points p and q, ql is an adjacent point of pixel point q, and τ1 is the maximum threshold value of the difference in color values between pixel point q and its adjacent point q l . When the spatial distance between any pair of pixel points among p, q, q l and q is between [L2, L1], a more stringent color threshold τ2 is set, where Dc(p, q) represents the difference in color values between the two pixel points p and q, and D G (p, q) represents the difference in gradient values between the two pixel points p and q, and D S (p, q) represents the difference in spatial distances between the two pixel points p and q, and q l and q also have a similar comparison relationship. L1 is the maximum spatial arm length distance, and L2 is the more stringent spatial arm length distance.

[0023] Furthermore, the "four-step optimization strategy" in step (5) is specifically as follows:

[0024] Left-right consistency check, iterative local voting, adjustment of disparity discontinuity regions, and sub-pixel optimization.

[0025] Compared with the prior art, the present invention has the following remarkable advantages:

[0026] The present invention proposes a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation. First, the method sets an adaptive threshold to divide the image into strong texture regions, weak texture regions, and textureless regions. In the strong texture regions, cost calculation combining AD and Census is adopted, and in the weak texture and textureless regions, cost calculation combining AD and gradient is adopted; in the cost aggregation part, in order to improve the traditional cross-arm region construction process that cannot dynamically adjust the arm length according to different pixel points, in the construction process of this method, the constraint condition of the gray value is weakened, and at the same time, a multi-threshold gradient constraint condition is added to increase the cross-arm length in the weak texture and textureless regions, which can better optimize the aggregation in the weak texture and textureless regions and achieve the purpose of significantly reducing the false matching rate; finally, the final disparity map is obtained through disparity calculation and multi-step disparity optimization.

[0027] Experimental results show that the method proposed by the present invention reduces the average false matching rate of standard stereo image pairs on the Middlebury test platform by 5.66%. Compared with the stereo matching methods using a single cost function and aggregation method, the method of the present invention has strong adaptability to different texture regions and good robustness.

[0028] The method of the present invention has good robustness, can better adapt to different scenarios, and can also be applied in the development trends of different fields such as 3D map reconstruction, target detection, unmanned driving, and virtual reality, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the flowchart of the method of the present invention.

[0030] Figure 2 (a) The cost function combining AD and Census, (b) The cost function combining AD and gradient, (c) The experimental result graph of the cost calculation stage in the method proposed by the present invention.

[0031] Figure 3 Schematic diagram of the construction of the cross-arm support domain in the method proposed by the present invention; (a) First vertical then horizontal, (b) First horizontal then vertical.

[0032] Figure 4 (a) The support arm constructed by the CBCA method in the weak texture area; (b) The experimental result graph of the cost aggregation stage in the method proposed by the present invention in the weak texture area; (c) The support arm constructed by the CBCA method in the strong texture area; (d) The experimental result graph of the cost aggregation stage in the method proposed by the present invention in the strong texture area.

[0033] Figure 5 Parallax maps of different cost functions for the Aloe image under different illuminations, where (a) Left image; (b) Right image; (c) Standard parallax map; (d) Combining SAD and Census; (e) Combining AD and Census; (f) Combining AD and gradient; (g) The method of the present invention.

[0034] Figure 6 Parallax maps of different cost functions for the Baby1 image under different illuminations, where (a) Left image; (b) Right image; (c) Standard parallax map; (d) Combining SAD and Census; (e) Combining AD and Census; (f) Combining AD and gradient; (g) The method of the present invention.

[0035] Figure 7 Parallax maps of different cost functions for the Wood1 image under different illuminations, where (a) Left image; (b) Right image; (c) Standard parallax map; (d) Combining SAD and Census; (e) Combining AD and Census; (f) Combining AD and gradient; (g) The method of the present invention.

[0036] Figure 8 Parallax maps of different cost functions for the Aloe image under different exposures, (a) Left image; (b) Right image; (c) Standard parallax map; (d) Combining SAD and Census; (e) Combining AD and Census; (f) Combining AD and gradient; (g) The method of the present invention.

[0037] Figure 9Disparity maps of Baby1 image under different exposure conditions with different cost functions: (a) left image; (b) right image; (c) standard disparity map; (d) combination of SAD and Census; (e) combination of AD and Census; (f) combination of AD and gradient; (g) method of the present invention.

[0038] Figure 10 Disparity maps of Wood1 image under different exposure conditions with different cost functions: (a) left image; (b) right image; (c) standard disparity map; (d) combination of SAD and Census; (e) combination of AD and Census; (f) combination of AD and gradient; (g) method of the present invention.

[0039] Figure 11 Disparity maps of Aloe images with different cost functions without distortion, (a) left image; (b) right image; (c) standard disparity map; (d) combination of SAD and Census; (e) combination of AD and Census; (f) combination of AD and gradient; (g) method of the present invention.

[0040] Figure 12 Disparity maps of Baby1 image with different cost functions without distortion, (a) left image; (b) right image; (c) standard disparity map; (d) combination of SAD and Census; (e) combination of AD and Census; (f) combination of AD and gradient; (g) method of the present invention.

[0041] Figure 13 Disparity maps of Wood1 image with different cost functions without distortion, (a) left image; (b) right image; (c) standard disparity map; (d) combination of SAD and Census; (e) combination of AD and Census; (f) combination of AD and gradient; (g) method of the present invention.

[0042] Figure 14 Aggregate disparity maps of Cones images under different cross-arm construction methods, (a) left image; (b) right image; (c) standard disparity map; (d) disparity map under CBCA construction; (e) the method in this paper; (f) disparity map after post-processing.

[0043] Figure 15 Aggregated disparity maps of Teddy images under different cross-arm construction methods, (a) left image; (b) right image; (c) standard disparity map; (d) disparity map under CBCA construction; (e) the method proposed in this paper; (f) disparity map after post-processing. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0045] See also Figure 1As shown in the figure, a stereo matching method based on texture region adaptive dynamic cost calculation and aggregation, the method comprising the following steps:

[0046] Step 1: Input of the rectified images of the left and right eyes; in this step, it is necessary to rectify the images collected by the left and right binocular cameras, and use the rectified images as the input for the calculation in Step 2.

[0047] Step 2: Based on Step 1, calculate the initial cost value by the cost calculation method proposed by the present invention, divide different texture regions, and establish a three-dimensional cost matrix DSI (Disparity Space Image).

[0048] After obtaining the rectified images input in Step 1, using the cost calculation method proposed by the present invention, first divide the strong texture region, weak texture region and textureless region in the image by Equation (1) or Equation (2), and calculate the AD, Census and gradient cost values in different texture regions according to Equations (3)-(11), and perform normalization processing by Equation (12) to obtain the cost value in the initial stage, and update it to the three-dimensional cost matrix DSI for the calculation in the subsequent cost aggregation stage. The experimental result graph is shown in Figure 2 , from Figure 2 it can be seen that the cost calculation method proposed by the present invention can adapt to regions with different textures and can achieve good results. At the same time, Figures 5 - 7 the experimental result graphs in the cost calculation stage under different illuminations, different exposures and full pixels respectively.

[0049] The matching cost calculation is to compare the correlation between the pixel points to be matched and the candidate pixel points. The smaller the cost between two pixel points, the greater the correlation between the two points (that is, the greater the probability that the two points are corresponding points). Usually, a three-dimensional matrix DSI (Disparity Space Image) is used to store the result of the cost calculation. The size of DSI is H×W×D, where H and W are the height and width of the image respectively, and D is the disparity range.

[0050] First, it is necessary to distinguish the strong texture, weak texture and textureless regions in the image, which are respectively represented as Us, Uw, Uq. Add the values of n pixel points around any pixel point and take the average value. Commonly, n can be set to four directions or eight directions, and a judgment threshold τ is set. The calculation formula is as follows:

[0051]

[0052] Equation (1) is to make a comparison and judgment by using the average gray value of surrounding pixel points and the threshold τ, and Equation (2) is to make a comparison and judgment by using the average color values of three channels of surrounding pixel points and the threshold τ. flag is the judgment flag bit. If it is 1, then this pixel point belongs to Us; otherwise, it belongs to Uw and Uq.

[0053] After the above calculation, in the strong texture area, a cost calculation function combining AD and Census is used, and in the weak texture and textureless areas, a cost calculation function combining AD and gradient is used.

[0054] The main strategy of the AD function is to continuously compare the pixel values of two points in the left and right images, take out the pixel value of a certain pixel point in the image to be matched, and at the same time look for the pixel point with the minimum cost value within the disparity range in the other image to be indexed. The calculation expression is as follows:

[0055] C AD (p,q) = |G L (p) - G R (q)| (3)

[0056]

[0057] Equations (3) and (4) are the cost calculation formulas for the gray value and color value of pixel points respectively. L and R represent the two images of the left and right cameras respectively, and p and q represent the pixel points in the left and right images.

[0058] The Census function can well detect the structural features of the local area, has high matching accuracy, and is strong in adaptability to external factors such as illumination. Its main strategy is to define a rectangular window with both length and width being odd numbers, and then use this rectangular window to traverse the entire area of the image to be matched and the image to be indexed, and set the pixel value of the center point of the window as the central value. At the same time, compare the pixel values of other pixels in the window except the center point with the central value. If it is greater than the central value, it is recorded as 1; otherwise, it is recorded as 0. Map the obtained values into a bit string, and then compare the Hamming distance of the bit strings of the left and right images to obtain the cost value.

[0059]

[0060] C Census = Hamming(B p , B q ) (7)

[0061] In the above three equations, p represents the center point, q represents other pixel points within the neighborhood of the center point, U is the neighborhood of the center point within the window, B_p is the bit string obtained at the center computer, and substituting the bit strings of the obtained left and right images into Equation (7) to obtain the Hamming distance is the Census cost value.

[0062] In this method, the gradient function mainly uses the Sobel operator to solve the gradient values of image pixels. The main idea is as follows:

[0063]

[0064] |G L | = |G x *L| + |G y *L| (10)

[0065] |G R | = |G x *R| + |G y *R| (11)

[0066] In the above formula, G_x and G_y are the operators in the x and y directions respectively. The operators in the two directions are convolved with the left image L and the right image R respectively to obtain the gradient amplitudes in the two directions. Here, for simplicity of calculation, the absolute values of the gradients in the two directions are added to obtain the total amplitude.

[0067] The method in this paper weights and fuses different cost functions to obtain the final cost calculation formula as follows:

[0068]

[0069] In the above formula, different cost functions are normalized, and different cost calculation functions are used according to the divided different regions. The value range of cost is [0, 2], and λAD, λCensus, and λGrad are the conversion coefficients for normalizing the three cost functions respectively.

[0070] Step 3: On the basis of Step 1 and Step 2, different cost aggregation strategies are adopted for different texture regions to construct an adaptive cross-arm support domain, and the aggregated three-dimensional cost matrix is obtained.

[0071] On the basis of the three-dimensional cost matrix DSI calculated in Step 2, further cost aggregation calculation is carried out. Formulas (13) - (16) are the constraint conditions in the process of constructing the cross-arm support domain. Different constraint conditions are adopted in different texture regions. Two constraint conditions of the difference in color values and the difference in the length of the spatial arm are set in the strong texture region; two constraint conditions of the gradient and the difference in the length of the spatial arm are set in the weak texture and textureless regions. In this way, the constructed cross-arm support domain can contain more pixel points to the greatest extent in the weak texture and textureless regions, so as to obtain a better aggregation effect. At the same time, a good support domain can also be constructed in the strong texture region. The final calculation formula is shown in Formula (17). The schematic diagram of cross-arm construction is shown in Figure 3 , and the experimental result diagram of cross-arm construction is shown in Figure 4 , and the cost aggregation result diagram is shown in Figure 8In Figures (d) and (e), they are the disparity map after CBCA aggregation and the disparity map obtained by the cost aggregation method proposed in the present invention, respectively.

[0072] The cost aggregation calculation method is as follows:

[0073] Traditional cost aggregation uses a unified cross-arm region to construct constraint conditions, that is, in weak texture, no texture regions, and strong texture regions, a unified arm length construction method is used. Such a construction method cannot adapt to complex and changeable environments, and may also increase the incorrect assignment of pixel costs in aggregation, resulting in an increase in the final false matching rate. For example, the arm length setting is too small in weak texture and no texture regions, which will lead to inaccurate pixel cost values at the center point. Therefore, the method in this paper weakens the constraint condition of the difference between pixel values (color values or grayscale values) during the construction of the cross-arm, and adds the constraint condition of the difference in gradient amplitude, making the construction arm length longer in weak texture and no texture regions of the picture and improving the accuracy of aggregation.

[0074] The construction judgment conditions for the cross intersection domain construction in the traditional CBCA method are as follows:

[0075] D c (p,q) = max i＝R、G、B (I i (p) - I i (q)) < τ1 (13)

[0076] D c (q l ,q) < τ1 (14)

[0077] D S (p,q) = |p - q| < L (15)

[0078] In Equations (13) and (14), it is to limit the maximum value of the difference in color values between two pixels, that is, the central pixel p and the neighborhood pixel q, and at the same time limit the maximum value of the difference in color values between q and its adjacent point q l and set the threshold τ1; in Equation (15), it is to limit the spatial position length of two adjacent pixels and set the threshold L.

[0079] Subsequently, in order to make the cross intersection region contain more pixels, the spatial length is expanded to L1 > L, and at the same time a more strict color threshold τ2 is set. The added judgment condition expression based on the above is as follows:

[0080] Dx(p,q) < τ2 if L2 < D S (p,q) < L1 (16)

[0081] When the spatial distance between two pixel points p and q in the above formula is between [L2, L1], the threshold of the color value difference is set to τ2 (τ2 < τ1), so as to obtain a larger arm length.

[0082] Based on dividing different texture regions in the cost calculation process of the method in this paper, different judgment conditions are set in different regions. Two constraint conditions of the color value difference and the spatial arm length difference are set in the strong texture region; two constraint conditions of the gradient and the spatial arm length difference are set in the weak texture and textureless regions. The specific expressions are as follows:

[0083]

[0084] Where D G (p,q) = |D G (p) - D G (q)| represents the absolute value of the difference in gradient values between two pixel points, and τ3 and τ4 are the gradient thresholds within the ranges of two broken arm lengths respectively.

[0085] The construction result of the dynamic cross-arm region based on the texture region is as Figure 3 shown Figure 3 (a) and Figure 3 (b), the light-colored regions are the dynamic support domains of pixel points, and the dark-colored regions are the vertical and horizontal arms during the construction process. There are two strategies for constructing the support domain, that is, vertical first and then horizontal or horizontal first and then vertical, as shown in Figure 3 (a) and Figure 3 (b) respectively.

[0086] From Figure 4 (a) and (b), it can be seen that in the weak texture region, the arm length of the cross support domain constructed by the traditional CBCA method is less than the cross arm length proposed in this paper, that is, the support domain constructed by this paper in the weak texture region can contain more pixel points; at the same time, in the strong texture region, the support domain constructed by this paper is not much different from the support domain constructed by the traditional CBCA method, and both can achieve good results, as shown in Figure 4 (c) and (d).

[0087] Step 4: Apply the WAT strategy to the three-dimensional cost matrix DSI after Step 3, and calculate the corresponding disparity value, so as to obtain the final two-dimensional disparity matrix for the post-processing of disparity optimization in Step 5.

[0088] Step 5: Perform post-processing on the two-dimensional disparity matrix obtained in Step 4, which is mainly divided into four-step strategies, namely left-right consistency check, iterative local voting, adjustment of disparity discontinuity regions, and sub-pixel optimization. The experimental results are shown in Figure 8 (f) in

[0089] Step 6: Output the final disparity map.

[0090] After the above steps, the final disparity map will be obtained. By calculating its epipolar distance from the binocular camera, the distance value in the real world will be obtained.

Claims

1. A stereo matching method based on texture region adaptive dynamic cost calculation and aggregation, characterized in that It includes the following steps: Step (1): Input the rectified binocular images; Step (2): Cost calculation: Divide the image into strong texture regions, weak texture regions, and textureless regions. In the strong texture regions, combine AD and Census for cost calculation. In the weak texture regions and textureless regions, combine AD and gradient for cost calculation; Step (3): Cost aggregation: During the construction of the cross arm, weaken the constraint condition of the difference in pixel values and increase the constraint condition of the difference in gradient magnitudes, so that the constructed arm length becomes longer in the weak texture regions and textureless regions of the image; Step (4): Use the WTA strategy to select the minimum cost value and calculate the corresponding disparity value; Step (5): Post-processing of disparity optimization: Apply a four-step optimization strategy to the disparity values obtained in Step (4) to obtain an optimized disparity map; Step (6): Output the final disparity map; Specifically, Step (3) is as follows: Set two constraint conditions of the difference in color values and the difference in spatial arm lengths in the strong texture regions; set two constraint conditions of the gradient and the difference in spatial arm lengths in the weak texture and textureless regions; the specific expressions are as follows: where D G (p,q) = |D G (p) - D G (q)| represents the absolute value of the difference in gradient values between two pixel points p and q, and q l is an adjacent point of pixel point q, τ1 is the maximum threshold value of the difference in color values between pixel point q and its adjacent point q l ; when the spatial distance between any pair of pixel points among p and q, q l and q is within [L2, L1], a more stringent color threshold τ2 is set, where Dc(p,q) represents the difference in color values between two pixel points p and q, and D G (p,q) represents the difference in gradient values between two pixel points p and q, and D S (p,q) represents the difference in spatial distance between two pixel points p and q, and q l and q also have a similar comparison relationship, L1 is the maximum spatial arm length distance, and τ3 and τ4 are respectively the gradient thresholds within the two broken arm length ranges; Specifically, the "four-step optimization strategy" in Step (5) is as follows: Left-right consistency check, iterative local voting, adjustment of disparity discontinuity regions, and sub-pixel optimization.

2. The method according to claim 1, wherein Specifically, in Step (2), "divide the image into strong texture regions, weak texture regions, and textureless regions" is as follows: The strong texture region, weak texture region, and textureless region are respectively denoted as Us, Uw, and Uq. Add the values of n pixels around any pixel and take the average, and set a judgment threshold τ. The calculation formula is as follows: Equation (1) is used to compare and judge the mean value of the gray values of the surrounding pixels with the threshold τ, and Equation (2) is used to compare and judge the mean value of the three-channel color values of the surrounding pixels with the threshold τ. flag is the judgment flag bit. If it is 1, then this pixel belongs to Us; otherwise, it belongs to Uw and Uq.

3. The method according to claim 2, wherein The specific formula for cost calculation in Step (2) is as follows: In the above formula, different cost functions are normalized, and different cost calculation functions are used according to different divided regions. The value range of cost is [0, 2], and λ AD , λ Census , λ Grad are the conversion coefficients for normalizing three cost functions respectively. C AD is the cost value calculated by the AD function, and C Census is the cost value calculated by the Census function, and C Grad is the cost value calculated by the gradient function.

Images