Binocular stereo matching method based on matching cost aggregation

Abstract

The invention provides a binocular stereo matching method based on matching cost aggregation. The binocular stereo matching method comprises the following steps: step 1, selecting an image; 2, constructing a parallax cube; 3, selecting a plurality of paths, and updating the parallax cube; 4, determining an optimal disparity value to obtain a final disparity map; according to the binocular stereo matching method based on matching cost aggregation, the real-time problem of three-dimensional reconstruction can be solved, and the binocular stereo matching method is mainly used for application scenes with high requirements for three-dimensional reconstruction efficiency, such as automatic driving. According to the method, the platform portability is considered, the resolving efficiency is emphasized, meanwhile, the excellent reconstruction quality is guaranteed, and the practicability is high.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a binocular stereo matching method based on matching cost aggregation. Background Art

[0002] The 3D reconstruction technology based on binocular vision imitates the principle of human eye perception of the 3D world, and infers the depth through the difference in the position of the same-name points (i.e. the same point in the real world) between the binocular images (called "parallax"): the parallax is large at close range and small at far distance. Estimating the parallax, that is, the process of finding matching points on the binocular images, is called stereo matching. It is the core step of binocular 3D reconstruction, and its accuracy and speed directly determine the effect and efficiency of reconstruction.

[0003] The classic stereo matching method is mainly patch matching. This is a matching method based on tilted support windows proposed by Michael Bleyer et al. in Stereo Matching with Slanted Support Windows, which can achieve good results for most scenes. However, due to the limitations of the algorithm itself, its computational efficiency is not high and it cannot be parallelized, which is not conducive to the reconstruction of dynamic scenes. On the other hand, the related results of stereo matching based on deep learning can achieve more delicate reconstruction effects, but its shortcomings are also obvious, such as unexplainable, unable to be finely tuned, high requirements on the hardware platform, and difficulty in achieving real-time reconstruction. Summary of the invention

[0004] In order to solve the real-time problem of 3D reconstruction.

[0005] The present invention provides a binocular stereo matching method based on matching cost aggregation, comprising the following steps:

[0006] Step 1: Select an image;

[0007] Step 2, construct a parallax cube;

[0008] Step 3, select multiple paths and update the parallax cube;

[0009] Step 4: Determine the optimal disparity value and obtain the final disparity map.

[0010] Furthermore, the specific process of step 1, selecting an image is:

[0011] Step 201: Use two cameras to capture two images, which are respectively recorded as image A and image B;

[0012] Step 202: Perform epipolar correction processing on image A and image B to obtain initial image A' and initial image B'.

[0013] Furthermore, the specific process of step 2, constructing the parallax cube is:

[0014] Step 301, performing Census transformation on the initial image A' and the initial image B' respectively to obtain the Census transformation values ​​of the initial image A' and the initial image B';

[0015] Step 302: Calculate the Hamming distance between the Census transformation values ​​of two pixels corresponding to the Census transformation values ​​of the initial image A' and the initial image B', that is,

[0016] C(u,v,d):=Hamming(C sl (u,v),C sr (ud,v)) (1);

[0017] Among them, u and v represent the horizontal and vertical coordinates of the pixel, d represents the disparity, C sl (u,v) and C sr (ud,v) represents the Census transformation result of a pair of points with the same name in the left and right camera images.

[0018] Step 303: Establish a matching cost function:

[0019]

[0020] Where d is a disparity map relative to the reference image, which can be expressed as d:{(i,j)→[d min ,d max ]|(i,j)∈I}, where I is the reference image; p: a coordinate in the reference image; q: the neighborhood N of pixel p p A pixel within d p ,d q : The disparity value assigned to pixels p and q; C data (p,d p ): The cost corresponding to the disparity value of pixel p; P1, P2: custom weights, where

[0021]

[0022] Where I represents the gray value, |I bp -I bq | represents the absolute value of the difference between the grayscale values ​​of two adjacent pixels. P2′ is generally set to a number much larger than P1;

[0023] Step 304: Create a parallax cube. The size of the parallax cube is expressed as:

[0024] W×H×(dmax -d min +1) (4);

[0025] Where the image size is W×H, and all possible values ​​of disparity d are defined by the following formula: Then, fill the disparity cube with the cost value calculated by the Census transformation;

[0026] Step 305: Based on formula (2), the total cost along direction r can be written as:

[0027]

[0028] Among them, pr represents the direction vector from the previous pixel to the next pixel along the path direction, i represents the path number, and C data (p, d) is the matching cost obtained by Census transformation, C r (pr,d-1) represents the path aggregation cost when the disparity of the previous pixel in the path is d-1; C r (pr,d+1) represents the path aggregation cost when the disparity of the previous pixel in the path is d+1; Represents the minimum value of all costs of the previous pixel in the path. In order to ensure the path cost C r Not exceed a certain numerical limit;

[0029] Step 306: Add the path aggregation costs obtained in each direction to obtain the total path aggregation cost S(p, d):

[0030]

[0031] in, Represents the aggregate cost of each path.

[0032] Furthermore, the specific process of step 3, selecting multiple paths and updating the parallax cube is: using any one of the three aggregation methods of 4 paths, 8 paths or 16 paths, gradually updating the values ​​in the parallax cube with the total path aggregation cost S(p, d), and finally obtaining a new cube.

[0033] Furthermore, the specific process of step 4, determining the optimal disparity value and obtaining the final disparity map is as follows: for each pixel of the reference image, the minimum value in the corresponding column of the disparity cube can be used as the optimal disparity of the pixel; stereo matching is completed.

[0034] The advantages of the present invention are as follows: the present invention provides a binocular stereo matching method based on matching cost aggregation, which can solve the real-time problem of three-dimensional reconstruction and mainly serves application scenarios with high requirements for three-dimensional reconstruction efficiency, such as autonomous driving. While considering platform portability and focusing on solution efficiency, the method also ensures excellent reconstruction quality and has strong practicality.

[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the flow chart of the binocular stereo matching method based on matching cost aggregation.

[0037] Figure 2 Flowchart for comparing gray levels for this Census transform.

[0038] Figure 3 This is the Hamming distance flow chart for the Census transformation value.

[0039] Figure 4 A schematic diagram of a parallax cube.

[0040] Figure 5 A schematic diagram of the aggregation path.

[0041] Figure 6 Schematic diagram of path aggregation in 8 directions.

[0042] Figure 7 Schematic diagram of the binocular stereo matching method processing the sampled image based on matching cost aggregation. DETAILED DESCRIPTION

[0043] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the specific implementation methods, structural features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and embodiments.

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0046] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0047] Example 1

[0048] This embodiment provides a Figure 1 to Figure 5 The binocular stereo matching method based on matching cost aggregation shown in FIG. 1 includes the following steps:

[0049] Step 1: Select an image;

[0050] Step 2, construct a parallax cube;

[0051] Step 3, select multiple paths and update the parallax cube;

[0052] Step 4: Determine the optimal disparity value and obtain the final disparity map.

[0053] Furthermore, the specific process of step 1, selecting an image is:

[0054] Step 201: Use two cameras to capture two images, which are respectively recorded as image A and image B;

[0055] Step 202: Perform epipolar correction on image A and image B to obtain initial image A' and initial image B'. Epipolar correction is performed on image A and image B using existing technology, which will not be described in detail in this example.

[0056] Furthermore, the specific process of step 2, constructing the parallax cube is:

[0057] Step 301, performing Census transformation on the initial image A' and the initial image B' respectively to obtain the Census transformation values ​​of the initial image A' and the initial image B';

[0058] Census transform is a parameter-free transform that only depends on relative grayscale. It is very invariant to changes in light intensity and is often used for optical flow calculation and disparity estimation. The most common Census transform uses a 3×3 window to compare the grayscale of the center pixel p with its 8 pixels. The comparison result can be defined by the following formula:

[0059]

[0060] The transformation result can be obtained by serializing the results in a certain order into an 8-bit bit string. Figure 2 shown.

[0061] Step 302: Calculate the Hamming distance between the Census transformation values ​​of two pixels corresponding to the Census transformation values ​​of the initial image A' and the initial image B', that is,

[0062] C(u,v,d):=Hamming(C sl (u,v),C sr (ud,v)) (1);

[0063] Among them, u and v represent the horizontal and vertical coordinates of the pixel, d represents the disparity, C sl (u,v) and C sr (ud,v) represents the Census transformation result of a pair of points with the same name in the left and right camera images.

[0064] The Hamming distance is the number of different corresponding bits of two bit strings. It is calculated by performing an OR operation on the two bit strings and then counting the number of bits in the OR operation result that are not 1, such as Figure 2 shown.

[0065] Because the Census transformation compares the relative grayscale, even if the brightness of the left and right images is inconsistent, a good matching effect can be obtained. In addition, because the Census transformation is a local window operation, it also has the advantage of high parallelism, and each pixel can be operated independently. This feature can be used to design a multi-threaded or GPU parallel computing model to achieve very high efficiency.

[0066] In the actual matching process, simply calculating the matching cost cannot produce a high-quality disparity map, and a cost aggregation step is required. The aggregated cost can more accurately reflect the matching similarity.

[0067] Step 303: Establish a matching cost function:

[0068] The existing method only considers local correlation and is very sensitive to noise, so it cannot be used directly to calculate the optimal disparity. Therefore, this algorithm uses the cost aggregation step to make the aggregated cost value more accurately reflect the correlation between pixels.

[0069] The global energy function is defined as

[0070] E(d)=E data (d)+E smooth (d)

[0071] Among them, E data is the data item, E smoothIt is a smoothing term, which is to make the disparity map meet the constraints of certain conditional assumptions, such as the continuity assumption of the scene surface. The smoothing term will penalize the situation where the disparity change of adjacent pixels exceeds a certain pixel (that is, increase the energy value). In this way, the stereo matching problem is transformed into an optimization problem.

[0072] Distinguish between disparity continuous and disparity discontinuous areas: Disparity continuity means that the disparity of pixels in the local range is very small (within 1 pixel), which is a continuous change trend; while disparity disparity means that the disparity of pixels in the local range is very different (more than 1 pixel), which is a sudden change trend. This local range is often a rectangular window (such as 3×3, 5×5). Based on this distinction criterion, the following matching cost function is established:

[0073]

[0074] Where d is a disparity map relative to the reference image, which can be expressed as d:{(i,j)→[d min ,d max ]|(i,j)∈I}, where I is the reference image; p: a coordinate in the reference image; q: the neighborhood N of pixel p p A pixel within d p ,d p : The disparity value assigned to pixels p and q; C data (p,d p ): The cost corresponding to the disparity value of pixel p; P1, P2: custom weights, where

[0075]

[0076] Where I represents the gray value, |I bp -I bq | represents the absolute value of the difference between the grayscale values ​​of two adjacent pixels. P2′ is generally set to a number much larger than P1. Its meaning is: if the brightness difference between a pixel and its neighboring pixels is large, then the pixel is likely to be located in a disparity discontinuous area. In this case, the disparity difference between the pixel and its neighboring pixels is allowed to exceed 1 pixel to a certain extent, and the penalty for exceeding 1 pixel is appropriately reduced.

[0077] The first term in the brackets in formula (2) is the data term, which means the accumulation of the matching cost of all pixels when the disparity map is d; the second and third terms are smoothing terms, which means the N matching cost of pixel p. pAll pixels q in the neighborhood are penalized. The second term penalizes the case where the disparity change between adjacent pixels is very small (1 pixel), and the penalty is small (P1 is small); the third term has a large penalty (P2>P1), and penalizes the case where the disparity change between adjacent pixels is large (greater than 1 pixel). Smaller penalty terms allow the algorithm to adapt to situations where the disparity change is small, such as tilted planes or continuous surfaces, and larger penalty terms allow the algorithm to correctly handle discontinuous disparity situations.

[0078] Step 304: Create a parallax cube, such as Figure 4 As shown, the size of the parallax cube is expressed as:

[0079] W×H×(d max -d min +1) (4);

[0080] Where the image size is W×H, and all possible values ​​of disparity d are defined by the following formula: Then, fill the disparity cube with the cost value calculated by the Census transformation;

[0081] Step 305: Based on formula (2), the total cost along direction r can be written as:

[0082]

[0083] Among them, pr represents the direction vector from the previous pixel to the next pixel along the path direction, and i represents the path number. data (p, d) is the matching cost obtained by Census transformation, C r (pr,d-1) represents the path aggregation cost when the disparity of the previous pixel in the path is d-1; C r (pr,d+1) represents the path aggregation cost when the disparity of the previous pixel in the path is d+1; Represents the minimum value of all costs of the previous pixel in the path. In order to ensure the path cost C r Not exceed a certain numerical limit;

[0084] The meaning of formula (5) is basically the same as that of formula (2): the first term is the matching cost obtained by Census transformation, which is generally calculated in the initialization stage and can be regarded as known; the second term is the smoothing term; the third term is to ensure that the path cost C r Not exceeding a certain upper limit, that is

[0085] C r ≤C max +P2

[0086] Among them, C max It is an artificially specified upper limit.

[0087] Let's explain the second term of (5): C r (pr,d-1) represents the path aggregation cost when the disparity of the previous pixel in the path is d-1; C r (pr,d+1) represents the path aggregation cost when the disparity of the previous pixel in the path is d+1; Represents the minimum value of all costs of the previous pixel in the path.

[0088] In the parallax cube, consider the case where d = k. Converging path C r (pr,k-1) and C r (pr,k+1) can be used Figure 5 Indicates. The arrow in the figure is the path direction r. If the current pixel is p, then the previous pixel is pr.

[0089] Step 306: Add the path aggregation costs obtained in each direction to obtain the total path aggregation cost S(p, d):

[0090]

[0091] in, Represents the aggregate cost of each path.

[0092] Furthermore, the specific process of selecting multiple paths and updating the parallax cube in step 3 is: using any one of the three aggregation methods of 4 paths (operating with the 4 pixels above, below, left and right of the current pixel), 8 paths or (operating with the 8 pixels around the current pixel) or 16 paths (operating with the 16 pixels around the current pixel), gradually updating the value in the parallax cube with the total path aggregation cost S(p, d), and finally obtaining a new cube. The more paths there are, the better the effect, but correspondingly, the operation takes longer. In practical applications, it is necessary to balance the pros and cons and use an appropriate number of paths. Figure 6 It is a schematic diagram of path aggregation in 8 directions.

[0093] Furthermore, the specific process of step 4, determining the optimal disparity value and obtaining the final disparity map is: for each pixel of the reference image, the minimum value in the corresponding column in the disparity cube can be used as the optimal disparity of the pixel; and stereo matching is completed.

[0094] Specifically, as the aggregation process proceeds, the values ​​in the disparity cube are gradually updated with the total path aggregation cost S(p,d), and finally a new cube is obtained. Next, for each pixel on the reference image, the minimum value in the corresponding column in the disparity cube can be used as the optimal disparity of the pixel (i.e., the Winner Takes All (WTA) algorithm). In this way, stereo matching is completed.

[0095] In summary, this embodiment provides a binocular stereo matching method based on matching cost aggregation, which can solve the real-time problem of 3D reconstruction and mainly serves application scenarios with high requirements for 3D reconstruction efficiency, such as autonomous driving. While considering platform portability and focusing on solution efficiency, this method also ensures excellent reconstruction quality and has strong practicality.

[0096] Example 2

[0097] Using the binocular stereo matching method based on matching cost aggregation shown in Example 1, Figure 7 The images shown are matched, where Figure 7 (a) is the image captured by the camera, Figure 7 (b) is the true value of the disparity map, Figure 7 (c) is the disparity map before aggregation, Figure 7 (d) is the disparity map after aggregation.

[0098] Through the above binocular stereo matching method based on matching cost aggregation, a high-precision and dense disparity map can be quickly obtained, laying a good foundation for the subsequent point cloud generation.

[0099] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A binocular stereo matching method based on matching cost aggregation, characterized in that: The steps include: Step 1: Select an image; Step 2, construct a parallax cube; Step 3, select multiple paths and update the parallax cube; Step 4: Determine the optimal disparity value and obtain the final disparity map.

2. A binocular stereo matching method based on matching cost aggregation as claimed in claim 1, characterized in that: The specific process of step 1, selecting an image is: Step 201: Use two cameras to capture two images, which are respectively recorded as image A and image B; Step 202: Perform epipolar correction processing on image A and image B to obtain initial image A' and initial image B'.

3. A binocular stereo matching method based on matching cost aggregation as claimed in claim 2, characterized in that: The specific process of step 2, constructing the parallax cube is: Step 301, performing Census transformation on the initial image A' and the initial image B' respectively to obtain the Census transformation values ​​of the initial image A' and the initial image B'; Step 302: Calculate the Hamming distance between the Census transformation values ​​of two pixels corresponding to the Census transformation values ​​of the initial image A' and the initial image B', that is, C(u,v,d):=Hamming(C sl (u,v),C sr (u-d,v)) (1); Among them, u and v represent the horizontal and vertical coordinates of the pixel, d represents the disparity, C sl (u, v) and C sr (ud, v) represents the Census transformation result of a pair of points with the same name in the left and right camera images. Step 303: Establish a matching cost function: Where d is a disparity map relative to the reference image, which can be expressed as d:{(i, j)→[d min , d max ]|(i, j)∈I}, where I is the reference image; p: a coordinate in the reference image; q: the neighborhood N of pixel p p A pixel within d p , d q : disparity value assigned to pixels p and q; C data (p,d p ): the cost corresponding to the disparity value of pixel p; P1, P2: custom weights, where Where I represents the gray value, |I bp -I bq | represents the absolute value of the difference between the grayscale values ​​of two adjacent pixels. P2′ is generally set to a number much larger than P1; Step 304: Create a parallax cube. The size of the parallax cube is expressed as: W×H×(d max -d min +1) (4); Where the image size is W×H, and all possible values ​​of disparity d are defined by the following formula: Then, fill the disparity cube with the cost value calculated by the Census transformation; Step 305: Based on formula (2), the total cost along direction r can be written as: Among them, pr represents the direction vector from the previous pixel to the next pixel along the path direction, i represents the path number, and C data (p, d) is the matching cost obtained by Census transformation, C r (pr, d-1) represents the path aggregation cost when the disparity of the previous pixel in the path is d-1; C r (pr, d+1) represents the path aggregation cost when the disparity of the previous pixel in the path is d+1; Represents the minimum value of all costs of the previous pixel in the path. In order to ensure the path cost C r Not exceed a certain numerical limit; Step 306: Add the path aggregation costs obtained in each direction to obtain the total path aggregation cost S(p, d): in, Represents the aggregate cost of each path.

4. The binocular stereo matching method based on matching cost aggregation according to claim 1, characterized in that: The specific process of step 3, selecting multiple paths and updating the parallax cube is: using any one of the three aggregation methods of 4 paths, 8 paths or 16 paths, gradually updating the values ​​in the parallax cube with the total path aggregation cost S(p, d), and finally obtaining a new cube.

5. The binocular stereo matching method based on matching cost aggregation according to claim 1, characterized in that: The specific process of step 4, determining the optimal disparity value and obtaining the final disparity map is: for each pixel of the reference image, the minimum value in the corresponding column of the disparity cube can be used as the optimal disparity of the pixel; and stereo matching is completed.

Images