Depth and width estimation method based on binocular stereo vision

Abstract

The application discloses a depth and width estimation method based on binocular stereo vision and belongs to the field of computer vision. The method uses a binocular stereo camera to acquire images, establishes a mathematical model by using the relationship between the depth of an observation target distance imaging plane and the parallax and the relationship between the observation point's approximate width and the parallax, calibrates the depth and the width respectively, and thus provides a method for solving the real distance from the observation point to the principal point of the camera in a world coordinate system. The method also considers the error caused by the different principal points of the left and right cameras due to the camera installation problem, and improves the calculation precision of the depth and the width by fitting.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a depth and width estimation method based on binocular stereo vision. Background Technology

[0002] In computer vision and robotics, binocular stereo vision is often required to construct 3D space. Binocular stereo vision is a technique that combines the principle of parallel-view imaging to construct two cameras with parallel and co-directional imaging planes to simulate human binocular vision. This technique samples the same object in a 3D scene to obtain two images, left and right. Based on a specific matching algorithm, the disparity of the object in the left and right images is calculated, and then the distance information of the object is obtained using triangulation, thereby reconstructing the 3D scene. Various techniques can be used to construct 3D space. For example, Structure in Motion Retrieval and Simultaneous Localization and Mapping (SLAM) are two such techniques. SLAM technology typically involves estimating the depth of the 3D scene to be mapped, which can be performed using a depth camera. However, depth cameras typically have limited range, relatively high power consumption, and may not work correctly in outdoor environments such as bright sunlight. In other cases, depth estimation can be performed without using a depth camera, such as based on spatial images. Binocular stereo vision technology now plays an irreplaceable role in fields such as autonomous driving, 3D audio-visual entertainment, and machine navigation. Summary of the Invention

[0003] This invention provides a method for depth and width calibration of observed targets using a binocular stereo vision camera. Images are acquired using a binocular stereo camera, and mathematical models are established based on the relationship between the depth of the observed target from the imaging plane and parallax, and the relationship between the desired width of the observed point and parallax. The depth and width are then calibrated accordingly, thus providing a method for calculating the true distance from the observed point to the camera's principal point in the world coordinate system. Furthermore, considering the errors caused by the different principal points of the left and right cameras due to camera installation issues, the accuracy of depth and width calculations is improved through fitting.

[0004] The technical solution of the present invention is as follows:

[0005] The depth and width estimation method based on binocular stereo vision involves the following steps:

[0006] 1) Depth calibration

[0007] 1-1) Set the camera placement position as the origin, use a plumb bob to ensure that the camera's imaging surface is perpendicular to the ground plane, and mark the origin position.

[0008] 1-2) Set a field of view angle of α degrees centered at the origin, and trisect the α-degree field of view angle. The range from -α / 2 to -α / 6 is the left region, the range from -α / 6 to α / 6 is the middle region, and the range from α / 6 to α / 2 is the right region. Also, set the radii of the regions to be a meters and b meters (b < a) respectively. Name the region with a radius from 0 to b meters as the front region, and the region from b to a meters as the rear region. There are three regions (left, middle, and right) in both the front and rear regions, totaling six regions.

[0009] 1-3) Collect M sampling points in each region, and use a total station to record the distance dis from the sampling point to the origin and the offset angle θ of the sampling point relative to the origin, and mark the position of each sampling point. Use M / 3 sampling points in each region for depth calibration, and the other 2M / 3 sampling points for width calibration.

[0010] 1-4) Place the depth calibration board at the positions of the marked sampling points respectively, and at the same time use a binocular camera to collect the images at these positions and calculate the disparity map.

[0011] 1-5) Use the MATLAB tool to draw a frame around the position of the depth calibration board in the disparity map obtained in 1-4), and calculate the mean value of the disparity within the framed area.

[0012] 1-6) Convert the distance dis from the sampling point to the origin and the offset angle θ of the sampling point relative to the origin obtained in 1-3) into the true depth value of the sampling point. The calculation formula is as follows:

[0013] Z

[0017] ,

[0016] , r ,

[0015] , true ,

[0020] ,

[0014] , ,

[0013] , , , l , , , true , , ,

[0019] , ,

[0018] , , = dis * cosθ

[0014] According to the Pythagorean theorem, the true depth value Z true is the distance dis from point to point multiplied by the cosine of the offset angle θ.

[0015] 1-7) Use the least squares method to fit and calculate the obtained disparity mean value and true depth value data to obtain the final depth calibration value. The specific steps are as follows:

[0016] a) Considering the error caused by the difference in the principal points of the left and right cameras due to the camera installation problem, use the principle of similar triangles to obtain the following relationship:

[0017]

[0018] After simplification, we get:

[0019]

[0020] where u0 and u0' are the abscissas of the principal points of the left and right cameras respectively, u l and u r are the abscissas of the imaging plane respectively, B is the camera baseline, and f is the camera focal length.

[0021] Generally, binocular ranging algorithms assume that the principal points of the left and right cameras are equal, i.e., u0 = u0'. However, in practice, due to camera installation issues, the principal points of the left and right cameras may not be equal. Therefore, let:

[0022] (-u0+u0')dx=Δd

[0023] (u l -u r )dx=d

[0024] Therefore, the relationship between the depth of the observed target from the imaging plane and the parallax is as follows:

[0025]

[0026] Where the depth is Z, from the above formula:

[0027]

[0028] make

[0029] Bf = t1

[0030] Δd=t2

[0031] Substituting the mean disparity and true depth data of the M / 3 pairs, we obtain the following overdetermined system of equations:

[0032]

[0033] make

[0034]

[0035]

[0036]

[0037] Then there is

[0038] AX = d

[0039] b) Solve the overdetermined system of equations using the least squares method.

[0040] make

[0041]

[0042] Right now

[0043] E(x) = ||AX - d|| 2

[0044] Solving

[0045]

[0046] X * This represents the optimal solution for t1 and t2 obtained after least squares fitting, which is also the optimal solution for Bf and Δd. Once Bf and Δd are obtained, the final depth calibration value Z can be calculated.

[0047] 2) Width calibration.

[0048] 2-1) Obtain the remaining 2M / 3 points for each region and use the least squares method to fit and solve for the width calibration value:

[0049] a) The relationship between width distance and parallax is:

[0050]

[0051] X represents the actual width of the object, u represents the x-coordinate of the center of the region in the image, d represents the parallax, and C represents the distance. x The x-coordinate of the optical center, f is the focal length, B is the baseline, and Δd is the error coefficient obtained in the depth calibration.

[0052] Let a1 = B, a2 = -BC x If a3 = -Δd, then the above expression becomes:

[0053]

[0054] After simplification, we get:

[0055] -ua1+xd+xa3=a2

[0056] Divide both sides of the above equation by a2, and let a'1 = a1 / a2, a'2 = 1 / a2, a'3 = a3 / a2, then:

[0057] -ua'1+xda'2+xa'3=1

[0058] Substituting the mean disparity and true depth data from the 2M / 3 sets, we obtain the following overdetermined nonhomogeneous equation system:

[0059]

[0060] make

[0061]

[0062]

[0063]

[0064] b) Solve the above overdetermined system of equations using the least squares method.

[0065] make

[0066]

[0067] Right now

[0068] E(x) = ||Ax - y|| 2

[0069] Solving

[0070]

[0071] x * These are the optimal solutions for a'1, a'2, and a'3 obtained after least squares fitting, which yields the optimal solutions for coefficients a1, a2, and a3. The final width calibration value X can then be calculated using a1, a2, and a3.

[0072] The beneficial effects of this invention are: This invention uses a binocular stereo camera to complete data acquisition and combines it with MATLAB to build a high-performance computing platform, scientifically and effectively processing the sampled data to accurately calculate the depth Z from the observation point to the pixel plane and the lateral width X of the pixel point from the observation point; this invention has the following characteristics:

[0073] (1) The system is easy to build, and the input data can be collected using a stereo binocular camera;

[0074] (2) The program is simple and easy to implement;

[0075] (3) Use mathematical modeling to solve the depth calibration and width calibration in two parts;

[0076] (4) It can perform multi-threaded operations, improving efficiency. Attached Figure Description

[0077] Figure 1 This is an overall flowchart of the present invention;

[0078] Figure 2 Flowchart for depth calibration;

[0079] Figure 3 A schematic diagram showing the marking of collection points in the area;

[0080] Figure 4 To draw a bounding box around the location of the depth calibration plate in the disparity map, a schematic diagram of the mean disparity within the bounding box is presented.

[0081] Figure 5 A graph showing the relationship between depth, width, and distance from the sampling point to the origin;

[0082] Figure 6 This is a schematic diagram of the principle of binocular ranging. Detailed Implementation

[0083] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0084] This invention designs a depth and width estimation method based on binocular stereo vision, such as... Figure 1 As shown, the specific steps are as follows:

[0085] 1) Depth calibration, the process is as follows: Figure 2 As shown, the details are as follows:

[0086] 1-1) Set the camera placement position as the origin, use a plumb bob to ensure that the camera's imaging surface is perpendicular to the ground plane, and mark the origin position.

[0087] 1-2) such as Figure 3 As shown, with the origin as the center, a 120-degree field of view is set. The 120-degree field of view is divided into three equal parts: -60 degrees to -20 degrees is the left region, -20 degrees to 20 degrees is the middle region, and 20 degrees to 60 degrees is the right region. Radius is set to 12 meters and 30 meters respectively. The region with radius from 0 to 12 meters is named the front region, and the region with radius from 12 to 30 meters is named the back region. The front and back regions each have three regions: left, middle, and right, so there are a total of six regions.

[0088] 1-3) Randomly collect 15 sampling points in each area, and use a total station to record the distance dis from the sampling point to the origin and the offset angle θ of the sampling point relative to the origin. Mark the position of each sampling point. In practice, the distance from the sampling point to the origin and the offset angle of the sampling point relative to the origin recorded by the total station are shown in Table 1 and Table 2. From the tables, we can see that there are 15 sampling points in each group of areas, and their distance and offset angle relative to the origin are recorded respectively, which are used to calculate the subsequent true depth value.

[0089] Table 1 Distance from sampling points to the origin

[0090] 5.32 5.188 5.087 5.031 5 5.029 5.085 5.183 5.327 6.376 6.22 6.1 6.03 6 6.03 6.1 6.22 6.38 8.51 8.296 8.134 8.037 8 8.04 8.14 8.3 8.514 10.638 10.7 10.167 10.044 10 10.05 10.18 10.39 10.645 12.78 12.44 12.19 12.05 12 12.07 12.24 12.5 12.85 14.904 14.529 14.245 14.076 14 14.072 14.237 14.52 14.89 15.793 15.562 15.25 15.067 15 15.071 15.268 15.573 15.796 21.285 20.747 20.334 20.082 20 20.085 20.323 20.738 21.726 26.612 25.916 25.414 25.103 25 25.096 25.394 25.908 26.602 31.947 31.11 30.501 30.135 30 30.102 30.462 31.062 31.933

[0091] Table 2. Offset angle of sampling points relative to the origin

[0092] 20.13 15.51 10.5 5.26 0 354.81 349.6 344.55 339.7 20 15.41 10.41 5.25 0 354.78 349.6 344.65 339.7 20.01 15.4 10.41 5.21 0 354.78 349.6 344.66 340 20 15.41 10.4 5.25 0 354.75 349.55 344.55 340 20.1 15.33 10.3 5.16 0 354.7 349.58 344.6 339.93 20.01 15.41 10.38 5.2 0 354.75 349.56 344.55 339.85 19.98 15.31 10.33 5.16 0 354.66 349.5 344.48 339.86 19.98 15.31 10.3 5.13 0 354.61 349.35 344.45 339.73 20.01 15.36 10.33 5.16 0 354.65 349.48 344.46 339.71 19.9 15.35 10.33 5.15 0 354.66 349.48 344.45 339.51

[0093] 1-4) Place the depth calibration plate at the marked sampling point locations, and simultaneously use a binocular camera to acquire images of those locations and calculate their disparity maps.

[0094] 1-5) Use 5 sampling points for depth calibration in each region, and another 10 sampling points for width calibration.

[0095] 1-6) Using MATLAB, draw a bounding box around the depth calibration plate position in the disparity map obtained in 1-4), and calculate the mean disparity within the bounding box area, such as... Figure 4As shown, the frame outlines the position of the left calibration plate, and the left toolbar displays information such as the maximum and minimum parallax, average parallax, and depth within the frame.

[0096] 1-7) Convert the distance dis from the sampling point to the origin and the offset angle θ of the sampling point relative to the origin obtained in 1-3) into the true depth value of the sampling point. The calculation principle is as follows: Figure 5 As shown:

[0097] Z ture =dis*cosθ

[0098] According to the right angle theorem shown in the figure, the true depth Z ture The distance dis from one point to another is the cosine of the offset angle θ.

[0099] 1-8) The obtained disparity mean and depth true values ​​are fitted using the least squares method to obtain the final depth calibration value.

[0100] a) Considering the error caused by the different principal points of the left and right cameras due to camera installation issues. According to Figure 6 The diagram showing the principle of binocular ranging can be used to derive the following relationship using the principle of triangular similarity:

[0101]

[0102] Simplifying, we get:

[0103]

[0104] Where u0 and u0' are the x-coordinates of the principal points of the left and right cameras, respectively. l and u r Here, B represents the x-coordinate of the imaging plane, B is the camera baseline, and f is the camera focal length.

[0105] Generally, binocular ranging algorithms assume that the principal points of the left and right cameras are equal, i.e., u0 = u0'. However, in practice, due to camera installation issues, the principal points of the left and right cameras may not be equal. Therefore, let:

[0106] (-u0+u0')dx=Δd

[0107] (u l -u r )dx=d

[0108] Therefore, the relationship between the depth of the observed target from the imaging plane and the parallax is as follows:

[0109]

[0110] Where the depth is Z, from the above formula:

[0111]

[0112] make

[0113] Bf = t1

[0114] Δd=t2

[0115] After substituting 5 pairs of parallax mean and depth true values, the following overdetermined equation system is obtained:

[0116]

[0117] make

[0118]

[0119]

[0120]

[0121] Then there is

[0122] AX = d

[0123] b) Solve the overdetermined system of equations using the least squares method.

[0124] make

[0125]

[0126] Right now

[0127] E(x) = ||AX - d|| 2

[0128] Solving

[0129]

[0130] X * This represents the optimal solution for t1 and t2 obtained after least squares fitting, which is also the optimal solution for Bf and Δd. Once Bf and Δd are obtained, the final depth calibration value Z can be calculated.

[0131] 2) Width calibration.

[0132] 2-1) Obtain the remaining 10 points for each region and use the least squares method to fit and solve for the width calibration value:

[0133] a) The relationship between width distance and parallax is:

[0134]

[0135] x is the actual width distance of the object, u is the image x-coordinate of the center of the region, d is the parallax, and C xThe x-coordinate of the optical center, f is the focal length, B is the baseline, and Δd is the error coefficient obtained in the depth calibration.

[0136] Let a1 = B, a2 = -BC x If a3 = -Δd, then the above expression becomes:

[0137]

[0138] After simplification, we get:

[0139] -ua1+xd+xa3=a2

[0140] Divide both sides of the above equation by a2, and let a'1 = a1 / a2, a'2 = 1 / a2, a'3 = a3 / a2, then:

[0141] -ua'1+xda'2+xa'3=1

[0142] Substituting the 10 sets of disparity mean and depth true values, we obtain the following overdetermined nonhomogeneous equation system:

[0143]

[0144] make

[0145]

[0146]

[0147]

[0148] b) Solve the above overdetermined system of equations using the least squares method.

[0149] make

[0150]

[0151] Right now

[0152] E(x) = ||Ax - y|| 2

[0153] Solving

[0154]

[0155] x * These are the optimal solutions for a'1, a'2, and a'3 obtained after least squares fitting, which yields the optimal solutions for coefficients a1, a2, and a3. The final width calibration value X can then be calculated using a1, a2, and a3.

Claims

1. A depth and width estimation method based on binocular stereo vision, characterized in that, The steps are as follows: 1) Depth calibration 1-1) Set the camera placement position as the origin, use a plumb bob to ensure that the camera imaging plane is perpendicular to the ground plane, and mark the origin position; 1-2) With the origin as the center, set the field of view angle of α degrees, divide the α-degree field of view angle into three equal parts. The area from -α / 2 to -α / 6 is the left area, the area from -α / 6 to α / 6 is the middle area, and the area from α / 6 to α / 2 is the right area; And set the radii of the areas as a meters and b meters respectively, where b < a. Name the area with a radius from 0 to b meters as the front area, and the area from b to a meters as the rear area. There are three areas on the left, middle, and right in both the front and rear areas, for a total of six areas; 1-3) Collect M sampling points in each area, and use a total station to record the distance dis from the sampling point to the origin and the offset angle θ of the sampling point relative to the origin, and mark the position of each sampling point; use M / 3 sampling points in each area for depth calibration, and the other 2M / 3 sampling points are used for width calibration; 1-4) Place the depth calibration board at the positions of the marked sampling points respectively, and at the same time use a binocular camera to collect the images at these positions and calculate the disparity map; 1-5) Use the MATLAB tool to draw a frame around the position of the depth calibration board in the disparity map obtained in 1-4), and calculate the mean value of the disparity within the framed area; 1-6) Convert the distance dis from the point of the sampling point to the origin and the offset angle θ of the sampling point relative to the origin obtained in 1-3) into the depth true value of the sampling point. The calculation formula is as follows: dis*cosθ; According to the right angle theorem, the true value of depth... The distance dis from one point to another is the cosine of the offset angle θ. 1-7) Use the least squares method to fit and calculate the obtained disparity mean value and depth true value data to get the final depth calibration value; the specific steps are as follows: a) Considering the error caused by the different principal points of the left and right cameras due to the camera installation problem, the following relationship is obtained using the principle of similar triangles: ； After simplification: ； in and These are the x-coordinates of the principal points of the left and right cameras, respectively. and Here, are the x-coordinates of the imaging plane, and B is the camera baseline. The focal length of the camera; Typical binocular ranging algorithms assume that the principal points of the left and right cameras are equal, i.e. = However, in objective practice, due to camera installation issues, the principal points of the left and right cameras are not equal, therefore: ； ； Therefore, the relationship between the depth of the observed target from the imaging plane and the disparity is: ； Where the depth is From the above equation, we get: ； Let ； ； After substituting M / 3 pairs of disparity mean value and depth true value data, the following overdetermined system of equations is obtained: ； Let ； ； ； Then there is ； b) Solve the overdetermined system of equations by the least squares method; Let ； That is ； The solution is ； That is, the result obtained after fitting using the least squares method. and The optimal solution, i.e. and The optimal solution is obtained; and Then the final depth calibration value Z can be calculated; 2) Width calibration; 2-1) Obtain the remaining 2M / 3 points in each area and use the least squares method to fit and solve for the width calibration value: a) The relationship between the width distance and the disparity is: ； The actual width distance of the object. Let d be the x-coordinate of the image center of this region, and d be the disparity. The x-coordinate of the optical center, B is the focal length, and B is the baseline. These are the error coefficients obtained during depth calibration; make =B, = , = Then the above expression becomes: ； After simplification: ； Divide both sides of the above equation ,make = / , =1 / , / ,but: ； Substitute 2M / 3 groups of disparity mean value and depth true value data to obtain the following overdetermined non-homogeneous system of equations: ； Let ； ； ； b) Solve the above overdetermined system of equations by the least squares method; Let ； That is ； The solution is ； That is, the result obtained after fitting using the least squares method. , and The optimal solution yields the coefficients. , and The optimal solution; through , and The final width calibration value can then be calculated. .

Images