Multi-view stereo matching optimization method based on multi-view geometry

Abstract

The invention discloses a multi-view stereo matching optimization method based on multi-view geometry. The method comprises the following steps: (1) acquiring wide baseline images of three frames of target areas and camera positioning data of the wide baseline images by using a monocular vision inertial positioning system; (2) carrying out stereo correction on the acquired three-frame wide baseline image by adopting a multi-view geometric method; (3) performing depth estimation on the corrected three frames of images by adopting an SGM algorithm; (4) calculating geometric constraints among the depth maps by adopting a multi-view geometric algorithm; and (5) a multi-view stereo matching optimization algorithm based on multi-view geometric constraints and an improved SGM algorithm is adopted, and the problem of close-shot shielding existing in wide-baseline stereo matching is improved while wide-baseline stereo matching large-range region high-precision measurement is ensured. The method can be used for solving the problem of serious close-range occlusion in a wide-baseline stereo matching algorithm, the accuracy of wide-baseline depth estimation is improved, and accurate surveying and mapping of large scenes with close-range occlusion such as runways are facilitated.

Description

Technical Field

[0001] The invention relates to a multi-view geometry-based multi-stereo matching optimization method, belonging to the technical field of industrial automatic measurement. Background Art

[0002] Stereo matching is to obtain the depth information of objects in the scene by using two cameras to capture images of the same scene. The basic principle is to use the parallax of the left and right cameras, that is, the position difference of the same object in the left and right images, to calculate the distance from the object to the camera. Generally, in three-dimensional measurement projects of large areas, the difference in the long-distance area in the binocular image is difficult to identify, resulting in a large error in the final measured three-dimensional structure. Therefore, wide-baseline binocular stereo vision enhances the parallax effect of the binocular camera and improves the measurement accuracy of long distances by constructing a virtual baseline that is much larger than the baseline of the binocular camera itself. However, wide-baseline binocular images cannot handle the problem of close-range occlusion, resulting in the expansion of the invalid depth point area near the close-range occluder in the depth map.

[0003] Wang Jie et al. proposed a stereo matching optimization method based on multi-cameras. They introduced an intermediate camera between the left and right cameras, and constrained the stereo matching of the left and right cameras through a fixed geometric relationship between the cameras to solve the occlusion problem of fixed baseline stereo matching. However, the geometric constraint of the fixed baseline is not suitable for wide baseline stereo matching and cannot solve the occlusion problem in the wide baseline case. Summary of the invention

[0004] To solve the above problems, the present invention provides a multi-view stereo matching optimization method based on multi-view geometry. The method has high measurement performance when measuring a large area, significantly reduces the occlusion effect of foreground occluders, can simultaneously ensure the measurement accuracy of near and far target areas, and has a certain robustness.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A multi-view stereo matching optimization method based on multi-view geometry comprises the following steps:

[0007] Step 1: Use a monocular visual inertial positioning system to collect three frames of wide baseline images and camera positioning data of the target area;

[0008] Step 2: Use multi-view geometry to estimate the pose of the camera corresponding to the three wide baseline images, and perform image correction based on the pose;

[0009] Step 3: Use the SGM algorithm to perform stereo matching on the rectified image pair, and use the camera positioning data to convert the disparity map into a depth map;

[0010] Step 4: Use multi-view geometry algorithm to calculate the geometric constraints between depth maps;

[0011] Step 5: Use a multi-view stereo matching optimization algorithm based on multi-view geometric constraints and an improved SGM algorithm to improve close-range occlusion during wide baseline stereo matching.

[0012] Furthermore, the step 1 specifically includes the following steps:

[0013] Step 1.1: Filter out the three-frame wide baseline image data combination that satisfies the forward intersection of all camera poses and target areas from the multi-frame data collected by the monocular visual inertial system;

[0014] Step 1.2: Filter out a three-frame wide baseline image data combination with a suitable baseline length from the three-frame wide baseline image data combination.

[0015] Furthermore, the step 2 specifically includes the following steps:

[0016] Step 2.1: Use bundle adjustment to estimate the pose of three frames of wide baseline images;

[0017] Step 2.2: Correct the image using a stereo correction method based on Euclidean geometry.

[0018] Furthermore, in step 2.1, the SIFT algorithm is used to extract image features, and feature matching is performed between the three frames of wide baseline images based on the nearest neighbor and next nearest neighbor distance ratio threshold of the SIFT feature descriptor, and then the RANSAC algorithm and epipolar constraints are used to remove mismatched features; based on the feature matching relationship, the camera pose of the first frame of wide baseline image is used as the reference coordinate system, and the PnP algorithm is used to calculate the relative pose between the cameras corresponding to the three frames of wide baseline images; and the bundle adjustment is used to optimize the relative pose between the cameras.

[0019] Furthermore, in the step 2.2, the three frames of wide baseline images are divided into narrow baseline image pairs and wide baseline image pairs. Assuming that the camera position of the three frames of wide baseline images moves from left to right, the three frames of wide baseline images are respectively recorded as I1, I2 and I3; the first frame of wide baseline image I1 and the second frame of wide baseline image I2 from the left constitute a narrow baseline image pair, and the first frame of wide baseline image I1 and the third frame of wide baseline image I3 from the left constitute a wide baseline image pair; stereo correction is performed on the narrow baseline image pair and the wide baseline image pair, respectively.

[0020] Furthermore, the step 3 specifically includes the following steps:

[0021] Step 3.1: Use the SGM algorithm to perform stereo matching on the wide and narrow baseline image pairs respectively to obtain a wide baseline disparity map and a narrow baseline disparity map;

[0022] Step 3.2: Use the positioning data to calculate the true baseline length of the wide and narrow baseline image pair, and convert the disparity map into a depth map using the following formula:

[0023]

[0024] Wherein, z represents the corrected pixel depth; B represents the baseline length of the image pair; f represents the focal length of the camera; and d represents the disparity of the pixel.

[0025] Furthermore, the step 4 specifically includes the following steps:

[0026] Step 4.1: Use the pinhole camera model to convert the corrected narrow baseline depth map into a point cloud in the reference coordinate system using the following formula:

[0027]

[0028] Among them, K nl is the left camera intrinsic reference in the corrected narrow baseline image pair; R nl is the rotation matrix of the left camera in the corrected narrow baseline image pair in the reference coordinate system; z1 is the depth of the pixel in the corrected narrow baseline depth map; (u, v) is the pixel coordinate of the pixel in the corrected narrow baseline image; (x c ,y c ,z c ) is the point cloud coordinate of the point in the corrected narrow baseline depth map in the reference coordinate system;

[0029] Step 4.2: Project the point cloud coordinates of the narrow baseline into the wide baseline depth map to obtain the geometric constraints between the depth maps, which are expressed by the following formula:

[0030]

[0031] Among them, K wl is the left camera intrinsic reference in the calibrated wide baseline image pair; R wl is the rotation matrix of the left camera in the corrected wide baseline image pair in the reference coordinate system; z2 is the depth of the pixel in the corrected wide baseline depth map, and (u', v') is the pixel coordinate of the pixel in the corrected wide baseline image.

[0032] Furthermore, the step 5 specifically includes the following steps:

[0033] Step 5.1: Calculate the pixel coordinates of the occluded points in the wide baseline depth map;

[0034] Step 5.2: Use the geometric constraints obtained in step 4.2 to limit the disparity search range of the occluded area in the wide baseline image pair, and use the following formula to perform dynamic range disparity search:

[0035]

[0036] Where r is the disparity search radius, which can be adjusted within a certain range; disparity is the disparity value of the wide baseline image;

[0037] Step 5.3: In the matching cost stage, the disparity cost value within the constraint radius is calculated through Census transformation; in the cost aggregation stage, the cost values ​​of the occluded area and the non-occluded area are fused using 8-path aggregation; in the disparity optimization stage, the winner-takes-all (WTA) strategy is adopted.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention screens the collected images and positioning data to ensure that the camera posture and the target area meet the forward intersection; uses a multi-view geometry method to correct the image to ensure the accuracy of stereo correction; uses a visual measurement method based on a wide baseline to enhance the parallax effect of the target area and improve the measurement accuracy of the long-distance target area.

[0040] 2. The present invention utilizes the depth information of the narrow-baseline binocular to constrain the measurement results of the wide-baseline binocular in the occluded area, significantly improving the serious problem of foreground occlusion effect in wide-baseline measurement, and realizing accurate measurement of foreground occlusion areas, which is beneficial to accurate mapping of large scenes with foreground occlusion such as runways. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is an overall flow chart of the multi-view stereo matching optimization method based on multi-view geometry provided by the present invention;

[0042] Figure 2 It is a set of three frames of wide baseline image data containing the target area to be measured;

[0043] Figure 3 is a set of rectified wide baseline image pairs and narrow baseline image pairs;

[0044] Figure 4 is the corrected wide baseline depth map;

[0045] Figure 5 is the optimized wide baseline depth map. DETAILED DESCRIPTION

[0046] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0047] The multi-view geometry-based multi-stereo matching optimization method provided by the present invention has the following overall process: Figure 1As shown, the specific steps include:

[0048] Step 1: Use a monocular visual inertial positioning system to collect wide baseline images and camera positioning data of the target area, and obtain Figure 2 , the specific steps are as follows:

[0049] Step 1.1: Filter out the three-frame wide baseline image data combination that satisfies the forward intersection of all camera poses and target areas from the multi-frame data collected by the monocular visual inertial system;

[0050] Step 1.2: Filter out a three-frame wide baseline image data combination with a suitable baseline length from the three-frame wide baseline image data combination.

[0051] Step 2: Use the multi-view geometry method to estimate the pose of the camera corresponding to the three frames of wide baseline images, and perform image correction based on the pose. The specific steps are as follows:

[0052] Step 2.1: Use the SIFT algorithm to extract image features, and perform feature matching between the three wide baseline images based on the nearest neighbor and next nearest neighbor distance ratio threshold of the SIFT feature descriptor. Then use the RANSAC algorithm and epipolar constraints to remove mismatched features. According to the feature matching relationship, use the camera pose of the first wide baseline image as the reference coordinate system, and use the PnP algorithm to calculate the relative poses between the cameras corresponding to the three wide baseline images. Use the bundle adjustment to optimize the relative poses between the cameras.

[0053] Step 2.2: Divide the three frames of wide baseline images into narrow baseline image pairs and wide baseline image pairs. Assume that the camera position of the three frames of wide baseline images moves from left to right. The three frames of wide baseline images are recorded as I1, I2 and I3 respectively; the first frame of wide baseline image I1 and the second frame of wide baseline image I2 from the left constitute a narrow baseline image pair, and the first frame of wide baseline image I1 and the third frame of wide baseline image I3 from the left constitute a wide baseline image pair; perform stereo correction on the narrow baseline image pair and the wide baseline image pair respectively.

[0054] The stereo-corrected wide baseline image pair and narrow baseline image pair obtained in this step are as follows: Figure 3 shown.

[0055] Step 3: Use the SGM algorithm to perform stereo matching on the rectified image pair, and use the camera positioning data to convert the disparity map into a depth map. The specific steps are as follows:

[0056] Step 3.1: Use the SGM algorithm to perform stereo matching on the wide and narrow baseline image pairs respectively to obtain a wide baseline disparity map and a narrow baseline disparity map;

[0057] Step 3.2: Use the positioning data to calculate the true baseline length of the wide and narrow baseline image pair, and convert the disparity map into a depth map using the following formula:

[0058]

[0059] Wherein, z represents the corrected pixel depth; B represents the baseline length of the image pair; f represents the focal length of the camera; and d represents the disparity of the pixel.

[0060] The stereo-corrected wide baseline depth map and narrow baseline depth map obtained in this step are as follows: Figure 4 shown.

[0061] Step 4: Use the multi-view geometry algorithm to calculate the geometric constraints between the depth maps. The specific steps are as follows:

[0062] Step 4.1: Use the pinhole camera model to convert the corrected narrow baseline depth map into a point cloud in the reference coordinate system using the following formula:

[0063]

[0064] Among them, K nl is the left camera intrinsic reference in the corrected narrow baseline image pair; R nl is the rotation matrix of the left camera in the corrected narrow baseline image pair in the reference coordinate system; z1 is the depth of the pixel in the corrected narrow baseline depth map; (u, v) is the pixel coordinate of the pixel in the corrected narrow baseline image; (x c ,y c ,z c ) is the point cloud coordinate of the point in the corrected narrow baseline depth map in the reference coordinate system;

[0065] Step 4.2: Project the point cloud coordinates of the narrow baseline into the wide baseline depth map to obtain the geometric constraints between the depth maps, which are expressed by the following formula:

[0066]

[0067] Among them, K wl is the left camera intrinsic reference in the calibrated wide baseline image pair; R wl is the rotation matrix of the left camera in the corrected wide baseline image pair in the reference coordinate system; z2 is the depth of the pixel in the corrected wide baseline depth map, and (u', v') is the pixel coordinate of the pixel in the corrected wide baseline image.

[0068] Step 5: Use a multi-view stereo matching optimization algorithm based on multi-view geometric constraints and an improved SGM algorithm. The specific steps are as follows:

[0069] Step 5.1: Calculate the pixel coordinates of the occluded points in the wide baseline depth map;

[0070] Step 5.2: Use the geometric constraints obtained in step 4.2 to limit the disparity search range of the occluded area in the wide baseline image pair, and use the following formula to perform dynamic range disparity search:

[0071]

[0072] Where r is the disparity search radius, which can be adjusted within a certain range; disparity is the disparity value of the wide baseline image;

[0073] Step 5.3: In the matching cost stage, the disparity cost value within the constraint radius is calculated through Census transformation; in the cost aggregation stage, the cost values ​​of the occluded area and the non-occluded area are fused using 8-path aggregation; in the disparity optimization stage, the winner-takes-all (WTA) strategy is adopted.

[0074] The optimized wide baseline depth map obtained in this step is as follows Figure 5 shown.

[0075] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical solutions composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A multi-view stereo matching optimization method based on multi-view geometry, characterized in that: The following steps are involved: Step 1: Use a monocular visual inertial positioning system to collect three frames of wide baseline images and camera positioning data of the target area; Step 2: Use multi-view geometry to estimate the pose of the camera corresponding to the three wide baseline images, and perform image correction based on the pose; Step 3: Use the SGM algorithm to perform stereo matching on the rectified image pair, and use the camera positioning data to convert the disparity map into a depth map; Step 4: Use multi-view geometry algorithm to calculate the geometric constraints between depth maps; Step 5: Use the above method to improve the near-field occlusion during wide baseline stereo matching.

2. The multi-view stereo matching optimization method based on multi-view geometry according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Filter out the three-frame wide baseline image data combination that satisfies the forward intersection of all camera poses and target areas from the multi-frame data collected by the monocular visual inertial system; Step 1.2: Filter out a three-frame wide baseline image data combination with a suitable baseline length from the three-frame wide baseline image data combination.

3. The multi-view stereo matching optimization method based on multi-view geometry according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Use bundle adjustment to estimate the pose of three frames of wide baseline images; Step 2.2: Correct the image using a stereo correction method based on Euclidean geometry.

4. The multi-view stereo matching optimization method based on multi-view geometry according to claim 3, characterized in that: The process of step 2.1 includes: using the SIFT algorithm to extract image features, and performing feature matching between the three frames of wide baseline images based on the nearest neighbor and next nearest neighbor distance ratio threshold of the SIFT feature descriptor, and then using the RANSAC algorithm and epipolar constraints to remove mismatched features; according to the feature matching relationship, using the camera pose of the first frame of wide baseline image as the reference coordinate system, using the PnP algorithm to calculate the relative pose between the cameras corresponding to the three frames of wide baseline images; and using the bundle adjustment method to optimize the relative pose between the cameras.

5. The multi-view stereo matching optimization method based on multi-view geometry according to claim 3, characterized in that: The process of step 2.2 includes: dividing the three frames of wide baseline images into narrow baseline image pairs and wide baseline image pairs, assuming that the camera position of the three frames of wide baseline images moves from left to right, the three frames of wide baseline images are respectively recorded as I1, I2 and I3; the first frame of wide baseline image I1 and the second frame of wide baseline image I2 from the left constitute a narrow baseline image pair, and the first frame of wide baseline image I1 and the third frame of wide baseline image I3 from the left constitute a wide baseline image pair; stereo correction is performed on the narrow baseline image pair and the wide baseline image pair respectively.

6. The multi-view stereo matching optimization method based on multi-view geometry according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3.1: Use the SGM algorithm to perform stereo matching on the wide and narrow baseline image pairs respectively to obtain a wide baseline disparity map and a narrow baseline disparity map; Step 3.2: Use the positioning data to calculate the true baseline length of the wide and narrow baseline image pair, and convert the disparity map into a depth map using the following formula: Wherein, z represents the corrected pixel depth; B represents the baseline length of the image pair; f represents the focal length of the camera; and d represents the disparity of the pixel.

7. The multi-view stereo matching optimization method based on multi-view geometry according to claim 6, characterized in that: The step 4 specifically includes the following steps: Step 4.1: Use the pinhole camera model to convert the corrected narrow baseline depth map into a point cloud in the reference coordinate system using the following formula: Among them, K nl is the left camera intrinsic reference in the corrected narrow baseline image pair; R nl is the rotation matrix of the left camera in the corrected narrow baseline image pair in the reference coordinate system; z1 is the depth of the pixel in the corrected narrow baseline depth map; (u, v) is the pixel coordinate of the pixel in the corrected narrow baseline image; (x c ,y c ,z c ) is the point cloud coordinate of the point in the corrected narrow baseline depth map in the reference coordinate system; Step 4.2: Project the point cloud coordinates of the narrow baseline into the wide baseline depth map to obtain the geometric constraints between the depth maps, which are expressed by the following formula: Among them, K wl is the left camera intrinsic reference in the calibrated wide baseline image pair; R wl is the rotation matrix of the left camera in the corrected wide baseline image pair in the reference coordinate system; z2 is the depth of the pixel in the corrected wide baseline depth map, and (u', v') is the pixel coordinate of the pixel in the corrected wide baseline image.

8. The multi-view stereo matching optimization method based on multi-view geometry according to claim 7, characterized in that: The step 5 specifically includes the following steps: Step 5.1: Calculate the pixel coordinates of the occluded points in the wide baseline depth map; Step 5.2: Use the geometric constraints obtained in step 4.2 to limit the disparity search range of the occluded area in the wide baseline image pair, and use the following formula to perform dynamic range disparity search: Where r is the disparity search radius, which can be adjusted within a certain range; disparity is the disparity value of the wide baseline image; Step 5.3: In the matching cost stage, the disparity cost within the constraint radius is calculated through Census transformation; in the cost aggregation stage, 8-path aggregation is used to fuse the cost values ​​of the occluded area and the non-occluded area; in the disparity optimization stage, the winner-takes-all WTA strategy is adopted.

Images