Spatio-Temporal Stereo Matching Method Based on Multi-Frame Speckles

Abstract

The present invention discloses a spatio-temporal stereo matching method based on multi-frame speckles, which includes projecting multiple speckle patterns onto the scene to be measured by a projector, synchronously collecting multiple frames of speckle images by left and right cameras, calculating the initial matching cost, and obtaining the initial disparity map of the scene to be measured through the WTA method. The matching cost is calculated by using a single-pixel matching method based on normalized mutual information, and the aggregated matching cost is obtained by using a cost aggregation algorithm based on guided filtering. The sub-pixel disparity value is obtained by using a sub-pixel optimization algorithm based on fractional difference for the aggregated matching cost. The occluded regions and mismatched points are removed to obtain the dense disparity map of the scene to be measured, and then the high-precision three-dimensional contour data of the scene to be measured is reconstructed according to the relative relationship between the image coordinate system and the camera coordinate system. Through the spatio-temporal stereo matching process from coarse to fine, the present invention reconstructs the high-precision three-dimensional contour of the scene to be measured according to the relationship between the image coordinate system and the camera coordinate system.

Description

Technical Field

[0001] The invention belongs to the technical field of optical measurement, and in particular relates to a spatiotemporal stereo matching method based on multi-frame speckles. Background Art

[0002] In recent decades, fast three-dimensional shape measurement technology has been widely used in various fields, such as intelligent monitoring, industrial quality control and three-dimensional face recognition. Among the many three-dimensional shape measurement methods, speckle projection profilometry is widely used in various fields. Due to its fast and efficient characteristics, it is highly favored. However, due to the limitation of the single-frame matching algorithm, its measurement accuracy is low. In addition, for three-dimensional measurement based on single-frame speckle projection, there is still a lack of a high-performance stereo matching algorithm based on single-frame speckle images to achieve high robustness and high-precision three-dimensional measurement. In order to improve the three-dimensional measurement accuracy and robustness of speckle projection profilometry, multiple speckle patterns can be projected, and multiple frames of different speckle images can be collected at the same time. More constraint information can be used to ensure the global uniqueness of the measured scene. These speckle patterns have a fixed displacement in space, thereby encoding the depth information of the measured scene in time and space, and then combined with a more robust time and space stereo matching method to obtain a high-precision three-dimensional profile of the measured scene, thereby improving the three-dimensional measurement accuracy.

[0003] Stereo matching has always been a hot topic in the field of computer vision. It uses the information of homonymous points between images from different perspectives to estimate the depth of the scene, and plays a vital role in image-based three-dimensional shape measurement. Depth estimation is accomplished using the disparity between homonymous points of each pixel, that is, the apparent displacement of the target pixel from the left image (reference image) to the right image (target image). Based on the various parameters obtained by the prior system epipolar correction, the disparity can be used to obtain the three-dimensional position of the target point in the scene.

[0004] In the past two decades, a lot of research work has been invested in this field, and great progress has been made. In general, the existing stereo matching algorithms are divided into two categories: global stereo matching algorithms and local stereo matching algorithms. The global method uses the principle of global energy minimization, comprehensively considers the data term (Data Term) that represents the similarity of the pixels themselves and the smooth term (Smooth Term) that represents the smoothness of adjacent pixels, and transforms the problem into minimizing the global energy function; at the same time, it uses the graph cut algorithm (GC), the belief propagation algorithm (BP), the dynamic programming algorithm (DP) and the scan line optimal algorithm to solve it. The completeness and accuracy of the obtained disparity map are relatively high, but the matching efficiency is extremely low. The local method is based on the correlation measure of the window to find the same-name pixels, and independently estimate the disparity of each pixel by comparing the features of the left and right images (usually the window around the pixel) to generate a locally smooth disparity image; using methods such as bilateral filter (BF) and guided filter (GF), it can generate good results for complex texture areas. However, if the window is located in a smooth area (lack of texture or repeated patterns), it is easy to get wrong disparity results. Therefore, for the spatiotemporal stereo matching method based on multi-frame speckle, in order to achieve high-precision three-dimensional measurement, there is currently a lack of a more robust spatiotemporal stereo matching method. Summary of the invention

[0005] In order to solve the above technical defects in the prior art, the present invention proposes a spatiotemporal stereo matching method based on multi-frame speckles, which reconstructs a high-precision three-dimensional contour of the scene to be measured through a coarse-to-fine spatiotemporal stereo matching process.

[0006] The technical solution to achieve the purpose of the present invention is: a spatiotemporal stereo matching method based on multi-frame speckle, comprising the following steps:

[0007] Step 1: Use a projector to project multiple speckle patterns onto the scene to be tested. The left and right cameras synchronously collect multiple speckle image pairs. The initial matching cost is calculated using the ZNCC-based local matching method, and the initial disparity map of the scene to be tested is obtained using the WTA method.

[0008] Step 2: Use the single-pixel matching method based on normalized mutual information to calculate the matching cost, and use the initial disparity map obtained in step 1 to calculate a more robust matching cost;

[0009] Step 3: Use the guided filtering-based cost aggregation algorithm to obtain the aggregated matching cost;

[0010] Step 4: Use the sub-pixel optimization algorithm based on fractional difference on the aggregated matching cost to obtain the sub-pixel disparity map by fusing the linear and parabolic weighted interpolation functions;

[0011] Step 5: Use left-right consistency check and median filtering to remove occluded regions and mismatched points, obtain a dense disparity map of the scene to be measured, and reconstruct the three-dimensional contour of the scene to be measured according to the relationship between the image coordinate system and the camera coordinate system.

[0012] Preferably, in step 1, for a pair of multi-frame speckle images I L,n and I R,n , the initial matching cost C(p, d) between any point p(x, y) in the left image and the matching point q(x - d, y) in the right image is expressed as:

[0013]

[0014] where r is the radius of the matching window; N represents the total number of images; d is the candidate disparity value; and are respectively the differences between the image intensities of each point in the left and right view windows and the average image intensity obtained within the left and right view windows;

[0015] Use the WTA method to obtain the initial disparity map I d (p):

[0016]

[0017] In the formula, d min is the minimum disparity value, and d max is the maximum disparity value.

[0018] Preferably, in step 2, the specific formula for calculating the matching cost using the single-pixel matching method based on normalized mutual information is:

[0019]

[0020] where is the mutual information between images ; represents the intensity value at the corresponding point p in the left and right matching images ; and represent the entropies of the left and right images respectively; is the joint entropy between the left and right images.

[0021] Preferably, first use the initial disparity map I d obtained in step 1 to warp the right view to obtain Use the warped right image to match the left view Calculate the mutual information values of all points, and then match the position of the point in the right image before warping with the left image The matching cost C is obtained at the point position mi (p, d):

[0022]

[0023] Preferably, in step 3, optimizing the matching cost using the cost aggregation algorithm based on guided filtering is specifically as follows:

[0024] Three frames of consecutive speckle images collected from the left view are used as the guidance map I = (I 1 , I 2 , I 3 ) T , for each disparity d, the cost value C(i, d) output after guided filtering is the weighted average of all pixels within the window centered on C mi (j, d):

[0025]

[0026] where ω i , ω j is the filtering window with a radius of w, w ≥ 3 and w is odd; μ k , ∑ k are respectively the average value vector and the covariance matrix of I within the window ω i or ω j ; U is the 3×3 identity matrix; |w| represents the number of pixels in the window; ε is the smoothness parameter.

[0027] Preferably, in step 4, the integer disparity value d p = d int (p) at each point p is obtained from the aggregated matching cost through WTA;

[0028] The sub-pixel disparity map is obtained using the sub-pixel optimization algorithm based on fractional difference, specifically as follows::

[0029]

[0030] F(x) is the interpolation function.

[0031] Preferably, the interpolation function is specifically as follows:

[0032]

[0033] where 0 ≤ α ≤ 1 is the weight coefficient; x = C(p, d p - 1) - C(p, d p ) / C(p, d p + 1) - C(p, d p ), C(p, d p ) represents d pThe cost value at point p.

[0034] Preferably, in step 5, using the sub-pixel disparity map obtained in step 4, the occlusion regions and mismatched points are removed by using the left-right consistency check operation, and the dense disparity map of the scene to be measured is obtained by the method of median filtering. According to the relationship between the image coordinate system and the camera coordinate system, the high-precision three-dimensional contour of the scene to be measured is reconstructed, and the three-dimensional coordinate values (X c , Y c , Z c ) of the scene in the camera coordinate system are obtained:

[0035]

[0036] where norm(·) represents the modulo operation, T represents the translation vector between the left and right cameras, u0 and v0 are the principal point coordinate parameters of the camera, f x and f y are the focal length parameters of the camera in the x and y directions, and d(i,j) is the sub-pixel disparity value at point (i,j).

[0037] Compared with the prior art, the present invention has the following remarkable advantages: The present invention proposes a more robust, efficient and high-precision spatio-temporal stereo matching method. This method encodes the depth information of the scene to be measured spatio-temporally by continuously projecting multiple frames of speckle patterns onto the scene to be measured, and there is a fixed displacement between these speckle patterns in space. Then, a coarse-to-fine spatio-temporal stereo matching algorithm is adopted to reconstruct the high-precision three-dimensional contour of the scene to be measured from multiple frames of speckle images collected synchronously. In the present invention, a more robust normalized mutual information stereo matching method is proposed, and by using the initial disparity map obtained in the previous step, the iterative calculation process of the traditional mutual information stereo matching method is eliminated, and a more robust matching cost is obtained. And a weighted interpolation function that combines a parabola and a linear function is proposed, and a sub-pixel optimization algorithm based on fractional difference is used to alleviate the "pixel locking" effect, make the distribution of sub-pixel values more uniform, and obtain more accurate sub-pixel disparity values. Experiments prove that the accuracy of the three-dimensional measurement implemented based on this method is about 57um, thus realizing high-precision three-dimensional measurement.

[0038] Other features and advantages of the present invention will be described in the following specification, and some of them will become obvious from the specification, or can be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are only for the purpose of illustrating specific embodiments, and are not considered as limiting the present invention. Throughout the drawings, the same reference numerals represent the same components.

[0040] Figure 1 It is a flowchart of a spatio-temporal stereo matching method based on multi-frame speckle. Detailed implementation manners

[0041] It is easy to understand that according to the technical solution of the present invention, without changing the essence of the present invention, those of ordinary skill in the art can imagine various implementation manners of the present invention. Therefore, the following detailed implementation manners and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as all of the present invention or regarded as a limitation or restriction on the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention will be specifically described below in conjunction with the drawings, where the drawings form a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0042] The inventive concept of the present invention is a spatio-temporal stereo matching method based on multi-frame speckle, including the following five steps:

[0043] Step 1: Build a three-dimensional measurement system based on speckle projection. First, use an infrared speckle-coded projector to project a speckle pattern onto the measured scene for spatio-temporal coding, and the image acquisition device synchronously acquires a series of consecutive speckle images. For the series of acquired speckle images, an initial matching cost is calculated using a local matching method based on ZNCC. For the corrected stereo speckle image pair I L,n and I R,n , the initial matching cost C(x, y, d) between any point (x, y) in the left image and the matching point (x - d, y) in the right image is expressed as:

[0044]

[0045] where r is the radius of the matching window; N represents the total number of images; d is the candidate disparity; and are the average values obtained within the left and right view windows subtracted from each pixel point in the left and right view windows.

[0046] Then, use the WTA method to obtain the initial disparity map I d (x, y):

[0047]

[0048] Step 2: Obtain the matching cost based on single-pixel matching. Use a more robust single-pixel-based normalized mutual information stereo matching method:

[0049]

[0050] where, For image I L,n and I R,n the mutual information; u = I L,n (p), v = I R,n (p) represents the intensity value of point p in the image; and respectively represent the entropies of the two images; is the joint entropy between the two images.

[0051] Then use the initial disparity map I d obtained in step 1 R,n to warp the right view I R,n to obtain I', eliminating the iterative calculation process of the traditional mutual information stereo matching method, matching the left view I L,n calculate the matching cost C mi (p, d) based on mutual information to obtain a more robust matching cost:

[0052]

[0053] Step 3: Obtain the matching cost value after cost aggregation. Use the cost aggregation algorithm based on guided filtering to optimize the matching cost to reduce false matches while retaining the edge details of the object. Use three frames of the continuous multi-frame speckle images collected from the left view as the guidance map I = (I 1 , I 2 , I 3 ). T For each disparity d, the cost value C(i, d) output after guided filtering is the weighted average of all pixels within the window centered on C mi (j, d):

[0054]

[0055] where ω i , ω j is the filtering window with a radius of w (w ≥ 3 and odd); and are the average value vector and covariance matrix of I within the window ω i or ω j ; U is the identity matrix of size 3×3; |w| represents the number of pixels in the window; ε is the smoothness parameter.

[0056] Step 4: Obtain the sub-pixel disparity value of the scene to be measured. After obtaining the integer disparity value d p = d int (p) at each point p by WTA for the aggregated matching cost, use a sub-pixel optimization algorithm based on fractional difference, which uses interpolation function F(x) fitting to achieve:

[0057]

[0058] By means of a weighted interpolation function that combines a parabola and a linear function, the "pixel locking" effect is alleviated, and the distribution of sub-pixel values becomes more uniform:

[0059]

[0060] where 0 ≤ α ≤ 1 is the weight coefficient; x = C(p, d p -1) - C(p, d p ) / C(p, d p +1) - C(p, d p ), and C(p, d p ) represents the cost value of point p at d p .

[0061] Step 5: Reconstruct the high-precision three-dimensional contour of the scene to be measured. Using the sub-pixel disparity map obtained in Step 4, perform left-right consistency check operations to remove occluded areas and mismatched points, and use median filtering to obtain a dense disparity map of the scene to be measured. According to the relationship between the image coordinate system and the camera coordinate system, reconstruct the high-precision three-dimensional contour of the scene to be measured to obtain the (X c , Y c , Z c ) coordinate values:

[0062]

[0063] where norm(·) represents the modulo operation, T represents the translation vector between the left and right cameras, u0 and v0 are the principal point coordinate parameters of the camera, f x and f y are the focal length parameters of the camera in the x and y directions, and d(i, j) is the sub-pixel disparity value at point (i, j).

[0064] As described above, the above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0065] It should be understood that in order to streamline the present invention and help those skilled in the art understand various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as meaning that the features included in the exemplary embodiments are all essential technical features of the patent claims of the present invention.

[0066] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be arranged in a device different from that of this embodiment. Different modules, units or components included in the device of the embodiment can be combined into one module, unit or component, or they can be divided into multiple sub-modules, sub-units or sub-components.

Claims

1. A spatio-temporal stereo matching method based on multi-frame speckle, characterized in that It includes the following steps: Step 1: Use a projector to project multiple speckle patterns onto the scene to be measured. The left and right cameras synchronously collect multiple frames of speckle image pairs. Calculate the initial matching cost using the local matching method based on ZNCC, and obtain the initial disparity map of the scene to be measured through the WTA method; Step 2: Calculate the matching cost using the single-pixel matching method based on normalized mutual information. The specific formula is: wherein, is the mutual information between images; between images; represents the intensity value at the corresponding point p in the left and right matching images; in the left and right matching images; and represent the entropies of the left and right images respectively; is the joint entropy between the left and right images; And using the initial disparity map obtained in step 1, a more robust matching cost is calculated. The specific process is as follows: First, use the initial disparity map I obtained in step 1 d to warp the right view to obtain Use the warped right image to match the left view Calculate the mutual information values of all points, and then match the right image before warping The point position and the left image The point position to obtain the matching cost C mi (p, d): Step 3: Use the cost aggregation algorithm based on guided filtering to obtain the aggregated matching cost; Step 4: Use the sub-pixel optimization algorithm based on score difference for the aggregated matching cost. By fusing the weighted interpolation functions of linear and parabola, obtain the sub-pixel disparity map, specifically: The aggregated matching cost is passed through WTA to obtain the integer disparity value d at each point p p = d int (p); Obtain the sub-pixel disparity map using the sub-pixel optimization algorithm based on score difference, specifically: F(x) is the interpolation function, specifically: Among them, 0 ≤ α ≤ 1 is the weight coefficient; x = C(p, d p -1) - C(p, d p ) / C(p, d p +1) - C(p, d p ), C(p, d p ) represents the cost value of point p at d p ; Step 5: Use left-right consistency check and median filtering to remove the occluded regions and mismatched points, obtain the dense disparity map of the scene to be measured, and reconstruct the three-dimensional contour of the scene to be measured according to the relationship between the image coordinate system and the camera coordinate system.

2. The spatio-temporal stereo matching method based on multi-frame speckles according to claim 1, wherein In step 1, for a multi-frame speckle image pair I L,n and I R,n , the initial matching cost C(p, d) between any point p(x, y) in the left image and the matching point q(x - d, y) in the right image is expressed as: Among them, r is the radius of the matching window; N represents the total number of images; d is the candidate disparity; and are respectively the differences between the image intensities of each point in the left and right view windows and the average image intensity obtained within the left and right view windows; Obtain the initial disparity map I using the WTA method d (p): where d min is the minimum parallax, and d max is the maximum parallax.

3. The spatio-temporal stereo matching method based on multi-frame speckles according to claim 1, wherein In Step 3, optimizing the matching cost using the cost aggregation algorithm based on guided filtering is specifically: Take three frames out of the consecutive multiple frames of speckle images collected from the left view as the guiding map I = (I 1 , I 2 , I 3 ). T , for each disparity d, the cost value C(i, d) output after guided filtering is the weighted average of all pixels within the window centered on C mi (j, d): where, ω i , ω j is a filtering window with a radius of w, where w ≥ 3 and is odd; μ k , ∑ k are the mean vector and covariance matrix of I in the window ω i or ω j respectively; U is a 3×3 identity matrix; |w| represents the number of pixels in the window; and ε is the smoothness parameter.

4. The spatio-temporal stereo matching method based on multi-frame speckles according to claim 1, characterized in that In step 5, using the sub-pixel disparity map obtained in step 4, the occlusion regions and mismatched points are removed by the left-right consistency check operation, and the dense disparity map of the scene to be measured is obtained by the method of median filtering. According to the relationship between the image coordinate system and the camera coordinate system, the high-precision three-dimensional contour of the scene to be measured is reconstructed, and the three-dimensional coordinate values (X c , Y c , Z c ) of the scene in the camera coordinate system are obtained: Among them, norm(·) represents the modulo operation, T represents the translation vector between the left and right cameras, u0 and v0 are the coordinate parameters of the principal points of the cameras, and f x and f y are the focal length parameters in the x and y directions of the camera, and d(i, j) is the sub-pixel disparity at the point (i, j).

Images