A spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array

Abstract

The application discloses a kind of space-time speckle projection three-dimensional imaging methods based on VCSEL projection array.First, integrate a group of small speckle projection module based on VCSEL to the measured scene projection space-time speckle pattern, binocular camera synchronous acquisition stereo speckle image.Stereo correction is carried out to the collected space-time speckle image using the calibration parameters of binocular camera.Process space-time speckle image using space-time matching algorithm based on Census transformation, and obtain the initial disparity map of the measured scene.Improve the measurement accuracy and spatial resolution of traditional space-time matching method using sub-pixel space-time matching algorithm based on stereo digital image correlation, to restore the fine profile of measured object.The application uses space-time matching algorithm based on Census transformation and sub-pixel space-time matching algorithm based on stereo digital image correlation to improve the measurement accuracy and spatial resolution of traditional space-time speckle matching method, and realizes high-resolution, high-precision three-dimensional imaging.

Description

Technical Field

[0001] This invention belongs to the field of optical measurement technology, specifically a spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array. Background Technology

[0002] Structured light projection profilometry, particularly fringe projection profilometry (FPP), is a mainstream high-precision non-contact 3D imaging technique widely used in manufacturing, basic research, and engineering applications. However, FPP methods are limited by complex and costly high-resolution spatial light modulation devices, posing a significant challenge to miniaturized and low-cost 3D imaging applications. On the other hand, speckle projection profilometry (SPP) employs highly integrated speckle projection devices based on vertical-cavity surface-emitting lasers (VCSELs). By projecting a single frame of speckle patterns, it can achieve rapid 3D reconstruction, opening new avenues for miniaturized mobile depth sensing applications such as scene reconstruction and face recognition. However, due to the poor performance of single-frame speckle matching and the complex reflection characteristics of the measured surface, SPP methods can only produce coarse 3D measurement results with low precision and low resolution. Summary of the Invention

[0003] The purpose of this invention is to propose a spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array.

[0004] The technical solution to achieve the objective of this invention is as follows: a spatiotemporal speckle projection three-dimensional imaging method based on a VCSEL projection array, comprising the following steps:

[0005] Step 1: Integrate a set of miniaturized speckle projection modules based on VCSEL to project spatiotemporal speckle patterns onto the scene under test, and simultaneously acquire stereo speckle images with a binocular camera.

[0006] Step 2: Use the calibration parameters of the binocular camera to perform stereo correction on the acquired spatiotemporal speckle image, and use the spatiotemporal matching algorithm based on Census transform to process the spatiotemporal speckle image to obtain the initial disparity map of the scene under test.

[0007] Step 3: Use a subpixel spatiotemporal matching algorithm based on stereo digital image correlation to determine the optimized subpixel matching result and recover the fine contour of the tested scene.

[0008] Preferably, the specific method for processing the spatiotemporal speckle image using a Census transform-based spatiotemporal matching algorithm to obtain the initial disparity map of the tested scene is as follows:

[0009] The Census transform based on a local spatiotemporal window is used to extract features from each pixel of the spatiotemporal speckle image.

[0010] According to the system's preset parallax range [D minD max The matching cost Cost(x,y,d) is obtained by calculating the Hamming distance between the feature vector of each pixel in the left image and the feature vectors of all candidate pixels in the right image.

[0011]

[0012] in, It is an XOR operation. BC(·) is used to count the number of "1"s in the XOR result, and d is any candidate disparity within the disparity range.

[0013] The initial disparity map D(x,y) is obtained by calculating using the winner-takes-all algorithm:

[0014]

[0015] Sub-pixel optimization is performed using nearest neighbor interpolation to obtain the optimized disparity map D. sub (x, y), the calculation process is as follows:

[0016]

[0017] Cost 1 (D(x,y)+1)=Cost(x,y,D(x,y)+1).

[0018] Preferably, the specific process of feature extraction is as follows:

[0019]

[0020]

[0021]

[0022] Where C(x,y) is the feature vector of the center pixel (x,y) of the speckle image. Indicates bitwise concatenation operation, I m (x,y) is the average light intensity of the local spatiotemporal window, and R and N are the spatial window radius and time length of the spatiotemporal matching, respectively.

[0023] Preferably, the matching cost Cost(x,y,d) is obtained by calculating the Hamming distance between the feature vector of each pixel in the left image and the feature vectors of all candidate pixels in the right image.

[0024]

[0025] in, It is an XOR operation. BC(·) is used to count the number of "1"s in the XOR result. R and N are the spatial window radius and time length of the spatiotemporal matching, respectively.

[0026] Preferably, the optimized sub-pixel matching result is determined using a sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation, and the fine contour of the tested scene is recovered, specifically as follows:

[0027] Step 3.1: Use the second-order shape function W(ξ;T) to describe the perspective transformation between the left camera reference subset centered at point p and the right camera target subset centered at point q:

[0028]

[0029]

[0030]

[0031] Where ξ = [x, y] T Represents the local coordinates within the subset, and T represents the deformation parameter vector [u, u] of the target window. x ,u y ,u xx ,u yy ,u xy ,v,v x ,v y ,v xx ,v yy ,v xy ] T (u,v) represents the sub-pixel matching result output by the sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation; u,u x ,u y Let v and v represent the horizontal displacement value and its partial derivatives in the horizontal and vertical directions, respectively. x ,v y U represents the vertical displacement value and its partial derivatives in the horizontal and vertical directions, respectively. xx ,u xy Representing u x The partial derivatives in the horizontal and vertical directions, v xx ,v xy They represent v respectively x The partial derivatives in the horizontal and vertical directions, u yy Represents u y The partial derivative in the vertical direction, v yy Represents v y The partial derivative in the vertical direction, (x p y p () represents the coordinates of point p;

[0032] Step 3.2: Combine the spatiotemporal ZNSSD method based on local spatiotemporal windows with a second-order shape function to quantitatively evaluate the similarity between the left and right spatiotemporal subsets:

[0033]

[0034]

[0035]

[0036] in, This represents the t-th speckle image captured by the left camera. Let represent the t-th speckle image captured by the right camera, and N represent the total number of speckle images captured by the camera; x represents the coordinates of the point to be matched in the left image. and The average intensity values ​​of the left and right spatiotemporal subsets are represented; W(ξ; ΔT) is the incremental deformation function of the left camera reference subset, and ΔT is the incremental deformation parameter vector.

[0037]

[0038] Where, [Δu,Δu x ,Δu y ,Δu xx ,Δu yy ,Δu xy ,Δv,Δv x ,Δv y ,Δv xx ,Δv yy ,Δv xy ] T is [u,u x ,u y ,u xx ,u yy ,u xy ,v,v x ,v y ,v xx ,v yy ,v xy ] T The incremental parameters are obtained by performing a first-order Taylor expansion of the vector with respect to ΔT:

[0039]

[0040] in, It is the gradient of the reference subset;

[0041] Solve for ΔT using the least squares method to minimize C ZNSSD (ΔT), that is

[0042]

[0043]

[0044] Step 3.3: Based on the calculated deformation parameter vector ΔT, update the second-order shape function W(ξ;T) of the target subset using the incremental shape function W(ξ;T) of the reference subset:

[0045] W(ξ;T)←W(ξ;T)W -1 (ξ;ΔT)

[0046] Step 3.4: Return to step 3.1 until ||ΔT||2≤0.001, stop the iterative calculation, and obtain the sub-pixel matching result (u,v) output by the sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation.

[0047] Preferably, the initial values ​​of W(ξ;T) are determined using the matching results obtained from the spatiotemporal matching method based on Census transform, according to the surface fitting algorithm:

[0048] p + W(ξ;T)=p + ξ - D sub (p+ξ)

[0049]

[0050] Among them, (a1,a2,a3,a4,a5,a6) and (b1,b2,b3,b4,b5,b6) are set to (u xx / 2,u yy / 2,u xy ,u x ,u y ,u) and (v xx / 2,v yy / 2,v xy ,v x ,v y ,v).

[0051] Compared with the prior art, the significant advantages of this invention are as follows: This invention uses a set of miniaturized speckle projection devices based on VCSEL to project spatiotemporal speckle patterns onto the scene under test. It improves the measurement accuracy and spatial resolution of traditional spatiotemporal speckle matching methods by using spatiotemporal matching algorithms based on Census transform and subpixel spatiotemporal matching algorithms based on stereo digital image correlation, thereby achieving high-resolution, high-precision three-dimensional imaging.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of a spatiotemporal speckle projection three-dimensional imaging method based on a VCSEL projection array.

[0054] Figure 2This is a basic schematic diagram of the spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array of the present invention. Detailed Implementation

[0055] A spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array includes the following steps:

[0056] Step 1: Integrate a set of miniaturized speckle projection modules based on VCSEL to project spatiotemporal speckle patterns onto the scene under test, and simultaneously acquire stereo speckle images with a binocular camera.

[0057] This step uses a miniaturized VCSEL-based speckle projection module to project speckle patterns. One miniaturized VCSEL-based speckle projection module can project one speckle pattern. This step integrates a set of 10 miniaturized VCSEL-based speckle projection modules in hardware, so 10 speckle patterns can be projected continuously onto the scene under test.

[0058] Step 2: Use the calibration parameters of the binocular camera to perform stereo correction on the acquired spatiotemporal speckle image, and use the spatiotemporal matching algorithm based on Census transform to process the spatiotemporal speckle image to obtain the initial disparity map of the scene under test.

[0059] Step 3: Improve speckle matching accuracy by using a subpixel spatiotemporal matching algorithm based on stereo digital image correlation, obtain optimized subpixel matching results, improve measurement accuracy and spatial resolution, and thus recover the fine contour of the measured object.

[0060] Preferably, step 2 uses the calibration parameters of the binocular camera to perform stereo correction on the acquired spatiotemporal speckle image. The spatiotemporal speckle image is then processed using a Census transform-based spatiotemporal matching algorithm to obtain the initial disparity map of the scene under test.

[0061] In the spatiotemporal matching method based on Census transform, the Census transform based on a local spatiotemporal window is first used to extract features from each pixel of the stereo-corrected spatiotemporal speckle image. The calculation process is expressed as follows:

[0062]

[0063]

[0064]

[0065] Where C(x,y) is the feature vector of the center pixel (x,y) of the speckle image. Indicates bitwise concatenation operation, I m (x,y) is the average light intensity of the local spatiotemporal window, and R and N are the spatial window radius and time length of the spatiotemporal matching, respectively.

[0066] According to the system's preset parallax range [D min D max The matching cost Cost(x,y,d) is obtained by calculating the Hamming distance between the feature vector of each pixel in the left image and the feature vectors of all candidate pixels in the right image. This cost is then used to estimate the similarity between each pixel in the left image and all candidate pixels in the right image.

[0067]

[0068] Among them, C L (x,y) is the feature vector of the center pixel (x,y) in the left speckle image, C R (xd,y) is the feature vector of the center pixel (xd,y) in the right speckle image. This is an XOR operation. BC(·) is used to count the number of "1"s in the XOR result, and d represents any candidate disparity within the disparity range. Then, the initial disparity map D(x,y) is calculated using the Winner-Take-All (WTA) algorithm.

[0069]

[0070] Then, the initial disparity map D(x,y) is optimized using nearest neighbor interpolation to obtain the optimized disparity map D. sub (x, y), the calculation process is as follows:

[0071]

[0072] Cost 1 (D(x,y)+1)=Cost(x,y,D(x,y)+1)

[0073] Preferably, step 3 utilizes a sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation to improve the measurement accuracy and spatial resolution of traditional spatiotemporal matching methods, thereby recovering the fine contour of the measured object: In the sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation, a second-order shape function W(ξ;T) is used to describe the perspective transformation between the left camera reference subset centered at point p (i.e., the local window of the left camera image) and the right camera target subset centered at point q:

[0074]

[0075]

[0076]

[0077] Where ξ = [x, y] TRepresents the local coordinates within the subset, and T represents the deformation parameter vector [u, u] of the target window. x ,u y ,u xx ,u yy ,u xy ,v,v x ,v y ,v xx ,v yy ,v xy ] T (u,v) represents the sub-pixel matching result output by the sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation, where u,u x ,u y Let v and v represent the horizontal displacement value and its partial derivatives in the horizontal and vertical directions, respectively. x ,v y U represents the vertical displacement value and its partial derivatives in the horizontal and vertical directions, respectively. xx ,u xy Representing u x The partial derivatives in the horizontal and vertical directions, v xx ,v xy They represent v respectively x The partial derivatives in the horizontal and vertical directions, u yy Represents u y The partial derivative in the vertical direction, v yy Represents v y Partial derivative in the vertical direction.

[0078] Then, the spatiotemporal ZNSSD method based on local spatiotemporal windows is combined with a second-order shape function to quantitatively evaluate the similarity between the left and right spatiotemporal subsets:

[0079]

[0080]

[0081]

[0082] Where x represents the coordinates of the point to be matched in the left figure. and This represents the average intensity value of the left and right spatiotemporal subsets. W(ξ; ΔT) is the incremental deformation function of the left camera reference subset, and ΔT is the incremental deformation parameter vector.

[0083]

[0084] To minimize the ZNSSD coefficients, a first-order Taylor expansion is performed on ΔT:

[0085]

[0086] in This is the gradient of the reference subset. The least squares method is used to solve for ΔT to minimize C. ZNSSD (ΔT), that is

[0087]

[0088]

[0089] Based on the calculated deformation parameter vector ΔT, the second-order shape function W(ξ;T) of the target subset is updated using the incremental shape function W(ξ;T) of the reference subset:

[0090] W(ξ;T)←W(ξ;T)W -1 (ξ;ΔT)

[0091] The above formula executes a subpixel spatiotemporal matching algorithm based on stereo digital image correlation. By iteratively calculating and updating W(ξ;T), the optimized subpixel matching result (u,v) is obtained, thereby recovering the fine contour of the measured object.

[0092] However, prior to this, the initial values ​​of W(ξ; T) can be obtained from the matching results obtained using the spatiotemporal matching method based on the Census transform, according to the surface fitting algorithm:

[0093] Specifically, the initial value of T is estimated using a least-squares surface fitting method. Combining the matching results obtained from the Census transform-based spatiotemporal matching method, the surface fitting algorithm yields the following:

[0094] p + W(ξ;T)=p + ξ - D sub (p+ξ)

[0095]

[0096] Where (a1,a2,a3,a4,a5,a6) and (b1,b2,b3,b4,b5,b6) are set to (u xx / 2,u yy / 2,u xy ,u x ,u y ,u) and (v xx / 2,v yy / 2,v xy ,v x ,v y ,v).

[0097] Finally, a subpixel spatiotemporal matching algorithm based on stereo digital image correlation is used to recover the fine contour of the measured object.

[0098] Figure 2 (a) shows a spatiotemporal speckle projection 3D imaging device based on a VCSEL projection array and its internal structure; Figure 2 (b) in the figure represents the spatiotemporal speckle projection and acquisition process; Figure 2 (c) in the figure is a data processing flowchart of the spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array.

[0099] Figure 2 (a) The spatiotemporal speckle projection 3D imaging device based on VCSEL projection array and its internal structure are shown. The device integrates a set of 10 miniaturized speckle projection modules based on VCSEL to form a VCSEL speckle projection array.

[0100] Figure 2 (b) The spatiotemporal speckle projection 3D imaging device based on VCSEL projection array was demonstrated to continuously project 10 speckle patterns onto the scene under test, and the left and right cameras simultaneously acquired 10 stereo speckle images.

[0101] Figure 2 (c) This section demonstrates how 10 stereo speckle images simultaneously acquired by the left and right cameras were stereo-corrected. A spatiotemporal matching algorithm based on Census transform was used to process the spatiotemporal speckle images, obtaining an initial disparity map of the measured scene. Then, a sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation was used to improve speckle matching accuracy, resulting in optimized sub-pixel matching results. This improved measurement accuracy and spatial resolution, thereby recovering the fine contours of the measured object. By comparing the 3D reconstruction results based on the initial disparity map with those based on the sub-pixel matching results, it can be observed that the measurement accuracy and spatial resolution of the 3D reconstruction results were significantly improved.

Claims

1. A spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array, characterized in that, Includes the following steps: Step 1: Integrate a set of miniaturized speckle projection modules based on VCSEL to project spatiotemporal speckle patterns onto the scene under test, and simultaneously acquire stereo speckle images with a binocular camera. Step 2: Use the calibration parameters of the binocular camera to perform stereo correction on the acquired spatiotemporal speckle image, and use the spatiotemporal matching algorithm based on Census transform to process the spatiotemporal speckle image to obtain the initial disparity map of the scene under test. Step 3: Utilize a sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation to determine the optimized sub-pixel matching results and recover the fine contours of the tested scene. Specifically: Step 3.1: Using the second-order shape function Description by points The left camera reference subset centered on the point and the point Perspective transformation between the right-side camera target subset centered on the subject: in Represents the local coordinates within a subset. The vector representing the deformation parameters of the target window , The subpixel matching result is output by a subpixel spatiotemporal matching algorithm based on stereo digital image correlation. These represent the horizontal displacement value and its partial derivatives in the horizontal and vertical directions, respectively. These represent the vertical displacement value and its partial derivatives in the horizontal and vertical directions, respectively. Represent Partial derivatives in the horizontal and vertical directions, Represent Partial derivatives in the horizontal and vertical directions, represent The partial derivative in the vertical direction, represent The partial derivative in the vertical direction, ( , ) represents a point The coordinates; Step 3.2: Combine the spatiotemporal ZNSSD method based on local spatiotemporal windows with a second-order shape function to quantitatively evaluate the similarity between the left and right spatiotemporal subsets: in, This represents the t-th speckle image captured by the left camera. represents the t-th speckle image captured by the right camera, and N represents the total number of speckle images captured by the camera; This represents the coordinates of the point to be matched in the left image. and This represents the average intensity value of the left and right spatiotemporal subsets; Incremental deformation function for the left camera reference subset: in, Incremental deformation parameter vector ,Right now for Incremental parameters, vector pairs Perform a first-order Taylor expansion: in, It is the gradient of the reference subset; Solve using the least squares method To minimize ,Right now : Step 3.3: Based on the calculated deformation parameter vector Using the incremental shape function of the reference subset To update the second-order shape function of the target subset : Step 3.4: Return to step 3.1 until... Stop the iterative calculation and obtain the sub-pixel matching result output by the sub-pixel spatiotemporal matching algorithm based on stereo digital image correlation. .

2. The spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array according to claim 1, characterized in that, The specific method for obtaining the initial disparity map of the tested scene by processing the spatiotemporal speckle image using a Census transform-based spatiotemporal matching algorithm is as follows: The Census transform based on a local spatiotemporal window is used to extract features from each pixel of the spatiotemporal speckle image. According to the system's preset parallax range The matching cost is obtained by calculating the Hamming distance between the feature vector of each pixel in the left image and the feature vectors of all candidate pixels in the right image. ; in, It is an XOR operation. Used to count the number of "1"s in the XOR result. For any candidate disparity within the disparity range; The initial disparity map is calculated using the winner-takes-all algorithm. : Sub-pixel optimization is performed using nearest neighbor interpolation to obtain the optimized disparity map. The calculation process is as follows: 。 3. The spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array according to claim 2, characterized in that, The specific process of feature extraction is as follows: in, The center pixel of the speckle image eigenvectors, This indicates a bitwise concatenation operation. It is the average light intensity within a local spatiotemporal window. and These represent the spatial window radius and time length for spatiotemporal matching, respectively.

4. The spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array according to claim 2, characterized in that, The matching cost is obtained by calculating the Hamming distance between the feature vector of each pixel in the left image and the feature vectors of all candidate pixels in the right image. Specifically: in, It is an XOR operation. Used to count the number of "1"s in the XOR result. and These represent the spatial window radius and time length for spatiotemporal matching, respectively.

5. The spatiotemporal speckle projection three-dimensional imaging method based on VCSEL projection array according to claim 1, characterized in that, The matching results obtained using the spatiotemporal matching method based on Census transform are used to determine the surface fitting algorithm. Initial values ​​for iteration: in, and Set as and .

Images