A precision control method of a binocular multi-view scanning structured light sensor

Abstract

The application discloses a precision control method of a binocular multi-view scanning structured light sensor, the sensor is composed of a left camera, a right camera and a DLP projector, the left camera or the right camera is selected as a reference camera, the other one is an auxiliary camera, and two-by-two combination forms a first system, a second system and a third system; an actual conversion matrix H between a reference camera coordinate system and a DLP projector coordinate system is calculated; an actual conversion matrix G between an auxiliary camera coordinate system and the DLP projector coordinate system is calculated, G=F*H; F represents a conversion matrix between two camera coordinate systems; the conversion matrices H, F and G are used for respectively reconstructing point clouds of the first system, the second system and the third system, and point cloud alignment is completed. The method does not need to be recalibrated, can eliminate the point cloud misalignment problem only by additional projection of the DLP projector, and is suitable for quick point cloud alignment and precision control of the sensor in an industrial field.

Description

Technical Field

[0001] This invention relates to the field of point cloud 3D reconstruction, and specifically to a method for precision control of a binocular multi-view scanning structured light sensor. Background Technology

[0002] Structured light 3D measurement is a widely used non-contact active measurement method in manufacturing. By projecting a specific coded and coded geometric pattern onto a target object, it acquires the spatial information of the target object using the principle of triangulation, enabling accurate and rapid calculation of the 3D information of the measured object. Based on the number of cameras, measurement sensors are divided into monocular and binocular systems. Compared to monocular systems, binocular structured light sensors consist of a binocular camera and a DLP (Digital Light Projector).

[0003] Binocular structured light sensors are mainly used in two ways:

[0004] Method 1 uses only DLP as the unit for feature projection, and all point cloud data is obtained through binocular processing. The problem with this method is that some point clouds are missing, and the completeness cannot be guaranteed. The reason for this problem is:

[0005] The workpiece has reflective areas, and the point cloud within these areas cannot be obtained;

[0006] The common field of view of a binocular camera cannot completely cover the entire area of ​​the workpiece (e.g., Figure 1c Point clouds in uncovered areas cannot be solved.

[0007] In method two, DLP is also used as a unit for point cloud computation, with the left camera and projector (e.g., Figure 1b ), left camera and right camera (e.g.) Figure 1c ), right camera and projector (e.g.) Figure 1d When combined in pairs, they form three systems (the first system, the second system, and the third system, such as...). Figure 1a The overall point cloud is composed of three point clouds reconstructed by three systems. This method solves the problem of incomplete point clouds in Method 1. However, due to the internal structure of the DLP projector, the projection position changes with temperature. Therefore, after a period of use, the DLP projector's projection position will change. Figure 2 As shown, changes in the projection position can lead to misalignment of the three point clouds, compromising accuracy. To address this, existing methods employ periodic on-site recalibration to re-acquire extrinsic parameters between the camera and the DLP projector, ensuring the accuracy of the reconstructed point cloud. However, this method is time-consuming and requires a calibration board at the workstation. Due to the complex industrial environment, the available calibration space is limited, hindering its implementation. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a precision control method for a binocular multi-view scanning structured light sensor. This method eliminates the need for recalibration in the industrial field; it only requires an additional DLP projector to project images, thereby eliminating the misalignment of the three point clouds and achieving precision control. This method is convenient to implement, has a fast processing speed, requires no manual intervention, and is suitable for rapid point cloud precision control of sensors in industrial settings.

[0009] The technical solution is as follows:

[0010] A method for precision control of a binocular multi-view scanning structured light sensor, wherein the binocular multi-view scanning structured light sensor consists of a left camera, a right camera and a DLP projector, wherein either the left camera or the right camera is selected as the reference camera and the other camera is selected as the auxiliary camera, wherein the reference camera and the projector constitute the first system, the reference camera and the auxiliary camera constitute the second system, and the auxiliary camera and the projector constitute the third system.

[0011] Includes the following steps:

[0012] 1) Calculate the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system using the following formula:

[0013]

[0014] Where M represents the intrinsic parameter matrix of the DLP projector, and C is the scaling factor of the DLP projector; coordinates (u i v i ), (X i Y i Z i Obtain the coordinates of each point through method A or method B, i = 1, 2, ..., S, where S represents the total number of points and S ≥ 6. Substitute the coordinates of each point into the above formula, solve the S equations simultaneously, and solve for matrix H.

[0015] Method A is as follows:

[0016] A DLP projector projects a feature image, which includes at least six non-collinearly distributed feature points, the coordinates (u... i v i ) represents the coordinates of each feature point in the feature image, which have been pre-stored in the DLP projector;

[0017] Both the left and right cameras acquire feature images, perform stereo matching, and obtain the three-dimensional coordinates (X, Y, F, Z) of each feature point in the reference camera coordinate system. i Y i Z i );

[0018] Method B is as follows:

[0019] ① A DLP projector projects an N-step phase-shifting fringe image group I, wherein each image in the N-step phase-shifting fringe image group I has horizontal fringe; then an N-step phase-shifting fringe image group II is projected, wherein each image in the N-step phase-shifting fringe image group II has vertical fringe; wherein the number of images in the N-step phase-shifting fringe image group I and the N-step phase-shifting fringe image group II is the same.

[0020] The left and right cameras simultaneously acquire N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II;

[0021] ② Perform phase unfolding on N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II acquired by the left camera to obtain left phase image I and left phase image II respectively;

[0022] S horizontal pixel coordinates u are obtained using the left phase map I. i-左 1. Obtain S vertical pixel coordinates v using the left phase map II. i-左 ;

[0023] Phase unfolding was performed on N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II acquired by the right camera to obtain right phase image I and right phase image II, respectively.

[0024] S horizontal pixel coordinates u are obtained using the right phase map I. i-右 1. Obtain the S vertical pixel coordinates v using the right phase map II. i-右 ;

[0025] For pixel (u) i-左 v i-左 ), (u i-右 v i-右 Perform stereo matching to obtain the three-dimensional coordinates (X). i Y i Z i );

[0026] Using pixels (u i-左 v i-左 Phase information or pixel (u) at the location i-右 v i-右 Find the phase information at the location, and locate the corresponding projection coordinates (u) of each pixel. i v i );

[0027] 2) Calculate the actual transformation matrix G = F × H between the auxiliary camera coordinate system and the DLP projector coordinate system; where F represents the transformation torque between the auxiliary camera coordinate system and the reference camera coordinate system obtained from the pre-calibration.

[0028] The point clouds of the first system, the second system, and the third system are reconstructed using transformation matrices H, F, and G respectively, to achieve point cloud alignment and precision control.

[0029] Furthermore, in method A, the feature points are round dots, crosshair dots, and corner dots.

[0030] Preferably, in method A, the number of feature points is 10 to 100, which are evenly distributed in the feature image.

[0031] Furthermore, in method B, S horizontal pixel coordinates u are obtained using the left phase map I. i-左 and phase ψ i-左 ;

[0032] S vertical pixel coordinates v are obtained using the left phase map II. i-左 and phase ξ i-左 ;

[0033] Using pixels (u i-左 v i-左 Phase information (ψ) at position ) i-左 ξ i-左 ), find phase information (ψ) i-左 ξ i-左 The corresponding projected coordinates (u) i v i );

[0034] Furthermore, in method B, the right phase map I is used to obtain the S horizontal pixel coordinates u. i-右 and phase ψ i-右 ;

[0035] S vertical pixel coordinates v are obtained using the right phase map II. i-右 and phase ξ i-右 ;

[0036] Using pixels (u i-右 v i-右 Phase information (ψ) at ) i-右 ξ i-右 ), find phase information (ψ) i-右 ξ i-右 The corresponding projected coordinates (u) i v i ).

[0037] Preferably, the optimization method is used to solve the formula and calculate the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system.

[0038] The accuracy control method for binocular multi-view scanning structured light sensors provided by this invention solves the problem of misalignment of point clouds reconstructed by a measurement system formed by dual cameras and a projector.

[0039] This method only requires the DLP projection feature image or stripe image, without recalibration, to quickly obtain the extrinsic parameters (transformation matrix) between the camera and the DLP, thus ensuring the maximum accuracy and completeness of the reconstructed point cloud.

[0040] The time taken is only in the millisecond range, making it easy to implement and enabling online real-time point cloud alignment and precision control. Attached Figure Description

[0041] Figure 1a A schematic diagram illustrating the theoretical process of reconstructing point clouds for the first, second, and third systems; assuming the left camera is the reference camera. Figure 1b This is a schematic diagram of the theory of reconstructing point clouds in the first system. The raised left side can be reconstructed in this system. Figure 1c This is a theoretical diagram for reconstructing point clouds in the second system. The two raised left and right sides cannot be reconstructed in this system. Figure 1d A schematic diagram of the point cloud reconstruction theory for the third system, showing that the right side of the protrusion can be reconstructed in this system;

[0042] Figure 2 A diagram illustrating the reason why misalignment occurs when reconstructing point clouds from three systems;

[0043] Figure 3 This is a schematic diagram of the feature image in a specific implementation method. Detailed Implementation

[0044] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] A method for precision control of a binocular multi-view scanning structured light sensor, as shown in Figure 1, comprises a left camera, a right camera, and a DLP projector. Either the left or right camera is selected as the reference camera, and the other camera is used as an auxiliary camera. The reference camera (in this embodiment, the left camera is selected as the reference camera) and the projector constitute the first system (e.g., ...). Figure 1b As shown), the reference camera and the auxiliary camera constitute the second system (as shown). Figure 1c As shown), the auxiliary camera and projector constitute a third system (such as...). Figure 1d (as shown);

[0046] Includes the following steps:

[0047] 1) Obtain coordinates (u) using method A or method B. i v i ), (X i Yi Z i );

[0048] Specifically, method A is:

[0049] A DLP projector projects a feature image, which includes at least six non-collinearly distributed feature points with coordinates (u...). i v i ) represents the coordinates of each feature point in the feature image (e.g. Figure 2 The feature points (pd) are pre-stored in the DLP projector; in practical applications, to calculate a more accurate actual transformation matrix H, the number of feature points is set to 10-100, which are evenly distributed in the feature image. The feature points are circles, crosshairs, and corners.

[0050] Both the left and right cameras acquire feature images, perform stereo matching, and obtain the three-dimensional coordinates (X, Y, F, Z) of each feature point in the reference camera coordinate system. i Y i Z i );

[0051] Method B is:

[0052] ① A DLP projector projects an N-step phase-shifting fringe image group I, wherein each image in the N-step phase-shifting fringe image group I has horizontal fringe; then an N-step phase-shifting fringe image group II is projected, wherein each image in the N-step phase-shifting fringe image group II has vertical fringe (perpendicular to the fringe direction in the N-step phase-shifting fringe image group I); wherein the number of images in the N-step phase-shifting fringe image group I and the N-step phase-shifting fringe image group II is the same;

[0053] The left and right cameras simultaneously acquire N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II;

[0054] For example, if a three-frequency four-phase sinusoidal fringe image is projected, then N-step phase-shifted fringe image group I includes 12 horizontal fringe images and N-step phase-shifted fringe image group II includes 12 vertical fringe images.

[0055] At this point, the left camera will capture 12 horizontal stripe images and 12 vertical stripe images, for a total of 24 stripe images, and the right camera will do the same.

[0056] ② Perform phase unfolding on N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II acquired by the left camera to obtain left phase image I and left phase image II respectively;

[0057] S horizontal pixel coordinates u are obtained using the left phase map I. i-左 1. Obtain S vertical pixel coordinates v using the left phase map II. i-左 ;

[0058] Then, phase unfolding is performed on the N-step phase shift fringe image group I and N-step phase shift fringe image group II acquired by the right camera to obtain right phase image I and right phase image II respectively;

[0059] S horizontal pixel coordinates u are obtained using the right phase map I. i-右 1. Obtain the S vertical pixel coordinates v using the right phase map II. i-右 ;

[0060] For pixel (u) i-左 v i-左 ), (u i-右 v i-右 Perform stereo matching to obtain the three-dimensional coordinates (X). i Y i Z i );

[0061] Using pixels (u i-左 v i-左 Phase information or pixel (u) at the location i-右 v i-右 Find the phase information at the location, and locate the corresponding projection coordinates (u) of each pixel. i v i );

[0062] Right now:

[0063] S horizontal pixel coordinates u are obtained using the left phase map I. i-左 and phase ψ i-左 ;

[0064] S vertical pixel coordinates v are obtained using the left phase map II. i-左 and phase ξ i-左 ;

[0065] Using pixels (u i-左 v i-左 Phase information (ψ) at ) i-左 ξ i-左 ), find phase information (ψ) i-左 ξ i-左 The corresponding projected coordinates (u) i v i ).

[0066] Alternatively, the right phase map I can be used to obtain the S horizontal pixel coordinates u. i-右 and phase ψ i-右 ;

[0067] S vertical pixel coordinates v are obtained using the right phase map II. i-右and phase ξ i-右 ;

[0068] Using pixels (u i-右 v i-右 Phase information (ψ) at ) i-右 ξ i-右 ), find phase information (ψ) i-右 ξ i-右 The corresponding projected coordinates (u) i v i ).

[0069] The one-to-one correspondence between phase information and projection coordinates is pre-recorded before the DLP projection of the N-step phase-shifted image; this is known information.

[0070] After completing either method A or method B, calculate the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system using the following formula:

[0071]

[0072] Where M represents the intrinsic parameter matrix of the DLP projector, and C is the scaling factor of the DLP projector; coordinates (u i v i ), (X i Y i Z i Obtain the coordinates of each point through method A or method B, i = 1, 2, ..., S, where S represents the total number of points and S ≥ 6. Substitute the coordinates of each point into the above formula, solve the S equations simultaneously, and solve for matrix H.

[0073] In practice, the optimization method can be used to solve the formula and calculate the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system.

[0074] To facilitate understanding, the following background information is provided: In a DLP projector, the image is generated by a DMD (Digital Micromirror Device). A DMD is a matrix of micromirrors (precise, miniature reflectors) arranged on a semiconductor chip, with each micromirror controlling a pixel in the projected image.

[0075] C represents the spatial dimension occupied by the side length of the micromirrors inside the DLP, in millimeters.

[0076] The intrinsic parameter matrix of a DLP projector includes the lens focal length, lens distortion coefficient, and projection center.

[0077] The actual transformation matrix H to be solved includes rotation and translation parameters.

[0078] 2) Calculate the actual transformation matrix G = F × H between the auxiliary camera coordinate system and the DLP projector coordinate system; where F represents the transformation torque between the auxiliary camera coordinate system and the reference camera coordinate system obtained from the pre-calibration.

[0079] The point clouds of the first, second, and third systems are reconstructed using transformation matrices H, F, and G respectively, to achieve point cloud alignment and precision control.

[0080] As one implementation method, the specific process is as follows:

[0081] In the first system, the epipolar matching method is used to find the corresponding pixel points in the reference camera image and the projector phase plane. The transformation matrix H is used to calculate the three-dimensional coordinates of each corresponding pixel point in the reference camera coordinate system to complete the three-dimensional reconstruction of the first system.

[0082] In the second system, the epipolar matching method is used to find the corresponding pixels in the reference camera image and the auxiliary camera image. The transformation matrix F is used to calculate the three-dimensional coordinates of each corresponding pixel in the reference camera coordinate system to complete the three-dimensional reconstruction of the second system.

[0083] In the third system, the epipolar matching method is used to find the corresponding pixels in the auxiliary camera image and the projector phase plane. The transformation matrix G is used to calculate the three-dimensional coordinates of each corresponding pixel in the auxiliary camera coordinate system, thus completing the three-dimensional reconstruction of the third system.

[0084] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and descriptive purposes. It is not intended to be exhaustive, nor to limit the invention to the precise forms disclosed; obviously, many changes and variations are possible in accordance with the foregoing teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical application, thereby enabling others skilled in the art to implement and utilize various exemplary embodiments of the invention, as well as their different alternatives and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A precision control method for a binocular multi-view scanning structured light sensor, wherein the binocular multi-view scanning structured light sensor consists of a left camera, a right camera and a DLP projector, wherein either the left camera or the right camera is selected as the reference camera and the other camera is selected as the auxiliary camera, wherein the reference camera and the projector constitute a first system, the reference camera and the auxiliary camera constitute a second system, and the auxiliary camera and the projector constitute a third system. Its features are, Includes the following steps: 1) Calculate the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system using the following formula: Where M represents the intrinsic parameter matrix of the DLP projector, and C is the scaling factor of the DLP projector; coordinates (u i v i ), (X i Y i Z i Obtain the coordinates of each point through method A or method B, i = 1, 2, ..., S, where S represents the total number of points and S ≥ 6. Substitute the coordinates of each point into the above formula, solve the S equations simultaneously, and solve for matrix H. Method A is as follows: A DLP projector projects a feature image, which includes at least six non-collinearly distributed feature points, the coordinates (u... i v i ) represents the coordinates of each feature point in the feature image, which have been pre-stored in the DLP projector; Both the left and right cameras acquire feature images, perform stereo matching, and obtain the three-dimensional coordinates (X, Y, F, Z) of each feature point in the reference camera coordinate system. i Y i Z i ); Method B is as follows: ① A DLP projector projects an N-step phase-shifting fringe image group I, wherein each image in the N-step phase-shifting fringe image group I has horizontal fringe; then an N-step phase-shifting fringe image group II is projected, wherein each image in the N-step phase-shifting fringe image group II has vertical fringe; wherein the number of images in the N-step phase-shifting fringe image group I and the N-step phase-shifting fringe image group II is the same. The left and right cameras simultaneously acquire N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II; ② Perform phase unfolding on N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II acquired by the left camera to obtain left phase image I and left phase image II respectively; S horizontal pixel coordinates u are obtained using the left phase map I. i-左 1. Obtain S vertical pixel coordinates v using the left phase map II. i-左 ; Phase unfolding was performed on N-step phase-shifted fringe image group I and N-step phase-shifted fringe image group II acquired by the right camera to obtain right phase image I and right phase image II, respectively. S horizontal pixel coordinates u are obtained using the right phase map I. i-右 1. Obtain the S vertical pixel coordinates v using the right phase map II. i-右 ; For pixel (u) i-左 v i-左 ), (u i-右 v i-右 Perform stereo matching to obtain the three-dimensional coordinates (X). i Y i Z i ); Using pixels (u i-左 v i-左 Phase information or pixel (u) at the location i-右 v i-右 Find the phase information at the location, and locate the corresponding projection coordinates (u) of each pixel. i v i ); 2) Calculate the actual transformation matrix G = F × H between the auxiliary camera coordinate system and the DLP projector coordinate system; where F represents the transformation torque between the auxiliary camera coordinate system and the reference camera coordinate system obtained from the pre-calibration. The point clouds of the first system, the second system, and the third system are reconstructed using transformation matrices H, F, and G respectively, to achieve point cloud alignment and precision control.

2. The accuracy control method for a binocular multi-view scanning structured light sensor as described in claim 1, characterized in that: In method A, the feature points are round dots, crosshair dots, and corner dots.

3. The accuracy control method for a binocular multi-view scanning structured light sensor as described in claim 1, characterized in that: In method A, the number of feature points is 10 to 100, and they are evenly distributed in the feature image.

4. The accuracy control method for a binocular multi-view scanning structured light sensor as described in claim 1, characterized in that: In method B, S horizontal pixel coordinates u are obtained using the left phase map I. i-左 and phase ψ i-左 ; S vertical pixel coordinates v are obtained using the left phase map II. i-左 and phase ξ i-左 ; Using pixels (u i-左 v i-左 Phase information (ψ) at ) i-左 ξ i-左 ), find phase information (ψ) i-左 ξ i-左 The corresponding projected coordinates (u) i v i ).

5. The accuracy control method for a binocular multi-view scanning structured light sensor as described in claim 1, characterized in that: In method B, the right phase map I is used to obtain the S horizontal pixel coordinates u. i-右 and phase ψ i-右 ; S vertical pixel coordinates v are obtained using the right phase map II. i-右 and phase ξ i-右 ; Using pixels (u i-右 v i-右 Phase information (ψ) at ) i-右 ξ i-右 ), find phase information (ψ) i-右 ξ i-右 The corresponding projected coordinates (u) i v i ).

6. The accuracy control method for a binocular multi-view scanning structured light sensor as described in claim 1, characterized in that: The formula is solved using an optimization method, and the actual transformation matrix H between the reference camera coordinate system and the DLP projector coordinate system is calculated.

Images