A real-time multi-camera multi-projector 3D imaging processing method and device
By optimizing the calibration and reconstruction methods of a multi-camera, multi-projector 3D measurement platform, the problems of system complexity and poor maintainability in multi-view reconstruction are solved, achieving efficient real-time 3D reconstruction and low-cost multi-view fusion.
Patent Information
- Application Number
- CN202310291143.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-23
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2043-03-23
AI Technical Summary
Existing 3D imaging technologies require multiple devices for multi-view reconstruction, resulting in high system complexity and poor maintainability, and dynamic real-time reconstruction is difficult.
By building a multi-camera, multi-projector 3D measurement platform, using a self-made stereo calibration plate and reconstruction standard parts, and combining the PMP method and triangulation method, the calibration parameters of the cameras and projectors are optimized to achieve multi-view point cloud fusion, reducing the number of devices and the calibration process.
It achieves real-time 3D reconstruction without point cloud stitching and multi-camera field of view overlap, reducing hardware costs, simplifying system structure, and improving maintainability and reconstruction speed.
Smart Images
Figure CN116363226B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of three-dimensional imaging technology, and in particular to a real-time multi-camera multi-projector 3D imaging processing method and device. BACKGROUND
[0002] In the field of machine vision, compared with traditional two-dimensional image processing technology, three-dimensional imaging technology has become a research hotspot in recent years due to its depth information feature, and is widely used in industrial production, life medicine, consumer electronics and other fields. With the development of technology and the iteration and upgrading of software and hardware, the precision, speed and range of three-dimensional measurement technology have been greatly improved. The current three-dimensional imaging technology mainly includes a structured light system calibration method based on a pseudo camera method and a phase measurement profilometry (PMP) based on structured light.
[0003] In the single camera single projector structured light system calibration method based on the pseudo camera method, Zhang's calibration method is first used to obtain camera calibration parameters, including camera internal parameters, distortion coefficients and target plane to camera external parameters, then the camera calibration parameters are used to give the projector the ability to acquire pictures, the projector is regarded as a reverse camera, and the camera calibration method is used to calibrate the projector, so as to obtain the overall system calibration parameters; in the PMP method, the projector projects a series of coded grating fringe patterns onto the measured object, and triggers the camera to collect these fringe patterns modulated by the object surface, and then calculates the three-dimensional coordinates of the object surface according to the depth information contained in the patterns combined with the overall system calibration parameters. However, due to the limitation of the field of view, the current three-dimensional imaging technology is mostly limited to single-view measurement, and the research on multi-view reconstruction of the same object is not perfect enough.
[0004] Currently, in the research of multi-view three-dimensional reconstruction, there are two commonly used schemes: the first is multi-view reconstruction based on point cloud splicing, that is, using a single structured light system, multiple reconstructions are performed under the change of the relative view angle of the device and the measured object, and a point cloud registration algorithm such as ICP (Iterative Closest Point) is used to splice the point clouds under different view angles into a whole, however, the number of measurements is large, the registration time is long, and dynamic real-time reconstruction cannot be completed; the second is a multi-light source three-dimensional measurement system composed of multiple cameras and multiple projectors, through system overall calibration, the point cloud data under different view angles is converted to the reference coordinate system, so as to obtain the directly merged multi-view point cloud data, compared with the first scheme, the whole time consumption of this scheme is short, and it is more suitable for high-speed measurement scene, but in order to realize the system overall high-precision calibration based on the light beam adjustment method, a large number of devices are needed to ensure that there is a large part of the overlapping field of view between adjacent cameras, in the actual production application process, more devices will lead to complex calibration process of the whole system, poor maintainability, and thus affect the practicability. SUMMARY
[0005] In view of the defects in the prior art, the purpose of the present application is to provide a real-time multi-camera multi-projector 3D imaging processing method and device, which does not need to increase the number of cameras to optimize the system overall calibration parameters, and has simple overall structure and strong maintainability.
[0006] To achieve the above purpose, the present application provides a real-time multi-camera multi-projector 3D imaging processing method, which specifically comprises the following steps:
[0007] A multi-camera multi-projector three-dimensional measurement platform composed of a computer and multiple single-camera single-projector measurement systems is built, and an industrial calibration board, a self-made stereo calibration board and a reconstruction standard part are created, the single-camera single-projector measurement system comprising one camera and one projector;
[0008] The camera and the projector in the single-camera single-projector measurement system are calibrated, and the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix of the projector coordinate system to the camera coordinate system are obtained;
[0009] Based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is obtained using the self-made stereo calibration board, taking any camera coordinate system as the reference coordinate system;
[0010] The reconstruction standard part is reconstructed based on the PMP method and the triangulation method, and the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereo constraint of the reconstruction standard part;
[0011] The overall calibration of the multi-camera multi-projector three-dimensional measurement platform is completed based on a reference coordinate system, the object to be measured is placed in a reconstruction range, a world coordinate system is established on the reference camera coordinate system, and a multi-view fusion point cloud of the object to be measured is obtained according to a position conversion matrix between the cameras.
[0012] On the basis of the above technical solution,
[0013] The multi-camera multi-projector three-dimensional measurement platform comprises two single-camera single-projector measurement systems, one of which comprises a first camera and a first projector, and the other of which comprises a second camera and a second projector.
[0014] The fields of view of the lenses of the first camera and the second camera are non-overlapping, and the working distances thereof are equal.
[0015] The working distances of the lenses of the first projector and the second projector are equal.
[0016] On the basis of the above technical solution,
[0017] The self-made three-dimensional calibration board is created by pasting the same calibration pattern printing paper on both sides of a wooden board with a flat surface and uniform thickness, and aligning the positions of the calibration pattern printing paper, to obtain a conversion matrix from a back target coordinate system of the self-made three-dimensional calibration board to a front target coordinate system of the self-made three-dimensional calibration board.
[0018] The reconstruction standard part is a high-precision matte ceramic flat plate and a high-precision matte ceramic ball.
[0019] On the basis of the above technical solution, the camera and the projector in the single-camera single-projector measurement system are calibrated to obtain the internal parameters and distortion coefficients of the camera and the projector, and a position conversion matrix from the projector coordinate system to the camera coordinate system, and the specific steps include:
[0020] Based on an industrial calibration board and Zhang's calibration method, feature points in an industrial calibration board calibration image captured by the camera are obtained, and an imaging model is established to solve the camera calibration parameters:
[0021]
[0022] wherein p represents a proportion factor, (u, v) is the pixel coordinates of the feature points in the industrial calibration board calibration image, (X W , Y W , Z W ) is the three-dimensional coordinates of the calibration points in the target coordinate system, A represents a 3*3 camera intrinsic parameter matrix, [R T] represents a conversion matrix from the target coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and T is a 3*1 translation matrix.
[0023] The calibration parameters of the camera are optimized using the LM algorithm, and the optimization target is the re-projection error of the feature point pixels of the calibration image, so as to obtain the distortion coefficient of the camera;
[0024] The internal parameters, distortion coefficients of the projector, and the conversion matrix of the target coordinate system to the projector coordinate system are calibrated;
[0025] Based on the conversion matrix of the target coordinate system to the camera coordinate system and the conversion matrix of the target coordinate system to the projector coordinate system, the position conversion matrix of the projector coordinate system to the camera coordinate system in a single camera and single projector measurement system is calculated.
[0026] On the basis of the above technical solutions,
[0027] When calibrating the camera, the calibration image on the industrial calibration board is a light gray dot calibration image;
[0028] When calibrating the projector, the projection pattern of the projector is a checkerboard calibration pattern, and the two-dimensional pixel coordinates of the calibration pattern feature points are pre-set, and the three-dimensional world coordinates of the calibration pattern feature points are calculated through the calibration parameters of the camera.
[0029] On the basis of the above technical solutions, based on the internal parameters and distortion coefficients of all cameras calibrated, using a self-made stereo calibration board, taking any camera coordinate system as a reference coordinate system, an initial estimation of the position conversion matrix of other cameras to the reference coordinate system is obtained, specifically including:
[0030] Based on the internal parameters and distortion coefficients of all cameras calibrated, using a self-made stereo calibration board, taking the first camera coordinate system as a reference coordinate system, an initial estimation of the position conversion matrix of the second camera coordinate system to the reference coordinate system is obtained, and the calculation method is:
[0031]
[0032] Wherein, the shooting side of the first camera is the front side of the self-made stereo calibration board, the shooting side of the second camera is the back side of the self-made stereo calibration board, [R' T'] represents the initial estimation of the position conversion matrix of the second camera coordinate system to the reference coordinate system, R' is a 3*3 orthogonal rotation matrix, T' is a 3*1 translation matrix, 0 is a 1*3 zero matrix, [R c1 T c1 ] represents the conversion matrix of the front target coordinate system of the self-made stereo calibration board to the first camera coordinate system, [R b T b ] represents the conversion matrix of the back target coordinate system of the self-made stereo calibration board to the front target coordinate system of the self-made stereo calibration board, [R c2 T c2R represents a conversion matrix from the reverse target coordinate system of the self-made stereoscopic calibration board to the second camera coordinate system.
[0033] On the basis of the above technical solution, the PMP method and the triangulation method are used to reconstruct the standard part, and the error of the initial estimation of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereoscopic constraint of the reconstructed standard part, wherein the PMP method and the triangulation method are used to reconstruct the standard part, and the specific steps include:
[0034] The computer controls the projector to project the coded grating image onto the surface of the reconstructed standard part, and the cameras in the multi-camera and multi-projector three-dimensional measurement platform sequentially collect the surface images of the reconstructed standard part.
[0035] The collected surface images are denoised based on the digital image processing method, and the denoised surface images are decoded to calculate the absolute phase values of all pixel points in the reconstruction range, and the decoding mode is complementary Gray code plus four-step phase shift.
[0036] The absolute phase values, the internal parameters of the cameras and the projectors, the distortion coefficients, and the position conversion matrix of the projector coordinate system to the camera coordinate system obtained through calibration are combined, and the three-dimensional point coordinates corresponding to all pixel points are solved by using the triangulation method.
[0037] On the basis of the above technical solution, the PMP method and the triangulation method are used to reconstruct the standard part, and the error of the initial estimation of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereoscopic constraint of the reconstructed standard part, and the specific steps include:
[0038] The high-precision matte ceramic plate is placed at the center of the reconstruction range, and the first camera is used to shoot the front surface of the high-precision matte ceramic plate, and the second camera is used to shoot the back surface of the high-precision matte ceramic plate.
[0039] Based on the PMP method and the triangulation method, the front surface point cloud and the back surface point cloud of the high-precision matte ceramic plate are obtained, and filtering processing is performed.
[0040] The reference coordinate system of the back surface point cloud of the high-precision matte ceramic plate is changed from the second camera coordinate system to the first camera coordinate system, and the calculation mode is as follows:
[0041]
[0042] Wherein, (X C21 ,Y C21 ,Z C21 ) represents the coordinates of the back surface point cloud of the high-precision matte ceramic plate in the first camera coordinate system, (X C2 ,Y C2 ,Z C2() represents the coordinates of the point cloud on the reverse side of a high-precision matte ceramic plate in the second camera coordinate system;
[0043] Based on the coordinates (X, X) of the front point cloud of the high-precision matte ceramic plate in the first camera coordinate system. C1 ,Y C1 Z C1 Based on point cloud processing algorithms, the system fits the front point cloud of a high-precision matte ceramic flat plate and outputs a plane equation.
[0044] Based on (X) C21 ,Y C21 Z C21 The variance of the distance to the fitting plane on the front of the high-precision matte ceramic plate is calculated, and the rotation matrix R′ in the initial values of the extrinsic parameters is optimized. The optimization method is as follows:
[0045] J T *J*h=J T E
[0046] Where J represents the Jacobian matrix of the rotation vector r′, the 3*3 orthogonal rotation matrix R′ is transformed by Rodrigues to obtain the 3*1 rotation vector r′, h represents the iteration vector of r′, and E represents the residual matrix.
[0047] Place the high-precision matte ceramic sphere at the center of the reconstruction area, and have the first camera capture the front of the high-precision matte ceramic sphere and the second camera capture the rear of the high-precision matte ceramic sphere.
[0048] A high-precision matte ceramic sphere was reconstructed using the PMP method and triangulation to obtain the coordinates (X, Y, F) of the point cloud in front of the high-precision matte ceramic sphere in the first camera coordinate system. G1 ,Y G1 Z G1 ), and the coordinates (X, X) of the point cloud behind the high-precision matte ceramic sphere in the second camera coordinate system. G2 ,Y G2 Z G2 );
[0049] Using the optimized rotation matrix R′ and the unoptimized translation matrix T′, the coordinates (X, Y, T) of the high-precision point cloud behind the matte ceramic sphere in the first camera coordinate system are obtained. G21 ,Y G21 Z G21 );
[0050] Based on point cloud processing algorithms, a high-precision spherical point cloud of a matte ceramic sphere is fitted, and the coordinates of the center of the sphere (X, Y, F) of the point cloud in front of the high-precision matte ceramic sphere in the first camera coordinate system are output. SC1 ,Y SC1 Z SC1 ), and the center coordinates (X) of the point cloud behind the high-precision matte ceramic sphere.SC2 Y SC2 Z SC2 ), and calculate the sphere center distance ΔT:
[0051]
[0052] Wherein, the sphere center distance ΔT is the error of T' on three components, and the error is corrected to obtain a high-precision translation matrix T, and the calculation method is T=T'+ΔT;
[0053] Output the optimized second camera coordinate system to the position transformation matrix [R T] of the first camera coordinate system.
[0054] On the basis of the above technical solutions,
[0055] When the PMP method and the triangulation method are used to reconstruct the reconstruction standard part, the trigger mode between the camera and the projector in the single camera single projector measurement system is mutual external trigger mode;
[0056] The mutual external trigger mode is that the camera is triggered to collect the projected image after the projector finishes projecting the current image, and the projector projects the next image after the camera finishes collecting the current projected image, and so on.
[0057] The application provides a kind of real-time multi-camera multi-projector 3D imaging processing device, comprising:
[0058] The building module is used to build a multi-camera multi-projector three-dimensional measurement platform composed of a computer and a plurality of single camera single projector measurement systems, and to create an industrial calibration board, a self-made stereo calibration board and a reconstruction standard part, wherein the single camera single projector measurement system includes one camera and one projector.
[0059] The calibration module is used to calibrate the camera and the projector in the single camera single projector measurement system, and to obtain the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix of the projector coordinate system to the camera coordinate system.
[0060] The acquisition module is used to obtain the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, using the self-made stereo calibration board, and taking any camera coordinate system as the reference coordinate system.
[0061] The reconstruction module is used to reconstruct the reconstruction standard part based on the PMP method and the triangulation method, and to minimize the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system based on the stereo constraint of the reconstruction standard part.
[0062] The execution module is used for completing overall calibration of the multi-camera multi-projector three-dimensional measurement platform based on a reference coordinate system, placing the object to be measured in a reconstruction range, establishing a world coordinate system on the reference camera coordinate system, and obtaining a multi-view fusion point cloud of the object to be measured according to a position conversion matrix between the cameras.
[0063] Compared with the prior art, the application has the advantages that: the number of pictures required for shooting is small, the reconstruction time is short, and real-time three-dimensional reconstruction of a dynamic scene can be realized because the point cloud splicing process is not required; meanwhile, the number of cameras required for increasing the optimization of overall calibration parameters of the system is not required, the number of devices required is small, the hardware cost is low, the overall structure of the system is simple, installation is flexible and convenient, and the maintenance is strong because the multi-cameras do not need to have an overlapping field of view. BRIEF DESCRIPTION OF DRAWINGS
[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0065] Figure 1 The flow chart of the real-time multi-camera multi-projector 3D imaging processing method in the embodiments of the present application is shown.
[0066] Figure 2 The structural schematic diagram of the multi-camera multi-projector three-dimensional measurement platform in the present application is shown.
[0067] Figure 3 The principle diagram for solving the position conversion matrix of the multi-camera is shown. DETAILED DESCRIPTION
[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments.
[0069] Referring to Figure 1 The embodiments of the present application provide a real-time multi-camera multi-projector 3D imaging processing method to solve the problems of long reconstruction time and complex system in three-dimensional imaging, and specifically include the following steps:
[0070] S1: build a multi-camera multi-projector three-dimensional measurement platform composed of a computer and a plurality of single-camera single-projector measurement systems, and create an industrial calibration board, a self-made stereo calibration board and a reconstruction standard part, wherein each single-camera single-projector measurement system comprises one camera and one projector; that is, in the usual case, the number of cameras and projectors in the built multi-camera multi-projector three-dimensional measurement platform is greater than or equal to two, and there is no need for field of view overlap between the cameras of different single-camera single-projector measurement systems. In the specific implementation process, the computer should be connected to the equipment in advance, the image acquisition function of the camera and the projection function of the projector should be tested, and the working distance should be adjusted.
[0071] In a possible implementation, referring to FIG. 1, Figure 2 The multi-camera multi-projector three-dimensional measurement platform in the present application comprises two single-camera single-projector measurement systems, one of which comprises a first camera and a first projector, and the other of which comprises a second camera and a second projector; the fields of view of the lenses of the first camera and the second camera have no overlap, and the working distances thereof are equal; the working distances of the lenses of the first projector and the second projector are equal; and the camera and the projector are placed in a face-to-face manner. Figure 2 In FIG. 1, C1 represents the first camera, C2 represents the second camera, P1 represents the first projector, and P2 represents the first projector. The spherical ball in the middle represents the object to be measured.
[0072] In the present application, the self-made stereo calibration board is created in the following manner: the same calibration pattern printing paper is pasted on both sides of a wooden board with a flat surface and uniform thickness, and the positions of the calibration pattern printing papers are aligned to obtain the conversion matrix from the back target coordinate system of the self-made stereo calibration board to the front target coordinate system of the self-made stereo calibration board; and the reconstruction standard part is a high-precision matte ceramic flat plate and a high-precision matte ceramic ball.
[0073] S2: calibrate the camera and the projector in the single-camera single-projector measurement system to obtain the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix from the projector coordinate system to the camera coordinate system;
[0074] In the process of calibrating the camera and the projector in the single-camera single-projector measurement system, the imaging models of the camera and the projector are similar to the pinhole imaging model. Specifically, the camera and the projector in the single-camera single-projector measurement system are calibrated to obtain the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix from the projector coordinate system to the camera coordinate system, and the specific steps include:
[0075] S201: based on the industrial calibration board and Zhang's calibration method, obtain the feature points in the industrial calibration board calibration image collected by the camera, and establish an imaging model to solve the camera calibration parameters:
[0076]
[0077] wherein, p represents a proportional factor, (u, v) is a feature point pixel coordinate in an industrial calibration board calibration image, (X W ,Y W ,Z W ) is a three-dimensional coordinate of a calibration point under a target coordinate system, Z W is 0 by default, A represents a 3*3 camera intrinsic parameter matrix, [R T] represents a conversion matrix from the target coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and T is a 3*1 translation matrix;
[0078] S202: using an LM algorithm (an algorithm for iteratively finding function extreme values) to optimize the calibration parameters of the camera, the optimization target being a feature point pixel re-projection error of the calibration image, so as to obtain the distortion coefficient of the camera;
[0079] S203: calibrating the internal parameters, the distortion coefficient of the projector, and the conversion matrix from the target coordinate system to the projector coordinate system; it needs to be noted that, in the projector calibration stage, the two-dimensional pixel coordinates of the calibration pattern feature points of the projector are pre-set, and the three-dimensional world coordinates of the calibration pattern feature points can be calculated by using the calibration parameters of the camera, the principle being the same as that of solving the camera calibration parameters according to the imaging model, and then the internal parameters, the distortion coefficient of the projector, and the conversion matrix from the target coordinate system to the projector coordinate system can be calibrated by using the method of camera calibration.
[0080] S204: based on the conversion matrix from the target coordinate system to the camera coordinate system and the conversion matrix from the target coordinate system to the projector coordinate system, the position conversion matrix from the projector coordinate system to the camera coordinate system in a single camera single projector measurement system is calculated.
[0081] In the present application, when the camera is calibrated, the calibration image on the industrial calibration board is a light gray dot calibration image; when the projector is calibrated, the projection pattern of the projector is a checkerboard calibration pattern; and in the projector calibration stage, the projector projects the calibration pattern onto the industrial calibration board, and separates the image required for projector calibration based on the digital image correlation method. Using the light gray calibration pattern can improve the success rate of extracting calibration points from the separated image, and different calibration point extraction methods (dot and checkerboard) can reduce the recognition interference when the two patterns overlap.
[0082] The calibration in the present application mainly includes two steps: initial estimation of the position conversion matrix of the multi-camera and parameter optimization based on the standard part stereo constraint. Through specific experiments, it is proved that the measurement accuracy of the multi-view reconstruction of the high-precision matte ceramic ball with a diameter of 25.4162 mm based on the present application is about 42.631 um, and high-precision multi-view three-dimensional reconstruction can be realized.
[0083] S3: based on the internal parameters and distortion coefficients of all cameras obtained by calibration, using a self-made stereo calibration board, taking any camera coordinate system as a reference coordinate system, obtaining an initial estimate of the position conversion matrix of other cameras to the reference coordinate system; the specific steps are as follows:
[0084] Referring to Figure 3 , based on the internal parameters and distortion coefficients of all cameras obtained by calibration, using a self-made stereo calibration board, taking the first camera coordinate system as the reference coordinate system, obtaining an initial estimate of the position conversion matrix of the second camera coordinate system to the reference coordinate system, and the calculation method is as follows:
[0085]
[0086] Wherein, the shooting side of the first camera is the front side of the self-made stereo calibration board, and the shooting side of the second camera is the back side of the self-made stereo calibration board. The initial estimate of the position conversion matrix of the reference coordinate system is R', which is a 3*3 orthogonal rotation matrix, T' is a 3*1 translation matrix, 0 is a 1*3 zero matrix, and [R c1 T c1 ] represents the conversion matrix from the front target coordinate system of the self-made stereo calibration board to the first camera coordinate system, [R b T b ] represents the conversion matrix from the back target coordinate system of the self-made stereo calibration board to the front target coordinate system of the self-made stereo calibration board, [R c2 T c2 ] represents the conversion matrix from the back target coordinate system of the self-made stereo calibration board to the second camera coordinate system.
[0087] S4: based on the PMP method and the triangulation method, the standard part is reconstructed, and according to the stereo constraint of the reconstructed standard part, the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is minimized;
[0088] In the present application, based on the PMP method and the triangulation method, the standard part is reconstructed, and according to the stereo constraint of the reconstructed standard part, the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is minimized, wherein the PMP method and the triangulation method are used to reconstruct the standard part, and the specific steps include:
[0089] S401: the computer controls the projector to project the encoded raster image onto the surface of the reconstructed standard part, and the cameras in the multi-camera multi-projector three-dimensional measurement platform sequentially collect the surface images of the reconstructed standard part;
[0090] S402: Based on the digital image processing method, the collected surface image is denoised, and the denoised surface image is decoded to calculate the absolute phase values of all pixel points in the reconstruction range, and the decoding mode is complementary Gray code plus four-step phase shift;
[0091] S403: Combined with the absolute phase value and the internal parameters, distortion coefficients of the camera and the projector obtained by calibration, and the position conversion matrix from the projector coordinate system to the camera coordinate system, the three-dimensional point coordinates corresponding to all pixel points are solved by using the triangulation method.
[0092] In the application, the reconstruction standard part is reconstructed based on the PMP method and the triangulation method, and the error of the initial estimation of the position conversion matrix of other cameras to the reference coordinate system is minimized according to the three-dimensional constraint of the reconstructed standard part, and the specific steps include:
[0093] S411: Place the high-precision matte ceramic plate at the center of the reconstruction range, and make the first camera shoot the front surface of the high-precision matte ceramic plate, and the second camera shoot the back surface of the high-precision matte ceramic plate;
[0094] S412: Based on the PMP (phase measurement profilometry) method and the triangulation method, the front surface point cloud and the back surface point cloud of the high-precision matte ceramic plate are obtained, and the filtering processing is carried out based on the point cloud processing algorithm;
[0095] S413: The reference coordinate system of the back surface point cloud of the high-precision matte ceramic plate is changed from the second camera coordinate system to the first camera coordinate system, and the calculation mode is:
[0096]
[0097] Wherein, (X C21 ,Y C21 ,Z C21 ) represents the coordinates of the back surface point cloud of the high-precision matte ceramic plate in the first camera coordinate system, (X C2 ,Y C2 ,Z C2 ) represents the coordinates of the back surface point cloud of the high-precision matte ceramic plate in the second camera coordinate system;
[0098] S414: According to the coordinates (X C1 ,Y C1 ,Z C1 ) of the front surface point cloud of the high-precision matte ceramic plate in the first camera coordinate system, and based on the point cloud processing algorithm, the front surface point cloud of the high-precision matte ceramic plate is fitted, and the plane equation is output;
[0099] S415: Based on (X C21 ,Y C21 ,Z C21distance variance of the reconstructed points (X
[0100] J T *J*h=J T E
[0101] where J represents the Jacobian matrix of the rotation vector r', the 3*3 orthogonal rotation matrix R' is converted into the 3*1 rotation vector r' through the Rodrigues transformation, h represents the iterative vector of r' (i.e. the iterative direction and step size of the rotation vector r' in each component), E represents the residual matrix, i.e. the distance variance calculated based on the current rotation vector, minimizing E is a nonlinear least squares problem, which can be solved through the Levenberg-Marquardt algorithm, so as to obtain the rotation matrix R' with high accuracy;
[0102] Under ideal conditions, the two converted planes are parallel to each other, but the manufacturing accuracy of the self-made stereo calibration board is greatly different from that of the commonly used industrial calibration board, and the second camera to the first camera
[0103] This can be based on minimizing the distance variance of the reconstructed points (X C21 ,Y C21 ,Z C21 ) on the back of the plate to the fitting plane on the front of the high-precision matte ceramic plate, so as to optimize the rotation matrix R' in the initial value of the external parameter, and the translation matrix T' only affects the average distance, and thus is not involved in the optimization.
[0104] S416: Place the high-precision matte ceramic ball at the center of the reconstruction range, and make the first camera shoot the front of the high-precision matte ceramic ball, and the second camera shoot the back of the high-precision matte ceramic ball;
[0105] S417: Reconstruct the high-precision matte ceramic ball based on the PMP method and the triangulation method, and obtain the coordinates (X G1 ,Y G1 ,Z G1 ) of the point cloud in front of the high-precision matte ceramic ball in the first camera coordinate system, and the coordinates (X G2 ,Y G2 ,Z G2 ) of the point cloud behind the high-precision matte ceramic ball in the second camera coordinate system;
[0106] S418: Use the optimized rotation matrix R' and the unoptimized translation matrix T' to obtain the coordinates (X G21 ,Y G21 ,Z G21); that is, referring to the above formula for changing the reference coordinate system of the back side point cloud of the high-precision matte ceramic flat plate from the second camera coordinate system to the first camera coordinate system, using the optimized rotation matrix R' and the unoptimized translation matrix T', the coordinates (X G21 G21 G21 ) of the back side point cloud of the high-precision matte ceramic ball in the first camera coordinate system are obtained.
[0107] S419: Based on the point cloud processing algorithm, the spherical point cloud of the high-precision matte ceramic ball is fitted, the spherical center coordinates (X SC1 SC1 SC1 ) of the front side point cloud of the high-precision matte ceramic ball in the first camera coordinate system are output, and the spherical center coordinates (X SC2 SC2 SC2 ) of the back side point cloud of the high-precision matte ceramic ball are calculated, and the spherical center distance ΔT is calculated:
[0108]
[0109] Wherein, the spherical center distance ΔT is the error of T' in three components, and the high-precision translation matrix T is obtained by correcting the error, and the calculation method is T=T'+ΔT;
[0110] S410: The position transformation matrix [R T] of the optimized second camera coordinate system to the first camera coordinate system is output.
[0111] S5: Based on the reference coordinate system, the overall calibration of the multi-camera multi-projector three-dimensional measurement platform is completed, the object to be measured is placed in the reconstruction range, the world coordinate system is established on the reference camera coordinate system, and the multi-view fusion point cloud of the object to be measured is obtained according to the position conversion matrix between the cameras.
[0112] That is, the object to be measured is placed in the center of the reconstruction range, and after the multi-view point cloud of the object to be measured is obtained, the multi-view fusion point cloud of the object to be measured can be output according to the above formula for changing the reference coordinate system of the back side point cloud of the high-precision matte ceramic flat plate from the second camera coordinate system to the first camera coordinate system, and the optimized [R T].
[0113] In the present application, when the reconstruction standard part is reconstructed based on the PMP method and the triangulation method, the trigger mode between the camera and the projector in the single camera single projector measurement system is mutual external trigger mode; the mutual external trigger mode is that the camera collects the projected image after the projector completes the projection of the current image, and the projector projects the next image after the camera completes the collection of the current projected image, and so on.
[0114] Specifically, in the present application, the first camera and the second camera are both JAIGO-5000M-USB cameras, the projector is Texas Instruments DLP LightCrafter4500, and the working distance of the camera and the projector is set to 550mm; the industrial calibration board used in the calibration stage is a 14*11 circular pattern calibration board with a center distance of 30mm, the self-made three-dimensional calibration board is an 11*8 circular pattern calibration board with a center distance of 20mm, the back of the self-made three-dimensional calibration board has the same size of calibration patterns and is aligned front and back; the encoding mode of the fringe sequence pattern used in the phase measurement stage is seven-bit complementary Gray code plus four-step phase shift; the high-precision matte ceramic flat plate used in the parameter optimization stage has a size of 150mm*150mm*5mm, a flatness of 0.05mm, and a high-precision matte ceramic ball diameter of 25.4162mm, and a ball roundness of 0.00105mm.
[0115] In the specific implementation process, when the high-precision matte ceramic ball is reconstructed, the measurement accuracy of the single-view reconstruction using the method of the present application is 40.439um and 39.193um respectively, and the measurement accuracy of the multi-view reconstruction is about 42.631um, which is close to the accuracy of single-view reconstruction, indicating the effectiveness and accuracy of the present application.
[0116] The present application is composed of multiple single-camera single-projector measurement systems, and there is no need for field of view overlap between the cameras of each single-camera single-projector measurement system; the Zhang calibration method is used to calibrate the cameras in a single single-camera single-projector measurement system, and the pseudo-camera method is used to calibrate the projector; the initial estimate of the position conversion matrix between multiple cameras is obtained through the three-dimensional calibration board, and the error is minimized based on the three-dimensional constraints of the standard parts; by establishing the world coordinate system on the reference camera coordinate system during the reconstruction process, the multi-view point cloud fusion is realized based on the position conversion matrix of other cameras to the reference camera. The present application does not need point cloud splicing process and does not need field of view overlap between multiple cameras, and under the condition of the same reconstruction range, high-precision real-time multi-view three-dimensional reconstruction can be realized using relatively less equipment, the overall structure of the system is simple, the hardware cost is low, the installation is flexible and convenient, and the maintainability is strong.
[0117] The real-time multi-camera multi-projector 3D imaging processing device provided by the embodiment of the present application comprises a building module, a calibration module, an acquisition module, a reconstruction module and an execution module.
[0118] The building module is used for building a multi-camera multi-projector three-dimensional measurement platform composed of a computer and a plurality of single-camera single-projector measurement systems, and creating an industrial calibration board, a self-made stereo calibration board and a reconstruction standard part, the single-camera single-projector measurement system comprising one camera and one projector; the calibration module is used for calibrating the camera and the projector in the single-camera single-projector measurement system, and obtaining the internal parameters of the camera and the projector, the distortion coefficients, and the position conversion matrix of the projector coordinate system to the camera coordinate system; the acquisition module is used for acquiring the initial estimation of the position conversion matrix of other cameras to the reference coordinate system based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, using the self-made stereo calibration board, and taking any camera coordinate system as a reference coordinate system; the reconstruction module is used for reconstructing the reconstruction standard part based on the PMP method and the triangulation method, and minimizing the error of the initial estimation of the position conversion matrix of other cameras to the reference coordinate system according to the stereo constraint of the reconstruction standard part; and the execution module is used for completing the overall calibration of the multi-camera multi-projector three-dimensional measurement platform based on the reference coordinate system, placing the object to be measured in the reconstruction range, establishing the world coordinate system on the reference camera coordinate system, and obtaining the multi-view fusion point cloud of the object to be measured according to the position conversion matrix between the cameras.
[0119] The above description is merely a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
[0120] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or a plurality of flows and / or blocks Figure 1 one flow or a plurality of flows and / or blocks
Claims
1. A real-time multi-camera multi-projector 3D imaging processing method, characterized in that, Specifically comprising the following steps: A multi-camera multi-projector three-dimensional measurement platform composed of a computer and a plurality of single-camera single-projector measurement systems is built, and an industrial calibration board, a self-made stereo calibration board and a reconstructed standard part are created, the single-camera single-projector measurement system comprising one camera and one projector; The camera and the projector in the single-camera single-projector measurement system are calibrated, and the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix of the projector coordinate system to the camera coordinate system are obtained; Based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, the self-made stereo calibration board is used to obtain the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system with any camera coordinate system as the reference coordinate system; The reconstructed standard part is reconstructed based on the PMP method and the triangulation method, and the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereo constraint of the reconstructed standard part; The overall calibration of the multi-camera multi-projector three-dimensional measurement platform is completed based on the reference coordinate system, the object to be measured is placed within the reconstruction range, the world coordinate system is established on the reference camera coordinate system, and the multi-view fusion point cloud of the object to be measured is obtained according to the position conversion matrix between the cameras; The multi-camera multi-projector three-dimensional measurement platform comprises two single-camera single-projector measurement systems, one of which comprises a first camera and a first projector, and the other of which comprises a second camera and a second projector; The fields of view of the lenses of the first camera and the second camera do not overlap, and the working distances are equal; The working distances of the lenses of the first projector and the second projector are equal; The reconstructed standard part is reconstructed based on the PMP method and the triangulation method, and the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereo constraint of the reconstructed standard part, wherein the reconstructed standard part is reconstructed based on the PMP method and the triangulation method, and the specific steps include: The computer controls the projector to project the coded grating image onto the surface of the reconstructed standard part, and the cameras in the multi-camera multi-projector three-dimensional measurement platform sequentially collect the surface images of the reconstructed standard part; The collected surface images are denoised based on the digital image processing method, and the denoised surface images are decoded to calculate the absolute phase values of all the pixel points within the reconstruction range, and the decoding mode is complementary Gray code plus four-step phase shift; The three-dimensional point coordinates corresponding to all the pixel points are solved by using the triangulation method in combination with the absolute phase values and the internal parameters, the distortion coefficients of the camera and the projector, and the position conversion matrix of the projector coordinate system to the camera coordinate system.
2. The real-time multi-camera multi-projector 3D imaging processing method according to claim 1, wherein: The self-made stereo calibration board is created by aligning the positions of the same calibration pattern printing papers on both sides of a wooden board with a flat surface and uniform thickness, and obtaining the conversion matrix from the back target coordinate system of the self-made stereo calibration board to the front target coordinate system of the self-made stereo calibration board. The reconstructed standard part is a high-precision matte ceramic flat plate and a high-precision matte ceramic ball.
3. A real-time multi-camera multi-projector 3D imaging processing method as claimed in claim 2, characterized in that, The camera and the projector in the single-camera single-projector measurement system are calibrated, and internal parameters and distortion coefficients of the camera and the projector are obtained, and a position conversion matrix of a projector coordinate system to a camera coordinate system is obtained, and the specific steps include: Based on the industrial calibration board and Zhang's calibration method, the feature points in the industrial calibration board calibration image collected by the camera are obtained, and an imaging model is established to solve the camera calibration parameters: wherein, represents a scale factor, are feature point pixel coordinates in the industrial calibration plate calibration image, is a three-dimensional coordinate of the calibration point under the target coordinate system, represents a 3*3 camera intrinsic matrix, represents a conversion matrix from the target coordinate system to the camera coordinate system, is a 3*3 rotation matrix, is a 3*1 translation matrix; The LM algorithm is used to optimize the camera calibration parameters, and the optimization target is the re-projection error of the feature point pixels of the calibration image, so as to obtain the distortion coefficient of the camera; The internal parameters and the distortion coefficient of the projector are calibrated, and a conversion matrix of a target coordinate system to a projector coordinate system is obtained; Based on the conversion matrix of the target coordinate system to the camera coordinate system and the conversion matrix of the target coordinate system to the projector coordinate system, a position conversion matrix of the projector coordinate system to the camera coordinate system in the single-camera single-projector measurement system is calculated.
4. The real-time multi-camera multi-projector 3D imaging processing method according to claim 3, characterized in that: When calibrating the camera, the calibration image on the industrial calibration board is a light gray dot calibration image; When calibrating the projector, the projection pattern of the projector is a checkerboard calibration pattern, and the two-dimensional pixel coordinates of the calibration pattern feature points of the projector are pre-set, and the three-dimensional world coordinates of the calibration pattern feature points are calculated through the calibration parameters of the camera.
5. A real-time multi-camera multi-projector 3D imaging processing method as claimed in claim 2, wherein, Based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, the initial estimation of the position conversion matrix of the other cameras to the reference coordinate system is obtained using the self-made stereo calibration board with any camera coordinate system as the reference coordinate system, and the specific steps include: Based on the internal parameters and the distortion coefficients of all the cameras obtained by calibration, the initial estimation of the position conversion matrix of the second camera coordinate system to the reference coordinate system is obtained using the self-made stereo calibration board with the first camera coordinate system as the reference coordinate system, and the calculation method is: Wherein, the shooting side of the first camera is the front side of the self-made stereo calibration board, and the shooting side of the second camera is the back side of the self-made stereo calibration board, represents the initial estimation of the position conversion matrix of the second camera coordinate system to the reference coordinate system, is a 3*3 orthogonal rotation matrix, is a 3*1 translation matrix, and 0 is a 1*3 zero matrix, represents the conversion matrix of the front target coordinate system of the self-made stereo calibration board to the first camera coordinate system, represents the conversion matrix of the back target coordinate system of the self-made stereo calibration board to the front target coordinate system of the self-made stereo calibration board, represents the conversion matrix of the back target coordinate system of the self-made stereo calibration board to the second camera coordinate system.
6. A real-time multi-camera multi-projector 3D imaging processing method as claimed in claim 5, characterized in that, The PMP method and the triangulation method are used to reconstruct the reconstructed standard part, and the error of the initial estimation of the position conversion matrix of the other cameras to the reference coordinate system is minimized according to the stereo constraint of the reconstructed standard part, and the specific steps include: The high-precision matte ceramic flat plate is placed at the center of the reconstruction range, and the first camera shoots the front of the high-precision matte ceramic flat plate, and the second camera shoots the back of the high-precision matte ceramic flat plate; The PMP method and the triangulation method are used to obtain the front point cloud and the back point cloud of the high-precision matte ceramic flat plate, and filtering processing is performed; The reference coordinate system of the back point cloud of the high-precision matte ceramic flat plate is changed from the second camera coordinate system to the first camera coordinate system, and the calculation method is: wherein, represents the coordinates of the back side point cloud of the high-precision matte ceramic flat plate in the first camera coordinate system, represents the coordinates of the back side point cloud of the high-precision matte ceramic flat plate in the second camera coordinate system; According to the coordinates of the front face point cloud of the high-precision matte ceramic flat plate in the first camera coordinate system And based on a point cloud processing algorithm, fitting the front face point cloud of the high-precision matte ceramic flat plate, outputting a plane equation; Based on The distance variance from the fitting plane of the high-precision matte ceramic flat plate front surface to the rotation matrix in the initial value of the external parameter is optimized , and the optimization mode is wherein, is the Jacobian matrix of the rotation vector is a 3*3 orthogonal rotation matrix is a 3*1 rotation vector obtained by a Rodrigues transformation , is the iteration vector is the iteration vector is the residual matrix; The high-precision matte ceramic ball is placed at the center of the reconstruction range, and the first camera shoots the front of the high-precision matte ceramic ball, and the second camera shoots the back of the high-precision matte ceramic ball; Reconstruct the high-precision matte ceramic ball based on the PMP method and the triangulation method to obtain coordinates of the front point cloud of the high-precision matte ceramic ball in the first camera coordinate system , and coordinates of the back point cloud of the high-precision matte ceramic ball in the second camera coordinate system ; Using the optimized rotation matrix and the unoptimized translation matrix , obtain the coordinates of the back point cloud of the high-precision matte ceramic ball in the first camera coordinate system ; Based on the point cloud processing algorithm, the spherical point cloud of the high-precision matte ceramic ball is fitted, and the center coordinates of the front point cloud of the high-precision matte ceramic ball in the first camera coordinate system are output , and the center coordinates of the rear point cloud of the high-precision matte ceramic ball , and the center distance : Wherein, the center of the sphere is apart from is The error on three components, the correction error can get high-precision translation matrix , the calculation method is ; outputting a position transformation matrix of the optimized second camera coordinate system to the first camera coordinate system .
7. The real-time multi-camera multi-projector 3D imaging processing method according to claim 6, characterized in that: In the reconstruction of the reconstruction standard part based on the PMP method and the triangulation method, the triggering mode between the camera and the projector in the single camera single projector measurement system is mutual external triggering mode; The mutual external triggering mode is that the camera is triggered to collect the projected image after the projector finishes projecting the current image, and the projector projects the next image after the camera finishes collecting the current projected image, and so on.
8. A real-time multi-camera multi-projector 3D imaging processing apparatus, characterized in that, It comprises: a building module for building a multi-camera multi-projector three-dimensional measurement platform composed of a computer and a plurality of single camera single projector measurement systems, and creating an industrial calibration board, a self-made stereo calibration board and a reconstruction standard part, wherein the single camera single projector measurement system comprises one camera and one projector; a calibration module for calibrating the camera and the projector in the single camera single projector measurement system, and obtaining the internal parameters, distortion coefficients of the camera and the projector, and the position conversion matrix from the projector coordinate system to the camera coordinate system; an acquisition module for acquiring the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system based on the internal parameters and distortion coefficients of all cameras obtained by calibration, using the self-made stereo calibration board, and taking any camera coordinate system as the reference coordinate system; a reconstruction module for reconstructing the reconstruction standard part based on the PMP method and the triangulation method, and minimizing the error of the initial estimate of the position conversion matrix of the other cameras to the reference coordinate system according to the stereo constraint of the reconstruction standard part; an execution module for completing the overall calibration of the multi-camera multi-projector three-dimensional measurement platform based on the reference coordinate system, placing the object to be measured within the reconstruction range, establishing the world coordinate system on the reference camera coordinate system, and obtaining the multi-view fusion point cloud of the object to be measured according to the position conversion matrix between the cameras; The multi-camera multi-projector three-dimensional measurement platform comprises two single camera single projector measurement systems, one of which comprises a first camera and a first projector, and the other of which comprises a second camera and a second projector; The fields of view of the lenses of the first camera and the second camera do not overlap, and the working distances are equal; The working distances of the lenses of the first projector and the second projector are equal; The computer controls the projector to project the encoded grating image onto the surface of the reconstruction standard part, and the cameras in the multi-camera multi-projector three-dimensional measurement platform collect the surface images of the reconstruction standard part in turn; The collected surface images are denoised based on the digital image processing method, and the denoised surface images are decoded to calculate the absolute phase values of all pixel points in the reconstruction range, and the decoding mode is complementary Gray code plus four-step phase shift. The internal parameters of the camera and the projector, the distortion coefficients, and the position conversion matrix from the projector coordinate system to the camera coordinate system are obtained by combining the absolute phase values and calibration, and all pixel point corresponding three-dimensional point coordinates are solved by using triangulation method.