Calibration and Point Cloud Fusion Methods for Multi-Camera Single-Projector 3D Reconstruction Systems

Abstract

A calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system is presented. This method involves arranging at least four cameras with multiple viewing angles and projecting a pattern onto the surface of the object to be measured using a projector. The projector-camera system and the extrinsic parameters of each camera are calibrated. Finally, 3D point cloud data is acquired and fused based on the calibration of the projector-camera system and the extrinsic parameters of each camera. This invention utilizes multi-camera joint calibration and data fusion to improve the accuracy and robustness of calibration, enhance the accuracy of point cloud fusion, and solve the accuracy problem of traditional 3D reconstruction methods when acquiring highly reflective, low-reflectivity objects.

Description

Technical Field

[0001] This invention relates to three-dimensional reconstruction technology, specifically to a calibration and point cloud fusion method for a multi-camera single-projector three-dimensional reconstruction system. Background Technology

[0002] In the fields of computer vision and image processing, 3D reconstruction technology has been widely applied in virtual reality, augmented reality, and industrial inspection. To achieve high-precision 3D reconstruction, traditional monocular systems consisting of a single camera and a single projector often fall short when dealing with objects exhibiting high reflectivity and low reflectivity. In particular, they cannot achieve high-precision 3D reconstruction of tiny objects (chips of approximately 200 micrometers) with locally high reflectivity and locally low reflectivity. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system. This method combines multi-camera joint calibration and data fusion, improving the accuracy and robustness of calibration, enhancing the accuracy of point cloud fusion, and solving the accuracy problem of traditional 3D reconstruction methods when acquiring objects with high reflectivity and low reflectivity.

[0004] To achieve the above technical objectives, the adopted technical solution is: a calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system, including the following steps:

[0005] Step 1: Select at least four cameras to cover all angles of the object under test and arrange them in a multi-view configuration; select a high-precision projector to project the pattern onto the surface of the object under test through a high-precision mirror.

[0006] Step 2: Calibration of the projector-camera system

[0007] Using a calibration board placed within the camera's field of view and projecting multiple Gray code patterns, at least four cameras are calibrated, and the intrinsic parameters of each camera are calculated. At the same time, the projector is precisely calibrated, and the extrinsic parameters between the camera and the projector are calculated.

[0008] Step 3: Calibrate the external parameters of each camera

[0009] Step 3.1: Using the corner coordinates of the calibration board and the extrinsic parameters between the camera and the projector, calculate the corrected corner coordinates of the calibration board using RANSC. Then, calculate the homography matrix between the corrected corner coordinates of the calibration board and the world coordinate system coordinates of the calibration board. Based on this homography matrix, optimize the corner coordinates of the calibration board.

[0010] Step 3.2: Based on the optimized calibration board corner coordinates in 3.1 and the pre-set world coordinate system coordinates of the calibration board corners, calculate the extrinsic parameters of each camera using the solvePnP model, and unify the extrinsic parameters between a certain camera and the other cameras as a reference coordinate system;

[0011] Step 4: 3D point cloud data acquisition and fusion

[0012] Step 4.1: After the projector projects the phase-shift code onto the surface of the object to be measured, multiple cameras simultaneously acquire the projected pattern. These patterns are then used to calculate the depth information using the phase-shift method, thereby generating preliminary 3D point cloud data for each camera.

[0013] Step 4.2: Based on the extrinsic parameters of each camera mentioned in Step 3.3, convert the preliminary 3D point cloud data acquired by all cameras to the reference coordinate system.

[0014] Step 4.3: Remove obvious noise from the preliminary 3D point cloud data obtained by all cameras in Step 4.2, which has been transformed to the reference coordinate system. Calculate the weight based on the image information from which each pixel originates, and modify the actual depth value within each pixel according to the weight.

[0015] Furthermore, the specific steps of step 2 are as follows:

[0016] Step 2.1: All cameras acquire multiple sets of calibration images from different positions within their respective viewpoints;

[0017] Step 2.2: Based on the multiple sets of calibration images acquired by each camera, use Zhang's calibration method to calibrate each camera using the brightest image in each set of calibration images to obtain the intrinsic parameters of all cameras; use the Gray code decoding formula to decode each set of calibration image data acquired by each camera.

[0018] Step 2.3: The decoded data is used as the pixel coordinates of the calibration board. For each corner point of the calibration board and the selected area around it, the Gray code value of each pixel in the selected area and the coordinates of the corresponding camera pixel are marked. Then, the noise is removed to obtain the correspondence between the projector pixels and the camera pixels.

[0019] Step 2.4: Using the correspondence between projector pixels and camera pixels obtained in Step 2.3, calculate the homography matrix between projector pixels and camera pixels, update the Gray code value of the calibration board corner point according to the homography matrix, and obtain the optimized correspondence between projector pixels and camera pixels.

[0020] Step 2.5: Approximate the projector model as a camera model and use Zhang's calibration method to perform precise calibration of the projector;

[0021] Step 2.6: Based on the optimized correspondence between projector pixels and camera pixels obtained in Step 2.4, calculate the fundamental matrix or essential matrix between the camera and the projector, and perform singular value decomposition on it to extract the extrinsic parameters between the camera and the projector.

[0022] Furthermore, the specific implementation method of step 2.1 is as follows: place the calibration board in the common field of view of all cameras, project multiple Gray code patterns onto the calibration board in sequence, and capture the Gray code patterns on the calibration board as a set of calibration images by all cameras. Adjust the position of the calibration board multiple times and repeat the projection of multiple Gray code patterns and camera capture.

[0023] Furthermore, the method for calculating the corrected calibration board corner coordinates in step 3.1 is as follows: using the calibration board corner coordinates and the extrinsic parameters between the camera and the projector, the three-dimensional coordinates of each calibration board corner are calculated. Then, a plane is fitted to all calibration board corners using RANSC. A straight line is drawn perpendicular to the fitted plane for all three-dimensional corners. The intersection of the plane and the straight line is regarded as the more accurate three-dimensional corner coordinates. According to the camera imaging formula, the positions of these more accurate three-dimensional corners in the image are calculated to obtain the corrected calibration board corner coordinates.

[0024] Furthermore, the method for removing obvious noise in step 4.3 is to remove points that not only have a large spatial distance from other points within the same pixel grid, but also have a large spatial distance from surrounding points.

[0025] Furthermore, the high-precision projector can project patterns onto the surface of the object being measured through a high-precision mirror.

[0026] The beneficial effects of this invention are as follows: This patent achieves complete 3D object reconstruction through calibration and point cloud fusion using a multi-angle camera and a single projector. Its innovation lies in utilizing homography matrix optimization of Gray code-encoded projector intrinsic parameter calibration, improving calibration accuracy; simultaneously, it optimizes corner detection of 2D images based on 3D information, enhancing image processing accuracy. Furthermore, the data fusion technology in the patent effectively addresses the shortcomings of single-view cameras in handling specular highlights, improving reconstruction quality. It can not only quickly and efficiently reconstruct the 3D structure of larger objects but also accurately capture and reconstruct the details of tiny objects, demonstrating the system's high precision and high resolution performance. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention;

[0028] Figure 2 This is a system diagram of the present invention;

[0029] Figure 3 To calibrate patterns for different camera perspectives;

[0030] Figure 4 Gray code pattern from the same camera;

[0031] Figure 5 This is a screenshot of the data processing software interface.

[0032] Figure 6 Comparison images of a chip and a human hand;

[0033] Figure 7 A reconstruction rendering using a single camera and single projector;

[0034] Figure 8 This is a rendering of the chip point cloud before fusion.

[0035] Figure 9 Image showing the result of point cloud fusion;

[0036] Figure 10 A reconstruction image of a single-camera, single-projector patch.

[0037] Figure 11 This is a multi-view fusion effect diagram of this application;

[0038] In the diagram: 1. The object to be tested, 2. Camera, 3. Projector, 4. High-precision mirror. Detailed Implementation

[0039] The preferred embodiments of the invention are given below with reference to the accompanying drawings to illustrate the technical solution of the invention in detail. The corresponding drawings will be provided for detailed explanation of the invention. It should be particularly noted that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit or restrict the invention.

[0040] This invention proposes an innovative camera projector calibration and fusion method in a multi-camera system. By combining the advantages of a multi-camera system, it optimizes the calibration and point cloud fusion process of a multi-camera single-projector system, enabling high-precision 3D reconstruction of objects with locally high reflectivity and locally low reflectivity characteristics. For example, it is suitable for tiny objects (e.g., chips approximately 200 micrometers in size). Figure 1 As shown, the method of the present invention includes the following steps: Step 1, selection and setup of a multi-camera 3D reconstruction system:

[0041] 1.1 Camera Setup and Selection:

[0042] Choose a high-resolution, low-noise industrial camera to ensure image quality. The camera should have good optical performance and stable imaging, providing consistent image quality under different lighting conditions.

[0043] At least four cameras should be used in a multi-view arrangement to cover as many angles of the object as possible. This multi-camera layout can effectively reduce occlusion and reflection problems caused by a single viewpoint, and improve the comprehensiveness and integrity of image acquisition.

[0044] Adjust and optimize the position and angle of the cameras. The position of each camera is evenly distributed around the calibration scene, and the height and tilt angle are adjusted according to the size and shape of the object being measured to ensure that each camera can clearly capture different parts of the object. It has been verified that for targets with relatively simple structures (such as chip boards), the four-camera system can ensure that each area of ​​the chip can be captured by at least two cameras simultaneously.

[0045] 1.2 Projector Setup and Selection:

[0046] Choosing a high-precision, stable projector ensures consistent projection quality under varying lighting conditions. The projector should also have high resolution and brightness to guarantee the clarity and contrast of the projected image.

[0047] To ensure sufficient field of view for subsequent test cameras directly above the system, the projector can be positioned on one side of the system center, reflecting the projected pattern onto the surface of the object under test via a high-precision mirror. This design ensures that the projected pattern covers the entire surface of the object under test while projecting without interfering with the camera's field of view. The projector's position and angle are optimized based on the object's size and the spatial layout of the calibration scene to ensure the uniformity and clarity of the projected pattern.

[0048] Step 2, Projector-Camera System Calibration:

[0049] 2.1 First, taking four cameras as an example, place the calibration board in the common field of view of all cameras. Use a projector to project multiple Gray code patterns sequentially, ensuring all cameras can capture these patterns synchronously. The Gray code patterns include both horizontal and vertical Gray code patterns, both positive and negative. If the projector has 960 rows, then the binary representation needs to be 2 to the power of 10, equal to 1024, requiring 10 images to represent the Gray code. Considering the positive and negative orientation, 20 horizontal Gray code images are needed. If the vertical orientation has 1440 columns, then 22 vertical Gray code images are needed. Finally, adding all-black and all-white images, each set contains 44 images, meaning each camera must acquire at least 44 Gray code images at a single location. To ensure calibration accuracy, the position of the calibration board needs to be adjusted, and image data acquisition repeated multiple times. After each adjustment of the calibration board position, continue projecting Gray code patterns and synchronously acquiring data to obtain multiple sets of calibration images from different positions and perspectives of the four cameras.

[0050] 2.2 Based on multiple sets of calibration images acquired by each camera, for the brightest image in each set (the camera image corresponding to pure white light projected by the projector), Zhang's calibration method is used, following the formula...

[0051] M i =K[R|T]m i

[0052] Calibrate each camera, where m i M represents the three-dimensional coordinates of the i-th point in the world coordinate system. i Let K represent the two-dimensional pixel position of the i-th point in the image coordinate system, K represent the intrinsic parameter matrix of the camera, R represent the rotation matrix, and T represent the translation vector. This yields the intrinsic parameters of all cameras.

[0053] Next, using the Gray code decoding formula, the calibration board image data acquired by each camera at different poses (calibration board in different positions) are decoded. Specifically, for each set of calibration board data, the brightness changes of each pixel in multiple images are analyzed. For the grayscale value of the same pixel location in different images, brightness below the average is recorded as 0, and brightness above the average is recorded as 1. Then, the Gray code decoding formula is applied:

[0054]

[0055] Among them G n B represents the Gray code value of the nth bit. n This represents the value of the nth binary bit. This represents the XOR operator, B0 = G0. It retrieves the unique encoded value for each pixel on the calibration board. These encoded values ​​will be used for subsequent coordinate mapping and geometric calculations.

[0056] Since the projector projects two sets of Gray code patterns, namely horizontal Gray code patterns and vertical Gray code patterns, each pixel can ultimately obtain two unique decoded values, corresponding to the horizontal Gray code and the vertical Gray code respectively.

[0057] 2.3. In the decoded data, a 40*40 pixel area around each calibration board corner is sampled. The selection of the region's pixels is based on the camera resolution; for example, here the camera's resolution is 5120*5120. The Gray code value of each pixel i within the sampled region is denoted as G(a). i b i ), a i b is the horizontal Gray code value. i The vertical Gray code value is represented by the camera pixel coordinates (x). i y i Let P be the corner point of the corner calibration plate. ij The sampling pixel region is R ijIts range is R ij ={(x, y)|x i -20≤x≤x i +20,y i -20≤y≤y i +20} Obtain the Gray code decoding value of each pixel within this region. Utilize the monotonically increasing or decreasing property of the Gray code decoding value in the row and column directions for noise reduction. Specifically, if the decoding value of a pixel i satisfies the following in the row direction:

[0058] G(a i-1 b i )≤=G(a i b i )≤G(a i+1 b i )

[0059] Simultaneously, the following conditions are met in the column direction:

[0060] G(a i b i-1 )≤=G(a i b i )≤G(a i b i+1 )

[0061] If the pixel is deemed valid, its decoded value is considered valid. Otherwise, it is marked as invalid and deleted. This method effectively removes noise interference and improves the accuracy of the decoded value. After denoising, the Gray code decoded value of each pixel (i.e., the projector imaging position) is obtained, along with the correspondence between the pixel and other points in space (camera pixels).

[0062] 2.4 The projector pixel P obtained through the above steps i With camera pixels p i The correspondence is calculated using the homography matrix formula: p i =HP i The 3×3 homography matrix between the projector pixels and the camera pixels is calculated. The homography matrix describes the geometric mapping between the projector coordinate system and the camera coordinate system. Using this homography matrix, more precise projector coordinates corresponding to each calibration board corner point can be calculated, i.e., the Gray code value G(a) of each calibration board corner point can be updated. i b i This step involves obtaining the optimized correspondence between projector pixels and camera pixels. The purpose of this step is to ensure that the position and angle of each pixel are optimally corrected through geometric transformation. Simultaneously, due to the Gray code G(a)... i b iThe code value corresponds to a single specific pixel position of the projector, and each corner point corresponds to a three-dimensional point in the world coordinate system. At this point, the relationship between multiple pixels of the projector and points in the world coordinate system is established.

[0063] 2.5 In this invention, the projector model is approximated as a camera model. The coordinate parameters of the projector are obtained by establishing a mapping relationship between the two-dimensional pixels of the projector and the actual three-dimensional points in space. That is, the projector model is approximated as a camera model, and the specific pixel positions of the projector at these corner points are determined using the precise Gray code decoding values ​​of each corner point on the calibration board. The tensor calibration method is then used to achieve precise calibration of the projector. Precise calibration of the projector is performed to improve the accuracy of obtaining the initial point cloud.

[0064] 2.6 During the relative position calibration process between the camera and the projector, the coordinates of the corner points of the calibration board in the camera image are extracted using a corner detection algorithm, based on the image captured by the camera containing the calibration board. Simultaneously, according to the specific pattern projected by the projector, the corresponding projector pixel coordinates are obtained through a decoding method, establishing the corner point correspondence between the camera image and the projector image. The optimized projector pixel P... i With camera pixels p i The correspondence between the camera and the projector is determined by calculating the fundamental or essential matrix between them, performing singular value decomposition, and extracting the rotation matrix, which is the extrinsic parameter between the camera and the projector.

[0065] Step 3: Calibrate the external parameters of each camera:

[0066] 3.1 First, obtain the original coordinates of the corner points and the extrinsic parameters between the camera and the projector through steps 2.2 and 2.3. Calculate the three-dimensional coordinates of each corner point. Then, for all corner points, use RANSC to fit a plane. Next, draw a straight line perpendicular to the fitted plane for all three-dimensional corner points. The intersection of the plane and the straight line is regarded as the more accurate three-dimensional corner point coordinates. According to the camera imaging formula, calculate the position of these more accurate three-dimensional corner points in the image to obtain the corrected corner points. The new coordinates of the corrected corner points are then used to calculate the homography matrix with the (x, y) values ​​of the calibration board corner points in the world coordinate system. Based on this homography matrix, the coordinates of the calibration board corner points in the image coordinate system are then optimized.

[0067] The specific implementation process is as follows: After obtaining the relatively accurate 3D coordinates of each corner point of the calibration board in the camera coordinate system, the RANSAC algorithm is used to further improve the coordinate accuracy. First, the RANSAC algorithm is used to fit the planar coordinates of the calibration board where the current point cloud is located, obtaining the coordinates of each point cloud on the calibration board plane. Specifically, assuming the coordinates of point cloud P in the camera coordinate system are (x, y, z), the equation of the calibration board plane is aX + bY + cZ + d = 0. The RANSAC algorithm is used to fit the calibration board plane parameters a, b, c, d based on the known camera coordinates of each corner point. Then, point cloud P is mapped onto the calibration board plane to obtain the projected coordinates p. p =(x p y p If the coordinates of the point are 0, update this point to the new corner coordinates in the image coordinate system. Then, based on the new image coordinates of each corner point and the (x, y) values ​​of their world coordinates, calculate the new homography mapping matrix H2, such that p i =H2P i Furthermore, using this homography matrix, more precise image coordinates are calculated for each calibrated contra-angle point. This process effectively reduces errors caused by insufficient projector resolution, thereby improving the accuracy of spatial coordinates.

[0068] 3.2. Based on the optimized calibration board corner coordinates from 3.1 and the pre-set actual 3D spatial coordinates of the calibration board corners, the solvePnP model is used to calculate the extrinsic parameter matrix between each camera. Specifically, only the pose relationships between three pairs of cameras—camera 1 and camera 2, camera 1 and camera 3, and camera 1 and camera 4—need to be calculated. Using this model and the positional relationships of the calibration board corners, the rotation matrix and translation vector between each pair of cameras, i.e., the extrinsic parameters, can be accurately solved.

[0069] Furthermore, to ensure simplicity and consistency in the calculation process, the extrinsic parameters between each camera and camera number 1 (which can be randomly selected) are used as the reference coordinate system. This means that the extrinsic parameters of all cameras are defined and transformed relative to camera number 1. In this way, it is ensured that the extrinsic parameters of each camera are described under the same reference system, thereby unifying the coordinate systems of all cameras. This step is crucial because it lays the foundation for subsequent point cloud data alignment and fusion. Unifying the extrinsic parameters not only simplifies the calculation process but also improves the consistency and accuracy of data processing.

[0070] Step 4: 3D point cloud data acquisition and fusion:

[0071] 4.1 Three-dimensional reconstruction using pure phase-shifted codes

[0072] For high-precision 3D reconstruction systems, the projector projects phase-shifted codes onto the surface of the object under test. Since the distance between the object and the projector is already highly defined, there is no need for the camera to actually capture the Gray code. The approximate range of the Gray code for each pixel in the image can be determined solely through pre-calibrated information (camera calibration and projector calibration). In other words, by decoding the phase-shifted code, the actual Gray code value can be uniquely determined based on the phase-shifted code value. Specifically, for a pixel (x... i y i The decoded value is calculated in advance and can only be a value between 6 and 9. Since the phase shift code occupies multiple rows of pixels in the projector image, 4-7 is a Gray code for one pixel period, and 8-10 is another group. When the phase shift code decoded value P is between 0 and π, the actual decoded value can be considered to be 8+2P / π. When the phase shift code decoded value is between π and 2π, the actual decoded value can be considered to be 6+2P / π.

[0073] After the projector projects the phase-shifted code, multiple cameras simultaneously capture the projected pattern. These patterns are then used to calculate depth information using the phase-shifting method, thereby generating preliminary 3D point cloud data. This method eliminates the time required for projecting Gray code patterns, enabling rapid and efficient acquisition of high-precision 3D data.

[0074] 4.2 Preliminary processing of point cloud data

[0075] Throughout the reconstruction process, depth maps and their corresponding point cloud information from all reconstruction records are retained. After reconstruction, based on the extrinsic parameters of each camera obtained in section 3.2, the preliminary 3D point cloud data acquired by all cameras are uniformly converted to the pre-defined coordinate system of camera 1. The purpose of this step is to unify all point cloud data into the same reference coordinate system to facilitate subsequent fusion processing.

[0076] 4.3 Fusion and Optimization of Point Cloud Data

[0077] After transforming all point cloud data to the coordinate system of camera 1, each pixel in the depth map of camera 1 may correspond to multiple different depth information from different cameras. To obtain more accurate 3D reconstruction results, this depth information needs to be processed and fused. Processing and fusion can be performed region by region to further improve reconstruction accuracy.

[0078] First, obvious noise is removed. Specifically, multiple point cloud images are reprojected onto the image acquired by camera 1, and outlier removal is performed on multiple points within the same pixel. Points that are not only spatially distant from other points within the same pixel grid but also spatially distant from surrounding points are removed. Such points are usually noise caused by acquisition errors or reflections; removing these points improves data accuracy.

[0079] Next, weighting parameters (weights) are calculated based on the image information sourced by each pixel, and the actual depth value within each pixel is modified according to these weights. The specific approach is as follows: For each pixel obtained from phase-shift coding, if the grayscale value of that pixel changes significantly across multiple images, its confidence level is higher, because a large change in grayscale value usually means that the measurement of that pixel is more reliable. Conversely, if the grayscale value shows obvious anomalies, such as reaching 255, it usually indicates that the point is a reflective point, its confidence level is low, and it may even need to be removed. Let: ΔI = I max -I min , where I max It is the maximum pixel brightness across multiple images corresponding to each point cloud, I min The minimum value is used, and the weight is calculated based on the difference in grayscale values:

[0080]

[0081] Based on the confidence level of each point cloud, a weighted average is calculated for multiple pixels within the same pixel cell in the final depth map to obtain the actual depth value of the point cloud at that location. Specifically, for multiple points within the same pixel cell, a weighted average is calculated based on the confidence level of each pixel. Points with higher confidence levels have a larger weight in the weighted average, while points with lower confidence levels have a smaller weight, or are even removed.

[0082] Z = ∑ i w i z i

[0083] Where Z represents the final depth, z i w represents the depth value at that point. i This indicates the weight of that point.

[0084] After the above processing, the fused point cloud data is finally obtained, which constitutes a complete 3D model. This data is not only highly accurate but also has low noise, providing a reliable foundation for subsequent chip precision calculations and processing.

[0085] Example 1

[0086] 1. Selection and setup of multi-camera 3D reconstruction system

[0087] 1.1 Camera Setup and Selection

[0088] In this embodiment, we selected the high-resolution, low-noise Hikvision MV-CH250-90TM-C-NF industrial camera to ensure image quality. This camera features a 25-megapixel resolution, is compatible with C-mount lenses, possesses excellent optical performance and stable imaging capabilities, and can provide consistent image quality under various lighting conditions.

[0089] Four cameras were selected and arranged in a multi-view configuration to cover all key perspectives of the object. This multi-camera layout effectively reduces occlusion and reflection problems caused by a single viewpoint, improving the comprehensiveness of image acquisition and data integrity.

[0090] In practice, we adjusted and optimized the position and angle of the cameras. Each camera was evenly distributed around the calibration scene, and its height and tilt angle were adjusted according to the size and shape of the object being measured to ensure that each camera could clearly capture different parts of the object.

[0091] 1.2 Projector Setup and Selection

[0092] Choose the DLP4710 signal projector, one of the most advanced projectors available today, with a 1080*1920 resolution that ensures clarity and contrast of the projected image.

[0093] To ensure sufficient field of view for subsequent test cameras directly above the system, we positioned the projector on one side of the system's center. The projector reflects the projected pattern onto the surface of the object under test using a high-precision mirror. This design ensures that the projected pattern covers the entire surface of the object while allowing projection without interfering with the camera's field of view.

[0094] The position and angle of the projector are optimized and adjusted according to the size of the object and the spatial layout of the calibrated scene to ensure the uniformity and clarity of the projected pattern.

[0095] The structure of the entire system is as follows Figure 2 As shown, the fourth camera is not displayed because it is obstructed by the middle camera.

[0096] 2. Calibration of the projector-camera system

[0097] 2.1 Calibration board setup and image acquisition

[0098] First, place the calibration board in the common field of view of all cameras. Project Gray code patterns sequentially using a projector, ensuring all cameras can capture these patterns synchronously. To ensure calibration accuracy, the position of the calibration board needs to be adjusted, and image data acquisition repeated multiple times. After each adjustment of the calibration board position, continue projecting Gray code patterns and synchronously acquiring data to obtain calibration images from multiple perspectives and different positions. Calibration images from different camera perspectives are shown below. Figure 3 As shown, the Gray code pattern of the same camera is as follows: Figure 4 As shown.

[0099] 2.2 Calibration of Camera Intrinsic Parameters

[0100] Based on the calibration board patterns acquired by each camera, Zhang's calibration method is used to calibrate each camera using the brightest full white light image in each dataset, obtaining the intrinsic parameters of all cameras. These intrinsic parameters include the camera's focal length, principal point coordinates, and lens distortion coefficients. Next, Gray code decoding formulas are used to decode the calibration board patterns acquired by each camera under different poses, obtaining a unique encoded value for each Gray code pattern on the calibration board. These encoded values ​​will be used for subsequent coordinate mapping and geometric calculations to calibrate the camera projector's intrinsic and extrinsic parameters, and to reconstruct the 3D point cloud based on these parameters. (Software interface shown is not provided in the original text.) Figure 5 As shown.

[0101] 2.3 Sampling and Denoising of Pattern Decoding Values

[0102] In the decoded pattern, a 40*40 pixel area surrounding each corner point is sampled to obtain the Gray code decoding value of each pixel within that area. Simultaneously, utilizing the monotonically increasing or decreasing property of these decoding values ​​in the row and column directions, they are filtered for noise reduction. Noise reduction effectively reduces noise interference and improves the accuracy of the decoded values. After noise reduction, the correspondence between the Gray code decoding value of each pixel (i.e., the projector imaging position) and other points in space is obtained.

[0103] 2.4 Calculation of the homography matrix

[0104] The correspondence between spatial points and pixels obtained through the above steps is used to calculate the homography matrix between these two sets of points. The homography matrix describes the geometric mapping between the projector coordinate system and the camera coordinate system. Using this matrix, more precise projector coordinates corresponding to each calibration board corner point can be calculated. The purpose of this step is to ensure that the position and angle of each pixel are optimally corrected through geometric transformation.

[0105] 2.5 Calibration of Projector Intrinsic Parameters

[0106] The projector model is approximated as a camera model. The spatial coordinates (pixel coordinates) acquired by each camera from different viewpoints are used as input. By comprehensively analyzing the data from multiple viewpoints, the intrinsic parameters of the projector are calculated. These intrinsic parameters include the projector's focal length, principal point coordinates, and projection distortion parameters. Through joint calculation of multi-camera data, the accuracy and stability of the projector's intrinsic parameters can be significantly improved.

[0107] 2.6 Calculation of external parameters

[0108] Based on the multi-camera calibration model, the extrinsic parameters of each camera and projector pair are further calculated. These extrinsic parameters describe the relative position and orientation of the camera and projector in space, including rotation matrices and translation vectors. Geometric constraints and optimization algorithms ensure the accuracy of the relative positional relationships between each camera and projector. Ultimately, a high-precision set of intrinsic and extrinsic parameters for the cameras and projector is obtained, providing a solid foundation for subsequent 3D reconstruction and point cloud fusion.

[0109] 3. Calibration of external parameters for each camera

[0110] 3.1 Obtaining Spatial Coordinates

[0111] First, based on the Gray code decoding results, the spatial coordinates of each corner point of the calibration board are calculated using the intrinsic parameters of each camera and projector, as well as the extrinsic parameters between the camera and projector, and a 3D reconstruction formula. The purpose of this step is to accurately determine the position of each corner point of the calibration board in 3D space using multi-view image data and calibration parameters. This method ensures that the acquired spatial coordinates have high accuracy and reliability.

[0112] 3.2 Precision Optimization

[0113] Due to the limitations of projector resolution, adjacent pixels in the pattern captured by the camera may obtain the same Gray code value, which affects the accuracy of spatial point coordinates. To improve accuracy, each calibration plate corner point needs to be corrected. This process effectively reduces the error caused by insufficient projector resolution, thereby improving the accuracy of spatial coordinates.

[0114] 3.3 Calculation of external parameters

[0115] Based on the image coordinates and corresponding 3D spatial coordinates of the calibration board corner points after decoding, the solvePnP model is used to calculate the extrinsic parameter matrix between each camera. Using this model, and leveraging the positional relationships of the calibration board corner points, the rotation matrix and translation vector between each pair of cameras are accurately solved.

[0116] To ensure simplicity and consistency in the calculation process, the extrinsic parameters between each camera and camera 1 (which can be randomly selected) are used as the reference coordinate system. This means that the extrinsic parameters of all cameras will be defined and transformed relative to camera 1. First, the relative extrinsic parameters between each pair of cameras are calculated, and these parameters describe the rotation and translation relationship from one camera coordinate system to another.

[0117] Then, these relative extrinsic parameters are converted to extrinsic parameters relative to camera 1. This ensures that the extrinsic parameters of each camera are described in the same reference frame, thus unifying the coordinate systems of all cameras. This step is crucial because it lays the foundation for subsequent point cloud data alignment and fusion. Unifying the extrinsic parameters not only simplifies the calculation process but also improves the consistency and accuracy of data processing.

[0118] 4. 3D point cloud data acquisition and fusion

[0119] 4.1 Three-dimensional reconstruction using pure phase-shifted codes

[0120] For high-precision 3D reconstruction systems, a projector projects phase-shifted codes onto the surface of the object under test. Since the distance between the object and the projector is already highly defined, there is no need for Gray code calibration; the specific position of each pixel in the image can be determined solely from pre-calibrated information. After the projector projects the phase-shifted codes, multiple cameras simultaneously acquire the projected patterns. These patterns are then used to calculate depth information using the phase-shifting method, thereby generating preliminary 3D point cloud data. This method eliminates the time required for projecting Gray code patterns, enabling rapid and efficient acquisition of high-precision 3D data.

[0121] To verify the system's accuracy, a 3D model of a tiny chip was reconstructed. This chip, with a radius of only 0.5 millimeters, is extremely small, placing very high demands on the system's resolution and accuracy. In the experiment, a projector still projected phase-shifted codes, and multiple cameras simultaneously captured patterns. The depth information of the chip surface was calculated using the phase-shifting method, generating its 3D point cloud data. After data processing and fusion, a high-precision 3D model of the chip was finally obtained. This experiment demonstrates that the system can not only quickly and efficiently reconstruct the 3D structure of larger objects, but also accurately capture and reconstruct the details of tiny objects, proving the system's high precision and high-resolution performance.

[0122] Chips such as Figure 6 As shown, the point cloud effect acquired by a single device is as follows: Figure 7 As shown.

[0123] 4.2 Preliminary processing of point cloud data

[0124] Throughout the reconstruction process, depth maps and their corresponding point cloud information from all reconstruction records are preserved. After reconstruction, based on the extrinsic parameters of each camera, the point cloud data acquired by all cameras are uniformly converted to the pre-defined coordinate system of camera 1. The purpose of this step is to unify all point cloud data into the same reference coordinate system to facilitate subsequent fusion processing.

[0125] 4.3 Fusion and Optimization of Point Cloud Data

[0126] After transforming all point cloud data to the coordinate system of camera 1, for the depth map of camera 1, each pixel may correspond to multiple different depth information from different cameras. In order to obtain more accurate 3D reconstruction results, these depth information need to be processed and fused.

[0127] First, remove obvious noise. Specifically, remove points that are not only spatially distant from other points within the same pixel grid, but also spatially distant from surrounding points. These points are usually noise caused by acquisition errors or reflections; removing them improves data accuracy.

[0128] Next, weighted parameters are calculated based on the image information from which each point originates. The specific approach is as follows: For each pixel obtained through phase-shift coding, the greater the variation in grayscale value across multiple images, the higher its confidence level, because a larger variation in grayscale value usually indicates a more reliable measurement of that pixel. Conversely, if the grayscale value shows a significant anomaly, such as reaching 255, it typically indicates that the point is a reflective point, with lower confidence, and it may even need to be removed.

[0129] Based on the confidence level of each point cloud, a weighted average is calculated for multiple points within the same pixel cell in the final depth map to obtain the actual depth value of the point cloud at that location. Specifically, for multiple points within the same pixel cell, a weighted average is calculated based on the confidence level of each point. Points with higher confidence levels have a larger weight in the weighted average, while points with lower confidence levels have a smaller weight or are even removed.

[0130] After the above processing, the fused point cloud data is finally obtained. This data is not only highly accurate but also has low noise, providing a reliable foundation for subsequent 3D reconstruction and applications.

[0131] The point cloud effects before and after fusion (right calibration area) are as follows: Figure 8 As shown in Figure 9, the fused point cloud effectively fills the highly reflective part of the previous camera, and the reconstruction error is extremely small, almost indistinguishable.

[0132] The effect of single-camera, single-projector patch reconstruction of chip bumps, such as... Figure 10 As shown in (a) and (b), the multi-camera single-projector patch reconstruction effect is as follows: Figure 11 As shown in (a) and (b), the comparison shows that in the case of smaller recognition, the reconstruction of the present application is more accurate. The difference of the reconstructed image after the change of the angle of a single camera is small and the shape is irregular. In contrast, the sphere shape of the present application is more accurate and the undulation is smoother, with higher resolution and more accurate recognition, regardless of the angle.

Claims

1. A calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system, characterized in that: The method comprises the following steps: Step 1, selecting at least four cameras to cover the multi-view arrangement of the object to be measured under each view angle; selecting a high-precision projector to project a pattern onto the surface of the object to be measured; Step 2, calibration of the projector-camera system calibrate at least four cameras by using a calibration board placed in the field of view of the camera and projecting multiple gray code patterns, calculate the intrinsic parameters of each camera, and at the same time, accurately calibrate the projector, calculate the extrinsic parameters between the camera and the projector; Step 3, extrinsic calibration of each camera Step 3.1, using the corner coordinates of the calibration board, the extrinsic parameters between the camera and the projector, and ransc to calculate the corrected corner coordinates of the calibration board, and using the homography matrix of the corrected corner coordinates of the calibration board and the world coordinate system coordinates of the calibration board, optimizing the corner coordinates of the calibration board according to the homography matrix; The method for correcting the corner coordinates of the calibration board is as follows: using the corner coordinates of the calibration board and the extrinsic parameters between the camera and the projector, calculating the three-dimensional coordinates of each corner of the calibration board, then fitting a plane using ransc for all corner points of the calibration board, making a straight line perpendicular to the fitted plane for all three-dimensional corner points, and taking the intersection point of the plane and the straight line as a more accurate three-dimensional corner coordinate, calculating the positions of these more accurate three-dimensional corner points in the image according to the camera imaging formula, and obtaining the corrected corner coordinates of the calibration board; Step 3.2, according to the optimized corner coordinates of the calibration board in 3.1 and the preset world coordinate system coordinates of the corner points of the calibration board, using solvePnP model to calculate the extrinsic parameters of each camera, and taking the extrinsic parameters between a certain camera and the remaining cameras as the reference coordinate system; Step 4, three-dimensional point cloud data acquisition and fusion Step 4.1, after the projector projects the phase shift code on the surface of the object to be measured, multiple cameras simultaneously acquire the projected pattern, and use the pattern to calculate the depth information by the phase shift method, thereby generating preliminary three-dimensional point cloud data of each camera; Step 4.2, according to the extrinsic parameters of each camera mentioned in step 3.3, converting the preliminary three-dimensional point cloud data obtained by all cameras to the reference coordinate system; Step 4.3, removing the obvious noise points from the preliminary three-dimensional point cloud data obtained by all cameras in step 4.2, calculating the weight according to the image information of each pixel point, and modifying the actual depth value in each pixel point according to the weight.

2. The multi-camera single-projector three-dimensional reconstruction system calibration and point cloud fusion method of claim 1, wherein: The specific steps of step 2 are as follows: Step 2.1, all cameras obtain multiple sets of calibration images at different positions under their respective view angles; Step 2.2, according to the multiple sets of calibration images collected by each camera, for each camera, the brightest one in each set of calibration images is selected, and Zhang's calibration method is used to calibrate each camera to obtain the intrinsic parameters of all cameras; using the gray code decoding formula, decode each set of calibration image data collected by each camera; Step 2.3, using the decoded data as the pixel coordinates of the calibration board, sampling each corner point of the calibration board and its surrounding selected region, marking the gray code value of each pixel in the selected region and the coordinates of the corresponding camera pixel points, then performing noise reduction processing to obtain the correspondence between the projector pixel points and the camera pixel points; Step 2.4, the homography matrix of the projector pixel points and the camera pixel points is calculated according to the correspondence between the projector pixel points and the camera pixel points obtained by step 2.3, the gray code values of the corner points of the calibration board are updated according to the homography matrix, and the optimized correspondence between the projector pixel points and the camera pixel points is obtained; Step 2.5, the projector model is approximated as a camera model, and the Zhang calibration method is used for accurate calibration of the projector; Step 2.6, according to the optimized correspondence between the projector pixel points and the camera pixel points obtained in step 2.4, the fundamental matrix or the essential matrix between the camera and the projector is calculated, and singular value decomposition is performed on the fundamental matrix or the essential matrix to extract the external parameter between the camera and the projector.

3. The multi-camera single-projector three-dimensional reconstruction system calibration and point cloud fusion method of claim 1, wherein: The specific implementation method of step 2.1 is to place the calibration board in the common field of view of all cameras, and the projector projects multiple gray code patterns onto the calibration board in turn, and all cameras capture the gray code patterns on the calibration board as a set of calibration images. Adjust the position of the calibration board multiple times, and repeat the projection of multiple gray code patterns and the capture of the camera.

4. The multi-camera single-projector three-dimensional reconstruction system calibration and point cloud fusion method of claim 1, wherein: The method for removing obvious noise points in step 4.3 is to remove points that not only have a large spatial distance with other points in the same pixel grid, but also have a large spatial distance with surrounding points.

5. The multi-camera single-projector three-dimensional reconstruction system calibration and point cloud fusion method of claim 1, wherein: The high-precision projector can project the pattern through high-precision mirror reflection to the surface of the measured feature.

Images