Calibration and point cloud fusion method of multi-camera single-projector three-dimensional reconstruction system

Abstract

The invention discloses a calibration and point cloud fusion method of a multi-camera single-projector three-dimensional reconstruction system. The method comprises the following steps: projecting a projection pattern to the surface of a to-be-measured special body through multi-view arrangement of at least four cameras and projectors; calibration of a projector-camera system and calibration of external parameters of all cameras are carried out, and finally three-dimensional point cloud data collection and fusion are carried out according to the calibration of the projector-camera system and the calibration of the external parameters of all the cameras. According to the method, multi-camera joint calibration and data fusion are utilized, the calibration precision and robustness are improved, the point cloud fusion precision is enhanced, and the precision problem of a traditional three-dimensional reconstruction method when high-reflection and low-reflectivity objects are collected is solved.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction technology, and in particular to a calibration and point cloud fusion method for a multi-camera single-projector three-dimensional reconstruction system. Background Art

[0002] In the field of computer vision and image processing, 3D reconstruction technology has been widely used in virtual reality, augmented reality, industrial inspection and other fields. In order to achieve high-precision 3D reconstruction, the traditional monocular system consisting of a single camera and a single projector is often unable to cope with the problem of high reflectivity and low reflectivity of the objects to be inspected. In particular, it is impossible to achieve high-precision 3D reconstruction of tiny objects (about 200 micron chips) with local high reflectivity and local low reflectivity. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system, multi-camera joint calibration and data fusion, to improve the accuracy and robustness of calibration, enhance the accuracy of point cloud fusion, and solve the accuracy problem of traditional 3D reconstruction methods when collecting high-reflectivity and low-reflectivity objects.

[0004] In order to achieve the above technical purpose, the technical solution adopted is: a calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system, comprising the following steps: Step 1: Select at least four cameras to cover all viewing angles of the object to be tested for multi-viewing arrangement; select a high-precision projector to reflect the projected pattern onto the surface of the object to be tested through a high-precision mirror; Step 2: Calibration of the projector-camera system Using a calibration plate placed in the camera's field of view and projecting multiple Gray code patterns, at least four cameras are calibrated to calculate the intrinsic parameters of each camera. At the same time, the projector is accurately calibrated to calculate the extrinsic parameters between the camera and the projector; Step 3: Calibrate the external parameters of each camera Step 3.1, using the calibration plate corner point coordinates, the external parameters between the camera and the projector, calculate the corrected calibration plate corner point coordinates through ransc, and optimize the calibration plate corner point coordinates according to the homography matrix of the corrected calibration plate corner point coordinates and the calibration plate world coordinate system coordinates; Step 3.2, according to the calibration plate corner point coordinates optimized in 3.1 and the world coordinate system coordinates preset by the calibration plate corner points, use the solvePnP model to calculate the extrinsic parameters of each camera, and unify the extrinsic parameters between a certain camera and the rest of the cameras as the reference coordinate system; Step 4: 3D point cloud data collection and fusion Step 4.1, after the projector projects the phase shift code on the surface of the object to be measured, multiple cameras simultaneously collect the projection pattern, and use these patterns to calculate the depth information through the phase shift method, thereby generating preliminary three-dimensional point cloud data of each camera; Step 4.2: According to the external parameters of each camera mentioned in step 3.3, the preliminary three-dimensional point cloud data acquired by all cameras are converted into the reference coordinate system; Step 4.3: Remove obvious noise from the preliminary 3D point cloud data obtained by all cameras and converted to the reference coordinate system obtained in step 4.2, calculate the weight based on the image information of each pixel source, and modify the actual depth value in each pixel based on the weight.

[0005] Further, the specific steps of step 2 are: Step 2.1, all cameras obtain multiple sets of calibration images at different positions under their respective viewing angles; Step 2.2, according to the multiple sets of calibration images collected by each camera, the brightest image in each set of calibration images of each camera is calibrated by using Zhang's calibration method to obtain the intrinsic parameters of all cameras; each set of calibration image data collected by each camera is decoded by using Gray code decoding formula; Step 2.3, the decoded data is used as the pixel coordinates of the calibration plate, and each corner point of the calibration plate and the selected area around it are sampled, and the Gray code value of each pixel in the selected area and the coordinates of the corresponding camera pixel point are marked, and then denoising is performed to obtain the corresponding relationship between the projector pixel point and the camera pixel point; Step 2.4, calculating the homography matrix of the projector pixel points and the camera pixel points through the correspondence between the projector pixel points and the camera pixel points obtained in step 2.3, updating the Gray code values ​​of the corner points of the calibration plate according to the homography matrix, and obtaining the optimized correspondence between the projector pixel points and the camera pixel points; Step 2.5, the projector model is approximately regarded as a camera model, and the projector is accurately calibrated using Zhang's calibration method; Step 2.6: According to the correspondence between the optimized projector pixel points and the camera pixel points obtained in step 2.4, the basic matrix or essential matrix between the camera and the projector is calculated, and the singular value decomposition is performed on it to extract the extrinsic parameters between the camera and the projector.

[0006] Furthermore, the specific implementation method of step 2.1 is to place the calibration plate in the public field of view of all cameras, project multiple Gray code patterns onto the calibration plate in sequence through a projector, capture the Gray code patterns on the calibration plate by all cameras as a set of calibration images, adjust the position of the calibration plate multiple times, and repeat the projection of multiple Gray code patterns and camera capture.

[0007] Furthermore, the method for calculating the coordinates of the corner points of the calibration plate after correction in step 3.1 is to use the coordinates of the corner points of the calibration plate and the external parameters between the camera and the projector to calculate the three-dimensional coordinates of each corner point of the calibration plate, and then use ransc to fit a plane for all the corner points of the calibration plate. All three-dimensional corner points are made into straight lines perpendicular to the fitted plane, and the intersection of the plane and the straight line is regarded as a more accurate three-dimensional corner point coordinate. According to the camera imaging formula, the positions of these more accurate three-dimensional corner points in the image are calculated to obtain the corrected coordinates of the corner points of the calibration plate.

[0008] Furthermore, the method for removing obvious noise points in step 4.3 is to remove points that have a large spatial distance not only from other points in the same pixel grid but also from surrounding points.

[0009] Furthermore, the high-precision projector can project the pattern onto the surface of the object to be measured through a high-precision mirror.

[0010] The beneficial effects of the present invention are as follows: the patent realizes complete reconstruction of three-dimensional objects through calibration and point cloud fusion by using a multi-angle camera and a single projector. Its innovation lies in the use of homography matrix to optimize the calibration of the projector internal parameters encoded by Gray code to improve the calibration accuracy; at the same time, the corner point detection of the two-dimensional image is optimized according to the three-dimensional information to enhance the accuracy of image processing. In addition, the data fusion technology in the patent effectively solves the shortcomings of single-view cameras in dealing with highlight problems and improves the reconstruction quality. It can not only quickly and efficiently reconstruct the three-dimensional structure of larger objects, but also accurately capture and reconstruct the details of tiny objects, proving the high-precision and high-resolution performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the present invention; Figure 2 is a system diagram of the present invention; Figure 3 Calibrate patterns for different camera perspectives; Figure 4 Gray code pattern for the same camera; Figure 5 This is the interface diagram of the data processing software; Figure 6 Compare images between the chip and the human hand; Figure 7 Reconstruct renderings for a single camera and single projector; Figure 8 This is the effect picture before chip point cloud fusion; Fig. 9 The effect picture after point cloud fusion; Fig.10 Reconstruction effect diagram for single camera single projector patch; Fig.11This is the effect diagram after multi-view fusion of this application; In the figure: 1. Object to be measured, 2. Camera, 3. Projector, 4. High-precision mirror. DETAILED DESCRIPTION

[0012] The preferred embodiments of the invention are given below in conjunction with the accompanying drawings to explain the technical solution of the present invention in detail. Here, the corresponding drawings are given to explain the present invention in detail. It should be particularly noted that the preferred embodiments described here are only used to illustrate and explain the present invention, and are not used to limit or restrict the present invention.

[0013] This paper proposes an innovative camera-projector calibration and fusion method in a multi-camera system. By combining the advantages of a multi-camera system, the calibration and point cloud fusion process of a multi-camera single-projector system is optimized to achieve high-precision 3D reconstruction of objects with local high-reflectivity and local low-reflectivity characteristics. For example: tiny objects (chips of about 200 microns), specifically, Figure 1 As shown, the method of the present invention includes the following steps: Step 1, selection and setting of a multi-camera 3D reconstruction system: 1.1、Arrangement and selection of cameras: Choose a high-resolution, low-noise industrial camera to ensure image quality. The camera should have good optical performance and stable imaging effects, and be able to provide consistent image quality under different lighting conditions.

[0014] Select at least four cameras for multi-viewing arrangement to cover as many view angles of the object as possible. This multi-camera layout can effectively reduce the occlusion and reflection problems caused by a single view angle and improve the comprehensiveness of image acquisition and data integrity.

[0015] Adjust and optimize the position and angle of the camera. The position of each camera is evenly distributed around the calibration scene, and the height and tilt angle are adjusted according to the size and shape of the object to be measured to ensure that each camera can clearly capture different parts of the object. It has been verified that for targets with simpler structures (such as chip boards), a four-camera system can ensure that each area of ​​the chip can be captured by at least two cameras at the same time.

[0016] 1.2. Arrangement and selection of projectors: Choose a high-precision, stable projector that can maintain a stable projection effect under different lighting conditions. The projector should have high resolution and brightness to ensure the clarity and contrast of the projected pattern.

[0017] To ensure that there is a field of view for the subsequent test camera directly above the system, the projector can be placed on one side of the center of the system, and the projection pattern is reflected to the surface of the object to be tested through a high-precision mirror. This design ensures that the projection pattern covers the entire surface of the object to be tested, and can be projected without interfering with the camera's field of view. The position and angle of the projector are optimized according to the size of the object and the spatial layout of the calibration scene to ensure the uniformity and clarity of the projection pattern.

[0018] Step 2: Calibration of the projector-camera system: 2.1. First, take four cameras as an example and place the calibration plate in the common field of view of all cameras. Use a projector to project multiple Gray code patterns in sequence and ensure that all cameras can capture these patterns synchronously. Gray code patterns include both horizontal and vertical Gray code patterns, both positive and negative Gray codes. If the number of projector rows is 960, then the binary number must be 2 to the tenth power, which is equal to 1024. Ten images are needed to present the Gray code. Then, considering the positive and negative, 20 images are needed for the horizontal Gray code pattern. If the vertical number is 1440 columns, 22 images are needed for the vertical Gray code. Finally, add all black and all white, and a group is 44 images, that is, each camera collects at least 44 Gray code images at one position. In order to ensure the accuracy of the calibration, it is necessary to adjust the position of the calibration plate and repeatedly collect image data. After each adjustment of the calibration plate position, continue to project the Gray code pattern and collect synchronously to obtain multiple sets of calibration images at different positions under the respective viewing angles of the four cameras.

[0019] 2.2. Based on the multiple sets of calibration images collected by each camera, for the brightest image in each set of calibration image data (the camera image corresponding to the pure white light emitted by the projector), Zhang’s calibration method is used, following the formula M i =K[R|T]m i Calibrate each camera, where m i Represents the three-dimensional coordinates of the i-th point in the world coordinate system, M i represents the two-dimensional pixel position of the i-th point in the image coordinate system, K represents the intrinsic parameter matrix of the camera, R represents the rotation matrix, T represents the translation vector, and the intrinsic parameter parameters of all cameras are obtained.

[0020] Next, the Gray code decoding formula is used to decode the calibration plate image data collected by each camera at different positions (calibration plate at different positions). Specifically, for each set of calibration plate data, the brightness changes of each pixel in multiple images are analyzed. For the grayscale values ​​of the same pixel position in different images, the brightness below the average value is recorded as 0, and the brightness above the average value is recorded as 1, and according to the Gray code decoding formula: Among them Gn Indicates the nth Gray code value, B n Represents the binary code value of the nth bit, Represents the XOR operator, B0=G0. Get the unique code value of each pixel on the calibration plate. These code values ​​will be used for subsequent coordinate mapping and geometric calculations.

[0021] Since the projector projects two groups of Gray code patterns, namely, a horizontal Gray code pattern and a vertical Gray code pattern, each pixel point can eventually obtain two unique decoding values, corresponding to the horizontal Gray code and the vertical Gray code, respectively.

[0022] 2.3. In the decoded data, a 40*40 pixel area around each corner point of the calibration plate is sampled. The selection of the area pixels is designed according to the camera resolution. For example, the camera pixel used here is 5120*5120. The Gray code value of each pixel point i in the sampling area is denoted as G(a i , b i ), a i is the horizontal Gray code value, b i is the vertical Gray code value, and the camera pixel coordinates are denoted by (x i ,y i ), let the corner point P of the angle calibration plate ij The sampling pixel area is R ij , whose range is R ij ={(x, y)|x i -20≤x≤x i +20,y i -20≤y≤y i +20} Get the Gray code decoding value of each pixel in the area. Use the Gray code decoding value to monotonically increase or decrease in the row and column directions to perform denoising. Specifically, if the decoding value of a pixel i satisfies the following in the row direction: G(a i-1 , b i )≤=G(a i , b i )≤G(a i+1 , b i ) At the same time, in the column direction: G(a i , b i-1 )≤=G(a i , b i )≤G(a i , b i+1 ) If the decoded value of the pixel is 0, the decoded value of the pixel is considered valid. Otherwise, it is marked as invalid and deleted. This method can effectively remove noise interference and improve the accuracy of the decoded value. After denoising, the corresponding relationship between the Gray code decoded value of each pixel (i.e., the imaging position of the projector) and each point in space (pixel point of the camera) is obtained.

[0023] 2.4. The projector pixel point P obtained through the above steps i With camera pixel p i The corresponding relationship is calculated according to the homography matrix formula: p i =HP i , calculate the 3×3 homography matrix between the projector pixel and the camera pixel. The homography matrix is ​​used to describe the geometric mapping relationship between the projector coordinate system and the camera coordinate system. Using this homography matrix, we can further calculate the more accurate projector coordinates corresponding to each calibration plate corner point, that is, update the Gray code value G(a) of each calibration plate corner point. i , b i ), the correspondence between the optimized projector pixel and the camera pixel is obtained. The purpose of this step is to ensure that the position and angle of each pixel are optimized through geometric transformation. At the same time, since each Gray code G(a i , b i ) code value corresponds to a single specific pixel position of the projector, and each corner point corresponds to a three-dimensional point in the world coordinate system. At this time, the relationship between multiple pixels of the projector and the midpoint of the world coordinate system is established.

[0024] 2.5. In the present invention, the projector model is approximately regarded as a camera model, and the coordinate parameters of the projector are obtained by establishing a mapping relationship between the two-dimensional pixel points of the projector and the three-dimensional points in the actual space. That is, the projector model is approximately regarded as a camera model, and the precise Gray code decoding value of each corner point of the calibration plate is used to determine the specific pixel position of the projector at these corner points, and the Zhang calibration method is used to achieve accurate calibration of the projector. The projector is accurately calibrated in order to improve the accuracy of obtaining the initial point cloud.

[0025] 2.6. In the process of relative position calibration between the camera and the projector, the image containing the calibration plate is taken by the camera, and the coordinates of the corner points of the calibration plate on the camera image are extracted using the corner detection algorithm. At the same time, according to the specific pattern projected by the projector, the projector pixel coordinates corresponding to the corner points are obtained through the decoding method, and the corresponding relationship between the corner points of the camera image and the projector image is established. The optimized projector pixel point P i With camera pixel p i The corresponding relationship between the camera and the projector is calculated, and the basic matrix or essential matrix between the camera and the projector is decomposed and the rotation matrix, that is, the external parameter between the camera and the projector, is extracted.

[0026] Step 3: Calibrate the external parameters of each camera: 3.1. First, obtain the original coordinates of the corner points and the external parameters between the camera and the projector through steps 2.2 and 2.3, calculate the three-dimensional coordinates of each corner point, and then use ransc to fit a plane for all corner points. Then, make straight lines perpendicular to the fitted plane for all three-dimensional corner points. The intersection of the plane and the straight line is regarded as a more accurate three-dimensional corner point coordinate. According to the camera imaging formula, calculate the position of these more accurate three-dimensional corner points in the image to obtain the corrected corner points. The new coordinates of the corrected corner points are then used to calculate the homography matrix with the (x, y) values ​​of the world coordinate system coordinates of the calibration plate corner points. Based on this homography matrix, optimize the coordinates of the calibration plate corner points in the image coordinate system.

[0027] The specific implementation process is: after obtaining the more accurate three-dimensional coordinates of each corner point of the calibration plate in the camera coordinate system, use the RANSC algorithm to further improve the coordinate accuracy. First, use the RANSAC algorithm to fit the plane coordinates of the calibration plate where the current point cloud is located to obtain the coordinates of each point cloud on the plane of the calibration plate. Specifically, assume that the coordinates of the point cloud P in the camera coordinate system are (x, y, z), and the plane equation of the calibration plate is aX+bY+cZ+d=0. Use the RANSAC algorithm to fit the calibration plate plane parameters a, b, c, d based on the known coordinates of each corner point in the camera coordinate system. Then map the point cloud P to the plane of the calibration plate to obtain the projection coordinates p p =(x p ,y p , 0), update this point to the coordinates of the new corner point in the image coordinate system. Then calculate the new homography matrix H2 based on the new image coordinates of each corner point and the (x, y) values ​​of the world coordinates of each corner point, so that p i =H2P i , and through the homography matrix, the more accurate coordinates of each calibrated anti-corner point in the image coordinate system are calculated. This process effectively reduces the error caused by insufficient projector resolution, thereby improving the accuracy of spatial coordinates.

[0028] 3.2. According to the calibration plate corner point coordinates optimized in 3.1 and the actual 3D space coordinates preset by the calibration plate corner points, use the solvePnP model to calculate the extrinsic parameter matrix between each camera. Specifically, only the pose relationship between the three groups of cameras, camera 1 and camera 2, camera 1 and camera 3, and camera 1 and camera 4, needs to be calculated. Through this model, the position relationship of the calibration plate corner points is used to accurately solve the rotation matrix and translation vector, i.e., the extrinsic parameters, between each pair of cameras.

[0029] In addition, to ensure the simplicity and consistency of the calculation process, the external parameters between each camera and camera No. 1 (which can be randomly selected) are uniformly used as the reference coordinate system. This means that the external parameters of all cameras will be defined and transformed relative to camera No. 1. In this way, it is ensured that the external parameters of each camera are described in the same reference system, thereby unifying the coordinate systems of all cameras. This step is crucial because it lays the foundation for the subsequent point cloud data alignment and fusion. Unifying the external parameters not only simplifies the calculation process, but also improves the consistency and accuracy of data processing.

[0030] Step 4: 3D point cloud data collection and fusion: 4.1. 3D reconstruction using pure phase-shift coding For high-precision 3D reconstruction systems, the projector will project a phase-shift code onto the surface of the object to be measured. Since the distance between the object to be measured and the projector is highly determined, there is no need for the camera to actually capture the Gray code. The approximate range of the Gray code for each pixel in the image can be determined only through pre-calibrated information (camera calibration and projector calibration), that is, through decoding the phase-shift code. Based on the phase-shift code value, the actual Gray code value can be uniquely determined. Specifically, for a pixel point (x i ,y i ), its decoded value has been pre-calculated and can only be a value between 6 and 9. Since the phase-shift code occupies multiple rows of pixels in the projector image, 4-7 is a pixel period Gray code, and 8-10 is another group. When the phase-shift code decoded value P is between 0-∏, the actual decoded value can be considered to be 8+2P / ∏. When the phase-shift code decoded value is between ∏-2∏, the actual decoded value can be considered to be 6+2P / ∏.

[0031] After the projector projects the phase-shift code, multiple cameras simultaneously capture the projected pattern, and use these patterns to calculate depth information through the phase-shift method to generate preliminary 3D point cloud data. This method can save the time of projecting Gray code patterns and quickly and efficiently obtain high-precision 3D data.

[0032] 4.2 Preliminary processing of point cloud data During the entire reconstruction process, the depth map and its corresponding point cloud information in all reconstruction records are retained. After the reconstruction is completed, according to the external parameters of each camera obtained in 3.2, the preliminary 3D point cloud data acquired by all cameras are uniformly converted to the preset coordinate system of camera 1. The purpose of this step is to unify all point cloud data into the same reference coordinate system for subsequent fusion processing.

[0033] 4.3 Fusion and optimization of point cloud data After all point cloud data are converted to the coordinate system of camera 1, for the depth map of camera 1, each pixel may correspond to multiple different depth information from different cameras. In order to obtain more accurate 3D reconstruction results, these depth information need to be processed and fused. Processing and fusion can be performed in different regions to further improve reconstruction accuracy.

[0034] First, remove obvious noise points. The specific method is to reproject multiple point clouds into the image collected by camera 1, and remove outliers for multiple points in the same pixel, removing those points that have a large spatial distance not only to other points in the same pixel grid, but also to surrounding points. Such points are usually noise points caused by acquisition errors or reflections, and removing these points can improve the accuracy of the data.

[0035] Next, the weighting parameter (weight) is calculated based on the image information from which each pixel comes, and the actual depth value within each pixel is modified based on the weight. The specific idea is as follows: For each pixel obtained by the phase shift code, if the grayscale value of the pixel changes more in multiple pictures, its confidence level is higher, because a larger grayscale value change usually means that the measurement of the pixel is more reliable. On the contrary, if the grayscale value is obviously abnormal, such as reaching 255, it usually means that the point is a reflective point, and its confidence level is low, or even needs to be removed. Let: ΔI=I max -I min , where I max is the maximum value of pixel brightness in multiple images corresponding to each point cloud, I min is the minimum value, and the weight is calculated according to the gray value difference: Based on the confidence of each point cloud, a weighted average calculation is performed on multiple pixels in the same pixel grid in the final depth map to obtain the actual depth value of the point cloud at that location. Specifically, for multiple points in the same pixel grid, weights are performed according to the confidence of each pixel. Points with high confidence have a larger weight in the weighted average, while points with low confidence have a smaller weight or are even eliminated, that is: Z=∑ i w i z i Where Z represents the final depth, z i Indicates the depth value of the point, w i Represents the weight of the point.

[0036] After the above processing, the fused point cloud data is finally obtained, that is, a complete 3D model is established. These data are not only highly accurate, but also have few noise points, providing a reliable basis for subsequent chip accuracy calculation and processing.

[0037] Example 1 1. Selection and setup of multi-camera 3D reconstruction system 1.1 Camera Arrangement and Selection In this example, we selected the high-resolution, low-noise MV-CH250-90TM-C-NF Hikvision industrial camera to ensure image quality. The camera has a 25-megapixel resolution, can be adapted to C-mount lenses, has excellent optical performance and stable imaging effects, and can provide consistent image quality under different lighting conditions.

[0038] Four cameras are selected for multi-viewing arrangement to cover all key viewpoints of the object. This multi-camera layout can effectively reduce the occlusion and reflection problems caused by a single viewpoint and improve the comprehensiveness of image acquisition and data integrity.

[0039] In the specific operation, we adjusted and optimized the position and angle of the camera. The position of each camera is evenly distributed around the calibration scene, and the height and tilt angle are adjusted according to the size and shape of the object to ensure that each camera can clearly capture different parts of the object.

[0040] 1.2 Projector Arrangement and Selection Choose the DLP4710 signal projector, which is one of the most advanced projectors currently available, with a resolution of 1080*1920, which can ensure the clarity and contrast of the projected pattern.

[0041] In order to leave a field of view for the subsequent test camera directly above the system, the projector is placed on one side of the center of the system. The projector reflects the projection pattern onto the surface of the object to be tested through a high-precision mirror. This design ensures that the projection pattern covers the entire surface of the object to be tested and can be projected without interfering with the camera's field of view.

[0042] The position and angle of the projector are optimally adjusted according to the size of the object and the spatial layout of the calibration scene to ensure the uniformity and clarity of the projected pattern.

[0043] The structure of the whole system is as follows Figure 2 As shown, the fourth camera is blocked by the middle camera and is not shown.

[0044] 2. Calibration of the projector-camera system 2.1 Calibration board setting and image acquisition First, place the calibration plate in the common field of view of all cameras. Use a projector to project Gray code patterns in sequence and ensure that all cameras can capture these patterns synchronously. To ensure the accuracy of the calibration, it is necessary to adjust the position of the calibration plate and repeatedly collect image data. After each adjustment of the calibration plate position, continue to project Gray code patterns and synchronously collect data to obtain calibration images at multiple viewing angles and different positions. The calibration images at different camera viewing angles are shown in Figure 1. Figure 3 As shown, the same camera Gray code pattern is as follows Figure 4 shown.

[0045] 2.2 Calibration of camera intrinsic parameters According to the calibration plate patterns collected by each camera, for the brightest all-white light image in each data set, Zhang's calibration method is used to calibrate each camera to obtain the intrinsic parameters of all cameras. These intrinsic parameters include the focal length of the camera, the coordinates of the principal point, and the lens distortion coefficient. Then, the Gray code decoding formula is used to decode the calibration plate patterns collected by each camera in different postures to obtain the unique encoding value of each Gray code pattern on the calibration plate. These encoding values ​​will be used for subsequent coordinate mapping and geometric calculations to calibrate the internal and external parameters of the camera projector, and the software interface for reconstructing the three-dimensional point cloud based on the internal and external parameters is as follows Figure 5 shown.

[0046] 2.3 Sampling and denoising of pattern decoding values In the decoded pattern, the 40*40 pixel area around each corner point is sampled to obtain the Gray code decoding value of each pixel in the area. At the same time, these decoded values ​​are filtered and denoised by taking advantage of the fact that they have the property of monotonically increasing or decreasing in the row and column directions. By denoising, noise interference can be effectively reduced and the accuracy of the decoded value can be improved. After denoising, the corresponding relationship between the Gray code decoding value of each pixel (i.e., the imaging position of the projector) and each point in space is obtained.

[0047] 2.4 Calculation of the homography matrix The correspondence between the spatial points and the pixel points obtained in the above steps is used to calculate the homography matrix between the two sets of points. The homography matrix is ​​used to describe the geometric mapping relationship between the projector coordinate system and the camera coordinate system. Using this matrix, more accurate projector coordinates corresponding to each calibration plate corner point can be further calculated. The purpose of this step is to ensure that the position and angle of each pixel point are optimally corrected through geometric transformation.

[0048] 2.5 Calibration of projector internal parameters The projector model is approximately regarded as a camera model, and the spatial coordinates-pixel coordinates obtained by each camera at different viewing angles are used as input. The multi-view data is used for comprehensive analysis to calculate the intrinsic parameters of the projector. These internal parameters include the focal length of the projector, the coordinates of the principal point, and the projection distortion parameters. Through the joint calculation of multi-camera data, the accuracy and stability of the projector's internal parameters can be significantly improved.

[0049] 2.6 Calculation of extrinsic parameters Based on the multi-camera calibration model, the extrinsic parameters of each camera and projector are further calculated. The extrinsic parameters are used to describe the relative position and posture of the camera and projector in space, including the rotation matrix and translation vector. Through geometric constraints and optimization algorithms, the relative position relationship between each camera and projector is ensured to be accurate. Finally, a set of high-precision intrinsic and extrinsic parameters of the camera and projector are obtained, providing a solid foundation for subsequent 3D reconstruction and point cloud fusion.

[0050] 3. Calibration of external parameters of each camera 3.1 Acquisition of spatial coordinates First, based on the Gray code decoding results, the spatial coordinates corresponding to each corner point of the calibration plate are calculated using the internal parameters of each camera and projector and the external parameters between the camera and the projector using the 3D reconstruction formula. The purpose of this step is to accurately determine the position of each corner point of the calibration plate in 3D space through multi-view image data and calibration parameters. In this way, the acquired spatial coordinates can be ensured to have high accuracy and reliability.

[0051] 3.2 Accuracy Optimization Due to the limitation of the projector resolution, adjacent pixels in the pattern captured by the camera may obtain the same Gray code value, which will affect the accuracy of the spatial point coordinates. To improve the accuracy, it is necessary to correct each corner point of the calibration plate. This process effectively reduces the error caused by insufficient projector resolution, thereby improving the accuracy of the spatial coordinates.

[0052] 3.3 Calculation of extrinsic parameters According to the decoded image coordinates of the corner points of the calibration plate and the corresponding three-dimensional space coordinates, the solvePnP model is used to calculate the extrinsic parameter matrix between each camera. Through this model, the position relationship of the corner points of the calibration plate is used to accurately solve the rotation matrix and translation vector between each pair of cameras.

[0053] To ensure the simplicity and consistency of the calculation process, the extrinsic parameters between each camera and camera 1 (which can be randomly selected) are uniformly used as the reference coordinate system. This means that the extrinsic parameters of all cameras will be defined and transformed relative to camera 1. First, for each pair of cameras, the relative extrinsic parameters between them are calculated. These parameters describe the rotation and translation relationship from one camera coordinate system to another camera coordinate system.

[0054] Then, these relative extrinsic parameters are converted to extrinsic parameters relative to camera 1. In this way, the extrinsic parameters of each camera are described in the same reference system, thereby unifying the coordinate systems of all cameras. This step is crucial because it lays the foundation for subsequent point cloud data alignment and fusion. Unifying extrinsic parameters not only simplifies the calculation process, but also improves the consistency and accuracy of data processing.

[0055] 4. 3D point cloud data collection and fusion 4.1 3D reconstruction using pure phase-shift coding For high-precision 3D reconstruction systems, the projector projects a phase-shift code onto the surface of the object to be measured. Since the distance between the object to be measured and the projector is highly determined, there is no need to use Gray code for calibration. The specific position of each pixel in the image can be determined only by the pre-calibrated information. After the projector projects the phase-shift code, multiple cameras simultaneously capture the projected pattern, and use these patterns to calculate the depth information through the phase shift method, thereby generating preliminary 3D point cloud data. This method can save the time of projecting Gray code patterns and quickly and efficiently obtain high-precision 3D data.

[0056] To verify the accuracy of the system, an attempt was made to reconstruct a three-dimensional model of a tiny chip. The radius of the chip is only 0.5 mm, which is extremely small, and the resolution and accuracy of the system are very high. In the experiment, the projector still projects phase-shift codes, and multiple cameras synchronously collect patterns. The depth information of the chip surface is calculated by the phase-shift method, and its three-dimensional point cloud data is generated. After data processing and fusion, a high-precision three-dimensional model of the chip is finally obtained. This experiment shows that the system can not only quickly and efficiently reconstruct the three-dimensional structure of larger objects, but also accurately capture and reconstruct the details of tiny objects, proving the high-precision and high-resolution performance of the system.

[0057] Chip such as Figure 6 As shown in the figure, the point cloud effect collected by a single device is as follows Figure 7 shown.

[0058] 4.2 Preliminary processing of point cloud data During the entire reconstruction process, the depth map and its corresponding point cloud information in all reconstruction records are retained. After the reconstruction is completed, the point cloud data acquired by all cameras are uniformly converted to the preset coordinate system of camera 1 according to the external parameters of each camera. The purpose of this step is to unify all point cloud data into the same reference coordinate system for subsequent fusion processing.

[0049] 4.3 Fusion and optimization of point cloud data After all point cloud data are converted to the coordinate system of camera 1, for the depth map of camera 1, each pixel may correspond to multiple different depth information from different cameras. In order to obtain more accurate 3D reconstruction results, these depth information need to be processed and fused.

[0050] First, remove obvious noise points. The specific method is to remove those points that have a large spatial distance not only from other points in the same pixel grid, but also from surrounding points. Such points are usually noise points caused by acquisition errors or reflections. By removing these points, the accuracy of the data can be improved.

[0051] Next, the weighting parameters are calculated based on the image information of each point. The specific idea is as follows: for each pixel point obtained by the phase shift code, if the grayscale value of the pixel changes more in multiple images, its confidence is higher, because a larger grayscale value change usually means that the measurement of the pixel point is more reliable. On the contrary, if the grayscale value is obviously abnormal, such as reaching 255, it usually means that the point is a reflective point, its confidence is low, and it may even need to be removed.

[0052] Based on the confidence of each point cloud, a weighted average is calculated for multiple points in the same pixel grid in the final depth map to obtain the actual depth value of the point cloud at that location. Specifically, for multiple points in the same pixel grid, weights are applied according to the confidence of each point. Points with high confidence have a larger weight in the weighted average, while points with low confidence have a smaller weight or are even eliminated.

[0053] After the above processing, the fused point cloud data is finally obtained. These data are not only highly accurate but also have few noise points, providing a reliable basis for subsequent 3D reconstruction and application.

[0054] The point cloud effects before and after fusion (calibrated area on the right) are as follows Figure 8 , as shown in Figure 9: It can be seen that the fused point cloud effectively fills the highly reflective part of the previous camera, and the reconstruction error is extremely small and almost indistinguishable.

[0055] The effect of single camera and single projector on chip bump surface reconstruction, such as Fig.10 (a)(b) shows the reconstruction effect of multi-camera single projector patch. Fig.11 As shown in (a)(b), by comparison, it can be seen that in a more subtle recognition situation, the reconstruction of the present application is more accurate. After the single camera angle changes, the reconstructed image has little difference and the shape is irregular. However, regardless of the angle, the shape of the sphere of the present application is more accurate, the undulations are smoother, it has a higher resolution, and the recognition is more accurate.

Claims

1. A calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system, characterized by: The following steps are involved: Step 1: Select at least four cameras to cover all viewing angles of the object to be measured and arrange them in multiple viewing angles; Select a high-precision projector to project the projection pattern onto the surface of the object to be tested; Step 2: Calibration of the projector-camera system Using a calibration plate placed in the camera's field of view and projecting multiple Gray code patterns, at least four cameras are calibrated to calculate the intrinsic parameters of each camera. At the same time, the projector is accurately calibrated to calculate the extrinsic parameters between the camera and the projector; Step 3: Calibrate the external parameters of each camera Step 3.1, using the calibration plate corner point coordinates, the external parameters between the camera and the projector, calculate the corrected calibration plate corner point coordinates through ransc, and optimize the calibration plate corner point coordinates according to the homography matrix of the corrected calibration plate corner point coordinates and the calibration plate world coordinate system coordinates; Step 3.2, according to the calibration plate corner point coordinates optimized in 3.1 and the world coordinate system coordinates preset by the calibration plate corner points, use the solvePnP model to calculate the extrinsic parameters of each camera, and unify the extrinsic parameters between a certain camera and the rest of the cameras as the reference coordinate system; Step 4: 3D point cloud data collection and fusion Step 4.1, after the projector projects the phase shift code on the surface of the object to be measured, multiple cameras simultaneously collect the projection pattern, and use these patterns to calculate the depth information through the phase shift method, thereby generating preliminary three-dimensional point cloud data of each camera; Step 4.2: According to the external parameters of each camera mentioned in step 3.3, the preliminary three-dimensional point cloud data acquired by all cameras are converted into the reference coordinate system; Step 4.3: Remove obvious noise from the preliminary 3D point cloud data obtained by all cameras and converted to the reference coordinate system obtained in step 4.2, calculate the weight based on the image information of each pixel source, and modify the actual depth value in each pixel based on the weight.

2. The calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system according to claim 1, characterized in that: The specific steps of step 2 are: Step 2.1, all cameras obtain multiple sets of calibration images at different positions under their respective viewing angles; Step 2.2, according to the multiple sets of calibration images collected by each camera, the brightest image in each set of calibration images of each camera is calibrated by using Zhang's calibration method to obtain the intrinsic parameters of all cameras; each set of calibration image data collected by each camera is decoded by using Gray code decoding formula; Step 2.3, the decoded data is used as the pixel coordinates of the calibration plate, and each corner point of the calibration plate and the selected area around it are sampled, and the Gray code value of each pixel in the selected area and the coordinates of the corresponding camera pixel point are marked, and then denoising is performed to obtain the corresponding relationship between the projector pixel point and the camera pixel point; Step 2.4, calculating the homography matrix of the projector pixel points and the camera pixel points through the correspondence between the projector pixel points and the camera pixel points obtained in step 2.3, updating the Gray code values ​​of the corner points of the calibration plate according to the homography matrix, and obtaining the optimized correspondence between the projector pixel points and the camera pixel points; Step 2.5, the projector model is approximately regarded as a camera model, and the projector is accurately calibrated using Zhang's calibration method; Step 2.6: According to the correspondence between the optimized projector pixel points and the camera pixel points obtained in step 2.4, the basic matrix or essential matrix between the camera and the projector is calculated, and the singular value decomposition is performed on it to extract the extrinsic parameters between the camera and the projector.

3. The calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system according to claim 1, characterized in that: The specific implementation method of step 2.1 is to place the calibration plate in the public field of view of all cameras, project multiple Gray code patterns onto the calibration plate in sequence, and all cameras capture the Gray code patterns on the calibration plate as a set of calibration images. The position of the calibration plate is adjusted multiple times, and multiple Gray code pattern projections and camera captures are repeated.

4. The calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system according to claim 1, characterized in that: The method for calculating the corrected calibration plate corner point coordinates in step 3.1 is to use the calibration plate corner point coordinates and the external parameters between the camera and the projector to calculate the three-dimensional coordinates of each calibration plate corner point, and then use ransc to fit a plane for all the calibration plate corner points. All three-dimensional corner points are drawn with straight lines perpendicular to the fitted plane. The intersection of the plane and the straight line is regarded as a more accurate three-dimensional corner point coordinate. According to the camera imaging formula, the positions of these more accurate three-dimensional corner points in the image are calculated to obtain the corrected calibration plate corner point coordinates.

5. The calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system according to claim 1, characterized in that: The method for removing obvious noise points in step 4.3 is to remove points that have a large spatial distance not only from other points in the same pixel grid but also from surrounding points.

6. The calibration and point cloud fusion method for a multi-camera single-projector 3D reconstruction system according to claim 1, characterized in that: The high-precision projector will project the pattern and reflect it onto the surface of the object to be tested through a high-precision mirror.

Images