A Calibration Method for a Three-Dimensional Reconstruction System for Bulk Cargo Pile Measurement

Abstract

The present invention discloses a three-dimensional reconstruction system for bulk cargo pile measurement and its calibration method, which relates to the field of computer vision technology and aims to improve the accuracy and efficiency of bulk cargo pile measurement. First, the Zhang-Zhengyou calibration method is used to accurately calibrate the camera to obtain the internal parameters and distortion coefficients of the camera. Secondly, the projector is regarded as an inverse camera model, and the inverse camera method is used to calibrate the projector to determine its internal parameters and distortion coefficients. Since the surface of the calibration device may not be completely flat in practice, the present invention further performs bundle adjustment on the camera calibration results, the projector calibration results, and the feature points on the calibration device simultaneously to optimize the system parameters and reduce the requirements for the calibration device. Through this comprehensive calibration method, the present invention can more accurately use the fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk cargo pile, thereby improving the measurement accuracy and reliability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a calibration method for a three-dimensional reconstruction system for bulk material pile measurement. Background Art

[0002] In the measurement and management of bulk material piles, accurately obtaining the three-dimensional information of the piles is crucial. Traditional measurement methods often rely on manual operations or simple automation devices, which are not only time-consuming and labor-intensive, but also difficult to achieve high-precision measurement. Therefore, seeking an efficient and accurate three-dimensional measurement technology for material piles has become an important requirement in the industry.

[0003] Structured light fringe projection, as a full-field three-dimensional measurement technology based on triangulation, provides a possibility to solve this requirement. By using a projection device to project a fringe pattern onto the surface of the material pile and using a camera to capture the fringe pattern modulated by the shape of the material pile, this technology can analyze and obtain the corresponding position relationship between the positions on the surface of the measured material pile in the camera-acquired image and the projection pattern. Combining the internal and external parameters of the pre-calibrated camera and projection device, the depth information of the surface of the material pile can be calculated using the triangulation principle, thereby achieving accurate measurement of its three-dimensional shape.

[0004] When applying structured light fringe projection to bulk material pile measurement, the calibration of the system is a key step. The calibration method based on stereo vision regards the projector as an inverse camera model and forms a stereo vision system with the camera. However, since the projector cannot capture the feature points on the surface of the material pile, how to accurately find the matching relationship between the pixel points on the projection pattern and the real three-dimensional coordinate points has become a challenge. In the prior art, methods such as spatially encoded color structured light and temporally encoded fringe patterns are used to find this matching relationship, but they each have limitations in terms of accuracy or cost.

[0005] In addition, the calibration fixtures in practical applications often cannot be absolutely smooth. In other words, the calibration fixture is not an absolute plane, which will affect the calibration accuracy. High-precision calibration fixtures are usually expensive, and by modifying the position of the feature points or using the bundle adjustment method to reduce the requirements for the calibration fixture, additional costs may be introduced or additional equipment may be required.

[0006] Although structured light fringe projection has significant potential in bulk material pile measurement, the existing system calibration methods and the selection of calibration fixtures still face a series of challenges. Therefore, aiming at the specific requirements of bulk material pile measurement, developing an accurate and economical system calibration method and the corresponding calibration fixture is of great significance for improving the application effect of structured light fringe projection in pile measurement. Summary of the Invention

[0007] In view of this, the present invention proposes a calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement, taking into account the non-planarity of the calibration instruments used. This not only reduces the requirements for the calibration instruments used, but also can obtain higher system calibration accuracy when using calibration instruments with the same precision for system calibration, providing a new idea for reducing the cost of system calibration.

[0008] To this end, the present invention adopts the following technical solutions:

[0009] The present invention provides a calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement, including:

[0010] S1. Perform camera calibration using Zhang Zhengyou calibration;

[0011] S2. Perform projector calibration using the inverse camera method;

[0012] S3. Perform bundle adjustment with the camera calibration, projector calibration results, and specific points in the initial world coordinate system as initial values; the bundle adjustment optimizes the initial values of the parameters to be optimized by minimizing the error function; the error function is:

[0013]

[0014] Among them, is the reprojection error of camera calibration; is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th camera-captured image; is the coordinate of the feature point j in the world coordinate system projected onto the camera pixel coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficient; θ c includes the rotation matrix R, translation vector t, internal parameter matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2 of the camera;

[0015] is the reprojection error of projector calibration; p p dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th projector pattern; is the coordinate of the bulk cargo point j in the world coordinate system projected onto the projector pixel coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficient; θ p includes the rotation matrix R p 、translation vector t p 、internal parameter matrix K p 、radial distortion coefficients kp1, kp2, k p 3 and tangential distortion coefficients p p 1, p p 2;

[0016] represent all feature points on calibration board images with m different poses; λ is the scale factor;

[0017] S4. Simultaneously adjust the coordinates of the feature points in the world coordinate system, the internal and external parameters, and the distortion coefficients of the camera and the projector by bundle adjustment for iterative optimization, and output the calibration result after the iteration terminates;

[0018] S5. Based on the calibration result, use fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk cargo pile.

[0019] Furthermore, perform bundle adjustment with the camera calibration result, the projector calibration result, and specific points in the initial world coordinate system as initial values, including:

[0020] Generate a sparse Jacobian matrix and set the iteration stop condition;

[0021] Calculate the scale factor λ:

[0022]

[0023] Calculate the error function;

[0024] Update the feature points in the world coordinate system, the camera calibration result, and the projector calibration result;

[0025] Judge whether the iteration termination condition is satisfied. If it is satisfied, output the calibration result; if not, return to the step of calculating the scale factor to continue the iteration.

[0026] Furthermore, generating a sparse Jacobian matrix includes:

[0027] Each column in the Jacobian matrix represents the parameter to be optimized, and each row represents each item in the error function;

[0028] The reprojection error of the horizontal coordinate of the camera is related to the internal parameter, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the first and third items of the translation vector of the camera, and set it to 1;

[0029] The reprojection error of the vertical coordinate of the camera is related to the internal parameter, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the second and third items of the translation vector of the camera, and set it to 1;

[0030] Set the sparse Jacobian matrix according to the correlation between the reprojection errors of the horizontal and vertical coordinates of the projector and the parameters to be optimized;

[0031] For the offset error of the feature points in the world coordinate system, it is only related to itself, and set the diagonal matrix to 1;

[0032] Set the remaining terms to 0.

[0033] Furthermore, the trust-region reflection algorithm is used to solve the error function.

[0034] Furthermore, Zhang Zhengyou calibration is used for camera calibration, including:

[0035] Given m calibration board images with different poses, each image containing n feature points, according to the camera model, when the internal parameters, external parameters, and distortion coefficients of the camera are known, the feature points on the calibration board in the world coordinate system are reprojected as a point in the camera pixel coordinate system;

[0036] The internal parameters, external parameters, and distortion coefficients of the camera are estimated by minimizing the reprojection error, and the reprojection error is:

[0037]

[0038] where p c dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th camera-captured image; is the coordinate of feature point j in the world coordinate system projected onto the camera pixel coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficients; θ c includes the rotation matrix R, translation vector t, internal parameter matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2 of the camera.

[0039] Furthermore, the inverse camera method is used for projector calibration, including:

[0040] The camera captures the surface of the calibration board, and the coordinates of the feature points on the calibration board in the camera-captured image are found through image processing algorithms;

[0041] Assume the resolution of the projector is H p ×W p , the projector projects a set of N-step vertical sinusoidal grating patterns with a stripe number of n v and a set of N-step horizontal sinusoidal grating patterns with a stripe number of n h onto the surface of the calibration board respectively to establish the correspondence between the horizontal and vertical coordinates on the projector pattern and the camera-captured image;

[0042] Use the camera to capture the image of the calibration board surface containing the sinusoidal grating pattern, solve the wrapped phase of each pixel point on the camera-captured image through phase-shifting profilometry, and correspondingly, the phase unwrapping algorithm can be used to unwrap the wrapped phase to obtain the unwrapped phase of each pixel point. The unwrapped phases obtained using vertical stripes and horizontal stripes are respectively denoted as φ v (x c ,yc ) and φ h (x c , y c );

[0043] The coordinates of the points on the picture captured by the camera are (x c , y c ), T , and the coordinates of the points on the projector pattern are (x p , y p ). T Solve the correspondence between the pixel points on the picture captured by the camera and the pixel points on the projector pattern:

[0044]

[0045] After establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern, use the camera calibration method to calibrate the projector;

[0046] Estimate the internal parameters, external parameters, and distortion coefficients of the projector by minimizing the reprojection error. The reprojection error is:

[0047]

[0048] where p p dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th projector pattern; is the coordinate projected from the stockpile point j in the world coordinate system to the projector pixel coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficients; θ p includes the rotation matrix R p of the projector, the translation vector t p , the internal parameter matrix K p , the radial distortion coefficients k p 1, k p 2, k p 3, and the tangential distortion coefficients p p 1, p p 2.

[0049] Furthermore, based on the calibration results, use the fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk stockpile, including:

[0050] Use the projection device to project the fringe pattern onto the surface of the stockpile, and use the camera to capture the fringe pattern modulated by the shape of the stockpile; Analyze and obtain the corresponding position relationship between the positions on the surface of the measured stockpile in the camera-acquired image and the projection pattern;

[0051] Combined with the calibrated internal and external parameters of the camera and the projection device in the calibration result, the depth information of the stockpile surface is calculated using the principle of triangulation, so as to achieve accurate measurement of its three-dimensional shape.

[0052] The above technical solution has the following beneficial effects: The present invention proposes a calibration method for a three-dimensional reconstruction system for bulk stockpile measurement, which simultaneously performs bundle adjustment on the calibration results of the camera, the calibration results of the projector, and the three-dimensional feature point coordinates in the world coordinate system. It can handle the influence of the uneven surface of the calibration device on the system calibration, improve the calibration accuracy of the system, reduce the requirements for the use of calibration devices, and bring new ideas for reducing the system cost. Compared with the reprojection errors of the camera and the projector when the method proposed by the present invention is not used for optimization, the reprojection error of the camera is reduced by 50.36%, and the reprojection error of the projector is reduced by 72.64%. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a process diagram of fringe projection profilometry in the prior art;

[0055] Figure 2 It is a flowchart of the system calibration method considering the unevenness of the calibration device in the embodiment of the present invention;

[0056] Figure 3 It is a schematic diagram of the pinhole model in the embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the influence of different distortions on the points in the pixel physical coordinate system in the embodiment of the present invention;

[0058] Figure 5 It is a schematic diagram of the projector "seeing" the feature points on the calibration board in the embodiment of the present invention;

[0059] Figure 6 It is a schematic diagram of the generation method of the sparse Jacobian matrix in the embodiment of the present invention;

[0060] Figure 7 It is a schematic diagram of the calibration board with different poses photographed by the camera in the embodiment of the present invention;

[0061] Figure 8 It is a schematic diagram of the vertical sine grating in different poses in the embodiment of the present invention;

[0062] Figure 9 Schematic diagram of horizontal sine grating in different postures in the embodiment of the present invention;

[0063] Figure 10 Schematic diagram of reprojection error of the camera and the projector before optimization in the embodiment of the present invention;

[0064] Figure 11 Schematic diagram of reprojection error of the camera and the projector after optimization in the embodiment of the present invention. Detailed implementation manners

[0065] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0067] A calibration method for a three-dimensional reconstruction system for bulk material pile measurement is proposed in the embodiment of the present invention. First, the camera is calibrated using Zhang Zhengyou calibration. Secondly, the projector is regarded as an inverse camera model, and the inverse camera method is used to calibrate the projector. Finally, considering the factor that the surface of the calibration fixture used is uneven, the calibration results of the camera, the calibration results of the projector, and the feature points of the calibration fixture used are simultaneously adjusted by bundle adjustment. At the same time, the calibration parameters of the camera and the projector and the coordinates of the three-dimensional feature points are optimized.

[0068] As Figure 2 shown, it specifically includes the following steps:

[0069] S1. Calibrate the camera using Zhang Zhengyou calibration.

[0070] When calibrating the camera, a most commonly used model is as Figure 3The pinhole model shown. Among them, O w -X w Y w Z w represents the world coordinate system. P(x w ,y w ,z w ) is a point on the world coordinate system. After the rigid body transformation of the point on the world coordinate system, it becomes a point on the camera coordinate system O c -X c Y c Z c . O c is the optical center. On the pixel plane at a distance f from the camera plane, there are the pixel physical coordinate system u0v0-xy and the pixel coordinate system O-uv.

[0071] The purpose of camera calibration is to estimate the internal parameter matrix, external parameter matrix, and distortion coefficient of the camera. The rigid body transformation from a point in the world coordinate system to the camera coordinate system can be formulated as:

[0072]

[0073] Among them, is the homogeneous coordinate description of a point P(x w ,y w ,z w ) T in the world coordinate system. The superscript T represents the transpose. is the homogeneous coordinate description of the point in the camera coordinate system. R represents a 3×3 rotation matrix described by Euler angles and has unit orthogonality. t is a 3×1 translation vector that describes the translation change from the world coordinate system to the camera coordinate system.

[0074] When the point on the world coordinate system is transformed to the camera coordinate system through the rigid body transformation, combined with the focal length information of the camera and the principle of pinhole imaging, it is transformed to the pixel physical coordinate system:

[0075]

[0076] Among them, f represents the focal length of the camera, z c is the z-axis coordinate of the point in the camera coordinate system. is the homogeneous coordinate description of the point in the pixel physical coordinate system.

[0077] The process of the point on the pixel physical coordinate system becoming a point on the pixel coordinate system can be formulated as:

[0078]

[0079] Among them, is the homogeneous description of a point in the pixel coordinate system. Δu and Δv are the sizes of a camera pixel along the u-axis and v-axis directions respectively. (u0, v0) are the coordinates of the principal point, which describes the position of the camera optical center on the pixel plane. s is the skew coefficient, which describes the inclination between two planes. When the image axes are not perpendicular, its value is not zero.

[0080] Combining Equation (2) and Equation (3), the transformation from the camera coordinate system to the pixel coordinate system can be obtained as:

[0081]

[0082] For convenience, the parameters such as the focal length of the camera, the position of the principal point, and the scale factor of the pixel are denoted as the internal parameter matrix K:

[0083]

[0084] In summary, the entire camera calibration process can be described by Equation (6), where 0 represents a 3×1 all-zero matrix.

[0085]

[0086] The above pinhole model describes the case without using a lens. However, in the actual imaging process, in order to obtain better imaging effects, a lens system is often added in front of the camera. Due to the non-ideal properties of the light passing through the lens system, the addition of the lens system will cause the position of the point p'=(x, y) T in the pixel physical coordinate system to change, resulting in image distortion. According to the causes of distortion, camera distortion can be divided into two types: radial distortion and tangential distortion. The effects of different distortions on the point p' in the pixel physical coordinate system are as Figure 4 shown.

[0087] The reason for radial distortion is that the shape of the lens itself will affect the propagation of light. It mainly includes two types: barrel distortion and pincushion distortion. Barrel distortion makes the area near the center of the image appear inflated, while pincushion distortion makes the area near the center appear contracted. This kind of distortion usually makes straight lines or curves appear bent or distorted in the image. Radial distortion can be described formulaically by Equation (7), where k1, k2, k3 are the radial distortion coefficients, which need to be estimated in camera calibration, and δ xr , δ yr are the radial distortion errors of the x-axis and y-axis respectively:

[0088]

[0089] Tangential distortion is mainly caused by the incomplete parallelism or asymmetric placement between lens components. This distortion makes the straight lines in the image become curved, and usually appears in the edge part of the image. Tangential distortion can be formulated as Equation (8), where p1 and p2 are tangential distortion coefficients and need to be estimated in camera calibration, and δ xt , δ yt are the tangential distortion errors of the x-axis and y-axis respectively:

[0090]

[0091] In summary, in the actual imaging process, the actual position p d '=(x d , y d ) T of the point p' on the pixel physical coordinate system can be described as:

[0092]

[0093] The coordinates of the point with camera distortion on the pixel coordinate system are:

[0094]

[0095] Currently, Zhang Zhengyou calibration method is one of the most commonly used calibration methods in computer vision. Given m calibration board pictures with different poses, each picture contains n feature points. According to the above camera model, when the internal parameters, external parameters and distortion coefficients of the camera are known, it is easy to reproject the feature points on the calibration board in the world coordinate system into a point on the camera pixel coordinate system. If the used internal parameters, external parameters and distortion coefficients of the camera are correct, the points reprojected into the camera pixel coordinate system should coincide with the feature point coordinates obtained by the image processing algorithm for the pictures captured by the camera. Therefore, the internal parameters, external parameters and distortion coefficients of the camera can be estimated by minimizing the reprojection error, and the reprojection error can be described as:

[0096]

[0097] where p c dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th picture captured by the camera. Generally, the corresponding pixel coordinates can be identified through the image processing algorithm. is the coordinate of the feature point j in the world coordinate system projected into the camera pixel coordinate system through the estimated internal parameter matrix, external parameter matrix and distortion coefficients. θ c includes the rotation matrix R, translation vector t, internal parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2 of the camera.

[0098] S2. Use the inverse camera method to calibrate the projector.

[0099] The projector can be regarded as the inverse model of the camera. Therefore, the above camera calibration method can be used to calibrate the projector. However, since the projector cannot actively capture the feature points on the stockpile surface, the key to applying the camera calibration method to projector calibration is how to enable the projector to see the feature points on the stockpile surface like a camera.

[0100] In the structured light three-dimensional reconstruction system, the system consists of a camera and a projector. The camera can be used as the "eye" of the projector to capture the feature points on the stockpile surface. Using the phase as a bridge, the coordinate of the feature points captured by the camera is converted to the projector pattern, thereby establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern. The principle of establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern is phase-shifting profilometry, and its schematic diagram is as Figure 5 shown. The specific process can be divided into four steps: First, the camera captures the surface of the calibration board, and the coordinates of the feature points on the calibration board in the image captured by the camera are found through the image processing algorithm. Second, assuming the resolution of the projector is H p ×W p , the projector projects a set of N-step vertical sinusoidal grating patterns with the number of stripes n v and a set of N-step horizontal sinusoidal grating patterns with the number of stripes n h onto the surface of the calibration board to establish the correspondence between the horizontal and vertical coordinates on the projector pattern and the image captured by the camera. Third, the camera captures the image of the calibration board surface containing the sinusoidal grating pattern, and the wrapped phase of each pixel point in the image captured by the camera is solved through phase-shifting profilometry. Correspondingly, the phase unwrapping algorithm can be used to unwrap the wrapped phase to obtain the unwrapped phase of each pixel point. The unwrapped phases obtained using the vertical stripes and horizontal stripes are denoted as φ v (x c ,y c ) and φ h (x c ,y c ), respectively. Fourth, the coordinates of the points on the image captured by the camera are (x c ,y c ) T , and the coordinates of the points on the projector pattern are (x p ,y p ) T . Then, the correspondence between the pixel points on the image captured by the camera and the pixel points on the projector pattern can be solved by Equation (12).

[0101]

[0102] After establishing the correspondence between the coordinates of feature points in the world coordinate system and on the projector pattern, the projector can be calibrated using the camera calibration method. The internal parameters, external parameters, and distortion coefficients of the projector are estimated by minimizing the reprojection error, which can be described as:

[0103]

[0104] where p p dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th projector pattern. is the coordinate projected onto the projector pixel coordinate system of the stockpile point j in the world coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficients. θ p includes the rotation matrix R p of the projector, the translation vector t p the internal parameter matrix K p the radial distortion coefficients k p 1, k p 2, k p 3 and the tangential distortion coefficients p p 1, p p 2.

[0105] The projector uses the camera as an "eye" to see the feature points on the surface of the stockpile, and only needs to decode the pre-encoded sinusoidal grating stripes, and it does not require a calibrated camera. Therefore, calibrating the projector by this method can effectively avoid the transmission of camera calibration errors to projector calibration and improve the calibration accuracy of the projector.

[0106] S3. Bundle adjustment is performed using the camera calibration, projector calibration results, and specific points in the initial world coordinate system as initial values.

[0107] If the same set of pictures is used during the calibration of the camera and the projector, the feature points used for camera and projector calibration have the same coordinates in the world coordinate system. Therefore, the calibration results of the camera and the projector can be used as initial values, and the bundle adjustment method can be used to optimize the calibration parameters of the camera and the projector, thereby further improving the calibration accuracy of the system. Considering that the calibration device used is not an absolute plane, the coordinates of the feature points in the world coordinate system are simultaneously adjusted by bundle adjustment with the internal parameters, external parameters, and distortion coefficients of the camera and the projector for iterative optimization, so that the value of the error function shown in Equation (14) is minimized.

[0108]

[0109] where, Denote all the feature points on m calibration board images with different poses. The error function shown in Equation (14) introduces new optimization parameters This indicates that the coordinates of the feature points in the world coordinate system also need to be optimized. Bundle adjustment optimizes the initial values of the parameters to be optimized by minimizing the error function shown in Equation (14). This is a large-scale non-linear least squares problem, which can be solved by the trust region reflection algorithm.

[0110] S4. Generate a sparse Jacobian matrix and set the iteration stop condition.

[0111] Since the coordinates of the feature points in the world coordinate system are optimized simultaneously, the computational cost of the entire optimization process will be very high and the calculation time will be too long. To accelerate the optimization process and reduce the computational cost, in the embodiments of the present invention, a sparse Jacobian matrix J is designed to accelerate the calculation process of numerical finite derivatives. In the MATLAB environment, the calculation of the sparse Jacobian matrix J is automatically completed by MATLAB, and only need to specify how to generate the sparse Jacobian matrix J.

[0112] When generating a sparse Jacobian matrix J, it is necessary to consider how to calculate the partial derivatives of each term in the error function with respect to each parameter to be optimized. In the sparse Jacobian matrix, the values of the partial derivatives of each term in the error function with respect to each parameter to be optimized that need to be calculated are set to 1, and those that do not need to be calculated are set to 0, which reduces the unnecessary system computational cost and thus improves the calculation speed. As Figure 6 shown, it shows the generation method of the sparse Jacobian matrix J, where the yellow ones represent 1 and the blue ones represent 0. Each column in the Jacobian matrix represents the parameter to be optimized, and each row represents each term in the error function. The reprojection error of the horizontal coordinate of the camera is related to the internal parameters of the camera, the distortion coefficients, the rotation matrix, the feature points in the world coordinate system, and the first and third terms of the translation vector, and it is set to 1. The reprojection error of the vertical coordinate of the camera is related to the internal parameters of the camera, the distortion coefficients, the rotation matrix, the feature points in the world coordinate system, and the second and third terms of the translation vector, and it is set to 1. Similarly, the sparse Jacobian matrix can be set according to the correlation between the reprojection errors of the horizontal and vertical coordinates of the projector and the parameters to be optimized. For the offset error of the feature points in the world coordinate system, it is only related to itself, so the diagonal matrix is set to 1.

[0113] S5. Calculate the scale factor;

[0114] λ is the scale factor, which is used to prevent the coordinates of the optimized feature points from deviating too much in the world coordinate system.

[0115] According to experience, the value of the scale factor λ is set as Equation (15):

[0116]

[0117] S6. Calculate the error function;

[0118] S7. Update the calibration results of the feature points, camera, and projector in the world coordinate system;

[0119] S8. Determine whether the iteration termination condition is satisfied. If it is satisfied, output the calibration results; if not, return to S5 to continue the iteration;

[0120] S9. Based on the calibration results, use fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk cargo pile.

[0121] In specific implementation, a projection device is used to project a fringe pattern onto the surface of the pile, and a camera is used to capture the fringe pattern modulated by the shape of the pile. This technology can analyze and obtain the corresponding position relationship between the positions on the surface of the measured pile in the camera-acquired image and the projection pattern. Combining the internal and external parameters of the calibrated camera and projection device, the depth information of the pile surface can be calculated using the principle of triangulation, thereby achieving accurate measurement of its three-dimensional shape.

[0122] To verify the effectiveness of the above system calibration method, 14 calibration board postures as shown in Figure 7 are used for calibration. Figure 7 Only 12 postures of the used calibration board are listed in Figure 10 . By using the corner recognition algorithm of Opencv, the position of the center of the round hole can be known, and then the Zhang Zhengyou calibration method is further used for camera calibration. The reprojection error of the camera is shown in Table 1, and the visualization result is as shown on the right side in

[0123] Table 1

[0124]

[0125]

[0126] After camera calibration, the inverse camera method is used to calibrate the projector of the system. Vertical and horizontal sinusoidal gratings with 12-step phase shifts are projected onto the surface of the calibration board respectively to determine the phase information of the feature points on the abscissa and ordinate respectively, thereby establishing the correspondence between the pixel points in the camera-captured image and the pixel points on the projector pattern. As shown in Figure 8 is one step of the vertical sinusoidal grating with 12-step phase shifts projected, Figure 9 is one step of the horizontal sinusoidal grating with 12-step phase shifts projected.

[0127] The reprojection error of the projector is shown in Table 1, and the visualization result is as shown in Figure 10As shown on the left side in the figure, its average reprojection error is 0.09 pixels. Subsequently, we used the calibration results of the camera and the projector and the feature points in the world coordinate system as the initial values for bundle adjustment optimization. The reprojection errors after optimization are shown in Table 2, and the visualization results of the reprojection errors are as shown in Figure 11 shown. The average reprojection errors of the optimized camera and projector are 0.019857 pixels and 0.024628 pixels respectively. Compared with before optimization, the reprojection error of the camera decreased by 50.36%, and the reprojection error of the projector decreased by 72.64%.

[0128] Table 2

[0129]

[0130] In the above embodiments, a system calibration method considering the unevenness of the calibration instrument is designed for the system calibration problem in phase-shifting profilometry. The calibration results of the camera, the calibration results of the projector, and the three-dimensional feature point coordinates in the world coordinate system are simultaneously adjusted by bundle adjustment, which can handle the influence of the uneven surface of the calibration instrument on system calibration, improve the calibration accuracy of the system, reduce the requirements for the calibration instrument used, and bring new ideas for reducing the system cost. Compared with the reprojection errors of the camera and the projector when the method proposed in the present invention is not used for optimization, the reprojection error of the camera decreased by 50.36%, and the reprojection error of the projector decreased by 72.64%.

[0131] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a three-dimensional reconstruction system used for bulk cargo pile measurement, characterized in that Including: S1. Perform camera calibration using Zhang Zheng-you calibration; S2. Perform projector calibration using the inverse camera method; S3. Use the camera calibration, projector calibration results, and feature points in the initial world coordinate system as initial values for bundle adjustment; the bundle adjustment optimizes the initial values of the parameters to be optimized by minimizing the error function; the error function is: Among them, is the reprojection error of camera calibration; p c dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th camera-captured image; is the coordinate of the feature point j in the world coordinate system projected onto the camera pixel coordinate system through the estimated intrinsic matrix, extrinsic matrix, and distortion coefficients; θ c includes the rotation matrix R, translation vector t, intrinsic matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2 of the camera; Reprojection error calibrated for the projector; p p dij The pixel coordinates of the corresponding point of the feature point j in the world coordinate system on the i-th projector pattern; The coordinates projected onto the projector pixel coordinate system of the stockpile point j in the world coordinate system through the estimated internal parameter matrix, external parameter matrix, and distortion coefficient; θ p Including the rotation matrix R of the projector p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1, k p 2, k p 3 and tangential distortion coefficient p p 1, p p 2; Denote all feature points on calibration board images with m different poses; λ is the scale factor; S4. Simultaneously perform bundle adjustment on the coordinates of the feature points in the world coordinate system, the internal and external parameters of the camera and projector, and the distortion coefficients to iteratively optimize, and output the calibration results after the iteration terminates; S5. Based on the calibration results, use fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk cargo pile.

2. The calibration method of a three-dimensional reconstruction system for bulk cargo pile measurement according to claim 1, characterized in that Using the camera calibration, projector calibration results, and feature points in the initial world coordinate system as initial values for bundle adjustment includes: Generate a sparse Jacobian matrix and set the iteration stop condition; Calculate the scale factor λ: Calculate the error function; Update the feature points in the world coordinate system, the camera, and the projector calibration results; Determine whether the iteration termination condition is satisfied. If satisfied, output the calibration results; if not satisfied, return to the step of calculating the scale factor to continue the iteration.

3. A calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement according to claim 2, characterized in that, Generating a sparse Jacobian matrix includes: Each column in the Jacobian matrix represents a parameter to be optimized, and each row represents each term in the error function; The reprojection error of the camera's abscissa is related to the internal parameters, distortion coefficients, rotation matrix, feature points in the world coordinate system, and the first and third terms of the translation vector of the camera, and set it to 1; The reprojection error of the camera's ordinate is related to the internal parameters, distortion coefficients, rotation matrix, feature points in the world coordinate system, and the second and third terms of the translation vector of the camera, and set it to 1; Set the sparse Jacobian matrix according to the correlation between the reprojection errors of the projector's abscissa and ordinate and the parameters to be optimized; For the offset error of the feature points in the world coordinate system, it is only related to itself, and set the diagonal matrix to 1; Set the remaining terms to 0.

4. A calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement according to claim 1, characterized in that Solve the error function through the trust region reflection algorithm.

5. A calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement according to claim 1, characterized in that, Performing camera calibration using Zhang Zheng-you calibration includes: Given m calibration board images with different poses, each image contains n feature points. According to the camera model, when the internal and external parameters and distortion coefficients of the camera are known, the feature points on the calibration board in the world coordinate system are reprojected as a point in the camera pixel coordinate system; Estimate the internal and external parameters and distortion coefficients of the camera by minimizing the reprojection error. The reprojection error is: where p c dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th camera-captured image; is the coordinate of the feature point j in the world coordinate system projected onto the camera pixel coordinate system through the estimated intrinsic matrix, extrinsic matrix, and distortion coefficients; θ c includes the rotation matrix R, translation vector t, intrinsic matrix K, radial distortion coefficients k1, k2, k3, and tangential distortion coefficients p1, p2 of the camera.

6. A calibration method for a three-dimensional reconstruction system for bulk material pile measurement according to claim 1, characterized in that Performing projector calibration using the inverse camera method includes: The camera captures the surface of the calibration board, and finds the coordinates of the feature points on the calibration board in the camera-captured image through image processing algorithms; Assume that the resolution of the projector is H p ×W p , and the projector projects a set of N-step vertical sinusoidal grating patterns with a stripe number of n v and a set of N-step horizontal sinusoidal grating patterns with a stripe number of n h on the surface of the calibration board to establish the correspondence between the horizontal and vertical coordinates on the projector pattern and the image captured by the camera; Use a camera to capture an image of a calibration board surface containing a sinusoidal grating pattern. Solve the wrapped phase of each pixel point on the captured image by phase-shifting profilometry. Use a phase unwrapping algorithm to unwrap the wrapped phase and obtain the unwrapped phase of each pixel point. Denote the unwrapped phases obtained using vertical stripes and horizontal stripes as φ v (x c ,y c ) and φ h (x c ,y c ); The coordinates of the points on the picture captured by the camera are (x c , y c ). T , and the coordinates of the points on the projector pattern are (x p , y p ). T , to solve the corresponding relationship between the pixel points on the picture captured by the camera and the pixel points on the projector pattern: After establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern, use the camera calibration method to calibrate the projector; Estimate the internal and external parameters and distortion coefficients of the projector by minimizing the reprojection error. The reprojection error is: where p p dij is the pixel coordinate of the corresponding point of the feature point j in the world coordinate system on the i-th projector pattern; is the coordinate projected from the stockpile point j in the world coordinate system to the projector pixel coordinate system through the estimated internal parameter matrix, external parameter matrix and distortion coefficient; θ p includes the rotation matrix R of the projector p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1, k p 2, k p 3 and tangential distortion coefficient p p 1, p p 2.

7. A calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement according to claim 1, characterized in that, Based on the calibration results, using fringe projection profilometry to perform three-dimensional reconstruction measurement on the bulk cargo pile includes: Use a projection device to project a stripe pattern onto the surface of the stockpile, and use a camera to capture the stripe pattern modulated by the shape of the stockpile; analyze and obtain the corresponding position relationship between the positions on the surface of the measured stockpile in the camera-acquired image and the projection pattern. Combined with the internal and external parameters of the calibrated camera and projection device in the calibration result, use the principle of triangulation to calculate the depth information of the stockpile surface, so as to realize the accurate measurement of its three-dimensional shape.

Images