Three-dimensional reconstruction system calibration method for bulk cargo pile measurement

Abstract

The invention discloses a three-dimensional reconstruction system for bulk material pile measurement and a calibration method thereof, relates to the technical field of computer vision, and aims to improve the accuracy and efficiency of bulk material pile measurement. Firstly, a Zhang Zhengyou calibration method is used to accurately calibrate a camera so as to obtain internal parameters and distortion coefficients of the camera. Secondly, the projector is regarded as an inverse camera model, and the projector is calibrated by adopting an inverse camera method so as to determine internal parameters and distortion coefficients of the projector; as the surface of the calibration device is possibly not completely flat in practice, the camera calibration result, the projector calibration result and the feature points on the calibration device are further subjected to binding adjustment at the same time, so that system parameters are optimized, and the requirement for the calibration device is reduced. Through the comprehensive calibration method, the fringe projection profilometry can be more accurately utilized to carry out three-dimensional reconstruction measurement on the bulk material pile, so that the measurement precision and reliability are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement. Background Art

[0002] In the measurement and management of bulk cargo stockpiles, it is crucial to accurately obtain the three-dimensional information of the stockpile. Traditional measurement methods often rely on manual operation or simple automated equipment, which are not only time-consuming and labor-intensive, but also difficult to achieve high-precision measurement. Therefore, seeking an efficient and accurate three-dimensional measurement technology for stockpiles has become an important demand in the industry.

[0003] Fringe projection profilometry, as a full-field 3D measurement technology based on triangulation, provides a possible solution to this demand. By using a projection device to project a fringe pattern onto the surface of the pile, and using a camera to capture the fringe pattern after the pile shape is modulated, this technology can analyze the corresponding position relationship between the camera captured image and the projection pattern at the position on the measured pile surface. Combined with the pre-calibrated internal and external parameters of the camera and projection device, the depth information of the pile surface can be calculated using the principle of triangulation, thereby achieving accurate measurement of its 3D shape.

[0004] When applying fringe projection profilometry to bulk cargo stockpile measurement, system calibration is a key step. The calibration method based on stereo vision regards the projector as an inverse camera model, which forms a stereo vision system with the camera. However, since the projector cannot capture the feature points on the surface of the stockpile, it is a challenge to accurately find the matching relationship between the pixel points on the projection pattern and the real three-dimensional coordinate points. In the prior art, methods such as spatially encoded color structured light and time-encoded fringe patterns are used to find this matching relationship, but they each have limitations in accuracy or cost.

[0005] In addition, the calibration tools in practical applications are often not absolutely smooth. In other words, the calibration tools are not absolutely flat, which will affect the calibration accuracy. High-precision calibration tools are usually expensive, and lowering the requirements for calibration tools by modifying the position of feature points or using binding adjustment methods may introduce additional costs or require additional equipment.

[0006] Although fringe projection profilometry has significant potential in bulk stockpile measurement, the existing system calibration methods and the selection of calibration instruments still face a series of challenges. Therefore, it is of great significance to develop an accurate and economical system calibration method and corresponding calibration instruments to improve the application effect of fringe projection profilometry in bulk stockpile measurement according to the specific needs of bulk stockpile measurement. Summary of the invention

[0007] In view of this, the present invention proposes a calibration method for a three-dimensional reconstruction system for bulk stockpile measurement, which takes the non-planarity of the calibration instrument used into account. This can not only reduce the requirements for the calibration instrument used, but also obtain higher system calibration accuracy when using calibration instruments with the same accuracy for system calibration, thereby providing a new idea for reducing the cost of system calibration.

[0008] To this end, the present invention adopts the following technical solutions:

[0009] The present invention provides a calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement, comprising:

[0010] S1. Use Zhang Zhengyou calibration to calibrate the camera;

[0011] S2, projector calibration using the inverse camera method;

[0012] S3, using the camera calibration, projector calibration results and the specific point in the initial world coordinate system as initial values ​​to perform bundling adjustment; the bundling adjustment optimizes the initial values ​​of the parameters to be optimized by minimizing the error function; the error function is:

[0013]

[0014] in, is the reprojection error for camera calibration; The pixel coordinates of the corresponding point on the image captured by the i-th camera are the feature point j in the world coordinate system; is the coordinate of feature point j in the world coordinate system projected to the camera pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ c Including the camera's rotation matrix R, translation vector t, intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2;

[0015] is the reprojection error calibrated by the projector; p p dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th projector pattern; is the coordinate of the stockpile point j in the world coordinate system projected to the projector pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ p Including the projector's rotation matrix R p , translation vector t p , internal parameter matrix K p , radial distortion coefficients kp1, kp2, k p 3 and the tangential distortion coefficient p p 1,p p 2;

[0016] represents all feature points on m calibration plate images of different postures; λ is the scale factor;

[0017] S4, the coordinates of the feature points in the world coordinate system are adjusted by bundling technique together with the intrinsic parameters, extrinsic parameters and distortion coefficients of the camera and projector to iteratively optimize, and the calibration result is output after the iteration is terminated;

[0018] S5. Based on the calibration results, fringe projection profilometry is used to perform three-dimensional reconstruction measurement of the bulk cargo pile.

[0019] Furthermore, the binding technique is adjusted using the camera calibration, the projector calibration results and a specific point in the initial world coordinate system as initial values, including:

[0020] Generate sparse Jacobian matrix and set iteration stop condition;

[0021] Calculate the scale factor λ:

[0022]

[0023] Calculate the error function;

[0024] Update the feature points, camera and projector calibration results in the world coordinate system;

[0025] Determine whether the iteration termination condition is met. If so, output the calibration result; if not, return to the step of calculating the scale factor and continue the iteration.

[0026] Furthermore, a sparse Jacobian matrix is ​​generated, including:

[0027] Each column in the Jacobian matrix represents the parameter to be optimized, and each row represents each term in the error function;

[0028] The reprojection error of the camera's horizontal coordinate is related to the camera's intrinsic parameters, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the first and third items in the translation vector, which are set to 1;

[0029] The reprojection error of the camera's ordinate is related to the camera's intrinsic parameters, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the second and third terms in the translation vector, which are set to 1;

[0030] A sparse Jacobian matrix is ​​set according to the correlation between the reprojection errors of the horizontal and vertical coordinates of the projector and the parameters to be optimized;

[0031] For the offset error of the feature point in the world coordinate system, it is only related to itself, and the diagonal matrix is ​​set to 1;

[0032] Set the remaining items to 0.

[0033] Furthermore, the error function is solved by a trust region reflective algorithm.

[0034] Furthermore, the camera calibration is performed using Zhang Zhengyou calibration, including:

[0035] Given m calibration plate images of different poses, each image contains n feature points. According to the camera model, knowing the camera's intrinsic parameters, extrinsic parameters and distortion coefficients, reproject the feature points on the calibration plate in the world coordinate system into a point in the camera's pixel coordinate system.

[0036] The camera's intrinsic parameters, extrinsic parameters, and distortion coefficients are estimated by minimizing the reprojection error. The reprojection error is:

[0037]

[0038] where p c dij The pixel coordinates of the corresponding point on the image captured by the i-th camera are the feature point j in the world coordinate system; is the coordinate of feature point j in the world coordinate system projected to the camera pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ c Including the camera's rotation matrix R, translation vector t, intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2.

[0039] Furthermore, the inverse camera method is used to calibrate the projector, including:

[0040] The camera captures the surface of the calibration plate, and uses the image processing algorithm to find the coordinates of the feature points on the calibration plate in the image captured by the camera;

[0041] Assume the resolution of the projector is H p ×W p , the projector projects a set of stripes with the number of n onto the surface of the calibration plate v An N-step vertical sinusoidal grating pattern and a set of n fringes h The N-step horizontal sinusoidal grating pattern is used to establish the corresponding relationship between the horizontal and vertical coordinates on the projector pattern and the image captured by the camera;

[0042] Use a camera to capture an image of a sinusoidal grating pattern on the surface of the calibration plate. Use phase shift profilometry to solve the wrapped phase of each pixel on the camera captured image. Correspondingly, the phase unwrapping algorithm can be used to unwrap the wrapped phase to obtain the unwrapped phase of each pixel. The unwrapped phases obtained using vertical stripes and horizontal stripes are denoted as φ v (x c ,yc ) and φ h (x c ,y c );

[0043] The coordinates of the point on the image captured by the camera are (x c ,y c ) T , the coordinates of the point on the projector pattern are (x p ,y p ) T , solve the correspondence between the pixels on the camera image and the pixels on the projector pattern:

[0044]

[0045] After establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern, the projector is calibrated using the camera calibration method;

[0046] The intrinsic parameters, extrinsic parameters and distortion coefficients of the projector are estimated by minimizing the reprojection error. The reprojection error is:

[0047]

[0048] where p p dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th projector pattern; is the coordinate of the stockpile point j in the world coordinate system projected to the projector pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ p Including the projector's rotation matrix R p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1,k p 2,k p 3 and the tangential distortion coefficient p p 1,p p 2.

[0049] Further, based on the calibration results, fringe projection profilometry is used to perform three-dimensional reconstruction measurement of the bulk cargo pile, including:

[0050] The stripe pattern is projected onto the surface of the pile using a projection device, and the stripe pattern modulated by the shape of the pile is captured using a camera; the corresponding position relationship between the position on the surface of the measured pile and the camera captured image and the projected pattern is analyzed;

[0051] The depth information of the surface of the pile is calculated by combining the intrinsic and extrinsic parameters of the camera and the projection device calibrated in the calibration result and utilizing the triangulation principle, thereby realizing accurate measurement of the three-dimensional shape thereof.

[0052] The above technical solution has the following beneficial effects: the present invention proposes a calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement, which simultaneously adjusts the camera calibration results, the projector calibration results, and the three-dimensional feature point coordinates in the world coordinate system by bundling, and can handle the impact of the uneven surface of the calibration instrument on the system calibration, improve the calibration accuracy of the system, reduce the requirements for the use of the calibration instrument, and bring new ideas for reducing the system cost. Compared with the reprojection error of the camera and projector when the method proposed by the present invention is not used for optimization, the reprojection error of the camera is reduced by 50.36%, and the reprojection error of the projector is reduced by 72.64%. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 It is a process diagram of fringe projection profilometry in the prior art;

[0055] Figure 2 A flow chart of a system calibration method considering unevenness of a calibration device in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a pinhole model in an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of the influence of different distortions on points in the pixel physical coordinate system in an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of feature points on the calibration plate “seen” by a projector in an embodiment of the present invention;

[0059] Figure 6 Schematic diagram of a method for generating sparse Jacobian moments in an embodiment of the present invention;

[0060] Figure 7 Schematic diagram of a calibration plate with different postures photographed by a camera in an embodiment of the present invention;

[0061] Figure 8 Schematic diagram of a vertical sinusoidal grating in different postures in an embodiment of the present invention;

[0062] Fig. 9 Schematic diagram of a horizontal sinusoidal grating in different postures in an embodiment of the present invention;

[0063] Fig.10 A schematic diagram of reprojection errors of a front camera and a projector optimized in an embodiment of the present invention;

[0064] Fig.11 Schematic diagram of the reprojection error of the optimized camera and projector in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] In an embodiment of the present invention, a calibration method for a three-dimensional reconstruction system for bulk cargo pile measurement is proposed. First, the camera is calibrated using Zhang Zhengyou calibration. Secondly, the projector is regarded as an inverse camera model and the inverse camera method is used to calibrate the projector. Finally, considering the uneven surface of the calibration tool used, the calibration results of the camera, the calibration results of the projector, and the feature points of the calibration tool are simultaneously adjusted by bundling. At the same time, the calibration parameters of the camera and the projector and the coordinates of the three-dimensional feature points are optimized.

[0068] like Figure 2 As shown, the specific steps include:

[0069] S1. Use Zhang Zhengyou calibration to calibrate the camera.

[0070] When calibrating a camera, the most commonly used model is Figure 3The pinhole model shown. w -X w Y w Z w represents the world coordinate system, P(x w ,y w ,z w ) is a point in the world coordinate system. After the rigid body transformation, the point in the world coordinate system becomes the camera coordinate system O c -X c Y c Z c A little bit up, O c The pixel plane at the focal length f from the camera plane contains the pixel physical coordinate system u0v0-xy and the pixel coordinate system O-uv.

[0071] The purpose of camera calibration is to estimate the camera's intrinsic matrix, extrinsic matrix, and distortion coefficients. The rigid body transformation from a point in the world coordinate system to the camera coordinate system can be formulated as:

[0072]

[0073] in, is a point P(x w ,y w ,z w ) T Homogeneous coordinate description of , where the superscript T indicates transposition, is the homogeneous coordinate description of the point in the camera coordinate system. R represents a 3×3 rotation matrix described by Euler angles, which has unit orthogonality. t is a 3×1 translation vector that describes the translation change from the world coordinate system to the camera coordinate system.

[0074] After the point in the world coordinate system is transformed to the camera coordinate system through rigid body transformation, it is transformed to the pixel physical coordinate system based on the focal length information of the camera and the pinhole imaging principle:

[0075]

[0076] Where f is the focal length of the camera, z c is the midpoint of the camera coordinate system The z-axis coordinate of It is the homogeneous coordinate description of the point in the pixel physical coordinate system.

[0077] Point on the pixel physical coordinate system The process of becoming a point in the pixel coordinate system can be formally described as:

[0078]

[0079] in, is the homogeneous description of the midpoint in the pixel coordinate system. Δu and Δv are the sizes of a camera pixel along the u-axis and v-axis respectively. (u0, v0) is the principal point coordinate, describing the position of the camera optical center on the pixel plane. s represents the skew coefficient, describing the inclination of the two planes directly. When the image axes are not perpendicular, its value is not zero.

[0080] Combining equations (2) and (3), we can get the transformation from the camera coordinate system to the pixel coordinate system as follows:

[0081]

[0082] For convenience, the parameters describing the focal length of the camera, the position of the principal point, and the scale factor of the pixel are recorded as the intrinsic parameter matrix K:

[0083]

[0084] In summary, the entire camera calibration process can be described as equation (6), where 0 represents a 3×1 all-zero matrix.

[0085]

[0086] The pinhole model above describes the situation where no lens is used. However, in the actual imaging process, in order to obtain better imaging effects, a lens system is often added in front of the camera. Due to the non-ideal properties of light passing through the lens system, the addition of the lens system will make the point p' = (x, y) in the pixel physical coordinate system T The position of the pixel changes, resulting in image distortion. According to the cause of the distortion, camera distortion can be divided into two types: radial distortion and tangential distortion. The influence of different distortions on the point p' in the pixel physical coordinate system is as follows: Figure 4 shown.

[0087] The reason for radial distortion is that the shape of the lens itself affects the propagation of light. There are two main types of radial distortion: barrel distortion and pincushion distortion. Barrel distortion makes the area near the center of the image appear to expand, while pincushion distortion makes the area near the center appear to shrink. This distortion usually makes straight lines or curves appear to bend or distort in the image. Radial distortion can be formulated as formula (7), where k1, k2, k3 are radial distortion coefficients, which need to be estimated in camera calibration. xr ,δ yr They are the radial distortion errors of the x-axis and y-axis respectively:

[0088]

[0089] Tangential distortion is mainly caused by the incomplete parallelism or asymmetric placement of lens components. This distortion will cause straight lines in the image to become curved, usually at the edge of the image. The tangential distortion can be formulated as Equation (8), where p1, p2 are the tangential distortion coefficients that need to be estimated in camera calibration, δ xt ,δ yt They are the tangential distortion errors of the x-axis and y-axis respectively:

[0090]

[0091] In summary, in the actual imaging process, the actual position p of point p' in the pixel physical coordinate system is d '=(x d ,y d ) T It can be described as:

[0092]

[0093] The coordinates of the point containing camera distortion in the pixel coordinate system are:

[0094]

[0095] At present, Zhang Zhengyou calibration method is the most commonly used calibration method in computer vision. Given m calibration plate images with different postures, each image contains n feature points. According to the above camera model, knowing the camera's intrinsic parameters, extrinsic parameters and distortion coefficients, it is easy to reproject the feature points on the calibration plate in the world coordinate system into a point in the camera pixel coordinate system. If the camera's intrinsic parameters, extrinsic parameters and distortion coefficients used are correct, the points reprojected into the camera's pixel coordinate system should coincide with the feature point coordinates obtained by the image processing algorithm for the camera-captured image. Therefore, the camera's intrinsic parameters, extrinsic parameters and distortion coefficients can be estimated by minimizing the reprojection error. The reprojection error can be described as:

[0096]

[0097] where p c dij is the pixel coordinate of the corresponding point on the image captured by the i-th camera of feature point j in the world coordinate system. Generally speaking, the corresponding pixel coordinates can be identified by image processing algorithms. θ is the coordinate of feature point j in the world coordinate system projected to the camera pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient. c Including the camera's rotation matrix R, translation vector t, intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2.

[0098] S2. Use the inverse camera method to calibrate the projector.

[0099] The projector can be regarded as the inverse model of the camera, so the above camera calibration method can be used to calibrate the projector. However, since the projector cannot actively capture the feature points on the surface of the pile, how to make the projector see the feature points on the surface of the pile like a camera is the key to using the camera calibration method for projector calibration.

[0100] In the structured light 3D reconstruction system, the system consists of a camera and a projector. The camera can be used as the "eyes" of the projector to capture the feature points on the surface of the pile. The phase is used as a bridge to convert the feature point coordinates captured by the camera to the projector pattern, thereby establishing the corresponding relationship between the feature point coordinates in the world coordinate system and the projector pattern. The principle of establishing the corresponding relationship between the feature point coordinates in the world coordinate system and the projector pattern is phase shift profilometry, as shown in the schematic diagram. Figure 5 The specific process can be divided into four steps: In the first step, the camera captures the surface of the calibration plate, and the coordinates of the feature points on the calibration plate in the camera captured image are found through the image processing algorithm. In the second step, assuming that the resolution of the projector is H p ×W p , the projector projects a set of stripes with the number of n onto the surface of the calibration plate v An N-step vertical sinusoidal grating pattern and a set of n fringes h The N-step horizontal sinusoidal grating pattern is used to establish the correspondence between the horizontal and vertical coordinates on the projector pattern and the camera captured image. The third step is to use the camera to capture the image of the calibration plate surface containing the sinusoidal grating pattern, and solve the wrapped phase of each pixel on the camera captured image by phase shift profilometry. Correspondingly, the phase unwrapping algorithm can be used to unwrap the wrapped phase to obtain the unwrapped phase of each pixel. The unwrapped phases obtained using vertical stripes and horizontal stripes are denoted as φ v (x c ,y c ) and φ h (x c ,y c ). Step 4: The coordinates of the point on the image captured by the camera are (x c ,y c ) T , the coordinates of the point on the projector pattern are (x p ,y p ) T , then the correspondence between the pixel points on the camera captured image and the pixel points on the projector pattern can be solved by formula (12).

[0101]

[0102] After establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern, the camera calibration method can be used to calibrate the projector. The intrinsic parameters, extrinsic parameters and distortion coefficients of the projector are estimated by minimizing the reprojection error. The reprojection error can be described as:

[0103]

[0104] where p p dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th projector pattern. θ is the coordinate of the stockpile point j in the world coordinate system projected to the projector pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient. p Including the projector's rotation matrix R p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1,k p 2,k p 3 and the tangential distortion coefficient p p 1,p p 2.

[0105] The projector uses the camera as its "eyes" to see the characteristic points on the surface of the pile. It only needs to decode the pre-encoded sinusoidal grating stripes, and does not require a calibrated camera. Therefore, by calibrating the projector using this method, the camera calibration error can be effectively avoided from being transferred to the projector calibration, thus improving the accuracy of the projector calibration.

[0106] S3. Perform binding adjustment using the camera calibration, projector calibration results and a specific point in the initial world coordinate system as initial values.

[0107] If the same set of images is used in the process of calibrating the camera and projector, the feature points used for the camera and projector calibration have the same coordinates in the world coordinate system. Therefore, the calibration results of the camera and projector can be used as the initial values, and the calibration parameters of the camera and projector can be optimized by the binding adjustment method, thereby further improving the calibration accuracy of the system. Considering that the calibration instrument used is not an absolute plane, the coordinates of the feature points in the world coordinate system are adjusted by the binding method together with the intrinsic parameters, extrinsic parameters and distortion coefficients of the camera and projector to iteratively optimize, so that the value of the error function shown in formula (14) is minimized.

[0108]

[0109] in, represents all the feature points on m calibration plate images of different postures. The error function shown in formula (14) introduces a new optimization parameter This means that the coordinates of the feature points in the world coordinate system must also be optimized. The bundling technique optimizes the initial values ​​of the optimization parameters by minimizing the error function shown in equation (14). This is a large-scale nonlinear least squares problem that can be solved using the trust region reflective algorithm.

[0110] S4. Generate a sparse Jacobian matrix and set the iteration stop condition.

[0111] Since the coordinates of the feature points in the world coordinate system are optimized at the same time, the computational cost of the entire optimization process is very large and the computation time is too long. In order to accelerate the optimization process and reduce the computational cost, a sparse Jacobian matrix J is designed in the embodiment of the present invention to accelerate the computational process of numerical finite derivatives. In the MATLAB environment, the calculation of the sparse Jacobian matrix J is automatically completed by MATLAB, and only the sparse Jacobian matrix J needs to be specified.

[0112] To generate a sparse Jacobian matrix J, we need to consider how to calculate the partial derivative of each item in the error function with respect to each parameter to be optimized. In the sparse Jacobian matrix, the value of the partial derivative of each item in the error function with respect to each parameter to be optimized that needs to be calculated is set to 1, and the value of the partial derivative that does not need to be calculated is set to 0, which reduces unnecessary system calculation overhead and improves the calculation speed. Figure 6 As shown, it shows the generation method of the sparse Jacobian matrix J, where yellow represents 1 and blue represents 0. Each column in the Jacobian matrix represents the parameters that need to be optimized, and each row represents each item in the error function. The reprojection error of the camera's horizontal coordinate is related to the camera's intrinsic parameters, distortion coefficients, rotation matrix, feature points in the world coordinate system, and the first and third items in the translation vector, which are set to 1. The reprojection error of the camera's vertical coordinate is related to the camera's intrinsic parameters, distortion coefficients, rotation matrix, feature points in the world coordinate system, and the second and third items in the translation vector, which are set to 1. Similarly, the sparse Jacobian matrix can be set according to the correlation between the reprojection error of the projector's horizontal and vertical coordinates and the parameters to be optimized. For the offset error of the feature point in the world coordinate system, it is only related to itself, so the diagonal matrix can be set to 1.

[0113] S5. Calculate the scale factor;

[0114] λ is a scale factor, which is used to prevent the coordinates of the optimized feature points from shifting too much in the world coordinate system.

[0115] According to experience, the value of the scale factor λ is set as formula (15):

[0116]

[0117] S6, calculating the error function;

[0118] S7, updating the calibration results of feature points, cameras and projectors in the world coordinate system;

[0119] S8, judging whether the iteration termination condition is met, if so, outputting the calibration result; if not, returning to S5 to continue iteration;

[0120] S9. Based on the calibration results, fringe projection profilometry is used to perform three-dimensional reconstruction measurement of the bulk cargo pile.

[0121] In the specific implementation, a projection device is used to project a stripe pattern onto the surface of the pile, and a camera is used to capture the stripe pattern after the pile shape is modulated. This technology can analyze the corresponding position relationship between the camera captured image and the projected pattern at the position on the measured pile surface. Combined with the calibrated internal and external parameters of the camera and projection device, the depth information of the pile surface can be calculated using the triangulation principle, thereby achieving accurate measurement of its three-dimensional shape.

[0122] In order to verify the effectiveness of the above system calibration method, the following Figure 7 The 14 calibration plate postures shown are used for calibration. Figure 7 Only 12 postures of the calibration plate used are listed in the figure. The corner point recognition algorithm of Opencv can determine the position of the center of the circular hole. Then the camera is calibrated using the Zhang Zhengyou calibration method. The reprojection error of the camera is shown in Table 1. The visualization results are shown in Fig.10 As shown in the right side of the figure, the average reprojection error is 0.04 pixels.

[0123] Table 1

[0124]

[0125]

[0126] After the camera calibration, the inverse camera method is used to calibrate the projector of the system. Vertical and horizontal sinusoidal gratings with 12 phase shifts are projected onto the surface of the calibration plate to determine the phase information of the feature points on the horizontal and vertical coordinates, respectively, so as to establish the correspondence between the pixel points of the camera image and the pixel points on the projector pattern. Figure 8 Shown is one step of a projected 12-step phase-shifted vertical sinusoidal grating. Fig. 9 is one step in the projected 12-step phase-shifted horizontal sinusoidal grating.

[0127] The reprojection error of the projector is shown in Table 1, and the visualization results are shown in Fig.10As shown on the left side of the figure, the average reprojection error is 0.09 pixels. We then use the calibration results of the camera and projector and the feature points in the world coordinate system as the initial values ​​for bundling optimization. The optimized reprojection error is shown in Table 2. The visualization results of the reprojection error are shown in Fig.11 As shown in Figure 2, the average reprojection errors of the optimized camera and projector are 0.019857 pixels and 0.024628 pixels respectively. Compared with the unoptimized state, the reprojection error of the camera is reduced by 50.36%, and the reprojection error of the projector is reduced by 72.64%.

[0128] Table 2

[0129]

[0130] In the above embodiment, a system calibration method considering the unevenness of the calibration tool is designed for the system calibration problem in phase shift profilometry. The results of camera calibration, projector calibration and the coordinates of three-dimensional feature points in the world coordinate system are adjusted by binding technique at the same time, which can deal with the influence of the uneven surface of the calibration tool on the system calibration, improve the calibration accuracy of the system, reduce the requirements for the use of calibration tools, and bring new ideas for reducing system costs. Compared with the reprojection error of the camera and projector when the method proposed by the present invention is not used for optimization, the reprojection error of the camera is reduced by 50.36%, and the reprojection error of the projector is reduced by 72.64%.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for calibrating a three-dimensional reconstruction system for bulk cargo stockpile measurement, characterized in that: include: S1. Use Zhang Zhengyou calibration to calibrate the camera; S2, projector calibration using the inverse camera method; S3, using the camera calibration, projector calibration results and the specific point in the initial world coordinate system as initial values ​​to perform bundling adjustment; the bundling adjustment optimizes the initial values ​​of the parameters to be optimized by minimizing the error function; the error function is: in, is the reprojection error of camera calibration; p c dij The pixel coordinates of the corresponding point on the image captured by the i-th camera are the feature point j in the world coordinate system; is the coordinate of feature point j in the world coordinate system projected to the camera pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ c Including the camera's rotation matrix R, translation vector t, intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2; is the reprojection error calibrated by the projector; p p dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th projector pattern; is the coordinate of the stockpile point j in the world coordinate system projected to the projector pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ p Including the projector's rotation matrix R p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1,k p 2,k p 3 and the tangential distortion coefficient p p 1,p p 2; represents all feature points on m calibration plate images of different postures; λ is the scale factor; S4, the coordinates of the feature points in the world coordinate system are adjusted by bundling technique together with the intrinsic parameters, extrinsic parameters and distortion coefficients of the camera and projector to iteratively optimize, and the calibration result is output after the iteration is terminated; S5. Based on the calibration results, fringe projection profilometry is used to perform three-dimensional reconstruction measurement of the bulk cargo pile.

2. A calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 1, characterized in that: The camera calibration, projector calibration results and specific points in the initial world coordinate system are used as initial values ​​for binding adjustment, including: Generate sparse Jacobian matrix and set iteration stop condition; Calculate the scale factor λ: Calculate the error function; Update the feature points, camera and projector calibration results in the world coordinate system; Determine whether the iteration termination condition is met. If so, output the calibration result; if not, return to the step of calculating the scale factor and continue the iteration.

3. A calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 2, characterized in that: Generate a sparse Jacobian matrix, including: Each column in the Jacobian matrix represents the parameter to be optimized, and each row represents each term in the error function; The reprojection error of the camera's horizontal coordinate is related to the camera's intrinsic parameters, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the first and third items in the translation vector, which are set to 1; The reprojection error of the camera's ordinate is related to the camera's intrinsic parameters, distortion coefficient, rotation matrix, feature points in the world coordinate system, and the second and third terms in the translation vector, which are set to 1; A sparse Jacobian matrix is ​​set according to the correlation between the reprojection errors of the horizontal and vertical coordinates of the projector and the parameters to be optimized; For the offset error of the feature point in the world coordinate system, it is only related to itself, and the diagonal matrix is ​​set to 1; Set the remaining items to 0.

4. A calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 1, characterized in that: The error function is solved by the trust region reflective algorithm.

5. The method for calibrating a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 1, characterized in that: Use Zhang Zhengyou calibration to calibrate the camera, including: Given m calibration plate images of different poses, each image contains n feature points. According to the camera model, knowing the camera's intrinsic parameters, extrinsic parameters and distortion coefficients, reproject the feature points on the calibration plate in the world coordinate system into a point in the camera's pixel coordinate system. The camera's intrinsic parameters, extrinsic parameters, and distortion coefficients are estimated by minimizing the reprojection error. The reprojection error is: where p c dij The pixel coordinates of the corresponding point on the image captured by the i-th camera are the feature point j in the world coordinate system; is the coordinate of feature point j in the world coordinate system projected to the camera pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ c Including the camera's rotation matrix R, translation vector t, intrinsic parameter matrix K, radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2.

6. A calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 1, characterized in that: Projector calibration using the inverse camera method, including: The camera captures the surface of the calibration plate, and uses the image processing algorithm to find the coordinates of the feature points on the calibration plate in the image captured by the camera; Assume the resolution of the projector is H p ×W p , the projector projects a set of stripes with the number of n onto the surface of the calibration plate v An N-step vertical sinusoidal grating pattern and a set of n fringes h The N-step horizontal sinusoidal grating pattern is used to establish the corresponding relationship between the horizontal and vertical coordinates on the projector pattern and the image captured by the camera; Use a camera to capture an image of a sinusoidal grating pattern on the surface of the calibration plate. Use phase shift profilometry to solve the wrapped phase of each pixel on the camera captured image. Correspondingly, the phase unwrapping algorithm can be used to unwrap the wrapped phase to obtain the unwrapped phase of each pixel. The unwrapped phases obtained using vertical stripes and horizontal stripes are denoted as φ v (x c ,y c ) and φ h (x c ,y c ); The coordinates of the point on the image captured by the camera are (x c ,y c ) T , the coordinates of the point on the projector pattern are (x p ,y p ) T , solve the correspondence between the pixels on the camera image and the pixels on the projector pattern: After establishing the correspondence between the feature point coordinates in the world coordinate system and the projector pattern, the projector is calibrated using the camera calibration method; The intrinsic parameters, extrinsic parameters and distortion coefficients of the projector are estimated by minimizing the reprojection error. The reprojection error is: where p p dij is the pixel coordinate of the corresponding point of feature point j in the world coordinate system on the i-th projector pattern; is the coordinate of the stockpile point j in the world coordinate system projected to the projector pixel coordinate system through the estimated intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient; θ p Including the projector's rotation matrix R p , translation vector t p , internal parameter matrix K p , radial distortion coefficient k p 1,k p 2,k p 3 and the tangential distortion coefficient p p 1,p p 2.

7. A calibration method for a three-dimensional reconstruction system for bulk cargo stockpile measurement according to claim 1, characterized in that: Based on the calibration results, fringe projection profilometry is used to perform three-dimensional reconstruction measurement of the bulk material pile, including: The stripe pattern is projected onto the surface of the pile using a projection device, and the stripe pattern modulated by the shape of the pile is captured using a camera; the corresponding position relationship between the position on the surface of the measured pile and the camera captured image and the projected pattern is analyzed; The depth information of the surface of the pile is calculated by combining the intrinsic and extrinsic parameters of the camera and the projection device calibrated in the calibration result and utilizing the triangulation principle, thereby realizing accurate measurement of the three-dimensional shape thereof.

Images