Camera calibration method, handling robot and storage medium

Abstract

This application discloses a camera calibration method, a handling robot, and a storage medium. The method includes: acquiring point cloud data collected by a depth camera; performing plane fitting on a calibration device based on the point cloud data to obtain multiple plane fitting results; calculating a first angle parameter of the depth camera in a reference coordinate system based on a portion of the multiple plane fitting results; calculating a second angle parameter of the depth camera in the reference coordinate system based on all the multiple plane fitting results; calculating the angle difference between the first angle parameter and the second angle parameter; if the angle difference meets a preset condition, then calculating the rotation and translation parameters of the depth camera in the reference coordinate system based on the multiple plane fitting results. This application can verify the tooling accuracy of the calibration device based on the angle difference. After determining that the tooling accuracy of the calibration device meets the conditions, subsequent calibration is performed, improving the accuracy of camera calibration, and the calculation process is simple.

Description

Technical Field

[0001] This application relates to the field of parameter calibration technology, and in particular to a camera calibration method, a handling robot, and a storage medium. Background Technology

[0002] Currently, automated handling robots use sensors such as depth cameras and vision cameras, as well as a series of recognition algorithms, to handle goods. In order to achieve navigation and positioning, it is necessary to calibrate the external parameters of each sensor. Conventional camera external parameter calibration is performed using high-precision calibration targets.

[0003] However, the method of calibrating external parameters based on calibration targets requires high precision in the tooling of the calibration targets. In practical applications, it is difficult to control the installation error of the calibration targets, and the installation error will affect the calibration accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one camera calibration method, a handling robot, and a storage medium.

[0005] The first aspect of this application provides a camera calibration method. The camera to be calibrated includes a depth camera, which is used to acquire three-dimensional spatial information. A calibration device is set relative to a preset reference coordinate system. The calibration device includes at least two different planes. The method includes: acquiring point cloud data acquired by the depth camera; performing plane fitting on the calibration device based on the point cloud data to obtain multiple plane fitting results; calculating the angle of the depth camera in the reference coordinate system based on a portion of the multiple plane fitting results to obtain a first angle parameter; calculating the angle of the depth camera in the reference coordinate system based on all the multiple plane fitting results to obtain a second angle parameter; calculating the angle difference between the first angle parameter and the second angle parameter; if the angle difference meets a preset condition, calculating the rotation and translation parameters of the depth camera in the reference coordinate system based on the multiple plane fitting results.

[0006] In one embodiment, the calibration device includes a horizontal plane and a vertical plane perpendicular to the horizontal plane; the calibration device is fitted with planes based on point cloud data to obtain multiple plane fitting results, including: filtering invalid point clouds from the point cloud data to obtain a target point cloud corresponding to the calibration device; and using a preset plane fitting algorithm to fit the target point cloud with planes to obtain a horizontal plane fitting result and a vertical plane fitting result.

[0007] In one embodiment, based on the horizontal plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain the first angle parameter, including: obtaining the horizontal plane normal vector in the depth camera coordinate system based on the horizontal plane fitting results; obtaining the horizontal plane normal vector in the reference coordinate system; constructing a horizontal normal vector matrix by combining the horizontal plane normal vector in the depth camera coordinate system and the horizontal plane normal vector in the reference coordinate system; and performing singular value decomposition on the horizontal normal vector matrix to calculate the roll angle and pitch angle of the depth camera in the reference coordinate system to obtain the first angle parameter.

[0008] In one embodiment, based on the horizontal plane fitting results and the vertical plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain the second angle parameter, including: obtaining the vertical plane normal vector in the depth camera coordinate system based on the vertical plane fitting results; obtaining the vertical plane normal vector in the reference coordinate system; constructing a comprehensive normal vector matrix by combining the horizontal plane normal vector and the vertical plane normal vector in the depth camera coordinate system and the horizontal plane normal vector and the vertical plane normal vector in the reference coordinate system; and performing singular value decomposition on the comprehensive normal vector matrix to calculate the roll angle and pitch angle of the depth camera in the reference coordinate system to obtain the second angle parameter.

[0009] In one embodiment, both the first angle parameter and the second angle parameter include a roll angle and a pitch angle; calculating the angle difference between the first angle parameter and the second angle parameter includes: calculating the difference between the roll angle in the first angle parameter and the roll angle in the second angle parameter to obtain a roll angle difference; calculating the difference between the pitch angle in the first angle parameter and the pitch angle in the second angle parameter to obtain a pitch angle difference; using the roll angle difference and the pitch angle difference as the angle difference; the method further includes: if both the roll angle difference and the pitch angle difference are less than a preset difference threshold, then it is determined that the angle difference meets a preset condition.

[0010] In one embodiment, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on multiple plane fitting results, including: obtaining the vertical plane normal vector in the depth camera coordinate system based on the vertical plane fitting results; and calculating the roll and pitch angles of the depth camera in the reference coordinate system based on the horizontal plane fitting results; calculating the yaw angle of the depth camera in the reference coordinate system based on the roll, pitch, and vertical plane normal vectors in the depth camera coordinate system; calculating the Z-axis offset of the depth camera in the reference coordinate system based on the roll, pitch, and yaw angles and the horizontal plane fitting results; calculating the X-axis and Y-axis offsets of the depth camera in the reference coordinate system based on the roll, pitch, and yaw angles and the vertical plane fitting results; and using the roll, pitch, and yaw angles as rotation parameters, and the Z-axis, X-axis, and Y-axis offsets as translation parameters.

[0011] In one embodiment, the method further includes: if the angle difference does not meet the preset conditions, generating an instruction or prompt to adjust the plane angle of the calibration device.

[0012] In one embodiment, the camera to be calibrated further includes a vision camera, and a vision marker is deployed on the calibration device; the method further includes: if the angle difference meets a preset condition, acquiring image data collected by the vision camera from the vision marker; calculating the pose of the vision camera in the vision marker coordinate system based on the image data to obtain a first pose; acquiring a preset pose transformation relationship between the vision marker coordinate system and the reference coordinate system; transforming the first pose to the reference coordinate system based on the pose transformation relationship to obtain a second pose; and obtaining the rotation and translation parameters of the vision camera in the reference coordinate system based on the second pose.

[0013] A second aspect of this application provides a camera calibration device, comprising: a plane fitting module for acquiring point cloud data collected by a depth camera, performing plane fitting on the calibration device based on the point cloud data to obtain multiple plane fitting results; a first angle calculation module for calculating the angle of the depth camera in a reference coordinate system based on a portion of the multiple plane fitting results to obtain a first angle parameter; a second angle calculation module for calculating the angle of the depth camera in a reference coordinate system based on all the multiple plane fitting results to obtain a second angle parameter; and a verification module for calculating the angle difference between the first angle parameter and the second angle parameter, and if the angle difference meets a preset condition, calculating the rotation and translation parameters of the depth camera in the reference coordinate system based on the multiple plane fitting results.

[0014] The third aspect of this application provides a transport robot, which includes a transport component, a camera to be calibrated, a memory, and a processor. A reference coordinate system is constructed relative to the transport component, and the processor is used to execute program instructions stored in the memory to implement the above-mentioned camera calibration method and obtain the rotation and translation parameters of the camera to be calibrated in the reference coordinate system.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described camera calibration method.

[0016] The above scheme acquires point cloud data from a depth camera, performs plane fitting on the calibration device based on the point cloud data, and obtains multiple plane fitting results. Based on a portion of the multiple plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain a first angle parameter. Based on all the multiple plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain a second angle parameter. The angle difference between the first angle parameter and the second angle parameter is calculated. If the angle difference meets a preset condition, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on the multiple plane fitting results. The tooling accuracy of the calibration device can be verified based on the angle difference. After confirming that the tooling accuracy of the calibration device meets the conditions, subsequent calibration can be performed, improving the accuracy of camera calibration. The calculation process is also simple.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a flowchart illustrating a camera calibration method in an exemplary embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a camera calibration scene shown in an exemplary embodiment of this application;

[0021] Figure 3 This is a schematic diagram illustrating the extraction of planar point clouds in an exemplary embodiment of this application;

[0022] Figure 4 This is a schematic diagram illustrating a calibration device with tooling errors, as shown in an exemplary embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating the coordinate systems in an exemplary embodiment of this application;

[0024] Figure 6 This is a block diagram illustrating a camera calibration device in an exemplary embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of a handling robot shown in an exemplary embodiment of this application;

[0026] Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] The camera calibration method provided in the embodiments of this application will be described below.

[0031] Please see Figure 1 , Figure 1 This is a flowchart illustrating a camera calibration method in an exemplary embodiment of this application. The camera calibration method can be specifically executed by the camera to be calibrated, a device on which the camera to be calibrated is deployed, or a server communicatively connected to the camera to be calibrated.

[0032] like Figure 1 As shown, the camera calibration method includes at least steps S110 to S140, which are described in detail below:

[0033] Step S110: Acquire point cloud data collected by the depth camera, and perform plane fitting on the calibration device based on the point cloud data to obtain multiple plane fitting results.

[0034] The camera to be calibrated in this application includes a depth camera, which is used to acquire three-dimensional spatial information. A calibration device is set relative to a preset reference coordinate system, and the calibration device includes at least two different planes.

[0035] For example, the calibration device includes a horizontal plane and a vertical plane perpendicular to the horizontal plane. The types of planes contained in the calibration device can be flexibly set according to the actual application scenario, and this application does not limit them.

[0036] The specific type of depth camera can be a time-of-flight (ToF) camera, a binocular camera, a structured light camera, etc., and this application does not limit this.

[0037] The depth camera collects environmental information to obtain point cloud data. Based on the point cloud data, the calibration device is fitted with a plane to obtain the plane fitting result for each plane corresponding to the calibration device. Each plane fitting result represents the spatial information of the corresponding plane perceived by the depth camera.

[0038] The algorithms used for plane fitting include, but are not limited to, Random Sample Consensus (RANSAC), region growing, and Hough transform, etc., and this application does not limit them.

[0039] Step S120: Based on a portion of the plane fitting results from multiple plane fitting results, calculate the angle of the depth camera in the reference coordinate system to obtain the first angle parameter.

[0040] The reference coordinate system is preset according to the operational requirements of subsequent equipment. For example, after the camera calibration is completed, the cargo handling mechanism of the handling robot is positioned based on the calibrated camera, and the reference coordinate system is the coordinate system corresponding to the cargo handling structure.

[0041] After the calibration device is assembled, the pose relationship between the calibration device and the reference coordinate system is known. Therefore, based on some of the plane fitting results from multiple plane fitting results, the spatial relationship between the depth camera and the calibration device is calculated, and then mapped to the reference coordinate system to obtain the angle of the depth camera in the preset reference coordinate system, which is used as the first angle parameter.

[0042] For example, the plane fitting results include horizontal plane fitting results and vertical plane fitting results. The angle of the depth camera in the preset reference coordinate system can be calculated based on the horizontal plane fitting results to obtain the first angle parameter. Of course, the angle of the depth camera in the preset reference coordinate system can also be calculated based on the vertical plane fitting results to obtain the first angle parameter.

[0043] The angles include, but are not limited to, one or more of the following: roll, pitch, and yaw.

[0044] Step S130: Based on all the plane fitting results in the multiple plane fitting results, calculate the angle of the depth camera in the reference coordinate system to obtain the second angle parameter.

[0045] Similarly, by combining all the plane fitting results, the spatial relationship between the depth camera and the calibration device is calculated, and then mapped to the reference coordinate system to obtain the angle of the depth camera in the preset reference coordinate system, which is used as the second angle parameter.

[0046] It should be noted that the angle calculation methods used in steps S120 and S130 can be the same or different. There is no order of execution between steps S120 and S130. That is, steps S120 can be executed first and then steps S130, or steps S130 can be executed first and then steps S120, or steps S120 and S130 can be executed simultaneously.

[0047] Step S140: Calculate the angle difference between the first angle parameter and the second angle parameter. If the angle difference meets the preset conditions, calculate the rotation and translation parameters of the depth camera in the reference coordinate system based on multiple plane fitting results.

[0048] It should be noted that, for ease of calculation, the first angle parameter and the second angle parameter contain angle values ​​of the same type. The angle difference is obtained by subtracting the angle values ​​of the same type between the first angle parameter and the second angle parameter.

[0049] For example, both the first angle parameter and the second angle parameter contain roll angle and pitch angle. The difference between the roll angle of the first angle parameter and the roll angle of the second angle parameter, and the difference between the pitch angle of the first angle parameter and the pitch angle of the second angle parameter, are used to obtain the angle difference value.

[0050] Of course, the first angle parameter and the second angle parameter can also contain other angle value types. For example, the first angle parameter and the second angle parameter can both contain only the roll angle, or the first angle parameter and the second angle parameter can both contain only the pitch angle, or the first angle parameter and the second angle parameter can both contain the roll angle, pitch angle and yaw angle at the same time. This application does not limit this.

[0051] Then, check whether the angle difference meets the preset conditions.

[0052] For example, it can detect whether the angle difference corresponding to each type of angle value is less than a preset difference threshold. If they are all less than the preset difference threshold, it is determined that the angle difference meets the preset condition.

[0053] For example, calculate the average value of the angle difference corresponding to each type of angle value to obtain the average difference value, and check whether the average difference value is less than the preset difference value threshold. If it is less than the preset difference value threshold, it is determined that the angle difference value meets the preset condition.

[0054] For example, the sum of the angle differences corresponding to each type of angle value is calculated to obtain the sum of differences. The sum of differences is then checked to see if it is less than a preset difference threshold. If it is less than the preset difference threshold, the angle difference is determined to meet the preset condition.

[0055] If the angle difference meets the preset conditions, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on multiple plane fitting results, and the calculated rotation and translation parameters are used as the calibration results of the depth camera.

[0056] Based on the above steps, the pose relationship between the depth camera and the calibration device can be calculated from all or part of the multiple plane fitting results. Since the pose relationship between the calibration device coordinate system and the reference coordinate system is known, the rotation and translation parameters of the depth camera in the reference coordinate system can be obtained, thus obtaining the calibration result of the depth camera.

[0057] In some implementations, if the angle difference does not meet the preset conditions, an instruction or prompt is generated to adjust the plane angle of the calibration device.

[0058] If the angle difference does not meet the preset conditions, it indicates that there is a tooling error in the calibration device. At this time, the calculation of subsequent calibration parameters will not continue. Instead, an instruction or prompt will be generated to adjust the plane angle of the calibration device in order to adjust the plane of the calibration device and reduce the tooling error.

[0059] For example, in the scenario of calibrating a camera on a transport robot, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of a camera calibration scene illustrated in an exemplary embodiment of this application, such as... Figure 2 As shown, the transport robot includes a transport component and cameras to be calibrated (depth camera and vision camera). A reference coordinate system is established relative to the transport component, such as... Figure 2 The reference coordinate system is constructed with the center point of the transported component as its origin. A calibration device is provided relative to the reference coordinate system, such as... Figure 2 The section within the bold black box represents the transport components; the red section represents the calibration device frame; and the green section represents the observation area of ​​the camera to be calibrated during the calibration process, which contains at least two distinct planes. Figure 2 The calibration device includes a horizontal plane and a vertical plane perpendicular to the horizontal plane. Of course, other methods can also be used to implement the calibration device, and this application does not limit this. Then, the rotation and translation parameters of the camera to be calibrated relative to the reference coordinate system can be calculated using the camera calibration method, and the calibration result of the camera to be calibrated can be obtained.

[0060] The camera calibration method provided in this application acquires point cloud data collected by a depth camera, performs plane fitting on the calibration device based on the point cloud data, and obtains multiple plane fitting results. Based on a portion of the multiple plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain a first angle parameter. Based on all the multiple plane fitting results, the angle of the depth camera in the reference coordinate system is calculated to obtain a second angle parameter. The angle difference between the first angle parameter and the second angle parameter is calculated. If the angle difference meets a preset condition, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on the multiple plane fitting results. The tooling accuracy of the calibration device can be verified based on the angle difference. After confirming that the tooling accuracy of the calibration device meets the conditions, subsequent calibration can be performed, thereby improving the accuracy of camera calibration. The calculation process is also simple.

[0061] The following describes some embodiments of this application in detail.

[0062] In some embodiments, the calibration device includes a horizontal plane and a vertical plane perpendicular to the horizontal plane; in step S110, the calibration device is fitted with planes based on point cloud data to obtain multiple plane fitting results, including the following steps S111 and S112.

[0063] Step S111: Filter invalid point clouds from the point cloud data to obtain the target point cloud corresponding to the calibration device.

[0064] Filter out invalid point clouds that do not belong to the calibration device to obtain the target point cloud corresponding to the calibration device.

[0065] Step S112: Use a preset plane fitting algorithm to perform plane fitting on the target point cloud to obtain the horizontal plane fitting result and the vertical plane fitting result.

[0066] For example, the RANSAC algorithm and least squares method can be used for plane fitting. Specifically, the RANSAC algorithm is used to obtain an initial plane model through random sampling. All points are substituted into the initial plane model to evaluate the quality of the plane model. The above steps are iterated to determine the optimal plane model. Using all points corresponding to the optimal plane model, a more accurate plane is refitted using the least squares method to obtain the plane fitting result.

[0067] For example, by minimizing the distance from a point to a plane using the least squares method, the residual equation is constructed as shown in Formula 1 below:

[0068]

[0069] Solving Equation 1 yields the horizontal plane fitting equation A. h X+B h Y+C h Z+Dh =0 and the fitting equation A for the vertical plane v X+B v Y+C v Z+D v =0, simultaneously obtaining the point cloud Planar corresponding to the horizontal and vertical planes respectively. h Planar v The horizontal plane fitting equation and the vertical plane fitting equation mentioned above are both expressed in the depth camera coordinate system.

[0070] For example, please see Figure 3 , Figure 3 This is a schematic diagram illustrating the extraction of planar point clouds in an exemplary embodiment of this application, as shown below. Figure 3 As shown, after the depth camera collects point cloud data from the horizontal and vertical planes of the calibration device, the equation expression of each plane is obtained through plane fitting. Based on this, the point cloud corresponding to the horizontal plane and the point cloud corresponding to the vertical plane can be divided.

[0071] The above-mentioned horizontal plane fitting equation and the point cloud corresponding to the horizontal plane are taken as the horizontal plane fitting result; the above-mentioned vertical plane fitting equation and the point cloud corresponding to the vertical plane are taken as the vertical plane fitting result.

[0072] Of course, the plane normal vector can also be used as the plane fitting result. The specific content of the plane fitting result can be determined according to the specific calculation method of the subsequent calibration parameters. This application does not limit this.

[0073] In some embodiments, step S120 calculates the angle of the depth camera in the reference coordinate system based on a portion of the multiple plane fitting results to obtain the first angle parameter, including: calculating the angle of the depth camera in a preset reference coordinate system based on the horizontal plane fitting results to obtain the first angle parameter, specifically including the following steps S121 to S124.

[0074] Step S121: Based on the horizontal plane fitting results, obtain the horizontal plane normal vector in the depth camera coordinate system.

[0075] The depth camera coordinate system refers to a coordinate system constructed with the depth camera as the origin.

[0076] For example, if the horizontal plane fitting equation in the depth camera coordinate system is expressed as A h X+B h Y+C h Z+D h If = 0, then the horizontal plane normal vector in the depth camera coordinate system is expressed as:

[0077] Step S122: Obtain the horizontal plane normal vector in the reference coordinate system.

[0078] For example, the normal vector of the horizontal plane in the reference coordinate system is represented as:

[0079] Step S123: Combine the horizontal plane normal vector in the depth camera coordinate system and the horizontal plane normal vector in the reference coordinate system to construct the horizontal normal vector matrix.

[0080] Based on the horizontal plane normal vector in the depth camera coordinate system and the horizontal plane normal vector in the reference coordinate system, a horizontal normal vector matrix is ​​constructed. The calculation method of the horizontal normal vector matrix is ​​shown in Formula 2 below:

[0081]

[0082] Where H1 is the horizontal normal vector matrix, The horizontal plane normal vector in the reference coordinate system. This is the horizontal plane normal vector in the depth camera coordinate system.

[0083] Step S124: Perform singular value decomposition on the horizontal normal vector matrix to calculate the roll and pitch angles of the depth camera in the reference coordinate system, and obtain the first angle parameters.

[0084] The horizontal normal vector matrix H1 is decomposed by Singular Value Decomposition (SVD), as shown in Formula 3 below:

[0085] H1 = UAV T (Formula 3)

[0086] Where U and V are orthogonal matrices,

[0087] The rotation matrix is ​​then expressed as Equation 4 below:

[0088] R = VU T (Formula 4)

[0089] Based on the transformation from the rotation matrix to Euler angles, the roll and pitch angles of the depth camera are obtained, denoted as α. T and β T It is important to distinguish that the superscript T in the relevant formulas of this application indicates the transpose operation, and the subscript T of the calculated angle or translation parameters indicates that they belong to the extrinsic parameters corresponding to the depth camera.

[0090] In some implementations, step S130 calculates the angle of the depth camera in the reference coordinate system based on all the plane fitting results among multiple plane fitting results to obtain the second angle parameter. This includes: calculating the angle of the depth camera in the reference coordinate system based on the horizontal plane fitting results and the vertical plane fitting results to obtain the second angle parameter, specifically including the following steps S131 to S134.

[0091] Step S131: Based on the vertical plane fitting results, obtain the vertical plane normal vector in the depth camera coordinate system.

[0092] For example, if the vertical plane fitting equation in the depth camera coordinate system is expressed as A v X+B v Y+C v Z+D v If = 0, then the vertical plane normal vector in the depth camera coordinate system is expressed as:

[0093] Step S132: Obtain the vertical plane normal vector in the reference coordinate system.

[0094] For example, the vertical plane normal vector in the reference coordinate system is represented as:

[0095] Step S133: Combine the horizontal and vertical plane normal vectors in the depth camera coordinate system and the horizontal and vertical plane normal vectors in the reference coordinate system to construct a comprehensive normal vector matrix.

[0096] Similar to step S123, a composite normal vector matrix is ​​constructed based on the horizontal and vertical plane normal vectors in the depth camera coordinate system and the horizontal and vertical plane normal vectors in the reference coordinate system. The calculation method of the composite normal vector matrix is ​​shown in Formula 5 below:

[0097]

[0098] Where H2 is the composite normal vector matrix, The vertical plane normal vector in the reference coordinate system. This is the vertical plane normal vector in the depth camera coordinate system.

[0099] Step S134: Perform singular value decomposition on the integrated normal vector matrix to calculate the roll and pitch angles of the depth camera in the reference coordinate system, and obtain the second angle parameters.

[0100] Similar to step S124, the roll and pitch angles of the depth camera are calculated and denoted as α′. T and β T ′.

[0101] Then, the angle difference between the first angle parameter and the second angle parameter is calculated.

[0102] In some embodiments, both the first angle parameter and the second angle parameter include roll angle and pitch angle; step S140 calculates the angle difference between the first angle parameter and the second angle parameter, including the following steps S1411 to S1413.

[0103] Step S1411: Calculate the difference between the roll angle in the first angle parameter and the roll angle in the second angle parameter to obtain the roll angle difference.

[0104] Specifically, for α T and α′ T Subtracting the two values ​​gives the roll angle difference.

[0105] Step S1412: Calculate the difference between the pitch angle in the first angle parameter and the pitch angle in the second angle parameter to obtain the pitch angle difference.

[0106] Specifically, for β T and β T Subtract the values ​​to obtain the pitch angle difference.

[0107] Step S1413: Use the roll angle difference and pitch angle difference as the angle difference.

[0108] Of course, it is also possible to calculate only one of the roll angle difference and pitch angle difference, or to calculate other angle differences, such as the yaw angle difference. This application does not limit this.

[0109] The system checks whether the angle difference meets preset conditions. For example, if both the roll angle difference and pitch angle difference are less than a preset difference threshold, then the angle difference is determined to meet the preset conditions.

[0110] If the angle difference does not meet the preset conditions, for example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a calibration apparatus with tooling errors, as shown in an exemplary embodiment of this application. Figure 4 As shown, ideally the horizontal plane is perpendicular to the vertical plane. However, due to installation errors, the horizontal plane is not perpendicular to the vertical plane. Therefore, it is necessary to generate corresponding adjustment instructions or prompts to adjust the horizontal plane to be perpendicular to the vertical plane.

[0111] If the angle difference meets the preset conditions, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on the fitting results of multiple planes.

[0112] The rotational parameters to be calculated for the depth camera include roll angle, pitch angle, and yaw angle, while the translational parameters to be calculated include X-axis offset, Y-axis offset, and Z-axis offset.

[0113] In some implementations, step S140 calculates the rotation and translation parameters of the depth camera in the reference coordinate system based on multiple plane fitting results, including steps S1421 to S1425 below.

[0114] Step S1421: Based on the vertical plane fitting results, obtain the vertical plane normal vector in the depth camera coordinate system, and based on the horizontal plane fitting results, calculate the roll angle and pitch angle of the depth camera in the reference coordinate system.

[0115] The calculation process for roll angle and pitch angle can be found in steps S121 to S124 or steps S131 to S134 mentioned above, and will not be repeated here.

[0116] Step S1422: Based on the roll angle, pitch angle, and vertical plane normal vector in the depth camera coordinate system, calculate the yaw angle of the depth camera in the reference coordinate system.

[0117] Specifically, if the roll angle α is calculated... T Pitch angle β T Then according to α T β T A new rotation matrix R′ is constructed, in which the yaw angle is set to 0.

[0118] Based on the new rotation matrix R′, the vertical plane normal vector in the depth camera coordinate system is... Projected onto the reference coordinate system, the projection result This is expressed as Formula 6 below:

[0119]

[0120] Then, the yaw angle can be obtained by using the following formula 7:

[0121]

[0122] Where, γ T The yaw angle of the depth camera in the reference coordinate system. and These are the values ​​corresponding to the X and Y axes of the reference coordinate system, respectively.

[0123] Step S1423: Based on the roll angle, pitch angle, yaw angle, and horizontal plane fitting results, calculate the Z-axis offset of the depth camera in the reference coordinate system.

[0124] Specifically, a complete rotation matrix is ​​constructed based on the roll angle, pitch angle, and yaw angle. The average coordinates of each point in the point cloud corresponding to the horizontal plane are calculated. The average coordinates corresponding to the horizontal plane are mapped to the reference coordinate system using the complete rotation matrix to obtain the Z-axis offset of the depth camera in the reference coordinate system.

[0125] For example, using the above α T β T and γ T Once the complete rotation matrix R″ is obtained, the Z-axis offset can be calculated using the following formula 8:

[0126]

[0127] Among them, z T To calculate the Z-axis offset of the depth camera in the reference coordinate system, Planar h S represents the point cloud corresponding to the horizontal plane. h The number of points in the horizontal plane. Here are the coordinates of the point, and the subscript z indicates the calculation of the Z-axis offset.

[0128] Step S1424: Based on the roll angle, pitch angle, yaw angle, and vertical plane fitting results, calculate the X-axis offset and Y-axis offset of the depth camera in the reference coordinate system.

[0129] Specifically, a complete rotation matrix R″ is constructed based on the roll angle, pitch angle, and yaw angle. The average coordinates of each point in the point cloud corresponding to the vertical plane are calculated. The average coordinates corresponding to the vertical plane are mapped to the reference coordinate system using the complete rotation matrix to obtain the X-axis offset of the depth camera in the reference coordinate system.

[0130] For example, the Z-axis offset can be obtained using the following formula 9:

[0131]

[0132] Where, x T To calculate the X-axis offset of the depth camera in the reference coordinate system, Planar v S represents the point cloud corresponding to the vertical plane. v The number of midpoints in the vertical plane, with the subscript x indicating the calculation of the X-axis offset, d Tx This is the distance between the origin of the reference coordinate system and the vertical plane.

[0133] In addition, the Y-axis offset is solved using the point cloud corresponding to the vertical plane.

[0134] Similarly, by projecting the coordinates of the midpoints of the vertical plane point cloud onto the complete rotation matrix R″, we can obtain the projected point set Planar.v ′={p v ′|p v ′=R″*p v ,p v ∈Planar v}, p v Let p′ be the coordinates of the midpoint of the vertical plane point cloud. v These are the coordinates of the projected point.

[0135] First calculate the right edge of the vertical plane, then planar v The point cloud data is sorted in ascending order of its Y-axis values ​​to obtain the first sorted result, Planar. v "| R ={p0,p1,...,p m The right edge point is searched from the first sorted result using a sliding window method.

[0136] For example, the sliding window width is 0.5cm, and i is initialized to 0, where i is the index of each point in the first sorted result, indicating that the traversal starts from the point with the smallest Y-axis value and iterates through the points in the first sorted result. The traversal rule is as follows:

[0137]

[0138] Based on the above traversal rules, after the right edge point search is completed, the right edge point set N obtained from the search is... i | R Calculate the equation of the right edge line of the vertical plane. The calculation formula is shown in Formula 10 below:

[0139]

[0140] Among them, (p) y This represents the component of the point along the Y-axis.

[0141] Similarly, calculate the left edge of the vertical plane, and then... v The point cloud data is sorted in descending order of its Y-axis values ​​to obtain the second sorting result, Planar. v "| L ={p0,p1,...,p m The left edge point is searched from the second sorting results using a sliding window method.

[0142] For example, if the sliding window width is 0.5cm, and i is initialized to 0, the traversal rule is:

[0143]

[0144]

[0145] Based on the above traversal rules, after the left edge point search is completed, the left edge point set N obtained from the search is... i | L Calculate the equation of the left edge line of the vertical plane. The calculation formula is shown in Formula 11 below:

[0146]

[0147] Among them, (p) y This represents the component of the point along the Y-axis.

[0148] Finally, based on the equations of the right and left edge lines of the vertical plane, the Y-axis offset y is obtained. T For the specific solution method, please refer to Formula 12 below:

[0149] y T =-(y L +y R ) / 2 (Formula 12)

[0150] Step S1425: Use roll angle, pitch angle, and yaw angle as rotation parameters, and use Z-axis offset, X-axis offset, and Y-axis offset as translation parameters.

[0151] The extrinsic parameter calibration results of the depth camera are expressed as: [x T y T z T α T β T γ T ] T .

[0152] In some embodiments, the camera to be calibrated further includes a vision camera, and a vision marker is deployed on the calibration device; the method further includes: if the angle difference meets a preset condition, acquiring image data collected by the vision camera from the vision marker; calculating the pose of the vision camera in the vision marker coordinate system based on the image data to obtain a first pose; acquiring a preset pose transformation relationship between the vision marker coordinate system and the reference coordinate system; transforming the first pose to the reference coordinate system based on the pose transformation relationship to obtain a second pose; and obtaining the rotation and translation parameters of the vision camera in the reference coordinate system based on the second pose.

[0153] For example, please see Figure 5 , Figure 5 This is a schematic diagram illustrating the coordinate systems in an exemplary embodiment of this application, as shown below. Figure 5 As shown, red indicates the visual landmark coordinate system (origin at O). Tag Blue indicates the reference coordinate system (origin at O). B Green indicates the visual camera coordinate system (origin at O). C ).

[0154] See also Figure 2 The visual markers are QR codes arranged according to preset rules. A visual camera captures the image data of the QR codes, and the pose of the visual camera in the coordinate system of the visual markers can be calculated based on this image data, represented as...

[0155] And according to such Figure 2 The tooling information, the pose of the visual marker coordinate system in the reference coordinate system is (The values ​​are pre-defined), and the following formula 13 exists:

[0156]

[0157] Among them, R x R y R z These are rotation matrices obtained by rotating the axes around the X, Y, and Z axes by a certain angle, respectively. Cx equals d Tx d cy The offset of the visual landmark coordinate system from the Y-axis of the reference coordinate system, d Cz This refers to the offset between the visual landmark coordinate system and the Z-axis of the reference coordinate system. See details... Figure 2 As shown.

[0158] The pose of the visual camera in the reference coordinate system is calculated using the following formula 14:

[0159]

[0160] Then, based on the transformation matrix, the rotation and translation parameters of the visual camera can be obtained, expressed as: [x C y C z C α C β C γ C ] T .

[0161] Figure 6 This is a block diagram illustrating a camera calibration device according to an exemplary embodiment of this application. Figure 6 As shown, the exemplary camera calibration device 600 includes:

[0162] The plane fitting module 610 is used to acquire point cloud data collected by the depth camera, and perform plane fitting on the calibration device based on the point cloud data to obtain multiple plane fitting results.

[0163] The first angle calculation module 620 is used to calculate the angle of the depth camera in the reference coordinate system based on a portion of the plane fitting results from multiple plane fitting results, and obtain the first angle parameter.

[0164] The second angle calculation module 630 is used to calculate the angle of the depth camera in the reference coordinate system based on all the plane fitting results in multiple plane fitting results, and obtain the second angle parameter.

[0165] The verification module 640 is used to calculate the angle difference between the first angle parameter and the second angle parameter. If the angle difference meets the preset conditions, the rotation and translation parameters of the depth camera in the reference coordinate system are calculated based on multiple plane fitting results.

[0166] It should be noted that the camera calibration device and the camera calibration method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs its operation have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the camera calibration device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0167] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a transport robot in an exemplary embodiment of this application. The transport robot 700 includes a memory 701, a processor 702, a transport component 703, and a camera 704 to be calibrated. The processor 702 is used to execute program instructions stored in the memory 701 to implement the steps in any of the above-described camera calibration method embodiments.

[0168] Specifically, processor 702 controls itself and memory 701 to implement the steps in any of the camera calibration method embodiments described above. Processor 702 can also be referred to as a Central Processing Unit (CPU). Processor 702 may be an integrated circuit chip with signal processing capabilities. Processor 702 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 702 can be implemented using integrated circuit chips.

[0169] It should be noted that in actual application scenarios, the handling robot 700 has many other components, which are not included in this application. Figure 7 The image shows all the components of the transport robot 700.

[0170] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the camera calibration method embodiments described above.

[0171] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0172] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A camera calibration method, characterized in that, The camera to be calibrated includes a depth camera, which is used to acquire three-dimensional spatial information. A calibration device is set relative to a preset reference coordinate system. The calibration device includes a horizontal plane and a vertical plane perpendicular to the horizontal plane. The method includes: The point cloud data acquired by the depth camera is obtained, and the calibration device is fitted with plane based on the point cloud data to obtain the horizontal plane fitting result and the vertical plane fitting result. Based on the horizontal plane fitting result, the horizontal plane normal vector in the depth camera coordinate system is obtained; the horizontal plane normal vector in the reference coordinate system is obtained; the horizontal plane normal vector in the depth camera coordinate system and the horizontal plane normal vector in the reference coordinate system are combined to construct a horizontal normal vector matrix; the horizontal normal vector matrix is ​​subjected to singular value decomposition to solve the angle of the depth camera in the reference coordinate system, and the first angle parameter is obtained. Based on the vertical plane fitting results, the vertical plane normal vector in the depth camera coordinate system is obtained, and the vertical plane normal vector in the reference coordinate system is also obtained. Combining the horizontal and vertical plane normal vectors in the depth camera coordinate system and the horizontal and vertical plane normal vectors in the reference coordinate system, a comprehensive normal vector matrix is ​​constructed. Singular value decomposition is performed on the comprehensive normal vector matrix to calculate the angle of the depth camera in the reference coordinate system, thus obtaining the second angle parameter. Calculate the angle difference between the first angle parameter and the second angle parameter. If the angle difference meets the preset conditions, then calculate the rotation and translation parameters of the depth camera in the reference coordinate system based on the horizontal plane fitting result and the vertical plane fitting result.

2. The method according to claim 1, characterized in that, The step of performing plane fitting on the calibration device based on the point cloud data to obtain horizontal plane fitting results and vertical plane fitting results includes: The point cloud data is filtered for invalid points to obtain the target point cloud corresponding to the calibration device. A preset plane fitting algorithm is used to perform plane fitting on the target point cloud to obtain horizontal plane fitting results and vertical plane fitting results.

3. The method according to claim 1, characterized in that, The step of performing singular value decomposition on the horizontal normal vector matrix to calculate the angle of the depth camera in the reference coordinate system and obtain the first angle parameter includes: Singular value decomposition is performed on the horizontal normal vector matrix to calculate the roll and pitch angles of the depth camera in the reference coordinate system, thereby obtaining the first angle parameters.

4. The method according to claim 3, characterized in that, The process of performing singular value decomposition on the composite normal vector matrix to calculate the angle of the depth camera in the reference coordinate system and obtain the second angle parameter includes: Singular value decomposition is performed on the integrated normal vector matrix to calculate the roll and pitch angles of the depth camera in the reference coordinate system, thereby obtaining the second angle parameters.

5. The method according to claim 1, characterized in that, Both the first angle parameter and the second angle parameter include roll angle and pitch angle; calculating the angle difference between the first angle parameter and the second angle parameter includes: Calculate the difference between the roll angle in the first angle parameter and the roll angle in the second angle parameter to obtain the roll angle difference; Calculate the difference between the pitch angle in the first angle parameter and the pitch angle in the second angle parameter to obtain the pitch angle difference; The roll angle difference and the pitch angle difference are used as the angle difference; The method further includes: If both the roll angle difference and the pitch angle difference are less than a preset difference threshold, then the angle difference is determined to meet the preset condition.

6. The method according to claim 1, characterized in that, The calculation of the rotation and translation parameters of the depth camera in the reference coordinate system based on the horizontal plane fitting results and the vertical plane fitting results includes: Based on the horizontal plane fitting results, the roll angle and pitch angle of the depth camera in the reference coordinate system are calculated. Based on the roll angle, the pitch angle, and the vertical plane normal vector in the depth camera coordinate system, the yaw angle of the depth camera in the reference coordinate system is calculated. Based on the roll angle, pitch angle, yaw angle, and horizontal plane fitting results, the Z-axis offset of the depth camera in the reference coordinate system is calculated. Based on the roll angle, pitch angle, yaw angle, and vertical plane fitting results, the X-axis offset and Y-axis offset of the depth camera in the reference coordinate system are calculated. The roll angle, pitch angle, and yaw angle are used as rotation parameters, and the Z-axis offset, X-axis offset, and Y-axis offset are used as translation parameters.

7. The method according to claim 1, characterized in that, The method further includes: If the angle difference does not meet the preset conditions, an instruction or prompt is generated to adjust the plane angle of the calibration device.

8. The method according to any one of claims 1 to 7, characterized in that, The camera to be calibrated also includes a vision camera, and the calibration device is equipped with visual markers; the method further includes: If the angle difference meets the preset conditions, then the image data acquired by the visual camera of the visual marker is obtained; Based on the image data, the pose of the visual camera in the visual landmark coordinate system is calculated to obtain the first pose; Obtain the pose transformation relationship between the pre-set visual marker coordinate system and the reference coordinate system; Based on the pose transformation relationship, the first pose is transformed into the reference coordinate system to obtain the second pose; Based on the second pose, the rotation and translation parameters of the visual camera in the reference coordinate system are obtained.

9. A transport robot, characterized in that, The transport robot includes a transport component, a camera to be calibrated, a memory, and a processor. A reference coordinate system is constructed relative to the transport component. The processor is used to execute program instructions stored in the memory to implement the steps in the method as described in any one of claims 1-8, and obtain the rotation parameters and translation parameters of the camera to be calibrated in the reference coordinate system.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images