Ground auxiliary line-based extrinsic parameter calibration method and device

Abstract

The application discloses a kind of external parameter calibration method and device based on ground auxiliary line, comprising: obtaining the image data of the aligned image of ground auxiliary line's RGB image and depth image;The coordinates of each mark point under the camera coordinate system are obtained, and straight line fitting is carried out under the camera coordinate system, to obtain the direction vector of fitting straight line;Ground plane equation is obtained, and according to the direction vector of the fitted straight line and the normal vector of ground plane, the rotation matrix representation of forklift coordinate system in camera coordinate system is calculated;The coordinates of the origin of forklift coordinate system in camera coordinate system are obtained;According to the rotation matrix representation of forklift coordinate system in camera coordinate system and the coordinates of the origin of forklift coordinate system in camera coordinate system, the transformation relationship of forklift coordinate system and camera coordinate system is obtained, and then the external parameter calibration of camera is completed.The application does not need other calibration tools or templates, easy to mass production, and the accuracy and robustness of external parameter calibration result are high.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of RGBD camera calibration, in particular to a ground auxiliary line based extrinsic parameter calibration method and device. BACKGROUND

[0002] The unmanned intelligent forklift is widely used in intelligent warehousing. In the process of transferring goods, the intelligent forklift often needs to fork the pallet (pallet) or trolley loaded with goods. Because the pallet (pallet) or trolley may be offset in position and rotation angle during placement in the warehouse.

[0003] Before the intelligent unmanned forklift forks the goods, the RGBD sensor installed on the intelligent forklift needs to identify the offset of the pallet (pallet) or trolley in the warehouse, and then the unmanned forklift automatically adjusts the corresponding position to fork the pallet (pallet) or trolley. Generally, the three-dimensional point cloud data collected by the RGBD sensor installed on the intelligent forklift is based on the camera coordinate system, so the processing and analysis of these point cloud data are also based on the results of the camera coordinate system. In the existing extrinsic parameter calibration method of the RGBD sensor camera, the extrinsic parameter is calculated in the order of Yaw, Pitch, Roll and Ty by setting white lines outside, but the calculation process is based on the idea of rotation matrix multiplication, which is gradually disassembled into the calculation of the rotation matrix of each degree of freedom. The calculation process is complex, and it also needs to transmit the position information of the common target from the calibrated reference depth camera to the to-be-calibrated camera, so as to realize the calibration of the spatial position extrinsic parameter of the to-be-calibrated camera. However, it needs to use one more calibrated reference depth camera, and the operation complexity is high, and there is an error in the calibration.

[0004] Therefore, there is an urgent need for a method that can simplify the extrinsic parameter calibration process, improve the accuracy and robustness of the extrinsic parameter calibration. SUMMARY

[0005] The present application provides a ground auxiliary line based extrinsic parameter calibration method and device to solve the technical problems of high complexity, low accuracy and low robustness of the extrinsic parameter calibration in the prior art.

[0006] In order to solve the above technical problems, the present application provides a ground auxiliary line based extrinsic parameter calibration method, comprising:

[0007] Obtain the image data of the aligned RGB image and depth image of the ground auxiliary line; wherein the ground auxiliary line coincides with the motion direction of the forklift, and a preset number of identification points are arranged on the ground auxiliary line;

[0008] Obtain the coordinates of each identification point in the camera coordinate system, fit a straight line to the identification points in the camera coordinate system, and obtain the direction vector of the fitted straight line.

[0009] obtaining a ground plane equation, and calculating a rotation matrix representation of the forklift coordinate system in the camera coordinate system according to a direction vector of the fitted straight line of the identification points and a normal vector of the ground plane;

[0010] obtaining coordinates of an origin of the forklift coordinate system in the camera coordinate system;

[0011] obtaining a transformation relationship between the forklift coordinate system and the camera coordinate system according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, and then completing the camera extrinsic calibration.

[0012] As a preferred solution, the obtaining of the coordinates of each identification point in the camera coordinate system specifically comprises:

[0013] obtaining coordinates of each identification point on the RGB image, and obtaining coordinates of each identification point in the camera coordinate system according to depth information corresponding to the depth image.

[0014] As a preferred solution, the straight line fitting of the identification points in the camera coordinate system and the obtaining of the direction vector of the fitted straight line specifically comprise:

[0015] fitting a straight line of the identification points in the camera coordinate system by using a least square method to obtain a fitted space straight line equation;

[0016] solving the fitted space straight line equation to obtain a direction vector of the fitted straight line of the identification points in the camera coordinate system.

[0017] As a preferred solution, the obtaining of the ground plane equation and the calculating of the rotation matrix representation of the forklift coordinate system in the camera coordinate system according to the direction vector of the fitted straight line of the identification points and the normal vector of the ground plane specifically comprise:

[0018] fitting a ground plane equation in the camera coordinate system according to the ground point cloud;

[0019] obtaining a normal vector of the ground plane equation according to the ground plane equation, obtaining a direction vector of the forklift coordinate system in combination with the direction vector of the fitted straight line of the identification points, and thus obtaining a representation of the direction vector of the forklift coordinate system in the camera coordinate system;

[0020] normalizing the direction vector of each coordinate axis of the forklift coordinate system in the camera coordinate system to obtain a rotation matrix representation of the forklift coordinate system in the camera coordinate system.

[0021] As a preferred solution, the obtaining of the coordinates of the origin of the forklift coordinate system in the camera coordinate system specifically comprises:

[0022] acquire the distance from one of the identification points to the origin of the forklift coordinate system;

[0023] In the camera coordinate system, according to the straight line equation fitted according to the identification points and the positional relationship between the origin of the forklift coordinate system and one of the identification points, the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system is obtained.

[0024] The distance from one of the identification points to the origin of the forklift coordinate system is substituted into the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system, and the coordinates of the origin of the forklift coordinate system in the camera coordinate system are calculated.

[0025] As a preferred solution, the acquisition of the coordinates of the origin of the forklift coordinate system in the camera coordinate system further comprises:

[0026] If the origin of the forklift coordinate system appears in the RGB image, the coordinates of the forklift coordinate system in the camera coordinate system are obtained according to the intersection of the ground plane equation and the ray passing through the origin of the camera coordinate system.

[0027] As a preferred solution, the transformation relationship between the forklift coordinate system and the camera coordinate system is obtained according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, and specifically:

[0028] According to the coordinates of the origin of the forklift coordinate system in the camera coordinate system, the translation vector of the forklift coordinate system to the camera coordinate system is calculated.

[0029] The rotation representation of the forklift coordinate system in the camera coordinate system and the translation vector are combined to obtain the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system.

[0030] According to the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system, the extrinsic parameters of the camera are obtained, and the extrinsic parameter calibration of the camera is completed.

[0031] Correspondingly, the application also provides an extrinsic parameter calibration device based on a ground auxiliary line, comprising: an image acquisition module, an identification point direction module, a rotation matrix module, an origin coordinate module and an extrinsic parameter calibration module.

[0032] The image acquisition module is used to acquire image data after the RGB image and the depth image of the ground auxiliary line are aligned, wherein the ground auxiliary line coincides with the motion direction of the forklift, and a preset number of identification points are arranged on the ground auxiliary line.

[0033] The identification point direction module is used to acquire the coordinates of each identification point in the camera coordinate system, to perform straight line fitting on the identification points in the camera coordinate system, and to obtain the direction vector of the fitted straight line.

[0034] The rotation matrix module is configured to obtain a ground plane equation, and calculate a rotation matrix representation of the forklift coordinate system in the camera coordinate system according to a direction vector of a straight line fitted by the identification points and a normal vector of the ground plane.

[0035] The origin coordinate module is configured to obtain coordinates of an origin of the forklift coordinate system in the camera coordinate system.

[0036] The extrinsic calibration module is configured to obtain a transformation relationship between the forklift coordinate system and the camera coordinate system according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, and then complete extrinsic calibration of the camera.

[0037] As a preferred solution, the coordinates of each identification point in the camera coordinate system are obtained in the following manner:

[0038] The coordinates of each identification point on the RGB image are obtained, and the coordinates of each identification point in the camera coordinate system are obtained according to depth information corresponding to the depth image.

[0039] As a preferred solution, the straight line fitting of the identification points in the camera coordinate system and the direction vector of the fitted straight line are obtained in the following manner:

[0040] The straight line fitting of the identification points in the camera coordinate system is performed by using the least square method, and a fitted space straight line equation is obtained.

[0041] The fitted space straight line equation is solved to obtain a direction vector of the identification point fitted straight line in the camera coordinate system.

[0042] As a preferred solution, the ground plane equation is obtained, and the rotation matrix representation of the forklift coordinate system in the camera coordinate system is calculated according to the direction vector of the identification point fitted straight line and the normal vector of the ground plane in the following manner:

[0043] The ground plane equation is fitted in the camera coordinate system according to the ground point cloud.

[0044] The normal vector of the ground plane equation is obtained according to the ground plane equation, the direction vector of the forklift coordinate system is obtained in combination with the direction vector of the identification point fitted straight line, and thus the representation of the direction vector of the forklift coordinate system in the camera coordinate system is obtained.

[0045] The direction vector of each forklift coordinate system coordinate axis represented in the camera coordinate system is normalized to obtain the rotation matrix representation of the forklift coordinate system in the camera coordinate system.

[0046] As a preferred solution, the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained in the following manner:

[0047] acquire the distance from one of the identification points to the origin of the forklift coordinate system;

[0048] in the camera coordinate system, according to the straight line equation fitted by the identification points and the positional relationship between the origin of the forklift coordinate system and one of the identification points, obtain the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system;

[0049] substitute the distance from one of the identification points to the origin of the forklift coordinate system into the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system, and calculate the coordinates of the origin of the forklift coordinate system in the camera coordinate system.

[0050] As a preferred solution, the acquisition of the coordinates of the origin of the forklift coordinate system in the camera coordinate system further comprises:

[0051] if the origin of the forklift coordinate system appears in the RGB image, according to the intersection point of the ground plane equation and the ray passing through the origin of the camera coordinate system, obtain the coordinates of the forklift coordinate system in the camera coordinate system.

[0052] As a preferred solution, according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, obtain the transformation relationship between the forklift coordinate system and the camera coordinate system, specifically:

[0053] according to the coordinates of the origin of the forklift coordinate system in the camera coordinate system, calculate the translation vector from the forklift coordinate system to the camera coordinate system;

[0054] combine the rotation representation of the forklift coordinate system in the camera coordinate system and the translation vector to obtain the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system;

[0055] according to the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system, obtain the extrinsic parameters of the camera, thereby completing the extrinsic parameter calibration of the camera.

[0056] Correspondingly, the application also provides a terminal device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor realizes the ground auxiliary line based extrinsic parameter calibration method according to any one of the above when executing the computer program.

[0057] Correspondingly, the application also provides a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the ground auxiliary line based extrinsic parameter calibration method according to any one of the above when running.

[0058] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0059] The technical scheme of the present application obtains image data of the aligned RGB image and depth image of the ground auxiliary line, obtains a direction vector of a straight line fitted based on the identification points through the coordinates of each identification point in the camera coordinate system, calculates the rotation matrix of the forklift coordinate system in the camera coordinate system after obtaining the ground plane equation, and obtains the transformation relationship between the forklift coordinate system and the camera coordinate system accurately and efficiently after obtaining the coordinates of the origin of the forklift coordinate system in the camera coordinate system, thereby calibrating the camera with high robustness and high accuracy.

[0060] Further, the present application does not need a calibration template, but only needs a ground auxiliary line to realize the external parameter calibration of the camera to the forklift coordinate system. Meanwhile, the calibration method of the present application is simple and easy to operate, does not need other calibration tools, is easy to mass-produce, fully utilizes the characteristics of the camera point cloud data in the process of calculation to perform plane fitting and geometric properties of spatial rigid body coordinate transformation, and makes the external parameter calibration result accurate and robust. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 : is a schematic diagram of the forklift coordinate system and the camera coordinate system established by the embodiment of the present application;

[0062] Figure 2 : is a step flowchart of the external parameter calibration method based on the ground auxiliary line provided by the embodiment of the present application;

[0063] Figure 3 : is a flowchart of the external parameter calibration method provided by the embodiment of the present application;

[0064] Figure 4 : is a structural schematic diagram of the external parameter calibration device based on the ground auxiliary line provided by the embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0066] Embodiment one

[0067] Please refer to Figure 1, which is a schematic diagram of establishing the forklift coordinate system and the camera coordinate system provided by the embodiment of the application. In the exemplary embodiment, the extrinsic parameters of the RGBD sensor relative to the body coordinate system of the intelligent forklift are accurately estimated through a ground auxiliary line along the running direction of the unmanned forklift and four identification points on the ground auxiliary line; wherein the Z c positive direction of the camera optical axis, the X c positive direction is from left to right, the Y c positive direction satisfies the right-hand rule. The origin of the forklift body coordinate system is on the ground surface, and the X F positive direction is the direction of the forklift advancing, the Y F positive direction is from left to right, the Z F positive direction satisfies the right-hand rule, and the vertical top surface is upward.

[0068] Please refer to Figure 2 , a ground auxiliary line-based extrinsic parameter calibration method provided by the embodiment of the application, comprising the following steps S101-S105:

[0069] Step S101: obtaining image data of an aligned RGB image and a depth image of a ground auxiliary line; wherein the ground auxiliary line coincides with the forklift motion direction, and a preset number of identification points are arranged on the ground auxiliary line.

[0070] In the embodiment, a straight line is arranged on the ground along the direction of the forklift picking up goods, that is, coincides with the motion direction of the forklift and is mainly used for guiding the forklift to calibrate the extrinsic parameters, and a plurality of identification points are arranged on the ground auxiliary line. For example, the number of identification points is preferably 4, thereby identifying the positions of P1, P2, P3 and P4 points on the straight line, and ensuring that the four identification points can appear in the image of the camera RGB, and the distances of the four points to the origin of the forklift coordinate system are measured as d1, d2, d3 and d4.

[0071] It should be noted that in the embodiment, the number of identification points is only required to be greater than or equal to 2, that is, two or more identification points can determine the equation of the straight line.

[0072] Step S102: obtaining the coordinates of each identification point in the camera coordinate system, fitting a straight line to the identification points in the camera coordinate system, and obtaining the direction vector of the fitted straight line.

[0073] As a preferred scheme of the embodiment, the coordinates of each identification point in the camera coordinate system are obtained as follows:

[0074] Obtaining the coordinates of each identification point on the RGB image, and obtaining the coordinates of each identification point in the camera coordinate system according to the depth information corresponding to the depth image.

[0075] It should be noted that, in the case that the RGB image and the depth image collected by the RGBD camera have been registered and aligned, the coordinates of the four identification points P1, P2, P3 and P4 in the camera coordinate system are found, and the depth information of the corresponding identification points. In this embodiment, it is assumed that the coordinates of the ground identification point after imaging on the RGB image are (ui, vi), i = 1, 2, 3, 4, and the depth of the identification point is zi. According to formula (1) and the ground plane equation Ax + By + Cz + D = 0:

[0076]

[0077] The coordinates of the ground identification point in the camera coordinate system are obtained by simultaneously solving the equations as

[0078] As a preferred scheme of the present embodiment, the identification points are fitted with a straight line in the camera coordinate system, and a direction vector of the fitted straight line is obtained, specifically:

[0079] In the camera coordinate system, the identification points are fitted with a straight line by the least square method, and a fitted space straight line equation is obtained; the fitted space straight line equation is solved to obtain a direction vector of the identification point fitting straight line in the camera coordinate system.

[0080] In this embodiment, as z i changes, it means that the ray with the camera coordinate system origin as the starting point and passing through the corresponding intersection point of the RGB image, and the intersection point of the ray and the ground fitting plane is the coordinate (x ci , y ci , z ci ) of the ground identification point in the camera coordinate system. It can be seen that the difference between any two points is the direction vector of the forklift coordinate system XF. In order to reduce the error, the least square method is used to fit the straight line of P1, P2, P3 and P4 in the camera coordinate system, so that the fitted space straight line equation is: That is, the direction vector of the straight line of the ground identification point in the camera coordinate system is obtained, which is also the representation of the forklift running direction X F vector in the camera coordinate system.

[0081] Step S103: obtaining the ground plane equation, and calculating the rotation matrix representation of the forklift coordinate system in the camera coordinate system according to the direction vector of the fitted straight line and the normal vector of the ground plane.

[0082] As a preferred scheme of the embodiment, the ground plane equation is obtained, and a rotation matrix representation of the forklift coordinate system in the camera coordinate system is calculated according to a direction vector of a fitted straight line of the identification point and a normal vector of the ground plane, specifically as follows:

[0083] According to the ground point cloud, a ground plane equation is fitted in the camera coordinate system; a normal vector of the ground plane equation is obtained according to the ground plane equation, and a representation of a direction vector of the forklift coordinate system in the camera coordinate system is obtained according to the direction vector of the forklift coordinate system; and the direction vector of each forklift coordinate system coordinate axis represented in the camera coordinate system is normalized to obtain a rotation matrix representation of the forklift coordinate system in the camera coordinate system.

[0084] It should be noted that, since the depth map may have holes or incorrect positions due to some reasons, the ground plane equation is fitted in the camera coordinate system according to the ground point cloud as Ax+By+Cz+D=0.

[0085] Further, in the embodiment, in combination with the fitted ground plane equation, the normal vector of the plane equation is (A, B, C) T , which is exactly the same as the representation of the direction vector of the forklift coordinate system Z F in the camera coordinate system, according to the right-hand rule X Fc =(m, n, p) T and Z Fc =(A, B, C) T , the cross product X Fc ×Z Fc is exactly the representation of the direction vector of the forklift coordinate system Y F in the camera coordinate system Y Fc . According to the representation of the forklift coordinate system X F Y F Z F in the camera coordinate system, the normalization of each direction vector is the rotation matrix from the forklift coordinate system to the camera coordinate system, that is,

[0086] Step S104: Obtain the coordinates of the origin of the forklift coordinate system in the camera coordinate system.

[0087] As a preferred scheme of the embodiment, the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained, specifically including:

[0088] acquire the distance from one of the identification points to the origin of the forklift coordinate system; in the camera coordinate system, according to the straight line equation fitted by the identification points and the positional relationship between the origin of the forklift coordinate system and one of the identification points, the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system is obtained; the distance from one of the identification points to the origin of the forklift coordinate system is substituted into the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system, and the coordinates of the origin of the forklift coordinate system in the camera coordinate system are calculated.

[0089] It should be noted that the origin of the forklift coordinate system cannot be seen from the camera, but the distance from the origin of the forklift coordinate system to the P1 point is known as d1, and the coordinates of the origin of the forklift coordinate system in the camera coordinate system are (x c0 ,y c0 ,z c0 ), which are substituted into the straight line equation together with the coordinates (x c1 ,y c1 ,z c1 ) of P1 in the camera coordinate system to obtain: At the same time, since , the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained as: Further, the coordinates of the origin of the forklift coordinate system in the camera coordinate system are solved as:

[0090]

[0091] As a preferred scheme of the embodiment, the acquisition of the coordinates of the origin of the forklift coordinate system in the camera coordinate system further includes:

[0092] If the origin of the forklift coordinate system appears in the RGB image, the coordinates of the forklift coordinate system in the camera coordinate system are obtained according to the intersection of the ground plane equation and the ray passing through the origin of the camera coordinate system.

[0093] It should be noted that the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained directly by fitting the ground plane equation and the intersection of the ray passing through the origin of the camera.

[0094] Step S105: According to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, the transformation relationship between the forklift coordinate system and the camera coordinate system is obtained, and the extrinsic calibration of the camera is completed.

[0095] As a preferred scheme of the embodiment, according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, the transformation relationship between the forklift coordinate system and the camera coordinate system is obtained, and the extrinsic calibration of the camera is completed.

[0096] According to the coordinates of the origin of the forklift coordinate system in the camera coordinate system, a translation vector from the forklift coordinate system to the camera coordinate system is calculated; the rotation representation of the forklift coordinate system in the camera coordinate system and the translation vector are combined to obtain a transformation relationship corresponding to the forklift coordinate system and the camera coordinate system; and according to the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system, the extrinsic parameters of the camera are obtained, so as to complete the extrinsic parameter calibration of the camera.

[0097] It should be noted that in the embodiment, according to the coordinates of the origin of the forklift coordinate system in the camera coordinate system, a translation vector T=(x c0 ,y c0 ,z c0 ) T .

[0098] In the embodiment, after obtaining the translation vector T=(x c0 ,y c0 ,z c0 ) T from the forklift coordinate system to the camera coordinate system, the rotation matrix R is combined, and the transformation relationship of the forklift coordinate system and the camera coordinate system is obtained.

[0099] In another embodiment of the present application, please refer to Figure 3 , the technical solution of the present application obtains the image data of the aligned RGB image and depth image of the ground auxiliary line, obtains the direction vector based on the fitted straight line of the identification point, calculates the rotation matrix representation of the forklift coordinate system in the camera coordinate system after obtaining the ground plane equation, and accurately and efficiently obtains the transformation relationship of the forklift coordinate system and the camera coordinate system after obtaining the coordinates of the origin of the forklift coordinate system in the camera coordinate system, thereby calibrating the extrinsic parameters of the camera with high robustness and high accuracy.

[0100] It can be understood that the embodiment of the present application can enable the intelligent forklift to better use the stack plate (pallet) or trolley pose result recognized based on the RGBD sensor, and convert the recognition information in the RGBD camera coordinate system into the body coordinate system of the intelligent forklift, thereby simplifying the calibration process while ensuring the accuracy and stability of the calibration result, and the embodiment has high accuracy and robustness, is simple to operate, and is easy to mass produce.

[0101] The above embodiment has the following effects:

[0102] Compared with the prior art, the embodiment of the present application does not need to set a calibration template, but only needs one ground auxiliary line to realize the external parameter calibration of the camera to the forklift coordinate system. Meanwhile, the calibration method of the present application is simple and easy to operate, does not need other calibration tools, is easy for batch production, fully utilizes the characteristics of the camera point cloud data in the process of calculation to perform plane fitting and geometric properties of spatial rigid body coordinate transformation, so that the external parameter calibration result is high in accuracy and robustness.

[0103] Embodiment two

[0104] Correspondingly, the present application further provides a ground auxiliary line-based external parameter calibration device, comprising: an image acquisition module 201, an identification point direction module 202, a rotation matrix module 203, an origin coordinate module 204 and an external parameter calibration module 205.

[0105] The image acquisition module 201 is used to acquire the image data of the aligned RGB image and depth image of the ground auxiliary line; wherein the ground auxiliary line coincides with the motion direction of the forklift, and a preset number of identification points are arranged on the ground auxiliary line.

[0106] The identification point direction module 202 is used to acquire the coordinates of each identification point in the camera coordinate system, perform straight line fitting on the identification points in the camera coordinate system, and obtain the direction vector of the fitted straight line.

[0107] The rotation matrix module 203 is used to acquire the equation of the ground plane, and calculate the rotation matrix representation of the forklift coordinate system in the camera coordinate system according to the direction vector of the fitted straight line and the normal vector of the ground plane.

[0108] The origin coordinate module 204 is used to acquire the coordinates of the origin of the forklift coordinate system in the camera coordinate system.

[0109] The external parameter calibration module 205 is used to obtain the transformation relationship between the forklift coordinate system and the camera coordinate system according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, and then complete the external parameter calibration of the camera.

[0110] As a preferred scheme of the present embodiment, the acquisition of the coordinates of each identification point in the camera coordinate system is specifically:

[0111] The coordinates of each identification point on the RGB image are acquired, and the coordinates of each identification point in the camera coordinate system are obtained according to the depth information corresponding to the depth image.

[0112] As a preferred scheme of the present embodiment, the straight line fitting on the identification points in the camera coordinate system and the obtaining of the direction vector of the fitted straight line are specifically:

[0113] In the camera coordinate system, a straight line is fitted to the mark points by a least square method to obtain a fitted space straight line equation; and the fitted space straight line equation is solved to obtain a direction vector of the mark point fitted straight line in the camera coordinate system.

[0114] As a preferred scheme of the embodiment, the ground plane equation is obtained, and a rotation matrix representation of the forklift coordinate system in the camera coordinate system is calculated according to the direction vector of the mark point fitted straight line and the normal vector of the ground plane, specifically as follows:

[0115] According to the ground point cloud, a ground plane equation is fitted in the camera coordinate system; a normal vector of the ground plane equation is obtained according to the ground plane equation, and a direction vector of the forklift coordinate system is obtained in combination with the direction vector of the mark point fitted straight line to obtain a representation of the direction vector of the forklift coordinate system in the camera coordinate system; and the direction vector of each forklift coordinate system coordinate axis represented in the camera coordinate system is normalized to obtain a rotation matrix representation of the forklift coordinate system in the camera coordinate system.

[0116] As a preferred scheme of the embodiment, the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained, specifically including:

[0117] The distance from one of the mark points to the origin of the forklift coordinate system is obtained; in the camera coordinate system, a straight line equation corresponding to the origin of the forklift coordinate system is obtained according to the mark point fitted straight line equation and the positional relationship between the origin of the forklift coordinate system and the one of the mark points; and the distance from the one of the mark points to the origin of the forklift coordinate system is substituted into the straight line equation corresponding to the origin of the forklift coordinate system in the camera coordinate system to calculate the coordinates of the origin of the forklift coordinate system in the camera coordinate system.

[0118] As a preferred scheme of the embodiment, the coordinates of the origin of the forklift coordinate system in the camera coordinate system are obtained, specifically further including:

[0119] If the origin of the forklift coordinate system appears in the RGB image, the coordinates of the forklift coordinate system in the camera coordinate system are obtained according to the intersection of the ground plane equation and the ray passing through the origin of the camera coordinate system.

[0120] As a preferred scheme of the embodiment, the transformation relationship between the forklift coordinate system and the camera coordinate system is obtained according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the origin of the forklift coordinate system in the camera coordinate system, specifically as follows:

[0121] According to the coordinates of the origin of the forklift coordinate system in the camera coordinate system, a translation vector of the forklift coordinate system to the camera coordinate system is calculated; the rotation representation of the forklift coordinate system in the camera coordinate system and the translation vector are combined to obtain a transformation relationship corresponding to the forklift coordinate system and the camera coordinate system; and the extrinsic parameters of the camera are obtained according to the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system, so that the extrinsic parameter calibration of the camera is completed.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0123] The embodiment of the present application has the following effects:

[0124] The technical solution of the present application obtains the image data of the aligned RGB image and depth image of the ground auxiliary line, obtains the direction vector of the straight line fitted based on the identification points through the coordinates of each identification point in the camera coordinate system, calculates the rotation matrix representation of the forklift coordinate system in the camera coordinate system after obtaining the ground plane equation, and can accurately and efficiently obtain the transformation relationship of the forklift coordinate system and the camera coordinate system after obtaining the coordinates of the origin of the forklift coordinate system in the camera coordinate system, and then calibrate the extrinsic parameters of the camera with high robustness and high accuracy.

[0125] Further, the present application does not need a calibration template, but only needs a ground auxiliary line to realize the extrinsic parameter calibration of the camera to the forklift coordinate system. Meanwhile, the calibration method of the present application is simple and easy to operate, does not need other calibration tools, is easy to mass-produce, fully utilizes the characteristics of the camera point cloud data in the process of calculation for plane fitting and geometric properties of spatial rigid body coordinate transformation, and makes the extrinsic parameter calibration result accurate and robust.

[0126] Embodiment three

[0127] Correspondingly, the present application also provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the extrinsic parameter calibration method based on a ground auxiliary line as described in any one of the above embodiments when executing the computer program.

[0128] The terminal device of this embodiment comprises a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. The processor implements each step in Embodiment One described above when executing the computer program, for example Figure 1The processor implements the functions of the modules / units in the above-mentioned apparatus embodiments when executing the computer program, such as the landmark direction module 202.

[0129] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the terminal device. For example, the rotation matrix module 203 is used to obtain the ground plane equation, combine the direction vector of the straight line fitted by the landmarks to obtain the direction vector of the forklift coordinate system, and calculate the rotation matrix of the forklift coordinate system in the camera coordinate system.

[0130] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The terminal device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of the terminal device, and does not constitute a limitation on the terminal device, and can include more or fewer components than the schematic diagram, or combine certain components, or different components, for example, the terminal device can also include an input / output device, a network access device, a bus, and the like.

[0131] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, and the like. The processor is the control center of the terminal device, and connects all parts of the terminal device through various interfaces and lines.

[0132] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the terminal device by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0133] The modules / units integrated in the terminal device can be stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0134] Embodiment Four

[0135] Correspondingly, the application further provides a computer readable storage medium, including a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located performs the ground auxiliary line based external parameter calibration method as any one of the above embodiments.

[0136] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above-described specific embodiments are merely examples of the present application and are not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for extrinsic parameter calibration based on ground auxiliary lines, applicable to forklifts, wherein the forklifts are equipped with RGBD sensor cameras, characterized in that, The application relates to a method for calibrating an external parameter of a camera of a forklift truck, and belongs to the field of forklift truck control. A forklift truck coordinate system is established based on the forklift truck, and a camera coordinate system is established based on the RGBD sensor camera; An RGB image and a depth image of a ground auxiliary line are acquired by the RGBD sensor camera, the RGB image and the depth image are aligned, wherein the ground auxiliary line is coincident with the moving direction of the forklift truck, and a preset number of mark points are arranged on the ground auxiliary line; The coordinates of each mark point in the camera coordinate system are acquired, specifically including: acquiring the coordinates of each mark point on the RGB image, and obtaining the coordinates of each mark point in the camera coordinate system according to the depth information corresponding to the depth image; A straight line fitting is performed on the mark points in the camera coordinate system, and a direction vector of the fitted straight line is obtained; A ground plane equation is acquired, and a rotation matrix representation of the forklift truck coordinate system in the camera coordinate system is calculated according to the direction vector of the fitted straight line and the normal vector of the ground plane; The coordinates of the origin of the forklift truck coordinate system in the camera coordinate system are acquired; The transformation relationship between the forklift truck coordinate system and the camera coordinate system is obtained according to the rotation matrix representation of the forklift truck coordinate system in the camera coordinate system and the coordinates of the origin of the forklift truck coordinate system in the camera coordinate system, and the external parameter calibration of the camera is completed, specifically including: A translation vector from the forklift truck coordinate system to the camera coordinate system is calculated according to the coordinates of the origin of the forklift truck coordinate system in the camera coordinate system; The rotation matrix representation of the forklift truck coordinate system in the camera coordinate system and the translation vector are combined to obtain the transformation relationship between the forklift truck coordinate system and the camera coordinate system; The external parameter of the camera is obtained according to the transformation relationship between the forklift truck coordinate system and the camera coordinate system, and the external parameter calibration of the camera is completed.

2. The method of claim 1, wherein, The straight line fitting on the mark points in the camera coordinate system and the direction vector of the fitted straight line are specifically as follows: In the camera coordinate system, a straight line fitting is performed on the mark points by using the least square method to obtain a fitted space straight line equation; The fitted space straight line equation is solved to obtain the direction vector of the mark point fitted straight line in the camera coordinate system.

3. The method of claim 1, wherein, The ground plane equation is acquired, and the rotation matrix representation of the forklift truck coordinate system in the camera coordinate system is calculated according to the direction vector of the mark point fitted straight line and the normal vector of the ground plane, specifically as follows: The ground plane equation is fitted in the camera coordinate system according to the ground point cloud to obtain the ground plane equation; The normal vector of the ground plane equation is obtained according to the ground plane equation, the direction vector of the forklift truck coordinate system is obtained in combination with the direction vector of the mark point fitted straight line, and thus the direction vector of the forklift truck coordinate system in the camera coordinate system is obtained; The direction vectors of each coordinate axis of the forklift truck coordinate system in the camera coordinate system are normalized to obtain the rotation matrix representation of the forklift truck coordinate system in the camera coordinate system.

4. The method of claim 1, wherein, The coordinates of the origin of the forklift truck coordinate system in the camera coordinate system are acquired, specifically including: The distance from one of the mark points to the origin of the forklift truck coordinate system is acquired; In the camera coordinate system, according to the straight line equation fitted according to the mark points and the position relationship between the forklift coordinate system origin and one of the mark points, a straight line equation corresponding to the forklift coordinate system origin in the camera coordinate system is obtained; The distance from one of the mark points to the forklift coordinate system origin is substituted into the straight line equation corresponding to the forklift coordinate system origin in the camera coordinate system, and the coordinates of the forklift coordinate system origin in the camera coordinate system are calculated.

5. The method of claim 1, wherein, The coordinates of the forklift coordinate system origin in the camera coordinate system are obtained, and the method further comprises the following steps: If the forklift coordinate system origin appears in the RGB image, the coordinates of the forklift coordinate system in the camera coordinate system are obtained according to the intersection of the ground plane equation and the ray passing through the camera coordinate system origin.

6. A ground-based auxiliary line-based external parameter calibration device suitable for a forklift truck, wherein an RGBD sensor camera is arranged on the forklift truck. It comprises: An image acquisition module, a mark point direction module, a rotation matrix module, an origin coordinate module and an external parameter calibration module; The image acquisition module is used to establish a forklift coordinate system with the forklift as a reference, and a camera coordinate system with the RGBD sensor camera as a reference; the RGB image and the depth image of the ground auxiliary line are acquired by the RGBD sensor camera, and the RGB image and the depth image are aligned; wherein the ground auxiliary line coincides with the forklift movement direction, and a predetermined number of mark points are arranged on the ground auxiliary line; The mark point direction module is used to acquire the coordinates of each mark point in the camera coordinate system, specifically including: acquiring the coordinates of each mark point on the RGB image, and obtaining the coordinates of each mark point in the camera coordinate system according to the depth information corresponding to the depth image; the mark points are linearly fitted in the camera coordinate system, and the direction vector of the fitted straight line is obtained; The rotation matrix module is used to acquire the ground plane equation, and calculate the rotation matrix representation of the forklift coordinate system in the camera coordinate system according to the direction vector of the fitted straight line and the normal vector of the ground plane; The origin coordinate module is used to acquire the coordinates of the forklift coordinate system origin in the camera coordinate system; The external parameter calibration module is used to obtain the transformation relationship between the forklift coordinate system and the camera coordinate system according to the rotation matrix representation of the forklift coordinate system in the camera coordinate system and the coordinates of the forklift coordinate system origin in the camera coordinate system, and then complete the external parameter calibration of the camera, specifically including: According to the coordinates of the forklift coordinate system origin in the camera coordinate system, a translation vector from the forklift coordinate system to the camera coordinate system is calculated; The rotation matrix representation of the forklift coordinate system in the camera coordinate system and the translation vector are combined to obtain the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system; According to the transformation relationship corresponding to the forklift coordinate system and the camera coordinate system, the external parameters of the camera are obtained, so as to complete the external parameter calibration of the camera.

7. A terminal device, characterized by comprising: It comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor realizes the ground auxiliary line based external parameter calibration method according to any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program; wherein the computer program, when executed, controls a device where the computer readable storage medium is located to perform the ground auxiliary line based external parameter calibration method according to any one of claims 1-5.

Images