A fork truck linear motion-based external parameter calibration method and device

Abstract

This invention discloses a method and apparatus for extrinsic parameter calibration based on the linear motion of a forklift, comprising: acquiring first image data of a calibration template and obtaining a first transformation relationship when the forklift is in a first position; acquiring second image data of the calibration template and obtaining a second transformation relationship when the forklift is in a second position; obtaining a third transformation relationship between the coordinate systems of the first and second cameras based on the first and second transformation relationships; obtaining the direction vector and translation matrix of the forklift's motion direction based on the third transformation relationship and the origin of the first and second positions; calculating the direction vector of the forklift coordinate system based on the direction vector of the forklift's motion direction and the normal vector of the ground, thereby obtaining the rotation matrix of the camera; and combining the translation matrix to obtain the extrinsic parameters of the camera. This invention performs calibration during the linear movement of the forklift, solving the problems of high complexity, low efficiency, and low accuracy of extrinsic parameters in calibration operations, and is more in line with the actual application of forklifts.

Description

Technical Field

[0001] This invention relates to the field of RGBD camera calibration, and more particularly to an extrinsic parameter calibration method and apparatus based on the linear motion of a forklift. Background Technology

[0002] Unmanned intelligent forklifts have been widely used in the field of smart warehousing. In fully or semi-automated factory operations, unmanned intelligent forklifts play a crucial role in transferring and handling goods, typically moving pallets or trolleys loaded with goods from one location to another within the factory. However, pallets or trolleys are not always arranged ideally due to various reasons.

[0003] Before an unmanned forklift picks up a pallet or trolley, an RGBD sensor camera mounted on the forklift identifies and collects the 3D spatial point cloud and corresponding RGB 2D image data of the pallet or trolley. This data is then processed and analyzed to determine the position and orientation of the pallet or trolley. However, existing methods for calibrating the extrinsic parameters of RGBD sensor cameras use externally set white lines to calculate the extrinsic parameters in the order of Yaw, Pitch, Roll, and Ty. This calculation process is based on the idea of ​​multiplying rotation matrices, progressively breaking it down into rotation matrices for each degree of freedom, making the calculation complex. Furthermore, a pre-calibrated reference depth camera transmits position information of their common target to the camera to be calibrated, thus achieving the calibration of the spatial position extrinsic parameters of the camera to be calibrated. This method requires an additional pre-calibrated reference depth camera, has high operational complexity, and is prone to calibration errors.

[0004] Therefore, there is an urgent need for an external parameter calibration method for RGBD sensor cameras that can guarantee accuracy and efficiency. Summary of the Invention

[0005] This invention provides a method and apparatus for calibrating extrinsic parameters based on the linear motion of a forklift, in order to solve the technical problems of high complexity, low efficiency and low accuracy of extrinsic parameters in the prior art.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for calibrating extrinsic parameters based on the linear motion of a forklift, comprising:

[0007] When the forklift is in the first position, it acquires the first image data after aligning the first RGB image and the first depth image of the calibration template, and calculates the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data.

[0008] When the forklift is in the second position, it acquires second image data after aligning the second RGB image and the second depth image of the calibration template, and calculates the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template based on the second image data;

[0009] Based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained;

[0010] Based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, the coordinates of the origins of the first and second positions in the first camera coordinate system are obtained, and then the direction vector of the forklift's movement direction in the first camera coordinate system and the translation matrix from the forklift coordinate system to the camera coordinate system are obtained; wherein, the first position is the position before the forklift moves in a straight line, and the second position is the position after the forklift moves in a straight line.

[0011] Based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, the rotation matrix of the forklift coordinate system in the camera coordinate system is calculated, and the camera's extrinsic parameters are obtained based on the camera's rotation matrix and translation matrix.

[0012] As a preferred embodiment, the step of calculating the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data specifically involves:

[0013] Based on the first image data, the first plane equation of the calibration template is obtained;

[0014] The projection of the inner corner point of the calibration template onto the first image data is connected to the origin of the first camera coordinate system, and the coordinates of the intersection of the line and the first plane equation are used as the coordinate representation of the inner corner point of the calibration template in the camera coordinate system.

[0015] Based on the representation of the inner corner points of the calibration template in the calibration template coordinate system, the first transformation relationship from the first camera coordinate system to the calibration template coordinate system is obtained.

[0016] As a preferred embodiment, based on the second image data, a second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is calculated, specifically as follows:

[0017] Based on the second image data, the second plane equation of the calibration template is obtained;

[0018] The projection of the inner corner point of the calibration template onto the second image data is connected to the origin of the second camera coordinate system, and the coordinates of the intersection of the line and the second plane equation are used as the coordinate representation of the inner corner point of the calibration template in the camera coordinate system.

[0019] Based on the representation of the inner corner points of the calibration template in the calibration template coordinate system, the second transformation relationship from the second camera coordinate system to the calibration template coordinate system is obtained.

[0020] As a preferred embodiment, based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained, specifically as follows:

[0021] Based on the first transformation relationship and the second transformation relationship, the third transformation relationship between the first camera coordinate system and the second camera coordinate system is calculated when the forklift is located at the first position and the second position, respectively.

[0022] As a preferred embodiment, based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, the coordinates of the origins of the first and second positions in the first camera coordinate system are obtained, thereby obtaining the direction vector of the forklift's movement direction in the first camera coordinate system, and the translation matrix from the forklift coordinate system to the camera coordinate system, specifically:

[0023] Based on the third transformation relationship and the first origin of the forklift coordinate system at the first position, the coordinates of the first origin in the first camera coordinate system are calculated, thereby obtaining the translation matrix from the forklift coordinate system to the camera coordinate system.

[0024] Based on the origin of the forklift coordinate system at the first and second positions respectively, the forklift displacement distance is obtained, and based on the forklift displacement distance, the coordinates of the second origin in the first camera coordinate system are obtained;

[0025] Based on the coordinates of the first origin in the first camera coordinate system and the coordinates of the second origin in the first camera coordinate system, the direction vector of the forklift's movement direction in the first camera coordinate system is calculated.

[0026] As a preferred embodiment, the method for obtaining the normal vector of the ground plane in the first camera coordinate system specifically includes:

[0027] The ground point cloud data based on the first camera coordinate system is acquired and fitted to obtain the ground plane equation in the camera coordinate system, and then the normal vector of the ground plane in the first camera coordinate system is obtained.

[0028] As a preferred embodiment, the rotation matrix of the forklift coordinate system in the camera coordinate system is calculated based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system. Then, based on the camera's rotation and translation matrices, the camera's extrinsic parameters are obtained. Specifically:

[0029] Based on the normal vector of the ground plane in the first camera coordinate system, normalize each direction vector of the forklift coordinate system in the first camera coordinate system to obtain the rotation matrix from the forklift coordinate system to the camera coordinate system.

[0030] Based on the translation matrix from the forklift coordinate system to the camera coordinate system, and the rotation matrix from the forklift coordinate system to the camera coordinate system, the extrinsic parameters from the forklift coordinate system to the camera coordinate system are calculated, and then the extrinsic parameters of the camera are obtained.

[0031] Accordingly, an extrinsic parameter calibration device based on the linear motion of a forklift includes: a first position module, a second position module, a transformation relationship module, a direction vector module, and an extrinsic parameter module;

[0032] The first position module is used to acquire first image data after aligning the first RGB image and the first depth image of the calibration template when the forklift is in the first position, and to calculate the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data;

[0033] The second position module is used to acquire second image data after aligning the second RGB image and the second depth image of the calibration template when the forklift is in the second position, and to calculate the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template based on the second image data;

[0034] The transformation relationship module is used to obtain a third transformation relationship between the first camera coordinate system and the second camera coordinate system based on the first transformation relationship and the second transformation relationship;

[0035] The direction vector module is used to obtain the coordinates of the origins of the first and second positions in the first camera coordinate system based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, respectively, and then obtain the direction vector of the forklift's movement direction in the first camera coordinate system, as well as the translation matrix from the forklift coordinate system to the camera coordinate system; wherein, the first position is the position of the forklift before its straight-line movement, and the second position is the position of the forklift after its straight-line movement.

[0036] The extrinsic parameter module is used to calculate the rotation matrix of the forklift coordinate system in the camera coordinate system based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, and to obtain the extrinsic parameters of the camera based on the camera's rotation matrix and translation matrix.

[0037] As a preferred embodiment, the step of calculating the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data specifically involves:

[0038] Based on the first image data, the first plane equation of the calibration template is obtained;

[0039] The projection of the inner corner point of the calibration template onto the first image data is connected to the origin of the first camera coordinate system, and the coordinates of the intersection of the line and the first plane equation are used as the coordinate representation of the inner corner point of the calibration template in the camera coordinate system.

[0040] Based on the representation of the inner corner points of the calibration template in the calibration template coordinate system, the first transformation relationship from the first camera coordinate system to the calibration template coordinate system is obtained.

[0041] As a preferred embodiment, based on the second image data, a second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is calculated, specifically as follows:

[0042] Based on the second image data, the second plane equation of the calibration template is obtained;

[0043] The projection of the inner corner point of the calibration template onto the second image data is connected to the origin of the second camera coordinate system, and the coordinates of the intersection of the line and the second plane equation are used as the coordinate representation of the inner corner point of the calibration template in the camera coordinate system.

[0044] Based on the representation of the inner corner points of the calibration template in the calibration template coordinate system, the second transformation relationship from the second camera coordinate system to the calibration template coordinate system is obtained.

[0045] As a preferred embodiment, based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained, specifically as follows:

[0046] Based on the first transformation relationship and the second transformation relationship, the third transformation relationship between the first camera coordinate system and the second camera coordinate system is calculated when the forklift is located at the first position and the second position, respectively.

[0047] As a preferred embodiment, based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, the coordinates of the origins of the first and second positions in the first camera coordinate system are obtained, thereby obtaining the direction vector of the forklift's movement direction in the first camera coordinate system, and the translation matrix from the forklift coordinate system to the camera coordinate system, specifically:

[0048] Based on the third transformation relationship and the first origin of the forklift coordinate system at the first position, the coordinates of the first origin in the first camera coordinate system are calculated, thereby obtaining the translation matrix from the forklift coordinate system to the camera coordinate system.

[0049] Based on the origin of the forklift coordinate system at the first and second positions respectively, the forklift displacement distance is obtained, and based on the forklift displacement distance, the coordinates of the second origin in the first camera coordinate system are obtained;

[0050] Based on the coordinates of the first origin in the first camera coordinate system and the coordinates of the second origin in the first camera coordinate system, the direction vector of the forklift's movement direction in the first camera coordinate system is calculated.

[0051] As a preferred embodiment, the method for obtaining the normal vector of the ground plane in the first camera coordinate system specifically includes:

[0052] The ground point cloud data based on the first camera coordinate system is acquired and fitted to obtain the ground plane equation in the camera coordinate system, and then the normal vector of the ground plane in the first camera coordinate system is obtained.

[0053] As a preferred embodiment, the rotation matrix of the forklift coordinate system in the camera coordinate system is calculated based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system. Then, based on the camera's rotation and translation matrices, the camera's extrinsic parameters are obtained. Specifically:

[0054] Based on the normal vector of the ground plane in the first camera coordinate system, normalize each direction vector of the forklift coordinate system in the first camera coordinate system to obtain the rotation matrix from the forklift coordinate system to the camera coordinate system.

[0055] Based on the translation matrix from the forklift coordinate system to the camera coordinate system, and the rotation matrix from the forklift coordinate system to the camera coordinate system, the extrinsic parameters from the forklift coordinate system to the camera coordinate system are calculated, and then the extrinsic parameters of the camera are obtained.

[0056] Accordingly, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the extrinsic parameter calibration method based on the linear motion of a forklift as described in any of the preceding claims.

[0057] Accordingly, the present invention also provides a computer-readable storage medium comprising a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the extrinsic parameter calibration method based on the linear motion of a forklift as described in any of the preceding claims.

[0058] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0059] The technical solution of this invention calculates the transformation relationship between the first camera coordinate system and the second camera coordinate system to the calibration template coordinate system by acquiring corresponding image data when the forklift is in the first position and the second position, respectively. Based on this transformation relationship, the coordinates of the origin of the forklift coordinate system at the first position and the second position in the first camera coordinate system are calculated. This allows for the calculation of the direction vector of the forklift's movement direction in the first camera coordinate system and the translation matrix from the forklift coordinate system to the camera coordinate system. Furthermore, based on the normal vector of the ground plane in the first camera coordinate system, the rotation matrix of the forklift coordinate system in the camera coordinate system is obtained. Finally, based on the camera's rotation and translation matrices, the extrinsic parameters of the camera coordinate system are accurately and efficiently obtained.

[0060] This invention enables calibration during the straight-line movement of a forklift, significantly improving the efficiency of external parameter calibration in actual forklift operation. Furthermore, the entire calibration process requires minimal adjustment to the placement of the calibration template, does not need to be strictly correlated with the forklift coordinate system, and does not require measurement of the forklift's forward distance. This fundamentally eliminates the errors introduced by manual measurement of relevant parameters, reduces operational complexity, and thus achieves high-precision calibration. Attached Figure Description

[0061] Figure 1 : This is a schematic diagram illustrating the principle of external parameter calibration provided in an embodiment of the present invention;

[0062] Figure 2 : A schematic diagram of the forklift coordinate system provided in an embodiment of the present invention;

[0063] Figure 3 : A flowchart illustrating the steps of an extrinsic parameter calibration method based on forklift linear motion provided in an embodiment of the present invention;

[0064] Figure 4 : A schematic diagram of the forklift movement relationship provided in an embodiment of the present invention;

[0065] Figure 5 This is a flowchart for RGBD camera extrinsic parameter calibration provided in another embodiment of the present invention;

[0066] Figure 6 This is a schematic diagram of the external parameter calibration device based on the linear motion of a forklift provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] Please see Figure 1 This is a schematic diagram illustrating the principle of extrinsic parameter calibration in an embodiment of the present invention. In this embodiment, a calibration template based on a chessboard grid is used, wherein the coordinate system of the calibration template is X. b Y b Z b The origin of the coordinate system for the calibration template is defined as the first interior corner point of the top left corner of the chessboard grid, and the direction is X from left to right. b Positive direction, vertically downward is Y b Positive direction, satisfying the right-hand rule, forward is Z. b Positive. The camera coordinate system is X. c1 Y c1 Z c1 The direction is from left to right, designated X. c1 Positive direction, downward is Y c1 Positive direction, satisfying the right-hand rule, forward is Z. c1 Forward; the forklift moves forward a certain distance from position 1 to position 2, at which point the camera coordinate system is X. c2 Y c2 Z c2 The direction is defined relative to the first camera coordinate system X. c1 Y c1 Z c1 Consistent. The origin of the forklift coordinate system and the geometric center of the four wheels at the forklift fork are on the ground, X F The positive direction is the direction in which the forklift moves forward, Y. F The positive direction is from left to right, Z FThe positive direction satisfies the right-hand rule, perpendicular to the ground and upwards, such as... Figure 2 As shown.

[0070] Please refer to Figure 3 This invention provides a method for calibrating extrinsic parameters based on the linear motion of a forklift, comprising the following steps S101-S105:

[0071] Step S101: When the forklift is in the first position, acquire the first image data after aligning the first RGB image and the first depth image of the calibration template, and calculate the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data.

[0072] It should be noted that the RGBD sensor internally registers and aligns the RGB images and depth information it acquires, and outputs the aligned RGB images and depth images, which are the image data in the embodiments of this invention.

[0073] Furthermore, in this embodiment, when the forklift is at the first position 1, it acquires RGB and depth images of the ground using an RGBD sensor, assuming point O on the ground... F It did not appear in the RGB image field of view.

[0074] In this embodiment, the calibration template is fixed in advance, and there are no fixed conditions for the placement of the calibration template. That is, the calibration template has a certain pitch, roll or yaw angle relative to the forklift, the ground or the RGBD sensor, which can also achieve the technical solution of this embodiment.

[0075] In a preferred embodiment, the step of calculating the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data specifically involves:

[0076] Based on the first image data, the first plane equation of the calibration template is obtained; the projection of the interior corner point of the calibration template onto the first image data is connected to the origin of the first camera coordinate system, and the coordinates of the intersection of the connecting line and the first plane equation are used as the coordinate representation of the interior corner point of the calibration template in the camera coordinate system; based on the representation of the interior corner point of the calibration template in the calibration template coordinate system, the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template is obtained.

[0077] It should be noted that in this embodiment, the RGBD sensor acquires the RGB image and depth image of the calibration template, and the plane equation representing the calibration template, a1x + b1y + c1z + d1 = 0, can also be obtained. According to the pinhole imaging principle, the projection of the corner points of the checkerboard pattern of the calibration template onto the RGB image is related to the camera coordinate system X. c1 Y c1 Zc1 The coordinates of the intersection point of the ray formed by the line connecting the origin and the plane a1x+b1y+c1z+d1=0 are the coordinates of the corner points of the calibration template chessboard in the first camera coordinate system (x c1 ,y c1 ,z c1 The interior corner points of these calibration template chessboard grids are located in the X coordinate system of the calibration template. b Y b Z b The following is represented as (x b ,y b ,z b The X coordinate system of the calibration template can be calculated using the Iterative Closest Point (ICP) algorithm. b Y b Z b With respect to the camera coordinate system X c1 Y c1 Z c1 The transformation relationship (R1,T1) is shown in formula (1).

[0078]

[0079] Step S102: When the forklift is in the second position, acquire the second image data after aligning the second RGB image and the second depth image of the calibration template, and calculate the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template based on the second image data.

[0080] It should be noted that the forklift is moved forward in a straight line a certain distance until it reaches point O on the ground. F It appears in the RGB image field of view, and the forklift is currently located at position 2.

[0081] As a preferred embodiment, based on the second image data, a second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is calculated, specifically as follows:

[0082] Based on the second image data, the second plane equation of the calibration template is obtained; the projection of the interior corner point of the calibration template onto the second image data is connected to the origin of the second camera coordinate system, and the coordinates of the intersection of the connecting line and the second plane equation are used as the coordinate representation of the interior corner point of the calibration template in the camera coordinate system; based on the representation of the interior corner point of the calibration template in the calibration template coordinate system, the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is obtained.

[0083] It should be noted that RGB images and depth maps of the ground are acquired using an RGBD sensor, and the ground point cloud is fitted to the X-axis of the camera coordinate system. c2Y c2 Z c2 The equation of the plane is A²x + B²y + C²z + D² = 0. Similarly, based on the pinhole imaging model, the point O appearing on the RGB... F The intersection of the ray connecting the camera coordinate system origin to the plane A2x+B2y+C2z+D2=0 yields O. F The point is in the camera coordinate system X c2 Y c2 Z c2 The coordinates are denoted as (x oFc2 ,y oFc2 ,z oFc2 It can also calculate the X coordinate system of the calibration template when the forklift is in position 2. b Y b Z b With respect to the camera coordinate system X c2 Y c2 Z c2 The transformation relationship is shown in formula (2).

[0084]

[0085] Step S103: Based on the first transformation relationship and the second transformation relationship, obtain the third transformation relationship between the first camera coordinate system and the second camera coordinate system.

[0086] As a preferred embodiment, based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained, specifically as follows:

[0087] Based on the first transformation relationship and the second transformation relationship, the third transformation relationship between the first camera coordinate system and the second camera coordinate system is calculated when the forklift is located at the first position and the second position, respectively.

[0088] In this embodiment, according to formulas (1) and (2), the camera coordinate system X can be calculated when the forklift is in the second position 2 and the first position 1. c2 Y c2 Z c2 and camera coordinate system X c1 Y c1 Z c1 The transformation relationship is shown in formula (3).

[0089]

[0090] Step S104: Based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, obtain the coordinates of the origins of the first and second positions in the first camera coordinate system, and then obtain the direction vector of the forklift movement direction in the first camera coordinate system, and the translation matrix from the forklift coordinate system to the camera coordinate system; wherein, the first position is the position before the forklift moves in a straight line, and the second position is the position after the forklift moves in a straight line.

[0091] As a preferred embodiment, the steps involve obtaining the coordinates of the origins of the first and second positions in the first camera coordinate system based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, respectively, and then obtaining the direction vector of the forklift's movement direction in the first camera coordinate system, as well as the translation matrix from the forklift coordinate system to the camera coordinate system. Specifically:

[0092] Based on the first origin of the forklift coordinate system at the first position, calculate the coordinates of the first origin in the first camera coordinate system, thus obtaining the translation matrix from the forklift coordinate system to the camera coordinate system; based on the origins of the forklift coordinate system at the first and second positions, obtain the forklift displacement distance, and based on the forklift displacement distance, obtain the coordinates of the second origin in the first camera coordinate system; based on the coordinates of the first origin and the second origin in the first camera coordinate system, calculate the direction vector of the forklift's movement direction in the first camera coordinate system.

[0093] In this embodiment, the camera coordinate system X... c2 Y c2 Z c2 and camera coordinate system X c1 Y c1 Z c1 By understanding the transformation relationship, the origin O of the forklift can be determined. F In the camera coordinate system X c1 Y c1 Z c1 The coordinates P below oFc1 =(x oFc1 ,y oFc1 ,z oFc1 Thus, the translation matrix T from the forklift coordinate system to the camera coordinate system is obtained. * =(x oFc1 ,y oFc1 ,z oFc1 ) T .

[0094] It should be noted that you should refer to [link / reference]. Figure 4 Since the forklift traveled a certain distance in a straight line, the first camera's coordinate system X... c1 Y c1 Zc1 Second camera coordinate system X c2 Y c2 Z c2 Within space, only translation between forward and backward positions exists; point O F′ With point O F The distance between them is equal to the distance the forklift travels, therefore point O... F′ In the camera coordinate system X c1 Y c1 Z c1 The coordinates below must be equal to point O. F In the camera coordinate system X c2 Y c2 Z c2 The coordinates, i.e. (x oF′c1 ,y oF′c1 ,z oF′c1 )=(x oFc2 ,y oFc2 ,z oFc2 This is equivalent to obtaining two points O. F and O F′ In the camera coordinate system X c1 Y c1 Z c1 The coordinates of the forklift can be obtained by subtracting the coordinates of the two points. F In the camera coordinate system X c1 Y c1 Z c1 The direction vector below, i.e.

[0095] Step S105: Based on the direction vector of the forklift coordinate system in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, calculate the rotation matrix of the forklift coordinate system in the camera coordinate system, and then obtain the extrinsic parameters of the camera.

[0096] As a preferred embodiment, the method for obtaining the normal vector of the ground plane in the first camera coordinate system specifically includes:

[0097] The ground point cloud data based on the first camera coordinate system is acquired and fitted to obtain the ground plane equation in the camera coordinate system, and then the normal vector of the ground plane in the first camera coordinate system is obtained.

[0098] It should be noted that, in this embodiment, exemplarily, when the forklift is stopped at position 1, the RGB and depth images of the ground are acquired by the RGBD sensor, assuming point O on the ground... F It did not appear in the RGB image field of view. Based on the camera's intrinsic parameters, the ground information was converted into 3D point cloud data based on the camera coordinate system using formula (1):

[0099]

[0100] Plane fitting is performed on the ground point cloud to obtain the representation of the ground in the camera coordinate system X. c1 Y c1 Z c1 The equation of the plane is A1x + B1y + C1z + D1 = 0. Therefore, the plane of the ground in the camera coordinate system X is obtained. c1 Y c1 Z c1 The normal vector under is n1 = (A1, B1, C1). T .

[0101] As another preferred embodiment, when the forklift is in the first position 1, it is assumed that point O on the ground... F When the forklift appears in the RGB image field of view, i.e., at position 1, after fitting the equation A1x+B1y+C1z+D1=0 to the ground, point O can be directly calculated based on the pinhole imaging principle. F In the camera coordinate system X c1 Y c1 Z c1 The coordinates P below oFc1 =(x oFc1 ,y oFc1 ,z oFc1 ).

[0102] As a preferred embodiment, the step of calculating the rotation matrix of the forklift coordinate system in the camera coordinate system based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, and obtaining the camera's extrinsic parameters based on the camera's rotation and translation matrices, specifically:

[0103] Based on the normal vector of the ground plane in the first camera coordinate system, normalize each direction vector of the forklift coordinate system in the first camera coordinate system to obtain the rotation matrix from the forklift coordinate system to the camera coordinate system; based on the translation matrix from the forklift coordinate system to the camera coordinate system and the rotation matrix from the forklift coordinate system to the camera coordinate system, calculate the extrinsic parameters from the forklift coordinate system to the camera coordinate system, and then obtain the extrinsic parameters of the camera.

[0104] It should be noted that, in this embodiment, in the first camera coordinate system X... c1 Y c1 Z c1 Below, the ground normal vector n1 = (A1, B1, C1) T =X FC With the forklift's forward vector n2 = Z FC The cross product yields Y that satisfies the right-hand rule. FC The forklift coordinate system X was obtained. FCY FC Z FC The direction vector in each direction in the first camera coordinate system, when normalized, becomes the rotation matrix from the forklift coordinate system to the first camera coordinate system. After obtaining the X coordinate system from the forklift coordinate system to the camera coordinate system c1 Y c1 Z c1 After obtaining the rotation matrix, the translation matrix from the forklift coordinate system to the camera coordinate system can be used to finally obtain the extrinsic parameters from the forklift coordinate system to the camera coordinate system.

[0105] In another embodiment of the present invention, please refer to Figure 5 By acquiring corresponding image data when the forklift is in the first and second positions, respectively, the transformation relationship between the first and second camera coordinate systems and the calibration template coordinate system is calculated. This transformation relationship is then used to calculate the coordinates of the origin of the forklift coordinate system in the first camera coordinate system at the first and second positions, respectively. This allows for the calculation of the direction vector of the forklift's movement in the first camera coordinate system and the translation matrix from the forklift coordinate system to the camera coordinate system. Furthermore, based on the normal vector of the ground plane in the first camera coordinate system, the rotation matrix of the forklift coordinate system in the camera coordinate system is obtained. Finally, based on the camera's rotation and translation matrices, the extrinsic parameters of the camera coordinate system are accurately and efficiently obtained.

[0106] Understandably, in order for unmanned forklifts to accurately analyze the position and attitude of pallets or trolleys using 3D point cloud data based on the RGBD camera coordinate system, it is necessary to transform the recognition results of the RGBD sensor into the forklift coordinate system. This invention, without introducing human measurement errors, uses only a checkerboard calibration plate in conjunction with the linear movement of the forklift to quickly and accurately estimate the extrinsic parameters of the RGBD sensor relative to the unmanned forklift coordinate system, thereby effectively ensuring the accuracy and robustness of the unmanned forklift in picking up pallets or trolleys.

[0107] Furthermore, this embodiment of the invention has low requirements for the placement of the calibration template, does not require strict correlation with the forklift coordinate system, and does not require measurement of the forklift's forward travel distance. This fundamentally eliminates the errors introduced by manual measurement of relevant parameters, thus achieving high calibration accuracy. In addition, the method implemented in this embodiment is relatively simple, does not require rigorous training for calibration operators, and is easy to mass-produce.

[0108] Implementing the above embodiments has the following effects:

[0109] Compared with existing technologies, the technical solution of this invention can perform calibration during the straight-line movement of the forklift. In actual forklift operation, it can significantly improve the efficiency of external parameter calibration. Furthermore, the placement of the calibration template is less critical during the entire calibration process, and it does not need to be strictly correlated with the forklift coordinate system. Moreover, the distance the forklift moves forward does not need to be measured, which fundamentally eliminates the error introduced by manual measurement of relevant parameters and reduces the complexity of operation, thereby achieving high precision in calibration.

[0110] Example 2

[0111] Please see Figure 6 The present invention provides an extrinsic parameter calibration device based on the linear motion of a forklift, comprising: a first position module 201, a second position module 202, a transformation relationship module 203, a direction vector module 204, and an extrinsic parameter module 205.

[0112] The first position module 201 is used to acquire first image data after aligning the first RGB image and the first depth image of the calibration template when the forklift is in the first position, and to calculate the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data.

[0113] The second position module 202 is used to acquire second image data after aligning the second RGB image and the second depth image of the calibration template when the forklift is in the second position, and to calculate the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template based on the second image data.

[0114] The transformation relationship module 203 is used to obtain a third transformation relationship between the first camera coordinate system and the second camera coordinate system based on the first transformation relationship and the second transformation relationship.

[0115] The direction vector module 204 is used to obtain the coordinates of the origins of the first position and the second position in the first camera coordinate system according to the third transformation relationship and the origins of the forklift coordinate system at the first position and the second position respectively, and then obtain the direction vector of the forklift movement direction in the first camera coordinate system, and the translation matrix from the forklift coordinate system to the camera coordinate system; wherein, the first position is the position before the forklift moves in a straight line, and the second position is the position after the forklift moves in a straight line.

[0116] The extrinsic parameter module 205 is used to calculate the rotation matrix of the forklift coordinate system in the camera coordinate system based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, and to obtain the extrinsic parameters of the camera based on the camera's rotation matrix and translation matrix.

[0117] In a preferred embodiment, the step of calculating the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template based on the first image data specifically involves:

[0118] Based on the first image data, the first plane equation of the calibration template is obtained; the projection of the interior corner point of the calibration template onto the first image data is connected to the origin of the first camera coordinate system, and the coordinates of the intersection of the connecting line and the first plane equation are used as the coordinate representation of the interior corner point of the calibration template in the camera coordinate system; based on the representation of the interior corner point of the calibration template in the calibration template coordinate system, the first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template is obtained.

[0119] As a preferred embodiment, based on the second image data, a second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is calculated, specifically as follows:

[0120] Based on the second image data, the second plane equation of the calibration template is obtained; the projection of the interior corner point of the calibration template onto the second image data is connected to the origin of the second camera coordinate system, and the coordinates of the intersection of the connecting line and the second plane equation are used as the coordinate representation of the interior corner point of the calibration template in the camera coordinate system; based on the representation of the interior corner point of the calibration template in the calibration template coordinate system, the second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is obtained.

[0121] As a preferred embodiment, based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained, specifically as follows:

[0122] Based on the first transformation relationship and the second transformation relationship, the third transformation relationship between the first camera coordinate system and the second camera coordinate system is calculated when the forklift is located at the first position and the second position, respectively.

[0123] As a preferred embodiment, the steps involve obtaining the coordinates of the origins of the first and second positions in the first camera coordinate system based on the third transformation relationship and the origins of the forklift coordinate system at the first and second positions, respectively, and then obtaining the direction vector of the forklift's movement direction in the first camera coordinate system, as well as the translation matrix from the forklift coordinate system to the camera coordinate system. Specifically:

[0124] Based on the third transformation relationship and the first origin of the forklift coordinate system at the first position, the coordinates of the first origin in the first camera coordinate system are calculated, thereby obtaining the translation matrix from the forklift coordinate system to the camera coordinate system; based on the origins of the forklift coordinate system at the first and second positions, the forklift displacement distance is obtained, and based on the forklift displacement distance, the coordinates of the second origin in the first camera coordinate system are obtained; based on the coordinates of the first origin in the first camera coordinate system and the coordinates of the second origin in the first camera coordinate system, the direction vector of the forklift movement direction in the first camera coordinate system is calculated.

[0125] As a preferred embodiment, the method for obtaining the normal vector of the ground plane in the first camera coordinate system specifically includes:

[0126] The ground point cloud data based on the first camera coordinate system is acquired and fitted to obtain the ground plane equation in the camera coordinate system, and then the normal vector of the ground plane in the first camera coordinate system is obtained.

[0127] As a preferred embodiment, the step of calculating the rotation matrix of the forklift coordinate system in the camera coordinate system based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, and obtaining the camera's extrinsic parameters based on the camera's rotation and translation matrices, specifically:

[0128] Based on the normal vector of the ground plane in the first camera coordinate system, normalize each direction vector of the forklift coordinate system in the first camera coordinate system to obtain the rotation matrix from the forklift coordinate system to the camera coordinate system; based on the translation matrix from the forklift coordinate system to the camera coordinate system and the rotation matrix from the forklift coordinate system to the camera coordinate system, calculate the extrinsic parameters from the forklift coordinate system to the camera coordinate system, and then obtain the extrinsic parameters of the camera.

[0129] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] Implementing the embodiments of the present invention has the following effects:

[0131] The embodiments of the present invention can perform calibration during the straight-line movement of the forklift, which can significantly improve the efficiency of external parameter calibration in actual forklift operation. Furthermore, the entire calibration process has low requirements for the placement of the calibration template, does not need to be strictly correlated with the forklift coordinate system, and does not require measurement of the distance the forklift moves forward. This fundamentally eliminates the error introduced by manual measurement of relevant parameters to the calibration results and reduces the complexity of operation, thereby achieving high precision in calibration.

[0132] Example 3

[0133] Accordingly, the present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the extrinsic parameter calibration method based on the linear motion of a forklift as described in any of the above embodiments.

[0134] The terminal device in this embodiment includes a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps described in Embodiment 1 above, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiment, such as the extrinsic parameter module 205.

[0135] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device. For example, the extrinsic parameter module 205 is used to calculate the rotation matrix of the forklift coordinate system in the camera coordinate system based on the direction vector of the forklift's movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, and to obtain the extrinsic parameters of the camera based on the camera's rotation and translation matrices.

[0136] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0137] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0138] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0139] Wherein, if the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0140] Example 4

[0141] Accordingly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the extrinsic parameter calibration method based on the linear motion of a forklift as described in any of the above embodiments.

[0142] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for calibrating the external parameters based on the linear motion of a forklift, characterized in that, The method comprises the following steps: When the forklift is at a first position, first image data of a first RGB image and a first depth image of a calibration template after alignment is obtained, and a first transformation relationship from a first camera coordinate system to a coordinate system of the calibration template is calculated according to the first image data; wherein the first camera coordinate system is a coordinate system corresponding to a camera arranged at a preset position of the forklift when the forklift is at the first position; When the forklift is at a second position, second image data of a second RGB image and a second depth image of the calibration template after alignment is obtained, and a second transformation relationship from a second camera coordinate system to the coordinate system of the calibration template is calculated according to the second image data; wherein the second camera coordinate system is a coordinate system corresponding to a camera arranged at a preset position of the forklift when the forklift is at the second position; According to the first transformation relationship and the second transformation relationship, a third transformation relationship of the first camera coordinate system and the second camera coordinate system is obtained; According to the third transformation relationship and the origins of the forklift coordinate system at the first position and the second position respectively, coordinates of the origins of the first position and the second position in the first camera coordinate system are obtained, and then a direction vector of a moving direction of the forklift in the first camera coordinate system and a translation matrix of the forklift coordinate system to the camera coordinate system are obtained; wherein the first position is a position before the forklift moves linearly, and the second position is a position after the forklift moves linearly; According to the direction vector of the moving direction of the forklift in the first camera coordinate system and a normal vector of a ground plane in the first camera coordinate system, a rotation matrix of the forklift coordinate system in the camera coordinate system is calculated, and the extrinsic parameters of the camera are obtained according to the rotation matrix and the translation matrix of the camera.

2. The method for calibrating the external parameters based on the linear motion of the forklift according to claim 1, wherein, The first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template is calculated according to the first image data, and specifically comprises the following steps: A first plane equation of the calibration template is obtained according to the first image data; A projection of an internal corner point of the calibration template in the first image data is connected with an origin of the first camera coordinate system, and a coordinate of an intersection point of the connection line and the first plane equation is taken as a coordinate representation of the internal corner point of the calibration template in the camera coordinate system; The first transformation relationship from the first camera coordinate system to the coordinate system of the calibration template is obtained according to the representation of the internal corner point of the calibration template in the calibration template coordinate system.

3. The method of calibrating the external parameters based on the linear motion of the forklift according to claim 1, wherein, The second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is calculated according to the second image data, and specifically comprises the following steps: A second plane equation of the calibration template is obtained according to the second image data; A projection of an internal corner point of the calibration template in the second image data is connected with an origin of the second camera coordinate system, and a coordinate of an intersection point of the connection line and the second plane equation is taken as a coordinate representation of the internal corner point of the calibration template in the camera coordinate system; The second transformation relationship from the second camera coordinate system to the coordinate system of the calibration template is obtained according to the representation of the internal corner point of the calibration template in the calibration template coordinate system.

4. The method of calibrating the external parameters based on the linear motion of the forklift according to claim 1, wherein, According to the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is obtained, specifically as follows: According to the first transformation relationship and the second transformation relationship, a third transformation relationship between the first camera coordinate system and the second camera coordinate system is calculated when the forklift is located at the first position and the second position respectively.

5. The method of calibrating the external parameters based on the linear motion of the forklift according to claim 1, wherein, According to the third transformation relationship and the origins of the forklift coordinate system at the first position and the second position respectively, coordinates of the origins of the first position and the second position in the first camera coordinate system are obtained, and then a direction vector of the forklift movement direction in the first camera coordinate system and a translation matrix of the forklift coordinate system to the camera coordinate system are obtained, specifically as follows: According to the third transformation relationship and the first origin of the forklift coordinate system at the first position, coordinates of the first origin in the first camera coordinate system are calculated, and then a translation matrix of the forklift coordinate system to the camera coordinate system is obtained; According to the origins of the forklift coordinate system at the first position and the second position respectively, a forklift displacement distance is obtained, and according to the forklift displacement distance, coordinates of the second origin in the first camera coordinate system are obtained; According to the coordinates of the first origin in the first camera coordinate system and the coordinates of the second origin in the first camera coordinate system, a direction vector of the forklift movement direction in the first camera coordinate system is calculated.

6. The method for calibrating the external parameters based on the linear motion of the forklift according to claim 5, wherein, The method for obtaining the normal vector of the ground plane in the first camera coordinate system specifically includes: The ground plane equation in the camera coordinate system is obtained by obtaining and fitting the ground point cloud data based on the first camera coordinate system, and then the normal vector of the ground plane in the first camera coordinate system is obtained.

7. The method of calibrating the external parameters based on the linear motion of the forklift according to claim 6, wherein, According to the direction vector of the forklift movement direction in the first camera coordinate system and the normal vector of the ground plane in the first camera coordinate system, a rotation matrix of the forklift coordinate system in the camera coordinate system is calculated, and then the extrinsic parameters of the camera are obtained according to the rotation matrix and the translation matrix of the camera, specifically as follows: According to the normal vector of the ground plane in the first camera coordinate system, each direction vector of the forklift coordinate system in the first camera coordinate system is normalized to obtain a rotation matrix of the forklift coordinate system to the camera coordinate system; According to the translation matrix of the forklift coordinate system to the camera coordinate system and the rotation matrix of the forklift coordinate system to the camera coordinate system, the extrinsic parameters of the forklift coordinate system to the camera coordinate system are calculated, and then the extrinsic parameters of the camera are obtained.

8. A device for calibrating the external parameters based on the linear motion of a forklift, characterized in that, It includes: The first position module, the second position module, the transformation relationship module, the direction vector module and the extrinsic parameter module; The first position module is configured to, when the forklift is at the first position, obtain first image data obtained by aligning a first RGB image and a first depth image of a calibration template, and calculate a first transformation relationship between a first camera coordinate system and a coordinate system of the calibration template according to the first image data; wherein the first camera coordinate system is a coordinate system corresponding to a camera arranged at a preset position of the forklift when the forklift is at the first position. The second position module is configured to, when the forklift is at the second position, acquire second image data obtained by aligning a second RGB image and a second depth image of the calibration template, and calculate a second transformation relationship from a second camera coordinate system to a coordinate system of the calibration template according to the second image data; the second camera coordinate system is a coordinate system corresponding to a camera arranged at a preset position of the forklift when the forklift is at the second position. The transformation relationship module is configured to obtain a third transformation relationship between the first camera coordinate system and the second camera coordinate system according to the first transformation relationship and the second transformation relationship. The direction vector module is configured to obtain coordinates of origins of the first position and the second position in the first camera coordinate system according to the third transformation relationship and the origins of the forklift coordinate system at the first position and the second position, respectively, and further obtain a direction vector of the moving direction of the forklift in the first camera coordinate system and a translation matrix from the forklift coordinate system to the camera coordinate system; the first position is a position before the forklift moves linearly, and the second position is a position after the forklift moves linearly. The extrinsic parameter module is configured to calculate a rotation matrix of the forklift coordinate system in the camera coordinate system according to the direction vector of the moving direction of the forklift in the first camera coordinate system and a normal vector of the ground plane in the first camera coordinate system, and obtain the extrinsic parameters of the camera according to the rotation matrix and the translation matrix of the camera.

9. A terminal device, comprising: The computer readable storage medium comprises a computer program stored therein; when the computer program runs, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the extrinsic parameter calibration method based on linear motion of a forklift according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a computer program stored therein; when the computer program runs, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the extrinsic parameter calibration method based on linear motion of a forklift according to any one of claims 1-7.

Images