A method and device for external parameter calibration based on calibration template

Abstract

The present invention discloses a method and device for extrinsic parameter calibration based on a calibration template, comprising: collecting an RGB image and a depth map of the calibration template; aligning the RGB image and the depth map, and extracting inner corners of the RGB image based on a corner extraction algorithm; obtaining depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculating the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information; obtaining the coordinates of each inner corner point in the forklift coordinate system based on the relationship that the projection of a preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system; and calculating the calibration extrinsic parameters of the camera based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system. The present invention solves the technical problems in the prior art of the complex calibration process of the calibration template and the low accuracy and robustness of the extrinsic parameter calibration.

Description

Technical Field

[0001] The present invention relates to the technical field of RGBD camera calibration, and in particular to an extrinsic parameter calibration method and device based on a calibration template. Background Art

[0002] The rapid development and engineering implementation of Simultaneous Localization and Mapping (SLAM) technology has enabled the widespread application of autonomous driving technology in unmanned warehouses, smart factories, and intelligent logistics, gradually enabling intelligent robots to replace human labor and operations. Intelligent unmanned forklifts, a specific form of intelligent robots, play a significant role in the automated handling and transfer of freight on factory floors.

[0003] The existing forklift is calibrated using the RGBD camera extrinsic parameter calibration method, which requires three mutually perpendicular black and white checkerboard templates. The RGBD camera collects data and proposes the planes of the three calibration templates. The sum of the distances between the point set of the checkerboard corner feature points and the three planes is calculated, and the extrinsic parameters of the RGBD camera are obtained by gradient inference of the posture. However, a large number of calibration templates are used, and the placement of the calibration templates needs to be perpendicular to each other, which is a high requirement and the calibration environment is relatively difficult to meet. At the same time, the checkerboard corners on the RGB image in the calibration plate coordinate system are detected, and the three-dimensional coordinates of the corners in the depth camera coordinate system are calculated, thereby calculating the transformation relationship between the camera coordinate system and the calibration plate coordinate system. However, only the transformation relationship from the depth camera coordinate system to the calibration plate coordinate system is calculated. How to calculate the transformation relationship from the calibration plate to the robot and how to reduce the calibration error are not involved. In addition, the calculation complexity is high, and the accuracy and stability of the calibration results cannot be guaranteed.

[0004] Therefore, there is an urgent need for an external parameter calibration method that can simplify the calibration template calibration process and improve accuracy and robustness. Summary of the Invention

[0005] The present invention provides an extrinsic parameter calibration method and device based on a calibration template, so as to solve the technical problems in the prior art of complex calibration template calibration process and low accuracy and robustness of extrinsic parameter calibration.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides an extrinsic parameter calibration method based on a calibration template, comprising:

[0007] Collecting an RGB image and a depth map of a calibration template; wherein the projection of a preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system;

[0008] Aligning the RGB image and the depth map, and extracting inner corners of the RGB image according to a corner extraction algorithm;

[0009] Obtaining depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculating the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information;

[0010] Obtaining the coordinates of each inner corner point in the forklift coordinate system based on the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system;

[0011] The camera's calibration extrinsic parameters are calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system.

[0012] As a preferred solution, the depth information corresponding to each inner corner point extracted from the RGB image is obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information, specifically:

[0013] Acquire depth information corresponding to each inner corner point on the depth map according to the coordinates of the inner corner points of the calibration template;

[0014] The coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera.

[0015] As a preferred solution, before obtaining the depth information corresponding to each inner corner point on the depth map, the method further includes:

[0016] Perform plane fitting on the point cloud of the plane where the calibration template is located to obtain a plane equation of the point cloud of the calibration template fitted in the camera coordinate system.

[0017] As a preferred solution, the coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera, specifically:

[0018] According to the preset internal parameters of the camera, the pinhole imaging formula of the camera is obtained;

[0019] The coordinates of each inner corner point in the camera coordinate system are calculated according to the plane equation, the pinhole imaging formula, each inner corner point and its corresponding depth information.

[0020] As a preferred solution, the coordinates of each inner corner point in the forklift coordinate system are obtained according to the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, specifically:

[0021] The distance from the preset corner point to the ground is measured based on the coincidence between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system;

[0022] The coordinates of each inner corner on the calibration template in the forklift coordinate system are calculated based on the distance from the preset corner point to the ground and the preset size of the calibration template.

[0023] As a preferred solution, the calibration extrinsic parameters of the camera are calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, specifically:

[0024] According to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, the optimal transformation matrix model of the corresponding point cloud is obtained;

[0025] The optimal transformation matrix model is solved according to the iterative closest point algorithm to obtain the calibration extrinsic parameters of the camera.

[0026] As a preferred solution, the calibration template maintains the pitch angle, yaw angle and roll angle to be 0 in the forklift coordinate system.

[0027] Accordingly, the present invention also provides an external parameter calibration device based on a calibration template, comprising: an acquisition module, an inner corner point extraction module, a camera coordinate module, a forklift coordinate module and an external parameter calibration module;

[0028] The acquisition module is used to acquire the RGB image and depth map of the calibration template; wherein the projection of the preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system;

[0029] The inner corner point extraction module is used to align the RGB image and the depth map, and extract the inner corner points of the RGB image according to a corner point extraction algorithm;

[0030] The camera coordinate module is used to obtain the depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculate the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information;

[0031] The forklift coordinate module is used to obtain the coordinates of each inner corner point in the forklift coordinate system based on the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system;

[0032] The extrinsic parameter calibration module is used to calculate the extrinsic parameters of the camera according to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system.

[0033] As a preferred solution, the depth information corresponding to each inner corner point extracted from the RGB image is obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information, specifically:

[0034] Acquire depth information corresponding to each inner corner point on the depth map according to the coordinates of the inner corner points of the calibration template;

[0035] The coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera.

[0036] As a preferred solution, before obtaining the depth information corresponding to each inner corner point on the depth map, the method further includes:

[0037] Perform plane fitting on the point cloud of the plane where the calibration template is located to obtain a plane equation of the point cloud of the calibration template fitted in the camera coordinate system.

[0038] As a preferred solution, the coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera, specifically:

[0039] According to the preset internal parameters of the camera, the pinhole imaging formula of the camera is obtained;

[0040] The coordinates of each inner corner point in the camera coordinate system are calculated according to the plane equation, the pinhole imaging formula, each inner corner point and its corresponding depth information.

[0041] As a preferred solution, the coordinates of each inner corner point in the forklift coordinate system are obtained according to the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, specifically:

[0042] The distance from the preset corner point to the ground is measured based on the coincidence between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system;

[0043] The coordinates of each inner corner on the calibration template in the forklift coordinate system are calculated based on the distance from the preset corner point to the ground and the preset size of the calibration template.

[0044] As a preferred solution, the calibration extrinsic parameters of the camera are calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, specifically:

[0045] According to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, the optimal transformation matrix model of the corresponding point cloud is obtained;

[0046] The optimal transformation matrix model is solved according to the iterative closest point algorithm to obtain the calibration extrinsic parameters of the camera.

[0047] As a preferred solution, the calibration template maintains the pitch angle, yaw angle and roll angle to be 0 in the forklift coordinate system.

[0048] Correspondingly, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the external parameter calibration method based on the calibration template as described in any one of the above items when executing the computer program.

[0049] Accordingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the external parameter calibration method based on the calibration template as described in any one of the above items.

[0050] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0051] The technical solution of the present invention collects the RGB image and depth map of the calibration template, and after aligning the RGB image and the depth map, extracts the inner corner points of the RGB image, so that the depth information corresponding to each inner corner point extracted from the RGB image can be accurately obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information. At the same time, according to the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, the coordinates of each inner corner point in the forklift coordinate system are obtained, so that the calibration extrinsic parameters of the camera can be accurately calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, thereby realizing a simple and fast calibration method and making the extrinsic parameter calibration results highly accurate and robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of the steps of an extrinsic parameter calibration method based on a calibration template provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a forklift coordinate system and a camera coordinate system provided in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the external parameter calculation of the forklift coordinate system and the camera coordinate system provided in an embodiment of the present invention;

[0055] Figure 4 A flowchart of an extrinsic parameter calibration method based on a calibration template provided in another embodiment of the present invention;

[0056] Figure 5 This is a schematic structural diagram of an extrinsic parameter calibration device based on a calibration template provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] As a smart forklift's position constantly changes throughout its workspace, it needs to maintain real-time awareness of its changing dynamic and static environments, enabling it to automatically make informed decisions. Sensors installed on smart forklifts, such as vision sensors, wheel speedometers, and lidar, are crucial components in helping them perceive their external environment.

[0059] In an embodiment of the present invention, an RGBD sensor is used. As a visual sensor widely used on intelligent forklifts to perceive the external environment, the RGBD sensor adds one-dimensional depth information to a two-dimensional color image, thereby acquiring environmental data surrounding the intelligent forklift. This data is based on three-dimensional point cloud data in the camera coordinate system. Processing this point cloud data also yields results based on the camera coordinate system. To integrate the results of the RGBD sensor's analysis and processing into the intelligent forklift's perception of its surroundings, the intelligent forklift must understand how these camera-based analysis results are represented in its own coordinate system. This means transforming the camera coordinate system's analysis results into the intelligent forklift's body coordinate system.

[0060] Example 1

[0061] Please refer to Figure 1 , an extrinsic parameter calibration method based on a calibration template provided by an embodiment of the present invention, comprising the following steps S101-S105:

[0062] Step S101: collecting an RGB image and a depth map of a calibration template; wherein the projection of a preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system.

[0063] It should be noted that, preferably, the calibration template is a black and white checkerboard. In this embodiment, the RGB image and depth map of the calibration template are both collected and acquired by an RGBD sensor camera installed on a forklift.

[0064] In this example, see Figure 2 , which is the definition of the corresponding camera coordinate system and forklift coordinate system; camera coordinate system Z C The positive direction is the positive direction of the camera optical axis, X C The positive direction is from left to right, Y C The positive direction satisfies the right-hand rule, and the camera coordinate system is (X C ,Y C ,X C ); The origin of the forklift body coordinate system and the geometric center of the four wheels of the forklift fork end are on the ground, X F The positive direction is the direction of the forklift moving forward, Y F The positive direction is from left to right, Z F The positive direction satisfies the right-hand rule, the vertical top is upward, and the forklift coordinate system is (X F ,Y F ,Z F ).

[0065] As a preferred solution of this embodiment, the calibration template maintains the pitch angle, yaw angle and roll angle to be 0 in the forklift coordinate system.

[0066] For example, see Figure 3 , make the projection of the sixth column of inner corner points of the calibration template (preferably the column of inner corner points closest to the center of the calibration template) on the ground coincide with the origin of the forklift coordinate system, and try to keep the pitch angle, yaw angle and roll angle of the calibration template in the forklift coordinate system all zero.

[0067] Step S102: aligning the RGB image and the depth map, and extracting inner corners of the RGB image according to a corner extraction algorithm.

[0068] It should be noted that when the RGBD sensor collects image data, it has already completed the alignment of the RGB image and the depth map internally, and can directly use the corner extraction algorithm to extract the inner corner points of the RGB image. Preferably, the Harris algorithm is used to extract the inner corner points of the calibration template on the RGB image.

[0069] Step S103: obtaining depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculating the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information.

[0070] As a preferred solution of this embodiment, the depth information corresponding to each inner corner point extracted from the RGB image is obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information, specifically:

[0071] According to the coordinates of the inner corner points of the calibration template, the depth information corresponding to each inner corner point is obtained on the depth map; according to the depth information corresponding to each inner corner point and the preset internal parameters of the camera, the coordinates of each inner corner point in the camera coordinate system are calculated.

[0072] In this embodiment, based on the coordinates of each inner corner point of the checkerboard in the calibration template, the depth information z corresponding to each inner corner point is obtained on the depth map. ci , when the camera's intrinsic parameter matrix K is known, calculate the coordinates of each inner corner point in the camera coordinate system (x ci ,y ci ,z ci ), that is, the camera's internal parameter matrix K is used to obtain the camera's pinhole imaging formula, and the camera's pinhole imaging formula is used to calculate the coordinates of each inner corner point in the camera coordinate system:

[0073]

[0074] It should be noted that in the specific implementation process, it is common that there are holes in the point cloud or the position depth of the corresponding points of the inner corner points is incorrect, which will cause errors in the point calculation of the inner corner points in the camera coordinate system, resulting in large calculation errors and inaccurate coordinates of the corresponding inner corner points.

[0075] As a preferred solution of this embodiment, before obtaining the depth information corresponding to each inner corner point on the depth map, the method further includes:

[0076] Perform plane fitting on the point cloud of the plane where the calibration template is located to obtain a plane equation of the point cloud of the calibration template fitted in the camera coordinate system.

[0077] Specifically, according to the preset internal parameters of the camera, the pinhole imaging formula (1) of the camera is obtained; according to the plane equation, the pinhole imaging formula, each inner corner point and its corresponding depth information, the coordinates of each inner corner point in the camera coordinate system are calculated.

[0078] In this embodiment, the plane equation of the calibration template point cloud in the camera coordinate system is Ax+By+Cz+D=0, and it is assumed that the coordinates of any inner corner point after the chessboard is imaged on the RGB image are (u i ,v i ), the depth corresponding to the inner corner point is z i According to formula (1), the coordinates of this corner point on the chessboard in the camera coordinate system can be obtained as With z i The change of represents the ray starting from the origin of the camera coordinate system and passing through the corresponding inner corner point of the RGB image. The intersection of this ray and the fitting plane is the point (xci ,y ci ,z ci ), i = 1, 2, ..., N. Similarly, the same method can be used to obtain other corner points on the calibration template.

[0079] It can be understood that in order to solve the problem of incorrect point calculation of the above-mentioned inner corner points in the camera coordinate system, through this preferred solution, the point cloud of the plane where the template is located is plane fitted. In order to increase robustness, the RANSAC algorithm is used for plane fitting, thereby avoiding the problem of holes in the point cloud or incorrect position depth of the directly obtained corresponding points of the inner corner points.

[0080] Step S104: obtaining the coordinates of each inner corner point in the forklift coordinate system according to the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system.

[0081] As a preferred solution of this embodiment, the coordinates of each inner corner point in the forklift coordinate system are obtained according to the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, specifically:

[0082] Based on the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, the distance from the preset corner point to the ground is measured; based on the distance from the preset corner point to the ground and the preset size of the calibration template, the coordinates of each inner corner on the calibration template in the forklift coordinate system are calculated.

[0083] It should be noted that, since the projection of the preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system, the distance from the preset corner point in the sixth column of the calibration template to the ground is the distance to the origin of the forklift coordinate system. By manually measuring, the parameters of the distance from the preset corner point to the ground can be accurately obtained. As a result, the embodiment of the present invention only measures the parameters of the distance from one preset corner point to the ground in step S104, thereby achieving a simple and fast calibration method, easy operation and mass production of products.

[0084] In this embodiment, based on the placement of the current calibration template and the actual physical size of the calibration template checkerboard, the coordinates (x Fi ,y Fi ,z Fi ),i=1,2,…,N。

[0085] Step S105: Calculate the calibration extrinsic parameters of the camera according to the coordinates of each inner corner point in the camera coordinate system and the coordinates of each inner corner point in the forklift coordinate system.

[0086] As a preferred solution of this embodiment, the calibration extrinsic parameters of the camera are calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, specifically:

[0087] According to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, the optimal transformation matrix model of the corresponding point cloud is converted; according to the iterative closest point algorithm, the optimal transformation matrix model is solved to obtain the calibration extrinsic parameters of the camera.

[0088] Based on the above steps S103 and S104, the representation of each inner corner point of the calibration template in the camera coordinate system and the forklift coordinate system is obtained as (x ci ,y ci ,z ci ) and (x Fi ,y Fi ,z Fi ), which is equivalent to solving the optimal transformation matrix problem for two sets of known corresponding point clouds. The Iterative Closest Point (ICP) algorithm can be used to solve the transformation from the forklift coordinate system to the camera coordinate system, that is, Then solve the above formula to get the optimal solution, which is used as the calibration external parameter of the camera.

[0089] As can be seen, using a calibration template and selecting a suitable placement method, only a single parameter measurement is required to achieve extrinsic calibration of the RGBD sensor to the intelligent forklift's body coordinate system. This makes the calibration method simple and fast, easy to operate, and suitable for mass production. The calculation process fully utilizes the characteristics of the RGBD sensor's point cloud data to perform plane fitting and point cloud registration algorithms, resulting in highly accurate and robust extrinsic calibration results.

[0090] In this example, see Figure 4 , which is a flow of the extrinsic parameter calibration method implemented in another embodiment of the present invention. By placing the calibration template and measuring the distance from the preset inner corner point to the ground, the coordinates of the inner corner points of the calibration template in the forklift coordinate system can be accurately extracted. At the same time, the RGB image and depth map are collected, and the coordinates of the inner corner points of the calibration template in the camera coordinate system are extracted by extracting the inner corner points on the RGB image and fitting the point cloud plane of the calibration template. The external parameters of the RGBD sensor camera are accurately estimated through the ICP algorithm, thereby completing the calibration of the camera extrinsic parameters.

[0091] As another preferred solution of an embodiment of the present invention, during the placement of the calibration template, it is possible to not follow the principle that the projection of the inner corner point of the sixth column (center line) of the calibration template on the ground coincides with the origin of the forklift coordinate system, and it is possible to not follow the method of measuring the distance from the preset corner point to the ground. It is only necessary to obtain the coordinates of the inner corner point on the calibration template in the forklift coordinate system through manual testing, thereby realizing the calibration of the camera external parameters.

[0092] The implementation of the above embodiment has the following effects:

[0093] The technical solution of the present invention collects the RGB image and depth map of the calibration template, and after aligning the RGB image and the depth map, extracts the inner corner points of the RGB image, so that the depth information corresponding to each inner corner point extracted from the RGB image can be accurately obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information. At the same time, according to the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, the coordinates of each inner corner point in the forklift coordinate system are obtained, so that the calibration extrinsic parameters of the camera can be accurately calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, thereby realizing a simple and fast calibration method and making the extrinsic parameter calibration results highly accurate and robust.

[0094] Example 2

[0095] See also Figure 5 , which is an external parameter calibration device based on a calibration template provided by an embodiment of the present invention, including: an acquisition module 201, an inner corner point extraction module 202, a camera coordinate module 203, a forklift coordinate module 204 and a calibration external parameter module 205.

[0096] The acquisition module 201 is used to acquire the RGB image and depth map of the calibration template; wherein the projection of the preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system;

[0097] The inner corner point extraction module 202 is used to align the RGB image and the depth map, and extract inner corner points of the RGB image according to a corner point extraction algorithm;

[0098] The camera coordinate module 203 is configured to obtain, on the depth map, the depth information corresponding to each inner corner point extracted from the RGB image, and calculate the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information;

[0099] The forklift coordinate module 204 is configured to obtain the coordinates of each inner corner point in the forklift coordinate system based on the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system;

[0100] The extrinsic parameter calibration module 205 is used to calculate the extrinsic parameters of the camera according to the coordinates of each inner corner point in the camera coordinate system and the coordinates of each inner corner point in the forklift coordinate system.

[0101] As a preferred solution of this embodiment, the depth information corresponding to each inner corner point extracted from the RGB image is obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information, specifically:

[0102] According to the coordinates of the inner corner points of the calibration template, the depth information corresponding to each inner corner point is obtained on the depth map; according to the depth information corresponding to each inner corner point and the preset internal parameters of the camera, the coordinates of each inner corner point in the camera coordinate system are calculated.

[0103] As a preferred solution of this embodiment, before obtaining the depth information corresponding to each inner corner point on the depth map, the method further includes:

[0104] Perform plane fitting on the point cloud of the plane where the calibration template is located to obtain a plane equation of the point cloud of the calibration template fitted in the camera coordinate system.

[0105] As a preferred solution of this embodiment, the coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera, specifically:

[0106] According to the preset internal parameters of the camera, the pinhole imaging formula of the camera is obtained; according to the plane equation, the pinhole imaging formula, each inner corner point and its corresponding depth information, the coordinates of each inner corner point in the camera coordinate system are calculated.

[0107] As a preferred solution of this embodiment, the coordinates of each inner corner point in the forklift coordinate system are obtained according to the coincidence relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, specifically:

[0108] Based on the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, the distance from the preset corner point to the ground is measured; based on the distance from the preset corner point to the ground and the preset size of the calibration template, the coordinates of each inner corner on the calibration template in the forklift coordinate system are calculated.

[0109] As a preferred solution of this embodiment, the calibration extrinsic parameters of the camera are calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, specifically:

[0110] According to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, the optimal transformation matrix model of the corresponding point cloud is converted; according to the iterative closest point algorithm, the optimal transformation matrix model is solved to obtain the calibration extrinsic parameters of the camera.

[0111] As a preferred solution of this embodiment, the calibration template maintains the pitch angle, yaw angle and roll angle to be 0 in the forklift coordinate system.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0113] The implementation of the above embodiment has the following effects:

[0114] The technical solution of the present invention collects the RGB image and depth map of the calibration template, and after aligning the RGB image and the depth map, extracts the inner corner points of the RGB image, so that the depth information corresponding to each inner corner point extracted from the RGB image can be accurately obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information. At the same time, according to the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system, the coordinates of each inner corner point in the forklift coordinate system are obtained, so that the calibration extrinsic parameters of the camera can be accurately calculated based on the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, thereby realizing a simple and fast calibration method and making the extrinsic parameter calibration results highly accurate and robust.

[0115] Example 3

[0116] Correspondingly, the present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the external parameter calibration method based on the calibration template as described in any one of the above embodiments.

[0117] The terminal device of this embodiment includes: a processor, a memory, and a computer program and computer instructions stored in the memory and capable of running on the processor. When the processor executes the computer program, each step in the above embodiment 1 is implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiment, such as the camera coordinate module 203, are implemented.

[0118] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device. For example, the camera coordinate module 203 is used to obtain the depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculate the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information.

[0119] The terminal device may be a computing device such as a desktop computer, laptop, PDA, or cloud server. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a terminal device and does not limit the terminal device. The terminal device may include more or fewer components than shown, or a combination of certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, and the like.

[0120] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0121] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0122] If the module / unit integrated into the terminal device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process in the above-mentioned method embodiment by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0123] Example 4

[0124] Accordingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the external parameter calibration method based on the calibration template as described in any one of the above embodiments.

[0125] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A calibration template-based external parameter calibration method, characterized in that: include: Collecting an RGB image and a depth map of a calibration template; wherein the projection of a preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system; Aligning the RGB image and the depth map, and extracting inner corners of the RGB image according to a corner extraction algorithm; Obtaining depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculating the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information; The distance from the preset corner point to the ground is measured based on the coincidence between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system; Calculating the coordinates of each inner corner on the calibration template in the forklift coordinate system based on the distance from the preset corner point to the ground and the preset size of the calibration template; According to the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system, the optimal transformation matrix model of the corresponding point cloud is obtained; The optimal transformation matrix model is solved according to the iterative closest point algorithm to obtain the calibration extrinsic parameters of the camera.

2. The extrinsic parameter calibration method based on a calibration template according to claim 1, characterized in that: The depth information corresponding to each inner corner point extracted from the RGB image is obtained on the depth map, and the coordinates of each inner corner point in the camera coordinate system are calculated based on each inner corner point and its corresponding depth information, specifically: Acquire depth information corresponding to each inner corner point on the depth map according to the coordinates of the inner corner points of the calibration template; The coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera.

3. The external parameter calibration method based on the calibration template according to claim 2, characterized in that: Before obtaining the depth information corresponding to each inner corner point on the depth map, the method further includes: Perform plane fitting on the point cloud of the plane where the calibration template is located to obtain a plane equation of the point cloud of the calibration template fitted in the camera coordinate system.

4. The external parameter calibration method based on the calibration template according to claim 3, characterized in that: The coordinates of each inner corner point in the camera coordinate system are calculated based on the depth information corresponding to each inner corner point and the preset internal parameters of the camera, specifically: According to the preset internal parameters of the camera, the pinhole imaging formula of the camera is obtained; The coordinates of each inner corner point in the camera coordinate system are calculated according to the plane equation, the pinhole imaging formula, each inner corner point and its corresponding depth information.

5. The extrinsic parameter calibration method based on a calibration template according to any one of claims 1 to 4, characterized in that: The calibration template keeps the pitch angle, yaw angle and roll angle all zero in the forklift coordinate system.

6. An external parameter calibration device based on a calibration template, characterized in that: include: Acquisition module, interior corner extraction module, camera coordinate module, forklift coordinate module and external parameter calibration module; The acquisition module is used to acquire the RGB image and depth map of the calibration template; wherein the projection of the preset inner corner line of the calibration template coincides with the origin of the forklift coordinate system; The inner corner point extraction module is used to align the RGB image and the depth map, and extract the inner corner points of the RGB image according to a corner point extraction algorithm; The camera coordinate module is used to obtain the depth information corresponding to each inner corner point extracted from the RGB image on the depth map, and calculate the coordinates of each inner corner point in the camera coordinate system based on each inner corner point and its corresponding depth information; The forklift coordinate module is configured to measure the distance from the preset corner point to the ground based on the relationship between the projection of the preset inner corner line of the calibration template and the origin of the forklift coordinate system; and calculate the coordinates of each inner corner on the calibration template in the forklift coordinate system based on the distance from the preset corner point to the ground and the preset size of the calibration template; The calibration extrinsic parameter module is used to convert the coordinates of each inner corner point in the camera coordinate system and the coordinates in the forklift coordinate system to obtain the optimal transformation matrix model of the corresponding point cloud; according to the iterative closest point algorithm, the optimal transformation matrix model is solved to obtain the calibration extrinsic parameters of the camera.

7. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the calibration template-based extrinsic parameter calibration method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the calibration template-based extrinsic parameter calibration method according to any one of claims 1 to 5.

Images