A method and device for calibrating laser point cloud coordinates for automatic tower crane driving

Abstract

The embodiment of the present invention provides a method and device for calibrating the laser point cloud coordinates for automatic driving of a tower crane, the method comprising: identifying a first pixel in a camera image obtained by a camera, the first pixel being a pixel corresponding to a feature point in the surrounding environment of the tower crane; projecting the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel; obtaining calibration parameters between the laser coordinate system and the world coordinate system according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel. The technical solution of the present invention is applied in the automatic driving process of the tower crane, and the calibration parameters of the laser coordinate system and the world coordinate system of the tower crane laser radar can be automatically obtained regardless of any driving position of the tower crane, and used to obtain its world coordinates using the laser coordinates in the tower crane environment according to the calibration parameters, so as to calculate the obstacle contour during the automatic driving of the tower crane.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control, and in particular to a method and device for calibrating laser point cloud coordinates for automatic driving of a tower crane. Background Art

[0002] In order for a tower crane to complete intelligent (automatic) driving, it is necessary to reconstruct the environment around the tower crane. When using the laser radar SALM to perform three-dimensional reconstruction of the surrounding environment, it is necessary to know the posture relationship between the reconstructed point cloud map and the tower crane itself, that is, to use the external parameter matrix to convert the reconstructed point cloud map to the world coordinate system, and then convert it to the tower crane coordinate system.

[0003] The existing technology integrates the use of lidar and camera in converting point cloud maps into the world coordinate system. The first prerequisite is to correspond the spatial positions of the lidar and camera to ensure the unity of their spatial coordinates, thereby using the world coordinates and laser coordinates of the feature points to realize the calibration of the laser coordinate system and the world coordinate system.

[0004] The existing technology mainly uses a chessboard calibration plate to make the laser radar and camera see the calibration plate at the same time, and then manually selects multiple groups of corresponding points in the radar point cloud data and camera image data to calculate the external parameter information of the camera and the laser radar. However, during the driving process of the tower crane, the position of the laser radar is changing, and the laser coordinates of the laser point cloud are also related to the starting point of the map, so the above correspondence needs to be manually performed every time the map is built, resulting in the inability to achieve automatic calibration of the laser point cloud coordinates. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for laser point cloud coordinate calibration for automatic driving of a tower crane.

[0006] In the first aspect, an embodiment of the present invention provides a method for calibrating laser point cloud coordinates for automatic tower crane driving, comprising: identifying a first pixel in a camera image obtained by a camera, the first pixel being a pixel corresponding to a feature point in the surrounding environment of the tower crane; projecting the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel, the second pixel being a projection point of each point of the laser point cloud in the camera image, the laser point cloud being obtained by scanning a real world containing the feature points by a laser radar; obtaining calibration parameters between the laser coordinate system and the world coordinate system of the laser radar according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel, and being used to obtain its world coordinates using the laser coordinates in the tower crane environment according to the calibration parameters, so as to calculate the obstacle contour during automatic tower crane driving.

[0007] From the above, through the automatic recognition of feature points in the camera image and the projection of the laser point cloud into the camera image, the laser coordinates and world coordinates of several feature points are automatically obtained. Therefore, during the automatic driving of the tower crane, no matter where the laser radar on the tower crane is driven with the tower crane, the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar on the tower crane can be automatically obtained. The calibration parameters are used to obtain the world coordinates according to the laser coordinates in the tower crane environment, which can be used to calculate the obstacle contour and plan and correct the obstacle avoidance path of the tower crane during the automatic driving of the tower crane, thereby realizing the automatic driving of the tower crane.

[0008] In a possible implementation of the first aspect, the laser coordinate system of the laser point cloud is determined according to a starting position of mapping the laser point cloud.

[0009] From the above, the laser coordinate system of the laser radar on the tower crane is not only related to the installation position of the laser radar on the tower crane, but also to the position during the driving process of the tower crane. The installation position of the laser radar on the tower crane and the driving process of the tower crane jointly determine the starting position of the laser point cloud mapping. At each starting position of the laser point cloud mapping, the calibration parameters of the laser coordinate system of the laser radar on the tower crane and the world coordinate system can be automatically obtained, thereby avoiding manual calibration at each moving position of the tower crane.

[0010] In a possible implementation of the first aspect, the feature points include four corners and a center point of several feature graphs, and the feature graphs are chessboards or two-dimensional codes.

[0011] From the above, by taking the four corners and the center point of each feature map as feature points, the features of the feature map are used to accurately identify each feature point in the camera image.

[0012] In a possible implementation of the first aspect, the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar are obtained according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates of the second pixel corresponding to the first pixel, including: identifying the second pixel closest to each first pixel according to the pixel coordinates of the first pixel and the second pixel; obtaining the calibration parameters between the laser coordinate system and the world coordinate system according to the world coordinates of the feature point corresponding to each first pixel and the laser coordinates corresponding to the second pixel closest to the first pixel.

[0013] From the above, by matching the identified feature points with the projection points of the laser point cloud on the camera image, the calibration parameters of the laser coordinate system and the world coordinate system of the lidar can be accurately obtained through the world coordinates of the feature points and the corresponding laser coordinates.

[0014] In a possible implementation of the first aspect, the camera and the laser radar are installed at the same position and have matching fields of view.

[0015] From the above, by making the camera and the lidar have the same position and matching field of view, the external parameters of the camera coordinate system and the laser coordinate system are obtained.

[0016] In a possible implementation manner of the first aspect, the identifying a first pixel in a camera image obtained by the camera includes: accumulating and identifying feature maps in multiple frames of camera images to obtain pixels corresponding to feature points in the identified feature maps.

[0017] From the above, by accumulating and identifying feature maps in multiple frames of camera images, the impact of environment and weather on camera images can be solved and the recognition accuracy of feature points can be improved.

[0018] In a possible implementation manner of the first aspect, the method further includes: projecting the laser point cloud into the world coordinate system according to calibration parameters of the laser coordinate system and the world coordinate system to obtain the world coordinates of each point of the laser point cloud.

[0019] From the above, by obtaining the calibration parameters of the laser coordinate system and the world coordinate system, the world coordinates of each point in the laser point cloud are obtained for autonomous driving.

[0020] In the second aspect, an embodiment of the present invention provides a device for calibrating laser point cloud coordinates for automatic tower crane driving, including: a feature point recognition module, used to identify a first pixel in a camera image obtained by a camera, the first pixel being a pixel corresponding to a feature point in the surrounding environment of the tower crane; a point cloud projection module, used to project the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel, the second pixel being a projection point of each point of the laser point cloud in the camera image, the laser point cloud being obtained by scanning a tower crane environment containing the feature points by a laser radar; a calibration acquisition module, used to obtain calibration parameters between a laser coordinate system and a world coordinate system according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel, and used to obtain its world coordinates using the laser coordinates in the tower crane environment according to the calibration parameters, so as to calculate the obstacle contour during automatic tower crane driving.

[0021] From the above, through the automatic recognition of feature points in the camera image and the projection of the laser point cloud into the camera image, the laser coordinates and world coordinates of several feature points are automatically obtained. Therefore, during the automatic driving of the tower crane, no matter where the laser radar on the tower crane is driven with the tower crane, the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar on the tower crane can be automatically obtained. The calibration parameters are used to obtain the world coordinates according to the laser coordinates in the tower crane environment, which can be used to calculate the obstacle contour and plan and correct the obstacle avoidance path of the tower crane during the automatic driving of the tower crane, thereby realizing the automatic driving of the tower crane.

[0022] In a possible implementation of the second aspect, the laser coordinate system of the laser point cloud is determined according to a starting position of mapping the laser point cloud.

[0023] From the above, the laser coordinate system of the laser radar on the tower crane is not only related to the installation position of the laser radar on the tower crane, but also to the driving position of the tower crane. The installation position of the laser radar on the tower crane and the driving position of the tower crane jointly determine the starting position of the laser point cloud mapping. At each starting position of the laser point cloud mapping, the calibration parameters of the laser coordinate system of the laser radar on the tower crane and the world coordinate system can be automatically obtained, thereby avoiding manual calibration at each moving position of the tower crane.

[0024] In a possible implementation of the second aspect, the feature points include four corners and a center point of several feature graphs, and the feature graphs are chessboards or two-dimensional codes.

[0025] From the above, by taking the four corners and the center point of each feature map as feature points, the features of the feature map are used to accurately identify each feature point in the camera image.

[0026] In a possible implementation of the second aspect, the calibration acquisition module is specifically used to include: identifying the second pixel closest to each first pixel based on the pixel coordinates of the first pixel and the second pixel; and obtaining calibration parameters of the laser coordinate system and the world coordinate system based on the world coordinates of the feature point corresponding to each first pixel and the laser coordinates corresponding to the second pixel closest to the first pixel.

[0027] From the above, by matching the identified feature points with the projection points of the laser point cloud on the camera image, the calibration parameters of the laser coordinate system and the world coordinate system can be accurately obtained through the world coordinates of the feature points and the corresponding laser coordinates.

[0028] In a possible implementation of the second aspect, the camera and the laser radar are installed at the same position and have matching fields of view.

[0029] From the above, the camera and the laser radar are in the same position and have matching fields of view, so as to obtain the external parameters between the camera coordinate system and the laser coordinate system.

[0030] In a possible implementation of the second aspect, the feature point recognition module is specifically configured to obtain pixels corresponding to feature points in the recognized feature maps by cumulatively recognizing feature maps in multiple frames of camera images.

[0031] From the above, by accumulating and identifying feature maps in multiple frames of camera images, the impact of environment and weather on camera images can be solved and the recognition accuracy of feature points can be improved.

[0032] In a possible implementation of the second aspect, the world coordinate acquisition module is used to project the laser point cloud into the world coordinate system according to calibration parameters of the laser coordinate system and the world coordinate system to obtain the world coordinates of each point of the laser point cloud.

[0033] From the above, by obtaining the calibration parameters of the laser coordinate system and the world coordinate system, the world coordinates of each point in the laser point cloud are obtained, thereby performing automatic driving of the tower crane.

[0034] In a third aspect, an embodiment of the present invention provides a tower crane controller, comprising: the device described in any embodiment of the second aspect.

[0035] In a fourth aspect, an embodiment of the present invention provides a tower crane, comprising: the tower crane controller, camera and laser radar described in the third aspect.

[0036] In a fifth aspect, an embodiment of the present invention provides a computing device, comprising a bus; a communication interface connected to the bus; at least one processor connected to the bus; and at least one memory connected to the bus and storing program instructions, wherein when the program instructions are executed by the at least one processor, the at least one processor executes the method described in any embodiment of the first aspect.

[0037] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having program instructions stored thereon, wherein when the program instructions are executed by a computer, the computer executes the method described in any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a method for laser point cloud coordinate calibration for automatic driving of a tower crane according to a first embodiment of the present invention;

[0039] Figure 2 It is a flow chart of a second embodiment of a method for laser point cloud coordinate calibration for automatic driving of a tower crane according to the present invention;

[0040] Figure 3 It is a structural schematic diagram of a first embodiment of a device for laser point cloud coordinate calibration for automatic driving of a tower crane according to the present invention;

[0041] Figure 4 This is a structural schematic diagram of a second embodiment of a device for laser point cloud coordinate calibration for automatic driving of a tower crane according to the present invention;

[0042] Figure 5 Schematic diagram of the structure of the computing device according to various embodiments of the present invention. DETAILED DESCRIPTION

[0043] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third, etc." or module A, module B, module C, etc. are only used to distinguish similar objects, or to distinguish different embodiments, and do not represent a specific ordering of the objects. It can be understood that the specific order or sequence can be interchanged where permitted, so that the embodiments of the present invention described here can be implemented in an order other than that illustrated or described here.

[0045] In the following description, the numbers representing the steps, such as S110, S120, etc., do not necessarily mean that the steps must be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0047] The present invention is applied in the automatic driving process of a tower crane. A laser radar and a camera are installed on the tower crane. The laser radar obtains a laser point cloud of the tower crane environment, and the camera obtains an image of the tower crane environment. During the automatic driving process of the tower crane, the position of the laser radar changes with the driving process of the tower crane, and the calibration parameters between the laser coordinate system of the laser radar and the world coordinate system are also changing, and need to be automatically calibrated with the position of the laser radar.

[0048] In the calibration method of the embodiment of the present invention, the feature points in the camera image are automatically identified, and the feature points are corresponded to the pixel points of the laser point cloud projected onto the camera image to obtain the laser coordinates and world coordinates of the feature points; the calibration parameters between the laser coordinate system and the world coordinate system in the tower crane environment are obtained according to the laser coordinates and world coordinates of the feature points, and there is no need to perform manual calibration each time according to the laser radar position during the tower crane driving process. The laser coordinates of any laser point collected in the tower crane environment are converted into world coordinates using the calibration parameters.

[0049] During the process of controlling the tower crane driving by the tower crane controller, the calibration between the laser coordinate system and the world coordinate system is carried out automatically. The world coordinates of each point in the tower crane environment are determined using the currently obtained calibration parameters. After the contour points of obstacles around the tower crane are identified, the world coordinates of the contour points of each obstacle and the contour points of the hoisted object are used to plan or correct the obstacle avoidance path, and the tower crane hoisting trajectory points are planned. The world coordinates of the tower crane hoisting trajectory points are then converted into the axis coordinates of the movement of each axis of the tower crane, and each axis of the tower crane is controlled to realize automatic tower crane driving without human intervention.

[0050] Let's combine Figure 1 and Figure 2 Various method embodiments of laser point cloud coordinate calibration for automatic tower crane driving are introduced.

[0051] A method for calibrating laser point cloud coordinates for automatic tower crane driving, in embodiment 1, includes: identifying a first pixel in a camera image obtained by a camera, the first pixel being a pixel corresponding to a number of feature points in the surrounding environment of the tower crane; projecting the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel, the second pixel being a projection point of each point of the laser point cloud in the camera image, the laser point cloud being obtained by the laser radar scanning a real world of a tower crane environment containing the feature points; obtaining calibration parameters of a laser coordinate system and a world coordinate system according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel, so as to calibrate the laser coordinate system and the world coordinate system of the laser radar.

[0052] Figure 1 The flowchart of a first embodiment of a method for laser point cloud coordinate calibration for automatic tower crane driving is shown, including steps S110 to S140.

[0053] S110: Using a camera and a laser radar, respectively, a camera image and a laser point cloud of a tower crane environment including a plurality of feature points are obtained.

[0054] In some embodiments, the feature points are four corners and a center point in a number of feature graphs, the world coordinates of each feature point are known, and the world coordinates of the four corners and the center point of each AprilTag code are given by a total station or GNSS, etc. The posture of the AprilTag code graph or the black and white grid is not limited, the feature graph is a chessboard or a QR code, and the QR code can be an AprilTag code.

[0055] In some embodiments, the camera and the laser radar are in the same position, so that the calibration parameters between the camera coordinate system and the laser coordinate system are easy to obtain. In some embodiments, the field of view of the camera and the laser radar match, so that the camera image matches the laser point cloud.

[0056] Among them, the laser coordinates of each point in the laser point cloud are not only related to the installation position of the laser radar, but also to the starting position of the laser point cloud mapping (and the position of the tower crane driving).

[0057] S120: Identify first pixels in the camera image, and obtain pixel coordinates and world coordinates of each first pixel.

[0058] Among them, each first pixel is a pixel corresponding to a feature point in the tower crane environment, and the pixel coordinates of each first pixel are obtained according to the two-dimensional serial number of the first pixel in the camera image.

[0059] In some embodiments, machine learning, deep learning, or image processing is used to identify feature points in the camera image, thereby obtaining the world coordinates of the feature points corresponding to each first pixel.

[0060] In some embodiments, when the feature points are the four corners and the center point of the feature graph, at least five feature graphs are identified.

[0061] In some embodiments, by cumulatively recognizing feature maps in multiple frames of camera images, pixels corresponding to feature points in the recognized feature maps, namely, first pixels, are obtained, thereby improving recognition accuracy.

[0062] S130: Projecting the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain the pixel coordinates of the second pixel.

[0063] Among them, the second pixel is the projection point of each point of the laser point cloud in the camera image, the pixel coordinates of each second pixel correspond one-to-one to the two-dimensional serial number of the second pixel in the camera image, and the laser coordinates corresponding to each second pixel are the laser coordinates of the corresponding point in the laser point cloud.

[0064] In some embodiments, the parameters of the camera coordinates include camera intrinsic parameters, distortion parameters, and calibration parameters between the camera coordinate system and the laser coordinate system obtained in advance. The calibration parameters are also called extrinsic parameters between the camera coordinate system and the laser coordinate system.

[0065] S140: Obtain calibration parameters of the laser coordinate system and the world coordinate system of the laser radar according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel.

[0066] The calibration parameters of the laser coordinate system and the world coordinate system are used to convert the laser coordinates into world coordinates. The laser coordinate system changes with the starting position of the laser point cloud mapping (the starting position is determined according to the tower crane driving position and the installation position of the laser radar), and the calibration parameters also change with the starting position of the laser point cloud mapping. The process of obtaining the calibration parameters is also called the external parameter calibration of the laser radar.

[0067] In some embodiments, the second pixel closest to each first pixel is identified based on the pixel coordinates of the first pixel and the second pixel, and the second pixel closest to a first pixel and the first pixel can be considered to be the same feature point in the real environment of the tower crane; the calibration parameters of the laser coordinate system and the world coordinate system are obtained based on the world coordinates corresponding to each first pixel and the laser coordinates corresponding to the second pixel closest to the first pixel. In other embodiments, the grayscale of the pixel coordinates of the second pixel is set according to the point values ​​of each point in the laser point cloud, and the feature points therein are identified according to the grayscale of each second pixel, and matched with the first pixel; and the calibration parameters of the laser coordinate system and the world coordinate system are obtained based on the world coordinates corresponding to the first pixel and the laser coordinates of the feature points identified from the second pixel.

[0068] In some embodiments, the calibration parameters of the laser coordinate system and the world coordinate system form a 4*4 calibration matrix, and the expansion vector of the laser coordinates (the vector formed by the three-dimensional coordinates of the laser coordinates and the fourth dimension with a value of 1) and the calibration product are equal to the expansion vector of the world coordinates (the vector formed by the three-dimensional coordinates of the world coordinates and the fourth dimension with a value of 1), so solving the calibration parameters of the laser coordinate system and the world coordinate system requires at least (4*4 / 3 rounded up to 6) feature points. In some embodiments, in order to improve accuracy, 5 feature maps and 25 feature points are used.

[0069] In some embodiments, the laser point cloud is projected into the world coordinate system according to the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar to obtain the world coordinates of each point of the laser point cloud. In some of these embodiments, obstacles are identified and the world coordinates of each outline point of the obstacle are determined according to the world coordinates of each point in the tower crane environment; in other embodiments, the positions of each outline point of the obstacle are identified according to the laser point cloud of the tower crane environment to obtain the world coordinates of each outline point of the obstacle. Obstacle avoidance path design is performed based on the obtained world coordinates of each outline point of the obstacle and the world coordinates of each outline point of the hook or the hoisted object to perform automatic driving of the tower crane.

[0070] In summary, in a method for calibrating laser point cloud coordinates for automatic tower crane driving, in Embodiment 1, pixels corresponding to feature points in the real world in a camera image obtained by a camera are identified, and pixels corresponding to each point of the laser point cloud in the camera image are obtained by projecting the laser point cloud onto the camera image, and then the laser coordinates and world coordinates of the feature points are obtained by matching the two, and the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar are obtained accordingly. The technical solution of this embodiment is applied in the automatic driving process of the tower crane, and the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar can be realized regardless of the position of the laser radar on the tower crane in the driving of the tower crane, so that the world coordinates are obtained by using the laser coordinates in the tower crane environment according to the calibration parameters, so as to calculate the obstacle contour and plan and correct the obstacle avoidance path of the tower crane during the automatic driving of the tower crane, thereby realizing the automatic driving of the tower crane.

[0071] Embodiment 2 of a method for laser point cloud coordinate calibration for automatic driving of a tower crane is a detailed implementation of embodiment 1 of a method for laser point cloud coordinate calibration for automatic driving of a tower crane, and has all the advantages of embodiment 1 of a method for laser point cloud coordinate calibration for automatic driving of a tower crane.

[0072] Figure 2 The flowchart of a second embodiment of a method for laser point cloud coordinate calibration for automatic tower crane driving is shown, including steps S210 to S270.

[0073] S210: Obtain a camera image and a laser point cloud of a tower crane environment including at least 5 AprilTag code images by using a camera and a laser radar respectively.

[0074] The camera and lidar are installed on a tower crane with the same installation position and matching field of view, so that the camera image matches the laser point cloud. The appropriate size of the AprilTag code image can be selected according to the working distance and camera resolution.

[0075] The laser coordinates of each point in the laser point cloud are not only related to the installation position of the laser radar, but also to the starting position of the laser point cloud mapping. At the initial position of the laser radar mapping, keep the optical path between the AprilTag code camera and the laser radar unobstructed.

[0076] The four corners and the center point of each AprilTag code map are feature points. The world coordinates of each feature point are known, and the posture of the AprilTag code map is not limited. The world coordinates of the feature points of each AprilTag code map are given by a total station or GNSS.

[0077] The world coordinate system is defined with the center point of the tower base on the ground as the origin, the north is the positive x direction, the west is the positive y direction, and the vertical is the positive z direction. The world coordinates of any point are represented by (xw, yw, zw). The laser coordinates of any point in the laser point cloud are (xl, yl, zl), and the pixels of any point in the camera image are represented by (u, v).

[0078] S220: Identify at least five AprilTag code images from the camera image, and feature points in the identified code images constitute a first pixel set AUV.

[0079] The first pixel is a feature point in the identified code image, and the world coordinates of each first pixel are known in advance, and its pixel coordinates are obtained according to its two-dimensional serial number in the camera image. For example, AUV can be expressed as {(u i , v i , xw i , yw i , zw i ), i = 1, 2, ..., N}, (u i , v i ) is the pixel coordinate of the first pixel point i, (xw i , yw i , zw i ) is the world coordinate of the pixel coordinate of the first pixel point i.

[0080] By recognizing the AprilTag code images in multiple frames of camera images, each first pixel is obtained, and the recognition results of each frame of camera images are accumulated to solve the influence of the external environment, thereby improving the recognition accuracy.

[0081] S230: Project the laser point cloud into the camera image according to the camera intrinsic parameters and the extrinsic parameters between the camera coordinate system and the laser coordinate system to obtain a second pixel set BUV.

[0082] The second pixel is the projection point of each point of the laser point cloud in the camera image, and the pixel coordinates of each second pixel are obtained by projection using the camera intrinsic parameters and the extrinsic parameters between the camera coordinate system and the laser coordinate system.

[0083] Specifically, the process of projecting the laser point cloud onto the camera image can be expressed as formula (1).

[0084]

[0085] Among them, (xl, yl, zl) is the laser coordinate of the second pixel, f is the focal length of the camera, R1 is the matrix of the camera intrinsic parameters, RT1 is the matrix of the external parameters of the camera coordinate system and the laser coordinate system, and Zc is the depth coordinate in the camera coordinate system, also called the optical axis coordinate. Zc can be obtained according to RT1 and the laser coordinates (xl, yl, zl).

[0086] In some embodiments, the (u, v) obtained in equation (1) is also corrected using the distortion parameters of the camera.

[0087] For example, BUV can be expressed as {(u i , v i ,xl i ,yl i , zl i ), i = 1, 2, ..., N}, (u i , v i ) is the pixel coordinate of the second pixel point i, (xl i , yl i , zl i ) is the world coordinate of the pixel coordinate of the second pixel point i.

[0088] S240: searching in the set BUV for the second pixel point closest to each first pixel point in the set AUV, and establishing a correspondence between the world coordinates of the first pixel point and the laser coordinates of the second pixel point closest to it through the sequence number in the camera image.

[0089] The world coordinates of each first pixel point are known, and the laser coordinates of each second pixel point are also known.

[0090] Among them, the world coordinate system remains unchanged during the movement of the tower crane.

[0091] S250: Obtain automatic external parameter calibration of the laser radar's current laser coordinate system and the world coordinate system according to the correspondence between the world coordinate of the first pixel point and the laser coordinate of the second pixel point closest to it.

[0092] Among them, according to the world coordinates (xw, yw, zw) of the first pixel point and the laser coordinates (xl, yl, zl) of its nearest second pixel point, the equation group about each element in RT2 is established using formula (2), and RT2 is the external parameter from the world coordinate system to the laser coordinate system. It is also possible to obtain the external parameter from the laser coordinate system to the world coordinate system in reverse.

[0093]

[0094] Among them, because the first pixel point includes at least the 4 corners and the center point of the 5 AprilTag code images, a set of equations can be established through at least 25 points, and a numerical optimization method (such as least squares method) is used to solve the 4*4 transformation matrix RT2 between the calibration parameters of the laser coordinate system and the world coordinate system, thereby completing the automatic external parameter calibration of the laser coordinate system and the world coordinate system on the tower crane.

[0095] It should be emphasized that: in steps S210 to S250, the laser coordinate system of the laser point cloud changes with the starting position of the laser point cloud mapping, and the starting position is related to the tower crane driving position. The automatic external parameter calibration of the laser coordinate system of the laser radar and the world coordinate system is automatically performed with the tower crane driving position.

[0096] S260: Project the laser point cloud into the world coordinate system according to the calibration parameters of the current laser coordinate system and the world coordinate system of the laser radar, and obtain the world coordinates of each point in the tower crane environment.

[0097] Among them, the calibration parameters change with the driving position of the tower crane, so the calibration parameters of the current laser coordinate system and the world coordinate system of the laser radar are used to obtain the world coordinates of each point in the tower crane environment. Because the objects in the tower crane environment, especially obstacles, will also change during the driving process, the world coordinates of each point in the tower crane environment are obtained in time as the tower crane is driven.

[0098] S270: According to the world coordinates of each point in the tower crane environment, the world coordinates of each point of the obstacle outline during the automatic driving of the tower crane are obtained to perform automatic driving of the tower crane.

[0099] Among them, each point in the tower crane environment includes not only each point of the obstacle contour during the tower crane driving process, but also each point of the contour of the hook or the hoisted object. In some embodiments, obstacles are identified and the world coordinates of each point of the obstacle contour are determined according to the world coordinates of each point in the tower crane environment; in other embodiments, the positions of each point of the obstacle contour are identified according to the laser point cloud of the tower crane environment, and the world coordinates of each point of the obstacle contour are obtained. According to the world coordinates of each point of the obstacle contour during the tower crane driving process and the world coordinates of each point of the contour of the hook or the hoisted object, an obstacle avoidance path is designed to perform automatic tower crane driving.

[0100] Among them, after the obstacle avoidance path design is completed, the obstacle avoidance path is also corrected according to new obstacles appearing in the tower crane environment during the tower crane automatic driving process.

[0101] Among them, after the obstacle avoidance path correction during the tower crane driving process is completed, the world coordinates of the obstacle avoidance path are also converted into tower crane column coordinates, so as to realize unmanned automatic driving of the tower crane by controlling each axis of the tower crane.

[0102] Combine the following Figure 3 to Figure 4 An embodiment of a device for laser point cloud coordinate calibration for automatic driving of a tower crane is introduced in the present invention.

[0103] A device for calibrating laser point cloud coordinates for automatic driving of a tower crane. Embodiment 1. Calibration of the laser coordinate system and the world coordinate system of a laser radar of a tower crane. A method for calibrating laser point cloud coordinates for automatic driving of a tower crane. Embodiment 1. The method has all its advantages.

[0104] Figure 3 The structure of a first embodiment of a device for laser point cloud coordinate calibration for automatic driving of a tower crane is shown, including: a data acquisition module 310, a feature point recognition module 320, a point cloud projection module 330 and a calibration acquisition module 340.

[0105] The data acquisition module 310 is used to obtain a camera image and a laser point cloud of a tower crane environment containing a plurality of feature points using a camera and a laser radar. For its working principle and advantages, please refer to step S110 of the first embodiment of a method for calibrating laser point cloud coordinates for automatic tower crane driving.

[0106] The feature point recognition module 320 is used to recognize the first pixel in the camera image and obtain the pixel coordinates and world coordinates of each first pixel. For its working principle and advantages, please refer to step S120 of the first embodiment of a method for laser point cloud coordinate calibration for automatic tower crane driving.

[0107] The point cloud projection module 330 is used to project the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain the pixel coordinates of the second pixel. For its working principle and advantages, please refer to step S130 of the first embodiment of a method for laser point cloud coordinate calibration for automatic tower crane driving.

[0108] The calibration acquisition module 340 is used to obtain calibration parameters of the laser coordinate system and the world coordinate system according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel. For its working principle and advantages, please refer to step S140 of the first embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0109] Embodiment 2 of a device for calibrating laser point cloud coordinates for automatic driving of a tower crane is a detailed implementation method for calibrating laser coordinates of a laser point cloud applied to a tower crane, and the method of embodiment 2 of a method for calibrating laser point cloud coordinates for automatic driving of a tower crane has all its advantages.

[0110] Figure 4 The structure of a second embodiment of a device for laser point cloud coordinate calibration for tower crane automatic driving is shown, including: a data acquisition module 410, a feature point recognition module 420, a point cloud projection module 430, a pixel matching module 440, a calibration acquisition module 450, a world coordinate acquisition module 460 and a tower crane automatic driving module 470.

[0111] The data acquisition module 410 is used to obtain a camera image and a laser point cloud of the surrounding environment of the tower crane containing at least 5 AprilTag code images by using a camera and a laser radar. For its working principle and advantages, please refer to step S210 of the second embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0112] The feature point recognition module 420 is used to recognize at least 5 AprilTag code images from the camera image, and the feature points in the recognized code images constitute the first pixel set AUV. For its working principle and advantages, please refer to step S220 of the second embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0113] The point cloud projection module 430 is used to project the laser point cloud into the camera image according to the camera internal parameters and the external parameters between the camera coordinate system and the laser coordinate system to obtain the second pixel set BUV. For its working principle and advantages, please refer to step S230 of the second embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0114] The pixel matching module 440 is used to find the second pixel point closest to each first pixel point in the set AUV in the set BUV, and establish the corresponding relationship between the world coordinates of the first pixel point and the laser coordinates of the second pixel point closest to it through the serial number in the camera image. For its working principle and advantages, please refer to step S240 of the second embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0115] The calibration acquisition module 450 is used to obtain the automatic external parameter calibration of the laser coordinate system and the world coordinate system of the laser radar according to the correspondence between the world coordinate of the first pixel point and the laser coordinate of the second pixel point closest to it. For its working principle and advantages, please refer to step S250 of the second embodiment of a method for laser point cloud coordinate calibration for automatic tower crane driving.

[0116] The world coordinate acquisition module 460 is used to project the laser point cloud into the world coordinate system based on the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar, and obtain the world coordinates of each point in the tower crane environment. For its working principle and advantages, please refer to step S260 of the second embodiment of a method for calibrating laser point cloud coordinates for tower crane automatic driving.

[0117] The tower crane automatic driving module 470 is used to obtain the world coordinates of each point of the obstacle outline during the tower crane automatic driving according to the world coordinates of each point in the tower crane environment, so as to perform the tower crane automatic driving. For its working principle and advantages, please refer to step S270 of the second embodiment of a method for laser point cloud coordinate calibration for tower crane automatic driving.

[0118] The present invention also provides an embodiment of a tower crane controller, including: a device described in Embodiment 1 of a device for calibrating laser point cloud coordinates for automatic driving of a tower crane or a device described in Embodiment 2 of a device for calibrating laser point cloud coordinates for automatic driving of a tower crane.

[0119] The present invention provides a tower crane embodiment, comprising: a tower crane controller as described in a tower crane controller embodiment, a camera and a laser radar.

[0120] The embodiment of the present invention further provides a computing device, Figure 5 Detailed introduction.

[0121] The computing device 500 includes a processor 510 , a memory 520 , a communication interface 530 , and a bus 540 .

[0122] It should be understood that the communication interface 530 in the computing device 500 shown in this figure can be used to communicate with other devices.

[0123] The processor 510 may be connected to a memory 520. The memory 520 may be used to store the program code and data. Therefore, the memory 520 may be a storage unit inside the processor 510, or an external storage unit independent of the processor 510, or a component including a storage unit inside the processor 510 and an external storage unit independent of the processor 510.

[0124] Optionally, the computing device 500 may further include a bus 540. The memory 520 and the communication interface 530 may be connected to the processor 510 via the bus 540. The bus 540 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 550 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure is represented by only one line, but it does not mean that there is only one bus or one type of bus.

[0125] It should be understood that in the embodiment of the present invention, the processor 510 may adopt a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Alternatively, the processor 510 may adopt one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiment of the present invention.

[0126] The memory 520 may include a read-only memory and a random access memory, and provides instructions and data to the processor 510. A portion of the processor 510 may also include a nonvolatile random access memory. For example, the processor 510 may also store information on the device type.

[0127] When the computing device 500 is running, the processor 510 executes the computer-executable instructions in the memory 520 to perform the operating steps of the method embodiment.

[0128] It should be understood that the computing device 500 according to an embodiment of the present invention can correspond to the corresponding subjects executing the methods according to the various embodiments of the present invention, and the above-mentioned and other operations and / or functions of each module in the computing device 500 are respectively for implementing the corresponding processes of each method of the present embodiment, which will not be repeated here for the sake of brevity.

[0129] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it is used to execute the operating steps of the method embodiment.

[0136] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive table) of computer-readable storage media include, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0139] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A method for calibrating laser point cloud coordinates for automatic tower crane driving, characterized in that: include: Recognize a first pixel in a camera image obtained by a camera, where the first pixel is a pixel corresponding to a plurality of feature points in the surrounding environment of the tower crane; Projecting the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel, where the second pixel is a projection point of each point of the laser point cloud in the camera image, and the laser point cloud is obtained by scanning a tower crane environment containing the feature point by a laser radar; According to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel, a calibration parameter between the laser coordinate system and the world coordinate system of the laser radar is obtained, and the world coordinates thereof are obtained by using the laser coordinates in the tower crane environment according to the calibration parameter, so as to calculate the obstacle contour during automatic driving of the tower crane; Among them, laser radar and camera are installed on the tower crane, and the world coordinate system remains unchanged during the operation of the tower crane.

2. The method according to claim 1, characterized in that: The laser coordinate system of the laser point cloud is determined according to the starting position of the laser point cloud mapping.

3. The method according to claim 1, characterized in that: The feature points include four corners and a center point of several feature graphs, and the feature graphs are chessboards or two-dimensional codes.

4. The method according to claim 1, characterized in that: The method of obtaining the calibration parameters of the laser coordinate system and the world coordinate system of the laser radar according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel includes: identifying, based on pixel coordinates of the first pixels and the second pixels, a second pixel closest to each first pixel; The calibration parameters are obtained according to the world coordinates of the feature point corresponding to each first pixel and the laser coordinates corresponding to the second pixel closest to the first pixel.

5. The method according to claim 1, characterized in that: The camera and the laser radar are installed at the same position and have matching fields of view.

6. The method according to claim 3, characterized in that: The first pixel in the camera image obtained by the recognition camera includes: accumulating and recognizing feature maps in multiple frames of camera images to obtain pixels corresponding to feature points in the recognized feature map.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: The laser point cloud is projected into a world coordinate system according to the calibration parameters to obtain the world coordinates of each point of the laser point cloud.

8. A device for laser point cloud coordinate calibration for automatic tower crane driving, characterized in that: include: A feature point recognition module, used to recognize a first pixel in a camera image obtained by the camera, where the first pixel is a pixel corresponding to a feature point in the surrounding environment of the tower crane; A point cloud projection module, used to project the laser point cloud into the camera image according to the parameters of the camera coordinates to obtain a second pixel, where the second pixel is a projection point of each point of the laser point cloud in the camera image, and the laser point cloud is obtained by scanning the tower crane environment containing the feature point with a laser radar; A calibration acquisition module, used for obtaining calibration parameters between the laser coordinate system and the world coordinate system of the laser radar according to the world coordinates of the feature point corresponding to the first pixel and the laser coordinates corresponding to the second pixel corresponding to the first pixel, and used for obtaining its world coordinates using the laser coordinates in the tower crane environment according to the calibration parameters, so as to calculate the obstacle contour during automatic driving of the tower crane; Among them, laser radar and camera are installed on the tower crane, and the world coordinate system remains unchanged during the operation of the tower crane.

9. A computing device, characterized in that include, bus; A communication interface connected to the bus; at least one processor connected to the bus; as well as At least one memory is connected to the bus and stores program instructions, and when the program instructions are executed by the at least one processor, the at least one processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, and when the program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images