Tower crane working scene modeling method and device

Abstract

The invention discloses a tower crane working scene modeling method and device, and belongs to the technical field of computers. The tower crane working scene modeling method comprises the following steps: acquiring current frame point cloud data of a working scene of a tower crane and pose information of a laser radar, which are acquired by the laser radar arranged on a variable-amplitude trolley of the tower crane, in a rotating motion process of a big arm of the tower crane, obtaining angular velocity information of a big arm of the tower crane when the laser radar collects the point cloud data of the current frame; and analyzing the point cloud data of the current frame based on the angular velocity information and the pose information, and obtaining a working scene model corresponding to the point cloud data of the current frame. According to the working scene modeling method and device for the tower crane disclosed by the invention, by fusing the angular velocity information of the big arm of the tower crane when the laser radar collects the point cloud data, the motion trail and the environment characteristic coordinates of the tower crane can be more accurately estimated; and the accuracy of obtaining the working scene model of the tower crane is higher.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method and apparatus for modeling the working scene of a tower crane. Background Technology

[0002] In related technologies, modeling the working scene of tower cranes can provide a technical foundation for the autonomous driving of tower cranes. Currently, tower crane working scene modeling is usually achieved based on LiDAR-Inertial Odometry (LIO) systems. LIO-based tower crane working scene modeling generally includes: using a LiDAR mounted on the tower boom trolley to perform multi-angle scanning, collecting point cloud data of the tower crane's working site, analyzing the point cloud data to create a map, and obtaining an environmental map of the area below the tower crane. However, the above method has limitations in accuracy. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a method and apparatus for modeling tower crane working scenes, which can improve the accuracy of tower crane working scene modeling.

[0004] Firstly, this application provides a method for modeling the working scene of a tower crane, the method comprising: During the boom rotation of the tower crane, the current frame point cloud data of the tower crane's working scene and the pose information of the laser radar collected by the laser radar set on the luffing trolley of the tower crane are obtained, and the angular velocity information of the tower crane when the laser radar collects the current frame point cloud data is obtained. Based on the angular velocity information and the pose information, the current frame point cloud data is analyzed to obtain the working scene model corresponding to the current frame point cloud data.

[0005] According to the tower crane working scene modeling method of this application, by integrating the angular velocity information of the tower crane's boom when collecting point cloud data from LiDAR during the real-time positioning and map construction process based on LiDAR inertial odometry, the movement trajectory and environmental feature coordinates of the tower crane itself can be estimated more accurately, resulting in a more accurate tower crane working scene model and improving the accuracy of tower crane working scene modeling.

[0006] According to one embodiment of this application, the step of analyzing the current frame point cloud data based on the angular velocity information and the pose information to obtain the working scene model corresponding to the current frame point cloud data includes: For each point in the current frame point cloud data, based on the angular velocity information and the pose information, the position information corresponding to each point is obtained; the position information includes the transformation matrix between the upper arm coordinate system and the world coordinate system; Based on the location information corresponding to each point in the current frame point cloud data, the working scene model corresponding to the current frame point cloud data is obtained.

[0007] According to one embodiment of this application, obtaining the position information corresponding to each point based on the angular velocity information includes: Based on the angular velocity information, the angular velocity of the tower crane body and the angular velocity component corresponding to the rotation of the boom are obtained; Based on the angular velocity of the tower crane body and the pose information, a motion transformation is performed on each point to obtain the transformation matrix corresponding to each point; the transformation matrix is ​​used to indicate the position information corresponding to each point.

[0008] According to one embodiment of this application, after performing motion transformation on each point based on the angular velocity of the tower crane body and obtaining the transformation matrix corresponding to each point, the method further includes: Based on the position information of the target plane in the work scenario, the point cloud residual of each point is obtained; The transformation matrix corresponding to each point is updated by iterating based on the point cloud residual of each point.

[0009] According to one embodiment of this application, obtaining the working scene model corresponding to the current frame point cloud data based on the location information corresponding to each point in the current frame point cloud data includes: Based on the location information corresponding to each point in the current frame point cloud data and the location information of the target point in the historical work scene model, the work scene model corresponding to the current frame point cloud data is obtained; the historical work scene model is the work scene model corresponding to the point cloud data of each frame before the current frame; the target point is the point whose distance to each point in the current frame point cloud data is less than a target threshold.

[0010] According to one embodiment of this application, obtaining the angular velocity information of the tower crane when the lidar collects the current frame point cloud data includes: The angular velocity information collected by the inertial measurement unit set on the luffing trolley is obtained.

[0011] According to one embodiment of this application, when the inertial measurement unit is built into the lidar, the angle between the horizontal axis and the direction of gravity in the lidar coordinate system used by the lidar is less than an angle threshold.

[0012] According to one embodiment of this application, when the lidar and the inertial measurement unit are separately configured, the distance between the lidar and the inertial measurement unit is less than a distance threshold.

[0013] Secondly, this application provides a tower crane working scene modeling device, the device comprising: The acquisition module is used to acquire the current frame point cloud data of the tower crane's working scene and the pose information of the laser radar collected by the laser radar set on the luffing trolley of the tower crane during the boom rotation movement of the tower crane, and to acquire the angular velocity information of the tower crane when the laser radar collects the current frame point cloud data. The modeling module is used to analyze the current frame point cloud data based on the angular velocity information and the pose information, and obtain the working scene model corresponding to the current frame point cloud data.

[0014] According to the tower crane working scene modeling device of this application, by integrating the angular velocity information of the tower crane's boom when collecting point cloud data from the LiDAR during the real-time positioning and map construction process based on LiDAR inertial odometry, the device can more accurately estimate the tower crane's own motion trajectory and environmental feature coordinates, thereby obtaining a more accurate working scene model of the tower crane and improving the accuracy of tower crane working scene modeling.

[0015] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the tower crane working scene modeling method as described in the first aspect above.

[0016] Fourthly, this application provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the tower crane working scene modeling method as described in the first aspect above.

[0017] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the tower crane working scene modeling method as described in the first aspect.

[0018] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the tower crane working scene modeling method as described in the first aspect above.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the tower crane working scene modeling method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the tower crane working scene modeling device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] The tower crane working scene modeling method, tower crane working scene modeling device, electronic device and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0024] The tower crane working scenario modeling method can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0025] The tower crane working scene modeling method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the tower crane working scene modeling method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The tower crane working scene modeling method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0026] like Figure 1 As shown, the tower crane working scene modeling method includes steps 110 and 120.

[0027] In practical implementation, the tower crane (hereinafter referred to as "tower crane") working scene modeling method provided in this application embodiment is a high-precision Simultaneous Localization and Mapping (SLAM) solution suitable for the working environment of tower cranes, built on the basis of the LIO (Lidar-Inertial Odometry) system. SLAM is a technology that uses sensors to locate and build an environmental map in real time in an unknown environment. The core principle of SLAM technology is to simultaneously estimate the tower crane's own motion trajectory and environmental feature coordinates by fusing measurement data from sensors (such as LiDAR, cameras, and millimeter-wave radar).

[0028] Tower cranes typically consist of a vertical main tower and one or more horizontal jibs, resembling the shape of a tower. Tower cranes use electric or hydraulic systems to lift, move horizontally, and rotate loads.

[0029] In some embodiments, this application uses LiDAR to collect dense three-dimensional (3D) point cloud data of the tower crane body under rotational motion in real time, and integrates information such as the angular velocity and angular acceleration of the tower crane body. Then, it uses any LIO-SLAM algorithm (such as Fast LIO algorithm, Fast LIO2 algorithm, or Faster LIO algorithm) to estimate the pose and motion state of the tower crane in real time.

[0030] In some embodiments, in any of the LIO-SLAM algorithms described above, when the relative pose constraints between two frames of point clouds are obtained from the original point cloud information, the front-end odometry method can be used.

[0031] In some embodiments, the LIO-SLAM algorithm described above may be a tightly coupled Iterative Error-State Kalman Filter (IESKF) algorithm, etc.

[0032] It should be noted that when a tower crane is in operation, in order to ensure that personnel and other equipment maintain a safe distance from the tower crane, the working site of the tower crane needs to be confirmed to ensure that the working site has sufficient height and strength to accommodate the height and range of motion of the tower crane.

[0033] However, the working environment of tower cranes is a complex environment containing pedestrian and vehicular traffic and constantly generated fixed obstacles within a certain height around the lifting and landing positions, which is not conducive to the automatic driving of tower cranes. Therefore, it is necessary to construct an environmental map model of the area below the tower crane (i.e., a map model of the tower crane's working scene) through tower crane working scene modeling methods, so as to enable the tower crane to perform automatic driving based on the established map model and improve the working efficiency of the tower crane.

[0034] The tower crane working scene modeling method provided in this application can solve the problem of lack of three-dimensional perception in the tower crane working scene. By constructing a dense point cloud map of the tower crane's working area, a more accurate spatial model is provided for the tower crane's collision avoidance, and an environmental perception basis is provided for autonomous planning of the hoisting path. This significantly improves the safety of multi-tower collaboration and the efficiency of intelligent operation, laying the technical foundation for unmanned operation of tower cranes.

[0035] Step 110: During the rotational motion of the tower crane's boom, acquire the current frame point cloud data of the tower crane's working scene collected by the lidar set on the luffing trolley of the tower crane, and the pose information of the lidar. Also acquire the angular velocity information of the tower crane's boom when the lidar acquires the current frame point cloud data.

[0036] In actual implementation, the tower crane working scene modeling method provided in this application embodiment can be applied to model the working scene of a trolley luffing tower crane to obtain a map model of the tower crane working scene, i.e., a working scene model.

[0037] Understandably, each jib of a trolley-type luffing tower crane can be equipped with a luffing trolley. Luffing is achieved by moving the luffing trolley along the horizontal jib (i.e., the "horizontal arm").

[0038] In some embodiments, any horizontal jib of a tower crane can serve as the boom.

[0039] To implement the tower crane working scene modeling method provided in this application embodiment, a lidar can be installed on the luffing trolley of the tower crane's boom. In some embodiments, an inertial measurement unit (IMU) can also be installed on the luffing trolley of the tower crane's boom.

[0040] In some embodiments, the lidar can be mounted on a lidar bracket. The lidar bracket can be equipped with a rotating gimbal with multiple degrees of freedom capable of automatically adjusting the lidar's orientation.

[0041] During the rotation of the boom of a tower crane, the orientation of the lidar can be changed by rotating the pan-tilt head and moving it left, right, up, and down. The lidar then performs multi-angle scanning to obtain point cloud data from different directions.

[0042] It should be noted that the point cloud data obtained by the LiDAR in each scan is a frame of point cloud data. Each frame of point cloud data may include one or more points.

[0043] In actual operation, the lidar's pose information can be acquired during each scan. In some embodiments, the lidar's pose information may include position, velocity, and attitude. In some embodiments, attitude may include pitch angle, roll angle, and yaw angle.

[0044] In some embodiments, pitch angle, roll angle, and yaw angle can be obtained by rotating the gimbal around the X-axis, Y-axis, and Z-axis.

[0045] In actual operation, the angular velocity information of the tower crane's boom can also be acquired during each scan by the lidar. In some embodiments, the angular velocity information may include at least one of angular velocity and angular acceleration. Here, angular velocity refers to the angular velocity itself, and angular acceleration refers to the first derivative or first differential of the angular velocity (itself) with respect to time.

[0046] In some embodiments, angular velocity information can be acquired by an inertial measurement unit or other sensors.

[0047] Step 120: Analyze the current frame point cloud data based on angular velocity and pose information to obtain the working scene model corresponding to the current frame point cloud data.

[0048] In actual implementation, algorithms such as LIO-SLAM (e.g., the rotation-adaptive IESKF algorithm) can be used to analyze the current frame point cloud data based on the angular velocity information of the tower crane's boom and the pose information of the LiDAR when collecting the current frame point cloud data. This allows for the real-time estimation of the tower crane's pose and motion state, and the construction of a map of the tower crane's working environment, which serves as the working scene model corresponding to the current frame point cloud data.

[0049] For ease of description, in the various embodiments of this application, the coordinate system of the tower crane boom can be denoted as B (the origin is located at the luffing trolley), the coordinate system of the lidar can be denoted as L, and the coordinate system of the IMU (taking the IMU collecting the angular velocity information of the tower crane as an example) can be denoted as I.

[0050] It should be noted that the state vector X used in the execution of the LIO-SLAM algorithm can be represented as: (1).

[0051] The state vector X can include three parts: the pose information of the lidar, the IMU zero bias, and the arm angular velocity. The lidar pose information can include p, v, and q, where p represents position, v represents velocity, and q represents attitude. The IMU zero bias can include the zero bias b of the angular velocity acquired by the IMU. g and the zero bias of angular acceleration b a The angular velocity of the boom can include the angular velocity of the boom's rotation. .

[0052] It should be noted that, in addition to the conventional states such as the pose information of the lidar and the zero bias of the IMU, the state vector X in this embodiment also includes motion parameters specific to tower cranes (which may include the angular velocity of the boom rotation). This is considered as a state to be estimated. Based on this, the filter can simultaneously estimate the motion of the tower crane body and the rotational motion of the boom. During the filter's prediction phase, data obtained from the luffing trolley encoder can be used. By combining prior information with angular velocity data collected by the IMU, the motion trend of the tower crane in rotation can be predicted more accurately. Motion model fusion is performed, and the pose and motion state of the tower crane are estimated in real time based on this, thereby obtaining the working scene model corresponding to the current frame point cloud data.

[0053] It should be noted that the state vectors of traditional LIO systems do not include the rotational motion unique to tower cranes. Direct use of these vectors leads to inaccurate estimations of the tower crane's own motion trajectory and environmental feature coordinates, resulting in an inaccurate working scene model of the tower crane. However, the embodiments of this application include the tower crane's unique rotational motion in the state vectors. Using the LIO-SLAM algorithm, the tower crane's own motion trajectory and environmental feature coordinates can be estimated more accurately, thus obtaining a more accurate working scene model of the tower crane.

[0054] In some embodiments, the working scene model corresponding to the current frame point cloud data can be in the form of a globally consistent dense 3D point cloud map.

[0055] In some embodiments, the above-mentioned three-dimensional point cloud map can be stored using a kd-tree (k-dimensional tree) to enable fast point cloud lookup.

[0056] According to the tower crane working scene modeling method provided in the embodiments of this application, by integrating the angular velocity information of the tower crane's boom when collecting point cloud data from the LiDAR during the real-time positioning and map construction process based on LiDAR inertial odometry, the movement trajectory and environmental feature coordinates of the tower crane itself can be estimated more accurately, resulting in a more accurate tower crane working scene model and improving the accuracy of tower crane working scene modeling.

[0057] In some embodiments of this application, the current frame point cloud data is analyzed based on angular velocity information and pose information to obtain the working scene model corresponding to the current frame point cloud data, including: for each point in the current frame point cloud data, the position information corresponding to each point is obtained based on angular velocity information and pose information; the position information includes the transformation matrix between the upper arm coordinate system and the world coordinate system.

[0058] In actual implementation, point-by-point motion compensation can be performed, that is, motion compensation is performed for each point in the current frame point cloud data.

[0059] In some embodiments, for each point in the current frame point cloud data, the precise attitude of the tower crane at the acquisition time of that point can be calculated in reverse based on the timestamp of that point (representing the acquisition time of that point) and the currently estimated motion state of the tower crane (including pose information and angular velocity information), and then the point can be subjected to reverse motion transformation to correct its coordinates to the same reference coordinate system.

[0060] In some embodiments, points are typically corrected to the same reference coordinate system at the end of the current frame point cloud data.

[0061] It should be noted that the point cloud data acquired by the lidar uses the lidar coordinate system L, and the reference coordinate system can be the world coordinate system. This application does not limit the specific reference coordinate system used.

[0062] In some embodiments, the coordinates of each point in the current frame point cloud data can be corrected to the same reference coordinate system, which can be represented by a transformation matrix between the big arm coordinate system and the world coordinate system.

[0063] Based on the location information of each point in the current frame point cloud data, obtain the working scene model corresponding to the current frame point cloud data.

[0064] In actual execution, after obtaining the location information corresponding to each point in the current frame point cloud data, the location of each point in the current frame point cloud data can be added to the existing tower crane working scene model, thereby obtaining the working scene model corresponding to the current frame point cloud data.

[0065] In some embodiments, a map can be built based on the location information of each point in the current frame point cloud data, serving as the working scene model corresponding to the current frame point cloud data.

[0066] According to the tower crane working scene modeling method provided in the embodiments of this application, by performing motion compensation for each point in the current frame point cloud data based on the angular velocity information and pose information of the tower crane, the position of each point in the obtained point cloud data is more accurately corrected to the same reference coordinate system, and the accuracy of the tower crane working scene model is higher, which can improve the accuracy of tower crane working scene modeling.

[0067] In some embodiments of this application, position information corresponding to each point is obtained based on angular velocity information, including: obtaining the angular velocity of the tower crane body and the angular velocity component corresponding to the boom rotation based on angular velocity information.

[0068] In practical implementation, the rotational component of the angular velocity information of the tower crane's boom can be decoupled when the lidar acquires the current frame point cloud data. The rotation sensed by the sensor used to acquire the tower crane's angular velocity information includes both the movement of the tower crane body and the rotation of the boom. Therefore, the angular velocity information of the tower crane acquired by this sensor can be decomposed into the angular velocity of the tower crane body and the angular velocity component corresponding to the boom rotation.

[0069] The decomposition of the collected angular velocity information of the tower crane can be represented by the following tower crane rotational motion compensation model: (2).

[0070] in, The observed or measured value of the angular velocity of a tower crane, i.e., the angular velocity of the tower crane measured by sensors such as IMU; This indicates the angular velocity of the tower crane body; This represents the angular velocity component corresponding to the rotation of the boom of the tower crane, which is caused by the boom rotation. This represents the mounting matrix of sensors such as IMUs, to avoid point cloud distortion caused by rotational motion; The angular velocity of the boom rotation can be obtained in real time through the encoder of the luffing trolley; This indicates the zero bias of the angular velocity acquired by sensors such as IMUs; Indicates noise.

[0071] In some embodiments, motion decomposition and angular velocity fusion can be performed, utilizing the angular velocity data of the tower crane measured by sensors such as IMUs and the angular velocity of the tower crane boom rotation obtained in real time from the luffing trolley encoder. The angular velocity of the tower crane measured by sensors such as IMU is decomposed as shown in formula (2).

[0072] Based on the angular velocity and pose information of the tower crane body, motion transformation is performed on each point to obtain the transformation matrix corresponding to each point; the transformation matrix is ​​used to indicate the position information corresponding to each point.

[0073] In actual execution, during the motion transformation of each point in the current frame point cloud data, the precise attitude of the tower crane at the acquisition time of that point is calculated. Instead of directly using the angular velocity of the tower crane measured by sensors such as IMU, the angular velocity of the tower crane body obtained separately is used to eliminate the influence of the tower crane boom rotation on the precise attitude calculation, thereby obtaining a more accurate transformation matrix and more accurate position information corresponding to each point.

[0074] According to the tower crane working scene modeling method provided in the embodiments of this application, by separating the angular velocity of the tower crane body and the angular velocity component corresponding to the boom rotation, in calculating the precise attitude of the tower crane at the acquisition time of each point in the current frame point cloud data, the angular velocity measured by the IMU is decomposed into two parts: the tower crane body motion and the boom rotation. The angular velocity of the tower crane body is used instead of the angular velocity of the tower crane measured by sensors such as IMU, which avoids misidentifying the boom rotation of the tower crane as the motion of the tower crane body. This fundamentally improves the accuracy of pose estimation and provides accurate data for point cloud distortion compensation. The position of each point in the acquired point cloud data is more accurately corrected to the same reference coordinate system, resulting in a higher accuracy of the tower crane working scene model and improving the accuracy of tower crane working scene modeling.

[0075] In some embodiments of this application, after performing motion transformation on each point based on the angular velocity of the tower crane body and obtaining the transformation matrix corresponding to each point, the method further includes: obtaining the point cloud residual of each point based on the position information of the target plane in the working scene.

[0076] In practical implementation, planar residual constraints can be used in the point cloud distortion compensation step of the LIO-SLAM algorithm. Planar residual constraints mean that in the SLAM optimization process, the distance from a point to a plane is used as the residual instead of the point-to-point distance as the error.

[0077] In some embodiments, the working environment of a tower crane (e.g., a construction site) contains numerous planar structures (e.g., ground, floors, or walls). Prior knowledge of these planar structures can be used to map them to planes in the tower crane's working environment model. The target plane can be at least one of the aforementioned planar structures; that is, the target plane can include at least one plane in the tower crane's working environment.

[0078] In some embodiments, the point cloud residual of each point in the current frame point cloud data The planar structural features commonly found in tower crane scenarios can be used, expressed as follows: (3).

[0079] in, The normal vector of the target plane for local fitting (from typical working scenarios of tower cranes, such as ground or building facades). Indicates the coordinates of the center point of the target plane; This represents the transformation matrix between the upper arm coordinate system B and the world coordinate system; This indicates the coordinates of a point in the current frame point cloud data in the lidar coordinate system L. The superscript (L) indicates the lidar coordinate system L.

[0080] In some embodiments, feature extraction is performed on the distortion-compensated point cloud. Planar features (such as points in the plane and / or the plane's normal vector) can be extracted first to calculate the distance from the point to the plane. The calculation result is used as the point cloud residual. The transformation matrix corresponding to each point is updated by iterating based on the point cloud residual of each point.

[0081] In actual implementation, the transformation matrix obtained through point-by-point motion compensation can be used as the matrix in formula (3). The initial value is then used, and subsequent iterations are performed based on the point cloud residual for each point, with the transformation matrix corresponding to each point being... The transformation matrix is ​​updated to finally obtain the transformation matrix that satisfies the convergence condition. .

[0082] The tower crane working scene modeling method provided in the embodiments of this application obtains the point cloud residual of each point based on the position information of the target plane in the working scene, which is more in line with the geometric characteristics of the tower crane working scene, can provide stronger and more accurate constraints, can effectively reduce the cumulative drift of the odometer, and can improve the construction accuracy and efficiency of tower crane working scene modeling.

[0083] In some embodiments of this application, the working scene model corresponding to the current frame point cloud data is obtained based on the location information corresponding to each point in the current frame point cloud data. This includes: obtaining the working scene model corresponding to the current frame point cloud data based on the location information corresponding to each point in the current frame point cloud data and the location information of the target point in the historical working scene model; the historical working scene model is the working scene model corresponding to the point cloud data of each frame before the current frame; the target point is the point whose distance from each point in the current frame point cloud data is less than a target threshold.

[0084] In practice, incremental map management can be used to build locally consistent environment maps, generating high-precision, low-drift work scenario models, which are more suitable for further work such as path planning and obstacle avoidance for tower cranes.

[0085] In some embodiments, a local map based on a kd-tree can be maintained in real time. This map is a working scene model corresponding to the current frame point cloud data, and it is dynamically updated as the tower crane moves. Only the valid point cloud closest to the current state is retained. When updating the working scene model based on the current frame point cloud data, it is not necessary to perform calculations based on all historical point clouds. Only the current frame point cloud data and the valid point cloud closest to the current state are used for calculations. This ensures the real-time performance of the map while controlling the computational complexity, meeting the requirements of real-time operation.

[0086] In some embodiments, based on the current frame point cloud data and the working scene model corresponding to the point cloud data of each frame before the current frame (which can be called the "historical working scene model"), points in the historical working scene model whose distance from each point in the current frame point cloud data is less than a target threshold can be identified. These points are taken as target points (or valid points), and the point cloud formed by these target points (or valid points) (i.e., the aforementioned "valid point cloud") is the valid point cloud closest to the current state. That is, based on the distance between points in the historical point cloud and points in the current frame point cloud, the valid point cloud closest to the current state is determined.

[0087] According to the tower crane working scene modeling method provided in the embodiments of this application, by performing incremental map management, the real-time performance of tower crane working scene modeling can be guaranteed, and the computational complexity can be controlled to avoid excessively complex computations, thereby meeting the requirements of real-time operation.

[0088] In some embodiments of this application, obtaining the angular velocity information of the boom of the tower crane when the lidar collects the current frame point cloud data includes: obtaining the angular velocity information collected by the inertial measurement unit set on the luffing trolley.

[0089] In actual operation, the angular velocity information of the tower crane's boom when the lidar collects the current frame point cloud data can be collected by the inertial measurement unit set on the luffing trolley.

[0090] In some embodiments, the inertial measurement unit can acquire angular velocity information at extremely high frequencies (e.g., greater than or equal to 200 Hz).

[0091] In some embodiments, the inertial measurement unit (IMU) can be integrated with the lidar, i.e., the IMU is built-in. For example, a lidar with a built-in IMU can be used.

[0092] Given the characteristics of tower cranes, such as high-altitude vibration and high rotation speed, the lidar with built-in IMU needs to meet installation constraints.

[0093] The installation transformation matrix of the lidar with built-in IMU (representing the translation relationship from the upper arm coordinate system B to the lidar coordinate system L) satisfies: (4).

[0094] in, Represents the rotation matrix about the gravity axis; This indicates the installation offset.

[0095] In some embodiments, when the lidar has a built-in inertial measurement unit, the angle between the horizontal axis and the direction of gravity in the lidar coordinate system used by the lidar is less than an angle threshold.

[0096] In some embodiments, the core constraints of the lidar installation with a built-in IMU include: the X-axis in the lidar coordinate system L ( The angle between the direction of gravity and the direction of gravity must satisfy the following conditions. (Right now This is to suppress the impact of tower point cloud tilt on the system.

[0097] It should be noted that 5° is a preset angle threshold. The angle threshold is not limited to 5°. Usually, the angle threshold can be less than or equal to 5°. The angle threshold is a very small value, such as less than or equal to 5°.

[0098] In some embodiments, the inertial measurement unit (IMU) can be separate from the lidar, i.e., an external IMU. For example, a lidar without a built-in IMU can be used, with a separate IMU installed.

[0099] In some embodiments, when the lidar and the inertial measurement unit are set separately, the distance between the lidar and the inertial measurement unit is less than a distance threshold.

[0100] In some embodiments, for a lidar with an external inertial measurement unit, the calibration constraint equation can be expressed as: (5).

[0101] in, Let be the rotation matrix, representing the rotation relationship from the lidar coordinate system L to the IMU coordinate system I; This indicates the distance between the lidar and the IMU.

[0102] It should be noted that 0.01m is a preset distance threshold. The distance threshold is not limited to 0.01m. The positions of the lidar and the inertial measurement unit need to be set very close, so the distance threshold is a very small value, such as less than or equal to 0.01m.

[0103] The constraint relationship expressed by the above calibration constraint equation can eliminate the external parameter offset caused by the deformation of the steel structure of the tower crane, and ensure the consistency between the angular velocity acquired by the IMU and the point cloud motion.

[0104] According to the tower crane working scene modeling method provided in the embodiments of this application, the angular velocity information of the tower crane is collected by the inertial measurement unit set on the luffing trolley. It can realize the angular velocity information of the tower crane's boom when the point cloud data collected by lidar is fused. It can more accurately estimate the tower crane's own motion trajectory and environmental feature coordinates, and obtain a more accurate tower crane working scene model, thereby improving the accuracy of tower crane working scene modeling.

[0105] It should be noted that the tower crane working scene modeling method provided by any of the above embodiments of this application can output high-precision real-time pose and globally consistent dense 3D point cloud map.

[0106] High-precision real-time pose measurement of LiDAR refers to the precise position and orientation of the LiDAR point cloud in the global coordinate system (World Frame). = [R|t]. The high-precision real-time pose of lidar can be used to determine "where the point cloud is on the tower crane", that is, the positional relationship between the point cloud and the tower crane.

[0107] A globally consistent dense 3D point cloud map is the core output of this tower crane working scene modeling method. This 3D point cloud map is a 3D model containing all the details of the tower crane's surrounding environment (including buildings, steel bars, other tower cranes, and obstacles, etc.) with almost no cumulative error, and can be used as a model of the tower crane's working scene.

[0108] This 3D point cloud map can be used to understand the "working environment" of a tower crane, providing a model for calculating spatial relationships for the tower crane's collision avoidance system.

[0109] This 3D point cloud map can also be used to provide a computational environment for path planning of tower crane operations. Autonomous path planning algorithms can run on this high-precision map to find one or more collision-free paths from point A to point B.

[0110] The tower crane working scene modeling method provided in this application can be executed by a tower crane working scene modeling device. This application uses the tower crane working scene modeling device executing the tower crane working scene modeling method as an example to illustrate the tower crane working scene modeling device provided in this application.

[0111] This application also provides a tower crane working scene modeling device. For example... Figure 2 As shown, the tower crane working scene modeling device includes: an acquisition module 210 and a modeling module 220.

[0112] The acquisition module 210 is used to acquire the current frame point cloud data of the tower crane's working scene and the pose information of the laser radar collected by the laser radar set on the luffing trolley of the tower crane during the rotation movement of the tower crane's boom, and to acquire the angular velocity information of the tower crane's boom when the laser radar collects the current frame point cloud data. The modeling module 220 is used to analyze the current frame point cloud data based on angular velocity information and pose information, and obtain the working scene model corresponding to the current frame point cloud data.

[0113] According to the tower crane working scene modeling device provided in the embodiments of this application, by integrating the angular velocity information of the tower crane's boom when collecting point cloud data from the LiDAR during the real-time positioning and map construction process based on LiDAR inertial odometry, the device can more accurately estimate the tower crane's own motion trajectory and environmental feature coordinates, thereby obtaining a more accurate tower crane working scene model and improving the accuracy of tower crane working scene modeling.

[0114] In some embodiments, the modeling module 220 may include: The motion compensation unit is used to obtain the position information of each point in the current frame point cloud data based on angular velocity and pose information; the position information includes the transformation matrix between the upper arm coordinate system and the world coordinate system. The map building unit is used to obtain the working scene model corresponding to the current frame point cloud data based on the location information of each point in the current frame point cloud data.

[0115] In some embodiments, the motion compensation unit may include: The motion decomposition subunit is used to obtain the angular velocity of the tower crane body and the angular velocity components corresponding to the boom rotation based on angular velocity information. The motion transformation subunit is used to perform motion transformation on each point based on the angular velocity and pose information of the tower crane body, and obtain the transformation matrix corresponding to each point; the transformation matrix is ​​used to indicate the position information corresponding to each point.

[0116] In some embodiments, the modeling module 220 may further include: The residual processing unit is used to obtain the point cloud residual of each point based on the position information of the target plane in the working scene; and to iterate based on the point cloud residual of each point to update the transformation matrix corresponding to each point.

[0117] In some embodiments, the map building unit can be specifically used to obtain the working scene model corresponding to the current frame point cloud data based on the location information corresponding to each point in the current frame point cloud data and the location information of points in the working scene model corresponding to the point cloud data of previous frames that are less than a target threshold distance from each point in the current frame point cloud data.

[0118] In some embodiments, the acquisition module 210 may be specifically used to acquire angular velocity information collected by the inertial measurement unit set on the luffing trolley.

[0119] The tower crane working scene modeling device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0120] The tower crane work scene modeling device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0121] The tower crane working scene modeling device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0122] In some embodiments, such as Figure 3 As shown, this application embodiment also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the computer program is executed by the processor 301, it implements the various processes of the above-described tower crane working scene modeling method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0123] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0124] This application also provides a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described tower crane working scene modeling method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0125] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0126] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described tower crane working scene modeling method.

[0127] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0128] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described tower crane working scene modeling method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0129] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0130] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0132] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0133] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0134] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for modeling the working scene of a tower crane, characterized in that, include: During the boom rotation of the tower crane, the current frame point cloud data of the tower crane's working scene and the pose information of the laser radar collected by the laser radar set on the luffing trolley of the tower crane are obtained, and the angular velocity information of the tower crane when the laser radar collects the current frame point cloud data is obtained. Based on the angular velocity information and the pose information, the current frame point cloud data is analyzed to obtain the working scene model corresponding to the current frame point cloud data.

2. The tower crane working scene modeling method according to claim 1, characterized in that, The step of analyzing the current frame point cloud data based on the angular velocity information and the pose information to obtain the working scene model corresponding to the current frame point cloud data includes: For each point in the current frame point cloud data, based on the angular velocity information and the pose information, the position information corresponding to each point is obtained; the position information includes the transformation matrix between the upper arm coordinate system and the world coordinate system; Based on the location information corresponding to each point in the current frame point cloud data, the working scene model corresponding to the current frame point cloud data is obtained.

3. The tower crane working scene modeling method according to claim 2, characterized in that, The step of obtaining the position information corresponding to each point based on the angular velocity information includes: Based on the angular velocity information, the angular velocity of the tower crane body and the angular velocity component corresponding to the rotation of the boom are obtained; Based on the angular velocity of the tower crane body and the pose information, a motion transformation is performed on each point to obtain the transformation matrix corresponding to each point; the transformation matrix is ​​used to indicate the position information corresponding to each point.

4. The tower crane working scene modeling method according to claim 3, characterized in that, After performing motion transformation on each point based on the angular velocity of the tower crane body and obtaining the transformation matrix corresponding to each point, the method further includes: Based on the position information of the target plane in the work scenario, the point cloud residual of each point is obtained; The transformation matrix corresponding to each point is updated by iterating based on the point cloud residual of each point.

5. The tower crane working scene modeling method according to claim 2, characterized in that, The step of obtaining the working scene model corresponding to the current frame point cloud data based on the position information corresponding to each point in the current frame point cloud data includes: Based on the location information corresponding to each point in the current frame point cloud data and the location information of the target point in the historical work scene model, the work scene model corresponding to the current frame point cloud data is obtained; the historical work scene model is the work scene model corresponding to the point cloud data of each frame before the current frame; the target point is the point whose distance to each point in the current frame point cloud data is less than a target threshold.

6. The tower crane working scene modeling method according to any one of claims 1 to 5, characterized in that, The step of acquiring the angular velocity information of the tower crane when the lidar collects the current frame point cloud data includes: The angular velocity information collected by the inertial measurement unit set on the luffing trolley is obtained.

7. The tower crane working scene modeling method according to claim 6, characterized in that, When the inertial measurement unit is built into the lidar, the angle between the horizontal axis and the direction of gravity in the lidar coordinate system used by the lidar is less than the angle threshold.

8. The tower crane working scene modeling method according to claim 6, characterized in that, When the lidar and the inertial measurement unit are set separately, the distance between the lidar and the inertial measurement unit is less than a distance threshold.

9. A tower crane working scene modeling device, characterized in that, The acquisition module is used to acquire the current frame point cloud data of the tower crane's working scene and the pose information of the laser radar collected by the laser radar set on the luffing trolley of the tower crane during the boom rotation movement of the tower crane, and to acquire the angular velocity information of the tower crane when the laser radar collects the current frame point cloud data. The modeling module is used to analyze the current frame point cloud data based on the angular velocity information and the pose information, and obtain the working scene model corresponding to the current frame point cloud data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the tower crane working scene modeling method as described in any one of claims 1-8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the tower crane working scene modeling method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the tower crane working scene modeling method as described in any one of claims 1-8.

Images