A tower crane operation control system based on three-dimensional real-time modeling of complex scenes

Abstract

The present application relates to a kind of tower crane operation control system based on complex scene three-dimensional real-time modeling, the system is positioned to hook rough positioning by hook positioning and state sensing unit, utilize laser radar point cloud clustering algorithm to combine historical pose data and locate real-time position, synchronously with the aid of tower top camera AI visual tracking;Real-time point cloud data is converted into 3D voxel grid map, and global path is generated using 3DA algorithm, and after smoothing processing and trajectory optimization, hoisting track is output;Establish sling-hook double swing dynamics model, predict state sequence based on model predictive control algorithm, adjust control signal by feedforward compensation term and feedback correction term;Man-machine interaction and monitoring unit, real-time display tower crane cantilever angle, hook height, three-dimensional map and remote intervention to tower crane operating state.The system fuses multi-source data to construct high-precision three-dimensional map, realizes hook positioning and full-view tracking, improves hoisting safety and trajectory tracking accuracy through path planning and dynamics control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tower crane control, and in particular to a tower crane operation control system based on three-dimensional real-time modeling of complex scenes. BACKGROUND

[0002] The construction site environment is complex and dynamically changes fast, and there are various obstacles such as steel bars and scaffolds. The tower crane hook movement path is long, there are many blind spots in the field of vision, and there are many safety hazards.

[0003] The complexity of the construction site environment is reflected in three aspects: first, the environmental elements are complex, and there are static obstacles such as steel bars and scaffolds at the construction site, and dynamic occlusion will be caused by the rotation of the boom and the movement of the hook during the operation of the tower crane; second, the sensing conditions are poor, and strong light reflection and dust occlusion cause laser radar point cloud distortion and camera image blur, such as camera needing to process dust interference through adaptive threshold segmentation and morphological operation; third, the motion control challenge, the hook is easily occluded when moving between multiple scaffolds, and the double-swing motion caused by the flexibility of the sling is prone to collision in complex paths.

[0004] Traditional tower crane operation relies on manual operation, and there are problems such as strong experience dependence, limited precision, poor ability to cope with complex environments, high operation intensity and high safety risk. At the same time, the traditional anti-collision system relies on a single sensor and cannot intuitively display the dynamic changes in three-dimensional space in real time; the data visualization is insufficient, and it is difficult to display key parameters in real time; the efficiency of manual operation and collaborative work is low, and collisions are easily caused by misoperation. SUMMARY

[0005] In order to solve the problems mentioned in the background art, the present application provides a tower crane operation control system based on three-dimensional real-time modeling of complex scenes.

[0006] The tower crane operation control system based on three-dimensional real-time modeling of complex scenes provided by the present application adopts the following technical solution:

[0007] The tower crane operation control system based on three-dimensional real-time modeling of complex scenes comprises:

[0008] An environment perception and data acquisition unit is used to obtain scene point cloud data through a laser radar installed at the top of the tower crane, to collect hook movement parameters, and to obtain a visual image of the hook through a tower top camera;

[0009] A point cloud preprocessing and three-dimensional modeling unit is used to preprocess the original point cloud data of the laser radar, to generate high-quality point cloud data, and to construct and update a three-dimensional dynamic map in real time based on point cloud registration and fusion algorithms;

[0010] The hook positioning and state sensing unit is used for rough positioning of the hook position through IMU sensors, rope length sensors and boom angle parameters; is used for positioning the real-time position of the hook in the candidate area according to historical pose data through a laser radar point cloud clustering algorithm, and is used for synchronously utilizing AI vision of a tower top camera to dynamically track the hook in a full view angle.

[0011] The obstacle avoidance path planning unit is used for converting real-time point cloud data into a 3D voxel grid map, extracting a passable area and performing path channel inflation; is used for generating a collision-free global path by using a 3DA algorithm, smoothing the collision-free global path trajectory and performing trajectory-level optimization, and outputting a safe and smooth hoisting trajectory.

[0012] The hook motion control unit is used for establishing a sling-hook double-swing dynamics model, predicting a future state sequence according to real-time trajectory data based on a model predictive control algorithm; is used for adjusting a control signal through a feedforward compensation term and a feedback correction term based on the predicted future state sequence, and outputting a motion control signal containing speed and acceleration constraints, so as to suppress hook swing and track a target trajectory.

[0013] The human-computer interaction and monitoring unit is used for real-time display of a tower crane boom angle, hook height and a three-dimensional map, and supports a management personnel to monitor a site state through a visual interface; is used for integrating an emergency brake, mode switching control interface and remote intervention on a tower crane operation state.

[0014] Further, the hook position is roughly positioned through IMU sensors, rope length sensors and boom angle parameters, including: calculating the position of the boom end in a global coordinate system according to the boom length, pitch angle and rotation angle; performing coordinate transformation to convert the local offset into a global coordinate according to the displacement offset of the hook relative to the boom end calculated by the IMU, and obtaining the rough positioning coordinates of the hook.

[0015] Further, the real-time position of the hook is positioned in the candidate area according to historical pose data through a laser radar point cloud clustering algorithm, including: defining a candidate area with the rough positioning position of the hook in the global coordinate system as the center, extracting laser radar point cloud data located in the candidate area, performing spatial clustering on the point cloud in the candidate area, segmenting the point cloud into multiple dense point cloud blocks, constructing a hook motion trajectory prediction model by using historical pose data, predicting the possible pose of the hook at the current time, generating a predicted pose range, matching the centroid position of each point cloud block with the predicted pose range, screening out the point cloud blocks with a deviation from the predicted pose within a preset accuracy range, and determining the real-time accurate position of the hook according to the point cloud density and geometric shape features of the block.

[0016] Further, the AI vision of the overhead camera is used for full-view dynamic tracking of the hook, including: pre-processing the images collected by the camera to determine the metal profile of the hook by adaptive threshold segmentation, remove background noise, generate a binary target candidate area, input the pre-processed image into the EfficientNet-Lite model, perform data enhancement on the EfficientNet-Lite model, add a hook profile feature classifier to the full connection layer of the EfficientNet-Lite model, input the detection result of the EfficientNet-Lite model into the improved DeepSORT tracking framework, fuse the hook motion dynamics model and camera pose parameters, use double matching processing for occlusion scenes, use extended Kalman filtering to predict the three-dimensional position of the target, and then perform cross-frame association based on the Euclidean distance and apparent feature similarity, and continuously track the hook under the occlusion of the multi-layer scaffold.

[0017] Further, the real-time point cloud data is converted into a 3D voxel grid map, and the passable area is extracted and the path channel is inflated, including: first, set the voxel size of the 3D voxel grid map, divide the real-time laser radar point cloud data into corresponding voxel units according to the spatial coordinates, merge the historical frame point cloud through a sliding window, mark whether each voxel is occupied by an obstacle, then extract all unoccupied voxels as passable areas, and then inflate the path channel of the passable area in three dimensions according to the maximum swing length of the hook and the safety distance requirement, and expand the channel boundary by several times the safety distance along the motion direction of the hook and the space direction that may be involved in the swing, to form a smooth and continuous passable space.

[0018] Further, a 3DA algorithm is used to generate a collision-free global path, including: converting the current position and target position of the hook in the three-dimensional voxel grid map into grid coordinates as the start and end points of the path, taking the voxel grid as a node, calculating the heuristic cost and actual cost of each node to the target node, and gradually searching out a collision-free global polyline path from the start point to the end point without passing through the occupied voxels by the obstacle by preferentially expanding the node with the smallest cost function. The dynamically updated three-dimensional map is called in real time to obtain obstacle distribution information to avoid newly added obstacles.

[0019] Further, the trajectory of the collision-free global path is smoothed, including: for the polyline segment trajectory generated by the 3DA algorithm, setting the Bezier curve control points according to the endpoints of each polyline and the direction vectors of adjacent polylines, fitting a curve for each polyline, adjusting the control points to make the curve satisfy the first-order derivative continuity at the connection, eliminating the sharp corners of the polyline path, and generating a smooth trajectory with continuous curvature.

[0020] Further, trajectory-level optimization is performed, including: using a TEB algorithm to perform time parameterization optimization on the trajectory, taking the maximum speed, acceleration, acceleration constraint and hook swing amplitude limit of the tower crane operation as the optimization condition, and taking the minimum trajectory operation time, energy consumption and trajectory curvature fluctuation as the objective function; dynamically adjusting the time distribution and curve parameters of each section of the trajectory; outputting a smooth hoisting trajectory containing the speed and position instructions of each motion axis, and converting the trajectory into a control signal sequence of each actuator of the tower crane through kinematic inverse solution.

[0021] Further, a double-swing dynamics model of the sling-hook is established, including: the sling is equivalent to a massless rigid rod, a double-swing coupled dynamics model is constructed, the model input integrates the linear velocity of the luffing mechanism, the angular velocity of the slewing mechanism, the rope length change rate of the hoisting mechanism, and the boom pitch angle, the slewing angle, the sling length and the hook attitude data, the state vector is defined as the three-dimensional position and velocity of the hook in the global coordinate system, and the swing angle and swing angular velocity in the luffing plane and the slewing plane, and the state space equation considering the coupling of the boom motion is established based on the Lagrange mechanics principle, and the swing dynamics behavior of the hook in the three-dimensional space is represented.

[0022] Further, based on the model predictive control algorithm, the future state sequence is predicted according to the real-time trajectory data, including: taking the target trajectory coordinate sequence output by the obstacle avoidance path planning unit as the reference input, setting a fixed prediction time domain and a control time domain, in each control period, the current hook state and the tower mechanism motion parameters are used to predict the state sequence in the future prediction time domain through the double-swing dynamics model.

[0023] The application has the beneficial technical effects that: the application realizes all-around and dead-angle-free perception and dynamic obstacle avoidance of the tower crane operation environment through the fusion of complex scene three-dimensional real-time modeling and AI visual tracking technology, and the specific core effects include:

[0024] A high-precision three-dimensional map is constructed based on laser radar point cloud data, the obstacle distribution is updated in real time, the spatial relationship between the hook and the environment is accurately mastered, and the problem of insufficient perception of traditional single sensor is solved.

[0025] The hook is realized safe and smooth path planning through 3DA algorithm according to Bezier curve smoothing processing; the dynamics modeling and feedforward+feedback control based on model predictive control effectively suppress the hook swing, and improve the hoisting safety and trajectory tracking accuracy.

[0026] The multi-source data of laser radar, IMU and rope length sensor are fused to realize the rough positioning and precise positioning of the hook, and the AI visual tracking module (EfficientNet+DeepSORT) is used to realize the full-view dynamic tracking of the hook, and a comprehensive and high-stability perception and control system is constructed.

[0027] Particularly: the application realizes three-dimensional real-time modeling through multi-source data fusion and dynamic processing: the point cloud data is collected by the laser radar at the top of the tower crane, combined with the parameters such as the boom angle and the hook posture obtained by the IMU sensor, the rope length sensor and the like, pretreated through voxel grid filtering, statistical outlier removal and the like, and then the current frame point cloud is registered with the existing map through the iterative closest point algorithm, the sliding window is used to fuse the recent frame point cloud into the world coordinate system, a 3D voxel grid map containing obstacle distribution is generated, and when a new obstacle is detected, local map updating is triggered, and dynamic refreshing is realized at the level of per second. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A method flow chart for the tower crane operation control system based on complex scene three-dimensional real-time modeling of the application;

[0029] Figure 2 A simplified structure diagram of the tower crane in the application;

[0030] Figure 3 A trajectory curve diagram for the tower crane in the application;

[0031] Figure 4 A composition block diagram of the tower crane operation control system based on complex scene three-dimensional real-time modeling of the application. DETAILED DESCRIPTION

[0032] First of all, it should be pointed out that the 3DA algorithm of the application refers to a three-dimensional A path planning algorithm improved based on deep learning, which integrates the obstacle density into the heuristic function to optimize the path selection, so as to realize more efficient path search in a complex environment.

[0033] In the specific implementation, the application discloses a tower crane operation control system based on complex scene three-dimensional real-time modeling, like Figure 4 , the tower crane operation control system based on complex scene three-dimensional real-time modeling, comprising:

[0034] The environment perception and data acquisition unit 100 is used to acquire scene point cloud data through the laser radar installed at the top of the tower crane, to acquire hook motion parameters, and to acquire hook visual images through the tower top camera.

[0035] In the implementation, reference is made to Figure 2 , a multi-line laser radar 2 (such as a 32-line or 64-line mechanical laser radar) is installed at the top of the tower crane, which covers-30° to +15° in the vertical field of view angle and 360° in the horizontal full range, and collects point cloud data at a frequency of 10Hz. The laser radar 2 is fixed at the top of the tower.

[0036] For the acquisition of hook motion parameters, a dual-axis tilt sensor is installed at the root of the boom to measure the pitch angle, an absolute encoder is integrated at the slewing bearing to obtain the rotation angle, a wireline displacement sensor is installed at the shaft end of the wire rope drum to monitor the rope length change in real time, and a nine-axis IMU module (including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer with a sampling rate of 100 Hz) is integrated in the hook body to perceive the motion posture.

[0037] A high-definition camera 1 is deployed at the top of the tower, pointing 45° forward, directly forward, or 45° backward (the angle of the high-definition camera 1 can be adjusted), the lens is equipped with an infrared filter and an automatic aperture, and the video stream is transmitted to the edge computing unit through a gigabit Ethernet for processing by the AI vision tracking module. All sensors are time-stamped through a time synchronization module, and the laser radar point cloud data, sensor parameters, and camera video stream are transmitted in real time to the central processing unit through a gigabit industrial Ethernet to form a multi-source fused raw data set.

[0038] The point cloud preprocessing and three-dimensional modeling unit 200 is used for preprocessing the laser radar raw point cloud data to generate high-quality point cloud data, and for constructing and updating a three-dimensional dynamic map containing obstacle distribution and terrain features in real time based on point cloud registration and fusion algorithms.

[0039] The preprocessing of the laser radar raw point cloud data in the implementation is specifically filtering, using voxel grid filtering to reduce data density, and using statistical outlier removal algorithm to remove noise points to ensure reasonable sparsity of the point cloud. For point cloud distortion, motion compensation is performed using the synchronously collected IMU data to correct the position deviation caused by the tower crane micro-motion and achieve distortion removal.

[0040] The preprocessed point cloud enters the registration link, and the current frame of point cloud is matched with the existing three-dimensional map by means of the iterative closest point algorithm or the normal distribution transformation algorithm to estimate the tower crane pose. When fusing point clouds, a sliding window is set to retain only the effective point clouds of the recent frames, which are converted to the world coordinate system through pose transformation, and the overlapping areas are merged by voxel fusion to enhance the map density.

[0041] The real-time monitoring of point cloud data changes triggers local map updating when new obstacles are detected, and the incremental registration and fusion are used to update the changed areas to provide real-time and accurate three-dimensional spatial information for obstacle avoidance path planning and dynamic hook limiting.

[0042] The hook positioning and state sensing unit 300 is used for rough positioning of the hook position through an IMU sensor, a rope length sensor, and a boom angle parameter. In implementation, specifically, the IMU sensor is installed on the hook body, real-time three-axis acceleration and angular velocity data are collected, real-time attitude angle and linear acceleration information of the hook are obtained through attitude solving, and are used to represent the motion trend of the hook in the local coordinate system. The rope length sensor is integrated in the tower crane hoist system, the current effective length L of the hoisting rope is accurately obtained by measuring the number of revolutions of the drum or the displacement of the steel wire rope, the luffing angle θ (the angle between the boom and the horizontal plane) is collected in real time by the inclination sensor installed at the hinge point of the boom root, and the rotation angle φ (the rotation angle of the tower crane relative to the reference direction) is obtained through the angle encoder or absolute value encoder on the slewing bearing. Based on the above data, first, the position (X0, Y0, Z0) of the end of the boom in the global coordinate system is calculated according to the boom length R, the luffing angle θ, and the rotation angle φ, and the formula is X0 = R x cos θ x cos φ, Y0 = R x cos θ x sin φ, and Z0 = R x sin φ; then, the local offset (Δx, Δy, Δz) of the hook relative to the end of the boom is converted into global coordinates through coordinate transformation according to the rope length L and the preliminary solution of the IMU, and the rough positioning coordinates of the hook are obtained by superposition.

[0043] The hook positioning and state sensing unit 300 is used for positioning the real-time position of the hook in the candidate area according to historical pose data through a laser radar point cloud clustering algorithm. In implementation, specifically, first, the rough positioning position of the hook in the global coordinate system is calculated, a three-dimensional candidate area containing a prediction error range is defined with the rough positioning position as the center, and the point clouds in the area are extracted from the laser radar point cloud data. Then, the point clouds in the candidate area are subjected to spatial clustering processing, a Euclidean distance clustering algorithm is adopted, a distance threshold suitable for the point cloud characteristics of the hook is set, and the point clouds are segmented into multiple dense point cloud clusters. At the same time, a hook motion trajectory prediction model is constructed by using historical pose data, the possible pose of the hook at the current time is predicted through a Kalman filtering algorithm, and a prediction position range is generated. Then, the centroid position of each point cloud cluster is matched with the prediction pose range, the point cloud cluster with a deviation from the prediction pose within a preset accuracy range is screened out, the real-time accurate position of the hook is finally determined according to the point cloud density and geometric shape features (such as the point cloud distribution characteristics conforming to the small and compact hook), and accurate positioning of the hook in a complex environment is realized when the hook is blocked or moves rapidly.

[0044] The hook positioning and state sensing unit 300 is used for synchronous dynamic tracking of the full-view angle of the hook by AI vision of the tower top camera. In implementation, a multi-view vision system is installed on the tower top, the images collected by the camera are preprocessed by the edge computing unit, the adaptive threshold segmentation is used to highlight the metal profile of the hook, the background noise is removed by morphological opening operation, and the binary target candidate area is generated. The preprocessed image is input into the lightweight EfficientNet-Lite model, which introduces transfer learning in the training stage, performs data enhancement on the strong light reflection and dust shielding scenes commonly seen in tower operation, adds a hook profile feature classifier in the full connection layer, and controls the single-frame detection time within 200 ms, which can accurately output the pixel coordinates and size information of the hook head. The detection result is connected to the improved DeepSORT tracking framework, the hook motion dynamics model and the camera pose parameters are fused, the double matching strategy (appearance feature matching + motion trajectory prediction) is used to handle the shielding scene, the extended Kalman filter is used to predict the three-dimensional position of the target, the Euclidean distance and the apparent feature similarity are used to realize cross-frame association, and the tracking continuity of the hook under the shielding of the multi-layer scaffold is ensured.

[0045] The obstacle avoidance path planning unit 400 is used for converting real-time point cloud data into a 3D voxel grid map, extracting a passable area and performing path channel inflation. In implementation, first, the voxel size (such as 0.2m×0.2m×0.2m) of the 3D voxel grid map is set, the real-time laser radar point cloud data is divided into corresponding voxel units according to the spatial coordinates, the historical frame point clouds are merged through a sliding window (the window size can be set to the latest 10 frames of point cloud data), and each voxel is marked whether it is occupied by an obstacle. Then, all unoccupied voxels are extracted as passable areas, and low-height areas (ground debris area) with a height lower than 0.5m below the tower base plane and high-height areas (non-operation airspace) with a height higher than 2m above the maximum operation height of the tower are filtered out. Then, according to the maximum swing length of the hook (such as 50m of the sling length, swing range ±2m) and the safety distance requirement (such as an obstacle distance ≥1.5m), the path channel of the passable area is three-dimensionally inflated, the channel boundary is expanded by 1.2 times the safety distance (i.e. 1.8m) along the motion direction of the hook and the space direction possibly involved by the swing, and a smooth and continuous passable space is formed, providing a safety margin for subsequent path planning.

[0046] The obstacle avoidance path planning unit 400 is used for generating a collision-free global path by using a 3D A algorithm, smoothing the trajectory of the collision-free global path, and performing trajectory-level optimization, and outputting a safe and smooth hoisting trajectory.

[0047] In the implementation, the current position and the target position of the hook in the three-dimensional voxel grid map are first converted into grid coordinates as the start and end points of the path, the 3DA algorithm takes the voxel grid as the node, calculates the heuristic cost (using three-dimensional Euclidean distance) and the actual cost (adjacent node movement consumption) of each node to the target node, and gradually searches out the collision-free global broken line path from the start point to the end point without passing through the obstacle-occupied voxel by preferentially expanding the node with the minimum cost function. In the process, the dynamically updated three-dimensional map is called in real time to obtain the obstacle distribution information to avoid newly added obstacles.

[0048] For the broken line segment trajectory generated by the 3DA algorithm, the Bezier curve control points are set according to the end points of each broken line and the direction vectors of adjacent broken lines, the curve fitting is performed on each broken line, the control points are adjusted to make the curve satisfy the first derivative continuity at the connection, the sharp corners of the broken line path are eliminated, and the smooth trajectory with continuous curvature is generated. Referring to Figure 3 The transportation trajectory curve is "generating a smooth trajectory with continuous curvature".

[0049] On this basis, the TEB algorithm is used for time parameterization optimization of the trajectory, the maximum speed, acceleration, jerk constraints of the tower crane operation and the hook swing amplitude limit are used as optimization conditions, the trajectory running time, energy consumption and trajectory curvature fluctuation are minimized as the objective function, and the time distribution and curve parameters of each segment of the trajectory are dynamically adjusted, so that the optimized trajectory not only satisfies the mechanical motion characteristics of the tower crane, but also keeps the hook swing amplitude below the safety threshold during the hoisting process. Finally, a smooth hoisting trajectory containing the speed and position instructions of each motion axis is output, and the trajectory is converted into a control signal sequence of each actuator (luffing mechanism, slewing mechanism, hoisting mechanism) of the tower crane through kinematic inverse solution.

[0050] The hook motion control unit 500 is used to establish a sling-hook double-swing dynamics model, predict the future state sequence based on the model predictive control (MPC) algorithm, and predict the future state sequence according to real-time trajectory data.

[0051] In the implementation, the sling is equivalent to a massless rigid rod for the hook double-swing motion characteristics caused by the flexibility of the sling, a double-swing coupled dynamics model including the luffing direction and the slewing direction is constructed, the model input integrates the linear speed of the luffing mechanism, the angular velocity of the slewing mechanism, the rope length change rate of the hoisting mechanism, and the boom pitch angle (from the double-axis inclination sensor), the slewing angle (from the absolute encoder), the sling length (from the wire displacement sensor), and the hook attitude data (from the nine-axis IMU module). The state vector is defined as the three-dimensional position, velocity of the hook in the global coordinate system, and the swing angle, swing angle velocity in the luffing plane and the slewing plane, the state space equation considering the coupling of the boom motion is established based on the Lagrange mechanics principle, and the swing dynamics behavior of the hook in the three-dimensional space is represented.

[0052] The model predictive control (MPC) algorithm takes the target trajectory coordinate sequence output by the obstacle avoidance path planning unit as a reference input, sets a fixed prediction time domain (such as 100 ms in the future) and a control time domain (such as the first 20 ms), and in each control period (such as 10 ms), the current hook state (fusion of IMU attitude solution results, real-time data of the rope length sensor, and boom angle parameters) and tower mechanism motion parameters are used to predict the state sequence in the future prediction time domain through a double-swing dynamics model.

[0053] The cost function is designed as a weighted sum of the trajectory tracking error, the control input energy, and the amplitude of the hook swing angle, and the weight coefficients are dynamically adjusted according to the load mass (indirectly estimated by the rope length sensor), for example, the amplitude of the swing angle is increased to preferentially suppress the swing when the load is heavy. A quadratic programming solver is used to rollingly optimize the amplitude, rotation, and lifting mechanism control variables in the control time domain, and only the first cycle control command is output to the actuator, while the latest sensor feedback is received to update the model state in real time, ensuring that the hook accurately tracks the target trajectory and effectively suppresses the swing during complex motion.

[0054] The hook motion control unit 500 is used to output motion control signals containing speed and acceleration constraints by feeding forward compensation terms (based on the length of the hoisting rope and the estimated swing angle) and feedback correction terms (IMU real-time swing angle adjustment), to suppress the hook swing and track the target trajectory.

[0055] In implementation, the implementation of the feedforward compensation term is based on the effective length L of the hoisting rope collected by the hoisting rope length sensor and the acceleration data of each actuator (amplitude, rotation, and lifting) of the tower crane, and by constructing a hoisting rope-hook dynamics model, the estimated swing angle caused by the acceleration of the mechanism motion is calculated.

[0056] Specifically, when the amplitude mechanism moves laterally with acceleration ax, the rotation mechanism rotates with angular acceleration a, and the lifting mechanism moves vertically with acceleration az, the vector angle formed by the resultant tangential acceleration at and the acceleration of gravity g is the estimated swing angle feedforward , and the relationship is feedforward =arctan(at / g), where at=(ax²+(L×a)²) 1 / 2 .

[0057] According to the estimated swing angle, a compensation amount is superimposed in the speed and acceleration instructions of the amplitude, rotation, and lifting mechanisms, and the compensation amount is positively related to the swing angle and the length of the hoisting rope, so as to offset the trend of the hook swing caused by the acceleration of the mechanism in advance.

[0058] The feedback correction term uses a nine-axis IMU module integrated into the hook body to collect three-axis accelerometer data in real time. This data is then fused with gyroscope and magnetometer outputs via a complementary filtering algorithm to calculate the real-time swing angle of the hook relative to the direction of gravity. The deviation between the real-time swing angle and the ideal swing angle (set to 0°) is input to a proportional-integral-derivative (PID) controller. Based on preset control parameters (proportional coefficient Kp, integral coefficient Ki, and derivative coefficient Kd), the feedback correction is calculated. This correction is superimposed on the control signal output from the feedforward compensation term. Simultaneously, based on hard constraints such as the maximum speed of each actuator of the tower crane (e.g., maximum speed of the luffing mechanism, maximum angular velocity of the slewing mechanism, and maximum acceleration (e.g., maximum acceleration of the hoisting mechanism), a limiting algorithm generates the final motion control signal, which is applied to the drive motors of the luffing, slewing, and hoisting mechanisms to achieve dynamic suppression of hook sway and high-precision tracking of the target trajectory. The control signal output frequency is synchronized with the IMU sampling rate to ensure real-time response to hook attitude changes, keeping the swing angle within a safe threshold throughout the lifting process.

[0059] The Human-Machine Interaction and Monitoring Unit 600 is used to display the tower crane jib angle, hook height, and 3D map in real time, and supports managers to monitor the on-site status through a visual interface; it is also used to integrate emergency braking and mode switching control interfaces for remote intervention in the tower crane's operating status.

[0060] In practice, the system integrates and calculates real-time data on boom pitch angle (from a dual-axis tilt sensor at the boom root), slewing angle (from an absolute encoder on the slewing bearing), and rope length (from a wire-type displacement sensor at the drum shaft end) to obtain real-time cantilever angle and hook height information. Simultaneously, it utilizes point cloud preprocessing and a 3D modeling unit to generate a dynamic 3D map containing obstacle distribution. This data is transmitted to a monitoring server via gigabit industrial Ethernet. An interactive interface is built using a visualization software engine (such as Unity3D or WebGL) to render the tower crane operation scene in real-time as a 3D model. The cantilever angle is visually presented through the joint rotation of the virtual tower crane model, and the hook height is superimposed as a digital label on the 3D scene. The 3D map dynamically marks the position and outline of rebar, scaffolding obstacles, and other obstacles.

[0061] In the implementation of the remote intervention function, the monitoring interface sets a prominent red emergency brake button, and after clicking, the system immediately sends an emergency stop instruction to each actuator (amplitude, rotation, lifting mechanism), and at the same time triggers the hardware layer safety relay to cut off the power supply, ensuring that the tower crane stops all movements; the mode switching control interface supports switching between "manual operation", "automatic planning" and "semi-automatic mode". When switching from automatic mode to manual mode, the system automatically saves the current path planning state, and the operator can control the tower crane action through the virtual joystick or physical handle on the interface. At this time, the hook positioning and state sensing unit continues to monitor the hook position in real time. If it is detected that the safe distance from the obstacle is less than the threshold, the system automatically triggers the soft limit and issues an audible and visual alarm.

[0062] It is apparent that the system of the present application can be implemented by a computer program, and the computer program for implementing the system of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that the computer programs, when executed by the processor, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, and partially on a remote machine, or entirely on a remote machine or server.

[0063] As Figure 1 The control flow of the tower crane operation control system based on complex scene three-dimensional real-time modeling includes the following steps:

[0064] S100. Obtain scene point cloud data through the laser radar installed at the top of the tower crane, collect hook movement parameters, and obtain hook visual images through the tower top camera;

[0065] S200. Preprocess the original point cloud data of the laser radar to generate high-quality point cloud data; based on a point cloud registration and fusion algorithm, construct and real-time update a three-dimensional dynamic map containing obstacle distribution and terrain features;

[0066] S300. Coarsely position the hook position through the IMU sensor, rope length sensor, and hook arm angle parameters; based on historical pose data, position the real-time position of the hook in the candidate area through the laser radar point cloud clustering algorithm; simultaneously utilize the AI vision of the tower top camera for full-view dynamic tracking of the hook;

[0067] S400. Convert the real-time point cloud data into a 3D voxel grid map, extract the passable area and perform path channel dilation; generate a collision-free global path using a 3DA algorithm, smooth the collision-free global path trajectory, and perform trajectory-level optimization to output a safe and smooth hoisting trajectory;

[0068] S500. Establish a sling-hook double-swinging dynamics model, predict the future state sequence based on the model predictive control (MPC) algorithm and real-time trajectory data; output motion control signals containing speed and acceleration constraints through a feedforward compensation term (based on sling length and acceleration to estimate the swing angle) and a feedback correction term (IMU real-time swing angle adjustment), to suppress hook swing and track the target trajectory;

[0069] S600. Real-time display of tower crane cantilever angle, hook height and three-dimensional map, support management personnel to monitor the site state through the visual interface; used for integrated emergency braking, mode switching control interface, remote intervention on the tower crane operation state.

[0070] The application further discloses an electronic device, which comprises:

[0071] at least one processor; and

[0072] a memory connected with the at least one processor in communication; wherein,

[0073] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute one or more units of the tower crane operation control system based on three-dimensional real-time modeling of a complex scene or execute steps of one or more control processes of the tower crane operation control system based on three-dimensional real-time modeling of a complex scene.

[0074] The above are preferred embodiments of the application, but not limit the protection scope of the application, so: all equivalent changes made according to the principles of the application should be covered within the protection scope of the application.

Claims

1. A tower crane operation control system based on three-dimensional real-time modeling of complex scenes, characterized in that, Comprise: An environmental perception and data acquisition unit for acquiring scene point cloud data through a laser radar mounted at the top of the tower crane, for collecting hook motion parameters, and for acquiring a visual image of the hook through a tower top camera; A point cloud preprocessing and three-dimensional modeling unit for preprocessing raw point cloud data of the laser radar, for generating high-quality point cloud data, and for constructing and updating a three-dimensional dynamic map in real time based on point cloud registration and fusion algorithms; A hook positioning and state perception unit for coarsely positioning the hook position through IMU sensors, rope length sensors, and arm angle parameters, for positioning the real-time position of the hook within a candidate area based on historical pose data through a laser radar point cloud clustering algorithm, and for simultaneously utilizing AI vision of the tower top camera to dynamically track the full-view angle of the hook; An obstacle avoidance path planning unit for converting real-time point cloud data into a 3D voxel grid map, extracting a passable area, and performing path channel dilation, for generating a collision-free global path using a 3DA algorithm, smoothing the collision-free global path trajectory, and performing trajectory-level optimization to output a safe and smooth hoisting trajectory; A hook motion control unit for establishing a sling-hook double-swing dynamics model, predicting a future state sequence based on real-time trajectory data using a model predictive control algorithm, adjusting control signals based on the predicted future state sequence through a feedforward compensation term and a feedback correction term, and outputting motion control signals containing speed and acceleration constraints to suppress hook swing and track the target trajectory; A human-machine interaction and monitoring unit for real-time display of the tower crane jib angle, hook height, and three-dimensional map, supporting management personnel to monitor the on-site state through a visual interface, and for integrating emergency braking, mode switching control interfaces, and remote intervention on the tower crane operating state; Coarse positioning of the hook position through IMU sensors, rope length sensors, and arm angle parameters includes: calculating the position of the arm end in the global coordinate system based on the arm length, pitch angle, and rotation angle; and performing coordinate transformation to convert the local offset into global coordinates based on the rope length and the displacement offset of the hook relative to the arm end calculated by the IMU, to obtain the coarse positioning coordinates of the hook; Positioning the real-time position of the hook within a candidate area based on historical pose data through a laser radar point cloud clustering algorithm includes: defining a candidate area centered on the coarse positioning position of the hook in the global coordinate system, extracting laser radar point cloud data located in the candidate area, performing spatial clustering on the point cloud in the candidate area to segment the point cloud into multiple dense point cloud clusters, constructing a hook motion trajectory prediction model using historical pose data to predict the possible pose of the hook at the current time, generating a predicted pose range, matching the centroid position of each point cloud cluster with the predicted pose range, selecting point cloud clusters with a deviation from the predicted pose within a preset accuracy range, and determining the real-time accurate position of the hook based on the point cloud density and geometric shape features of the cluster.

2. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 1, characterized in that, AI vision of tower top camera for full-view dynamic tracking of hook, including: pre-processing the images collected by the camera to determine the hook metal profile by adaptive threshold segmentation, remove background noise, generate binary target candidate area, input the pre-processed image into the EfficientNet-Lite model, data augmentation for the EfficientNet-Lite model, add hook profile feature classifier to the full connection layer of the EfficientNet-Lite model, access the improved DeepSORT tracking framework from the detection result of the EfficientNet-Lite model, fuse the hook motion dynamics model and camera pose parameters, use double matching processing for occlusion scenes, use extended Kalman filter to predict the three-dimensional position of the target, then perform cross-frame association based on Euclidean distance and apparent feature similarity, and continuously track the hook under multi-layer scaffold occlusion.

3. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 2, characterized in that, Convert real-time point cloud data into a 3D voxel grid map, extract the passable area and perform path channel inflation, including: first, set the voxel size of the 3D voxel grid map, divide the real-time laser radar point cloud data into corresponding voxel units according to the spatial coordinates, merge the historical frame point cloud through a sliding window, mark each voxel as occupied or not occupied by obstacles, then extract all unoccupied voxels as passable areas, and then perform three-dimensional inflation on the path channel of the passable area according to the maximum swing length of the hook and the safety distance requirement, and expand the channel boundary by several times the safety distance in the direction of the hook movement and the space direction that may be involved in the swing, to form a smooth and continuous passable space.

4. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 3, characterized in that, Generate a collision-free global path using a 3DA algorithm, including: converting the current position and target position of the hook in the three-dimensional voxel grid map into grid coordinates as the start and end points of the path, taking the voxel grid as the node, calculating the heuristic cost and actual cost of each node to the target node, and gradually searching for a collision-free global polyline path from the start point to the end point without passing through occupied voxels by the obstacle by preferentially expanding the node with the smallest cost function, and calling the dynamically updated three-dimensional map in real time to obtain obstacle distribution information to avoid newly added obstacles.

5. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 4, characterized in that, Smooth the trajectory of the collision-free global path, including: for the polyline segment trajectory generated by the 3DA algorithm, setting the Bezier curve control points according to the endpoints of each polyline and the direction vectors of adjacent polylines, curve fitting for each polyline, adjusting the control points to make the curve satisfy the first-order derivative continuity at the connection, eliminating the sharp corners of the polyline path, and generating a smooth trajectory with continuous curvature.

6. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 5, characterized in that, Trajectory-level optimization, including: using the TEB algorithm to perform time parameterization optimization on the trajectory, taking the maximum speed, acceleration, acceleration constraint of the tower crane operation and the hook swing amplitude limit as the optimization conditions, and taking the minimization of the trajectory running time, energy consumption and trajectory curvature fluctuation as the objective function; dynamically adjusting the time distribution and curve parameters of each segment of the trajectory; outputting a smooth hoisting trajectory containing the speed and position instructions of each motion axis, and converting the trajectory into a control signal sequence of each actuator of the tower crane through kinematic inverse solution.

7. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 6, characterized in that, A double-swing dynamic model of the sling-hook is established, including: the sling is equivalent to a massless rigid rod, a double-swing coupling dynamic model is constructed, including the variable-amplitude direction and the rotation direction, the model inputs the linear velocity of the variable-amplitude mechanism, the angular velocity of the rotation mechanism, the rope length change rate of the lifting mechanism, and the data of the boom pitch angle, the rotation angle, the sling length and the hook posture, the state vector is defined as the three-dimensional position, the velocity of the hook in the global coordinate system, and the swing angle, the swing angular velocity in the variable-amplitude plane and the rotation plane, the state space equation considering the coupling of the boom motion is established based on the Lagrange mechanics principle, representing the swing dynamic behavior of the hook in the three-dimensional space.

8. The tower crane operation control system based on three-dimensional real-time modeling of complex scenes according to claim 7, characterized in that, Based on the model predictive control algorithm, the future state sequence is predicted according to the real-time trajectory data, including: taking the target trajectory coordinate sequence output by the obstacle avoidance path planning unit as the reference input, setting the fixed prediction time domain and the control time domain, in each control period, the current hook state and the tower mechanism motion parameters are used to predict the state sequence in the future prediction time domain through the double-swing dynamic model.

Images