An intelligent control method for tower crane hoisting based on reinforcement learning

Abstract

The present invention relates to a tower crane hoisting intelligent control method based on reinforcement learning. The method comprises the following steps: S1, constructing a simulated construction site; S2, constructing a minimum distance formula for the model; S3, adopting a dual tower crane path planning based on reinforcement learning. The specific steps of S3 are: constructing a policy network, setting an action space, a state space, and a reward function, wherein the action space satisfies the working range space of the model, the reward function is calculated based on the minimum distance formula, and training an intelligent agent based on a reinforcement learning algorithm; S4, performing intelligent control of tower crane hoisting based on the trained intelligent agent. Compared with the existing technology, the present invention has the advantages of realizing a multi-dimensional collaborative optimization mechanism, enabling the system to simultaneously meet the balanced requirements of safety, economy, and engineering efficiency in complex construction environments.

Description

Technical Field

[0001] The present invention relates to the technical field of tower crane hoisting intelligent control, and in particular to a tower crane hoisting intelligent control method based on reinforcement learning. Background Art

[0002] As core equipment in modern construction, tower cranes' operational safety and efficiency directly impact the overall progress of a project. With the construction industry's shift toward intelligent operations, traditional tower crane operation models that rely on manual experience are no longer able to meet the increasingly complex demands of construction. Multi-agent simulation technology offers new insights into tower crane operations, and existing research has attempted to simulate construction site dynamics through agent-based modeling. However, most of these models simplify the tower crane into a simple moving entity and fail to fully consider the multiple constraints inherent in actual operations. In particular, they lack quantitative analysis of key factors such as collision risk and energy optimization during the lifting process, resulting in significant discrepancies between simulation results and actual working conditions.

[0003] Reinforcement learning demonstrates unique advantages in solving complex decision-making problems, particularly those with Markov characteristics, such as tower crane path planning. Existing research has proposed mobile robot path planning methods based on reinforcement learning, but their application to tower crane operations faces three major challenges: First, the unique dynamics of tower crane motion (including the coordinated multi-degree-of-freedom of tower arm rotation, trolley movement, and hook raising and lowering), requiring specialized action space modeling; second, the dynamic nature of the construction site environment, with factors such as material stacking and other equipment movement leading to continuous state space updates; and finally, hoisting operations must simultaneously meet multi-objective optimization requirements, including safety, efficiency, and energy consumption, making traditional single-objective reward functions difficult to apply.

[0004] The core challenge of tower crane intelligence lies in real-time decision-making in dynamic environments. Unlike conventional mobile robots, tower crane operations are characterized by numerous temporary obstacles in the workspace, dynamically changing collision risks, strict time window constraints for lifting tasks, and the need to resolve task allocation conflicts when multiple towers work together. Existing methods have significant shortcomings in real-time collision detection and multi-objective optimization and balancing. In particular, they lack mechanisms to integrate construction specification knowledge into the decision-making process, resulting in low feasibility of planning solutions in practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a tower crane lifting intelligent control method based on reinforcement learning in order to realize a multi-dimensional collaborative optimization mechanism so that the system can simultaneously meet the balanced requirements of safety, economy and engineering efficiency in a complex construction environment.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] 1. A tower crane hoisting intelligent control method based on reinforcement learning, the method comprising the following steps:

[0008] S1. Construct a simulated construction site, including a rigid body model of a trolley-type luffing twin tower crane and other environmental entities, and construct the scope of the model's configuration space. The rigid body model of the trolley-type luffing twin tower crane includes a first tower crane and a second tower crane. Both tower cranes include a tower body, a tower arm, a slewing base, a tower crane base, a luffing trolley, a sling, a hook, and hoisted components.

[0009] S2, minimum distance formula for constructing the model;

[0010] S3, using reinforcement learning-based dual tower crane path planning. The specific steps of S3 are:

[0011] Build a policy network, set the action space, state space and reward function. The action space meets the working range of the model. The reward function is calculated based on the minimum distance formula. Use the reinforcement learning algorithm to train the two crane agents.

[0012] S4. Control the corresponding tower crane based on the trained intelligent agent.

[0013] Furthermore, the configuration space is:

[0014]

[0015] in, θ Refers to the angle between the tower arm and the X-axis of the local coordinate system, with counterclockwise being positive; r Refers to the distance from the luffing trolley to the tower arm's rotation center; h is the height difference between the hook and the tower arm rotation center; the subscripts 1 and 2 represent tower crane No. 1 and tower crane No. 2 respectively. The tower arm rotation center is the intersection of the rotation center axis of the tower body and the tower arm rotation plane. The tower arm rotation plane represents the plane swept by the center line of the tower arm when it rotates;

[0016] The local coordinate system satisfies:

[0017] Origin: The intersection of the tower body's central axis of rotation and the tower arm's rotation plane, that is, the center point where the tower arm rotates around the tower body;

[0018] Y-axis: vertical direction, coincides with the central axis of the tower, with upward as the positive direction, pointing to the sky;

[0019] X-axis: horizontal direction, a straight line passing through the origin and parallel to the main support direction of the tower crane base. The tower crane base is a flat plate base, which is a rectangular parallelepiped. The main support direction of the tower crane base is the direction of the long side of the contact surface between the rectangular parallelepiped and the ground.

[0020] Z axis: Horizontal direction, parallel to the short side of the cuboid's contact surface with the ground, and a straight line passing through the origin.

[0021] Furthermore, the range of the configuration space is:

[0022]

[0023] in, C is the configuration space, C= , the minimum limit distance and the maximum limit distance are r min and r max , the height difference from the tower arm rotation center to the ground is h c , subscripts 1 and 2 represent tower crane No. 1 and tower crane No. 2 respectively.

[0024] Furthermore, the specific steps of S2 are:

[0025] The other entities in the simulated construction site environment and the various components of the tower crane are fitted with a cubic structure that can only rotate around the Y axis. The axis-aligned directed bounding volumes corresponding to each entity and component are obtained. The minimum Y-axis distance and the minimum XZ-plane distance between the bounding volume of the hoisted component and the bounding volume of the obstacle are calculated. The root of the sum of the squares of the minimum Y-axis distance and the minimum XZ-plane distance is calculated as the minimum distance, and the formula for the minimum distance between the hoisted component and the obstacle is obtained.

[0026] If the hoisted component belongs to the No. 1 tower crane, the obstacle is the various components of the No. 2 tower crane or other environmental entities;

[0027] If the hoisted component belongs to the No. 2 tower crane, the obstacle is the various components of the No. 1 tower crane or other environmental entities.

[0028] Furthermore, the minimum distance on the Y axis is:

[0029] If the lowest point of the hoisting component is higher than the highest point of the obstacle, ,in, D y Indicates the minimum distance of the Y axis, Indicates the lowest point of the hoisted component. Indicates the highest point of the obstacle;

[0030] If the lowest point of the obstacle is higher than the highest point of the lifting component, ,in, Indicates the lowest point of the obstacle. Indicates the highest point of the hoisted component;

[0031] If there is overlap between the hoisted component and the obstacle in the y-axis direction, .

[0032] Furthermore, the minimum distance in the XZ plane is:

[0033]

[0034] in, E i ( s ) is a rectangle R 1st i The parametric equation of the edge ( s ∈ [0, 1]), F j ( t ) is a rectangle R 2nd j The parametric equation of the edge ( t ∈ [0, 1]), rectangle R 1 and rectangle R 2 are the projections of the bounding volume of the hoisted component and the bounding volume of the obstacle on the XZ plane, s Indicates the i The point on the edge is i The position ratio on the edge of the strip, t Indicates the j The point on the edge is j The position ratio on the edge of the strip.

[0035] Furthermore, the action space is:

[0036]

[0037] in, represents the action space, They correspond to tower arm rotation, trolley luffing and hook lifting respectively. Among them, tower arm rotation is the lifting arm, that is, the tower arm rotates horizontally around the center of the revolving base, changing the angle between the tower arm and the X-axis of the local coordinate system θ The slewing base connects the tower body and the tower arm; the trolley luffing is the axial translation movement of the luffing trolley along the tower crane boom, changing the distance from the luffing trolley to the tower arm rotation center. r The hook is lifted and lowered by moving the hook up and down along the wire rope of the sling, changing the height difference between the hook and the tower arm rotation center. h c ,

[0038] Angle θ Belongs to the range [-π, π] in the configuration space. For the No. 1 tower crane, the distance from the luffing trolley to the tower arm rotation center satisfies the range [ r 1min， r1max ]; For the No. 2 tower crane, the distance from the luffing trolley to the tower arm rotation center satisfies the range of the configuration space [ r 2min， r 2max ]; For tower crane No. 1, the height difference from the tower arm rotation center to the ground satisfies the range of the configuration space [ 0 ， h 1c ]; For the No. 2 tower crane, the height difference from the tower arm rotation center to the ground satisfies the range of the configuration space [ 0 ， h 2c ].

[0039] Furthermore, the state space is:

[0040]

[0041] in, represents the state space, Represents the position information of the tower crane itself, including the angle between the tower arm and the X-axis of the local coordinate system θ , the distance from the luffing trolley to the tower arm rotation center r The height difference from the hook to the tower arm rotation center h c , Represents the environmental state information perceived by the agent, including the location of the target point P The target points include the hook point and the installation point, the minimum value of the minimum distance between the hoisted component and each obstacle, and the interference information between the tower cranes. I d The interference information is the minimum distance between two rectangles of the operating spaces of the two tower cranes projected on the XZ plane.

[0042] Furthermore, the reward function includes the main line reward for the hook to find the task component and arrive at the material yard to load the task component onto the hook. R D1 , Installation mainline reward for transporting task components to the installation point R D2 , collision reward R D3 and a shaping reward function based on the minimum distance collision detection algorithm R c , where the task component represents the designated component to be hoisted;

[0043] When the distance between the hook point and the center point of the task component is less than the safety threshold, ,on the contraryR D1 =0;

[0044] When the distance between the center point of the task component and the installation point is less than the installation distance threshold, ,on the contrary R D2 =0;

[0045] When the minimum value of the minimum distance between the bounding volume of the task component and the bounding volume of each obstacle is less than the safety threshold, ,on the contrary ;

[0046] in c 1 c 2 c 3 is a constant;

[0047] The reward function R for: .

[0048] Furthermore, shaping the reward function R c for:

[0049]

[0050] in, Indicates the minimum value of the minimum distance between the bounding box of the hoisted component and the bounding box of each obstacle. Indicates the safety threshold of the lifting process.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] This invention proposes an intelligent simulation method for tower crane hoisting based on reinforcement learning, innovatively constructs a digital twin simulation environment that integrates knowledge in the construction field, and realizes intelligent decision-making for tower crane operations by establishing a composite reward function that includes multiple dimensions such as collision risk and timeliness. Compared with the existing technology that mainly focuses on single-objective optimization, this method is the first to transform the complex constraints in the construction scene into a quantifiable multi-dimensional evaluation system: in the collision risk dimension, not only static obstacle interference is considered, but also the dynamic factors of other moving lifting equipment are introduced to achieve real-time collision prediction and provide a high negative reward in the event of a collision. In terms of timeliness, a time-based reward and punishment mechanism is adopted. The minimum distance between the obstacle and the hoisted structure changes with time, and a shaping reward function is set in the reward. R c , R cWhen it is 0, it means that the distance between the hoisted structure and the obstacle exceeds the threshold. Once it is less than the threshold, the smaller the distance, the greater the negative reward, which in turn encourages the agent to control the hoisted structure to be as far away from the obstacle as possible, so as to meet the project progress requirements and avoid the safety hazards caused by the blind pursuit of speed. This multi-dimensional collaborative optimization mechanism enables the system to simultaneously meet the balanced needs of safety, economy and engineering efficiency in complex construction environments. Compared with existing technologies, this method realizes the real-time calculation of collision status throughout the hoisting process, safety warning based on motion dynamics, and multi-tower collaborative path planning under multiple constraints for the first time, providing a new technical solution for improving the safety and efficiency of tower crane operations.

[0053] In addition, the present invention fits other entities in the simulated construction site environment and various components of the tower crane with a cubic structure that can only rotate around the Y axis, and obtains axis-aligned directed bounding boxes corresponding to each entity and component, calculates the minimum distance on the Y axis and the minimum distance on the XZ plane between the bounding box of the hoisted component and the bounding box of the obstacle, and calculates the root of the sum of the squares of the minimum distance on the Y axis and the minimum distance on the XZ plane as the minimum distance. First, the bounding boxes of the tower arm, balance arm, hook and other hoisted parts of the tower crane, or other entities (such as other lifting equipment or building materials) in the simulated construction site of the present application will only rotate around the Y axis. At this time, the bounding box has the characteristics of both AABB (Axis-Aligned Bounding Box) and OBB (Oriented Bounding Box). The bounding volume of the present invention only rotates about the Y axis and has the same properties as AABB along the Y axis. The present invention can determine the minimum distance on the Y axis through simple distance calculation. On the zx plane, it has similar properties to OBB. Furthermore, the minimum distance formula for the XZ plane is adopted. The minimum distance between the edges of the zx projection of the bounding volume (the projection must be a rectangle) is calculated as the minimum distance on the XZ plane. This method is simpler than the traditional OBB collision detection method that requires projecting the cross product of three edge direction vectors and nine edge direction vectors onto the separating axis for judgment. It not only simplifies the calculation of the projection on the Y axis, but also requires only one projection on the zx plane. This allows for more real-time collision detection and timely adjustment of the crane status to reduce collision risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the flowchart of the axis-aligned directed bounding volume collision algorithm;

[0055] Figure 2 This is the collision calculation flow chart for the minimum distance in the hoisting scenario;

[0056] Figure 3Schematic diagram of the tower crane coordinate system;

[0057] Figure 4 Tower crane structure diagram;

[0058] In the figure, there are tower arm 1, slewing base 2, tower crane base 3, luffing trolley 4, lifting rope 5, hook 6, and tower body 7. DETAILED DESCRIPTION

[0059] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] To address the challenges of existing technologies, this paper provides a reinforcement learning-based intelligent simulation method for tower crane hoisting. This method aims to address the safety deficiencies, low efficiency, and poor adaptability to dynamic environments that plague existing tower crane hoisting operations. It also proposes a collision detection mechanism to address the difficulty of traditional collision detection methods in achieving accurate real-time calculations in dynamic environments. This method enables intelligent simulation of the hoisting process, enhances the tower crane path planning algorithm's ability to comprehensively optimize multiple objectives, including safety, efficiency, and energy consumption, and effectively resolves the issue of space conflicts when operating two tower cranes in coordinated operation.

[0061] The present invention proposes a tower crane hoisting intelligent control method based on reinforcement learning, which includes the following steps:

[0062] S1. Construct a simulated construction site, including a rigid body model of a trolley-type luffing twin tower crane and other environmental entities, and construct the scope of the model's configuration space. The rigid body model of the trolley-type luffing twin tower crane includes a first tower crane and a second tower crane. Both tower cranes include a tower body, a tower arm, a slewing base, a tower crane base, a luffing trolley, a sling, a hook, and hoisted components.

[0063] S2, minimum distance formula for constructing the model;

[0064] S3, using reinforcement learning-based dual tower crane path planning. The specific steps of S3 are:

[0065] Build a policy network, set the action space, state space and reward function, the action space meets the working range space of the model, the reward function is calculated based on the minimum distance formula, and the agent is trained based on the reinforcement learning algorithm;

[0066] S4. Perform intelligent control of tower crane hoisting based on the trained agent. The trained agent obtains the actual state vector and outputs the actual action to control the crane operation.

[0067] To achieve the above effects, the present invention provides an intelligent hoisting simulation method based on axis-aligned directed bounding volumes and reinforcement learning, comprising the following steps:

[0068] a. Digital simulation method of tower crane construction behavior,

[0069] b. Real-time collision detection method based on distance calculation;

[0070] c. Dual tower crane path planning based on reinforcement learning.

[0071] The steps for designing a. a digital simulation method for tower crane hoisting construction scenarios and construction behaviors include three components:

[0072] (1) Tower crane rigid body model and motion behavior;

[0073] (2) Tower crane configuration space;

[0074] (3) Implementation of tower crane construction behavior simulation.

[0075] In the rigid body model and motion behavior of the tower crane, the rigid body model of the trolley-type tower crane is divided into the following parts: tower crane foundation, tower body, tower arm, tower top, luffing trolley, sling, hook and hoisted object. The motion behavior of the tower crane is decomposed into three parts: tower arm rotation, trolley luffing and hook lifting. The details are as follows: (1) Tower arm rotation: The horizontal rotation of the crane arm around the center of the slewing base, with a motion range of [- π , π ]; (2) Trolley boom length adjustment: The boom length adjustment trolley moves horizontally along the axial direction of the tower crane boom, ranging from the minimum limit distance to the maximum limit distance of the trolley under the current working conditions; (3) Hook lifting: The hook moves horizontally up and down along the sling wire rope, ranging from 0 to the longest limit distance of the sling.

[0076] The designed tower crane configuration space, configuration space C Refers to the working range of the tower crane. For a twin tower crane, the two configuration spaces are cylindrical spaces with the rotation center of each tower crane as the origin, the maximum limit length of the crane arm as the radius, and the height from the ground to the rotation center as the height, expressed as:

[0077] (1)

[0078] in, θ Refers to the angle between the tower arm and the X-axis of the local coordinate system, with counterclockwise being positive; r Refers to the distance from the luffing trolley to the tower arm's rotation center; h It is the height difference from the hook to the rotation center; the subscripts 1 and 2 represent tower crane No. 1 and No. 2 respectively.

[0079] On Configuration Space C The tower arm can rotate a full circle. θ The value range is [- π , π ]; The luffing trolley can move along the tower crane boom from the minimum limit distance to the maximum limit distance, and the minimum limit distance and the maximum limit distance are respectively r min and r max ; Define the height difference from the rotation center to the ground as h c , then the configuration space of the double tower crane is C The range can be expressed as:

[0080]

[0081] This simulation implements tower crane construction behaviors, including three motion behaviors: tower boom rotation, trolley luffing, and hook raising and lowering. The tower crane's rotation center is the central axis of the tower, around which the tower boom and its attached components perform circular motion. Therefore, the RotateAround() function can be used to simulate the tower boom's rotation. The trolley's motion is translated along the direction of the boom. The trolley is set as a child of the tower boom and its position in the local coordinate system is altered using the transform.Translate() function. Similar to the trolley's luffing motion, the hook raising and lowering also utilizes the transform.Translate() function.

[0082] The distance-based collision detection method first analyzes the hoisting scene elements and construction requirements, classifying collision risks into two categories: the first is the collision risk between hoisted components and other environmental entities; the second is the collision risk between cranes. The second category of risk arises from the interference caused by multiple cranes operating simultaneously in the same area. This type of risk requires attention to the overlap between the cranes' operating spaces and the distance between their hoisted components. An axis-aligned directed bounding volume (VA-OBB), a cubic structure that can rotate only about the Y axis, is used to fit the structural elements in the scene.

[0083] The minimum distance Dmin between VA and OBB is decomposed into the minimum distance Dy on the Y axis and the minimum distance Dxz on the XZ plane. For the minimum distance on the Y axis: the lowest point of the hoisting component is higher than the highest point of the obstacle. ; The lowest point of the obstacle is higher than the highest point of the lifting component, ; There is overlap between the hoisting component and the obstacle in the y-axis direction, . For the minimum distance in the XZ plane, the minimum distance between the two rectangles projected by VA-OBB in the XZ plane is calculated. First, parameterize each edge and establish an edge distance function. Obtain candidate solutions by solving extreme points and verify their validity: the internal solution directly calculates the distance, and the boundary solution degenerates to the point edge distance. Finally, the minimum value of all edge pair combinations is what is sought. For two rectangles R1 and R2, the minimum distance calculation formula is:

[0084] (3)

[0085] Where Ei(s) is the parametric equation of the i-th side of rectangle R1 (s ∈ [0, 1]), and Fj(t) is the parametric equation of the j-th side of rectangle R2 (t ∈ [0, 1]).

[0086] The dual tower crane path planning based on reinforcement learning, the space design of tower crane path planning training mainly includes action space design, state space design and reward function design.

[0087] For action space design, the action space refers to the set of all valid actions in a given environment. The tower crane has three actions: tower arm rotation, trolley luffing, and hook lifting. Therefore, the policy network will output the action value , corresponding to the three actions of the crane. To enhance the numerical stability of the training process, they are scaled to [-1, 1].

[0088] The collection of state vectors in the state space is divided into two parts: , where S o Represents the position information of the tower crane itself, that is, the coordinate information in the configuration space . S e This represents the environmental state information perceived by the crane agent, including the location of the target point (the hook point and installation point of the hoisted component), collision distance information of the hoisted component, and interference information between cranes. To coordinate with the reward function and unify the state perception of the agent and the environment, we also focus on the distance difference between the hook point and the target point in the three degrees of freedom of the configuration space.

[0089] Regarding reward function design, rewards are numerical and concrete expressions of task objectives. The agent continuously improves its strategy based on reward feedback signals from the environment during exploration. The rewards for hoisting path planning include mainline rewards and auxiliary rewards. The mainline task is to transport the task component placed in the material yard to the installation point. The corresponding hook searches for the task component and arrives above the material yard to load the task component onto the hook to obtain the mainline reward R. D1 and the installation mainline reward R for transporting the task components to the installation point D2 .

[0090] In order to solve the problem of sparse rewards, we introduce sub-goal rewards R for the three degrees of freedom of tower arm rotation, trolley amplitude change and hook lifting. C1 、R C2 and R C3 As the agents approach the target point, the negative shaping rewards they receive decrease and reach their minimum when they reach the target point, guiding the agents to reach the target point as quickly as possible to avoid being punished.

[0091] In order to ensure that the generated hoisting path is collision-free, a shaping reward function Rc based on the minimum distance collision detection algorithm is introduced. The closer the agent is to the obstacle, the greater the negative penalty it will receive. When a collision occurs, a large negative reward R D3 is given to the agent to guide it in learning to avoid collisions.

[0092] (4)

[0093] The framework's intellectual contribution lies in describing hoisting path planning as a Markov process, allowing for the introduction of reinforcement learning for intelligent simulation and policy discovery. Ultimately, the process is trained using a proximal policy optimization algorithm, implemented using a platform, and validated through a case study. The effectiveness of the approach is demonstrated by evaluating the path, safety, and efficiency of the hoisting solution.

[0094] For crane configuration spaces No. 1 and No. 2, each is a cylindrical space with its own crane's rotation center as its origin, the maximum boom length as its radius, and the height from the ground to the rotation center as its height. The two cranes are independent and each has three degrees of freedom. Because the two crane configuration spaces overlap, collisions between the cranes must be considered during path planning.

[0095] Regarding the tower crane's spatial configuration parameters, its motion can be decomposed into boom rotation, trolley luffing, and hook raising and lowering, corresponding to the boom angle coordinates, trolley coordinates, and hook coordinates. θ is the angle between the boom and the local coordinate system's X-axis, with positive values ​​counterclockwise; r is the distance from the luffing trolley to the boom's rotation center; and h is the height difference between the hook and the rotation center.

[0096] The parameters of the coordinate system are:

[0097] Origin (O): The intersection of the central axis of rotation of the tower crane body and the rotation plane of the horizontal arm (tower arm), that is, the center point of the tower arm's rotation around the tower body.

[0098] Y-axis: vertical direction, coincides with the central axis of the tower, and the positive direction is upward (pointing to the sky).

[0099] X-axis: Horizontal direction, parallel to the main support direction of the tower crane base. Usually determined by the base structure.

[0100] Z axis: the lateral horizontal direction pointing to the direction in which the tower arm extends.

[0101] Prefabricated construction projects typically utilize precast concrete components with regular cubic structures. Therefore, collision calculations are typically performed using two box-type bounding boxes: AABBs (Axis-Aligned Bounding Boxes) or OBBs (Oriented Bounding Boxes). Because the bottom surface of hoisted components remains parallel to the ground during the hoisting process and does not rotate about the X and Z axes, axis-aligned directed bounding boxes (VA-OBBs) are well-suited to their motion characteristics. VA-OBBs exhibit the same properties as AABBs along the Y axis and are similar to OBBs in the X and Z planes. Using VA-OBBs to fit building components and mechanical assemblies in construction scenarios ensures a tight fit while significantly simplifying the calculation process.

[0102] According to the characteristics of VA-OBB, the minimum distance in three-dimensional space is decomposed into the minimum distance on the Y axis and the minimum distance on the XZ plane, and the VA-OBB Collider (axis-aligned directed bounding volume collider) algorithm is proposed. Figure 1 ).

[0103] Non-critical elements specifically refer to structural elements whose minimum distance from the lifting component is greater than a certain threshold (Dth). For the collision threshold, 15m is taken. In order to quickly distinguish between critical elements and non-critical elements in the scene, a spherical collision body with a large radius is generated for the lifting component in the simulation environment. ,here l b This refers to the diagonal length of the box-shaped bounding volume of the hoisting component. First, a Boolean operation is performed on all elements in the scene and the spherical collider. The minimum distance between all elements outside the spherical collider and the hoisting component is greater than the set threshold and will not affect the hoisting path. At a certain moment in the hoisting process, the initial screening process is for objects that are in contact with the surface or interior of the spherical collider. These are considered key elements and require minimum distance calculation.

[0104] Figure 2 This is the minimum distance collision calculation process in the hoisting scenario. Depending on the type of collision risk, the minimum distance calculation for collision problems can be divided into the minimum distance calculation between hoisting components and other environmental entities and the minimum collision distance calculation between tower cranes. For hook components, after preliminary collision screening, the minimum distance between two VA-OBBs is calculated. For tower cranes, it is divided into the minimum VA-OBB distance calculation between hoisting components and the minimum distance calculation in the XZ plane of the tower crane's operating space. If the minimum distance is less than the safety threshold, a collision will occur; otherwise, there is no collision risk. If a collision is identified, the dangerous component ID will also be given during the training process.

[0105] At a certain moment, the position of task component 1279093 of tower crane No. 2 is such that the lower edge of the component conflicts with element component No. 836247. Therefore, the lifting height of the tower crane should be increased to replan the path.

[0106] During the reinforcement learning training process, the network was trained for 6,000,000 steps using the PPO algorithm. Each round had a maximum step size of 10,000 steps, and the round ended when the maximum step size was reached or the minimum distance during the lifting process fell below a safety threshold. Two training policy models, TowerCrane1 and TowerCrane2, were trained and applied to intelligent decision-making guidance for cranes 1 and 2, respectively, to complete the lifting tasks. The lifting paths for the two components, IDs 836341 and 1279091, were obtained by completing the lifting tasks using cranes 1 and 2, respectively. A complete path includes the starting position of the hook, the starting position of the component, and the component installation position.

[0107] To further evaluate the performance of the reinforcement learning algorithm in generating hoisting paths, the time-varying changes in the tower crane's configuration coordinates were analyzed. Analysis of the changes in the configuration coordinates for hoisting schemes for components 836341 and 1279091 revealed smooth changes in the crane's configuration coordinates, with no noticeable sawtooth oscillations, which more closely met the requirements for continuous and stable hoisting.

[0108] The action space of the crane agent is defined as , in order to control the motion behavior of the tower crane. Through the action output values ​​of the six lifting tasks, operation instructions can be issued to the intelligent agent. The coordination ratio is a quantitative measure of multiple simultaneous movements, which can be used to infer the difficulty of executing the planned movement or path. The action coordination ratio is defined as The calculated motion coordination ratios of the 836339 and 1279099 components are 57.93% and 68% respectively, indicating that the hanging case is practical and feasible.

[0109] To test the tower crane agent's obstacle avoidance capabilities during the installation process, an obstacle exceeding the height of an existing structure was added to the original trajectory of task component 1279099. The intelligent decision-making model then controlled the crane to re-execute the component's lifting task, generating a new lifting path. The crane's luffing trolley moved inward, and the attached task component circumvented the red obstacle, safely reaching the component installation point. The results demonstrate that the trained tower crane intelligent decision-making model possesses excellent obstacle avoidance capabilities.

[0110] Figure 3 Schematic diagram of the crane coordinate system, showing some parameters of the local coordinate system.

[0111] The local coordinate system satisfies:

[0112] Origin: The intersection of the tower body's central axis of rotation and the tower arm's rotation plane, that is, the center point where the tower arm rotates around the tower body;

[0113] Y-axis: vertical direction, coincides with the central axis of the tower, with upward as the positive direction, pointing to the sky;

[0114] X-axis: horizontal direction, a straight line passing through the origin and parallel to the main support direction of the tower crane base. The tower crane base is a flat plate base, which is a rectangular parallelepiped. The main support direction of the tower crane base is the direction of the long side of the contact surface between the rectangular parallelepiped and the ground.

[0115] Z axis: Horizontal direction, parallel to the short side of the cuboid's contact surface with the ground, and a straight line passing through the origin.

[0116] Figure 4 This is a diagram of the tower crane structure, which shows the tower arm 1, slewing base 2, tower crane base 3, luffing trolley 4, lifting rope 5, hook 6, and tower body 7. The tower arm's rotation center and rotation axis are also marked.

[0117] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A tower crane hoisting intelligent control method based on reinforcement learning, characterized in that: The method comprises the following steps: S1. Construct a simulated construction site, including a rigid body model of a trolley-type luffing twin tower crane and other environmental entities, and construct the scope of the model's configuration space. The rigid body model of the trolley-type luffing twin tower crane includes a first tower crane and a second tower crane. Both tower cranes include a tower body, a tower arm, a slewing base, a tower crane base, a luffing trolley, a sling, a hook, and hoisted components. S2, minimum distance formula for constructing the model; S3, using reinforcement learning-based dual tower crane path planning. The specific steps of S3 are: Build a policy network, set the action space, state space and reward function. The action space meets the working range of the model. The reward function is calculated based on the minimum distance formula. Use the reinforcement learning algorithm to train the two crane agents. S4. Control the corresponding tower crane based on the trained intelligent agent; The specific steps of S2 are: The other entities in the simulated construction site environment and the various components of the tower crane are fitted with a cubic structure that can only rotate around the Y axis. The axis-aligned directed bounding volumes corresponding to each entity and component are obtained. The minimum Y-axis distance and the minimum XZ-plane distance between the bounding volume of the hoisted component and the bounding volume of the obstacle are calculated. The root of the sum of the squares of the minimum Y-axis distance and the minimum XZ-plane distance is calculated as the minimum distance, and the formula for the minimum distance between the hoisted component and the obstacle is obtained. If the hoisted component belongs to the No. 1 tower crane, the obstacle is the various components of the No. 2 tower crane or other environmental entities; If the hoisted component belongs to the No. 2 tower crane, the obstacle is the various components of the No. 1 tower crane or other environmental entities; The minimum distance on the Y axis is: If the lowest point of the hoisting component is higher than the highest point of the obstacle, ,in, D y Indicates the minimum distance of the Y axis, Indicates the lowest point of the hoisted component. Indicates the highest point of the obstacle; If the lowest point of the obstacle is higher than the highest point of the lifting component, ,in, Indicates the lowest point of the obstacle. Indicates the highest point of the hoisted component; If there is overlap between the hoisted component and the obstacle in the y-axis direction, ; The minimum distance in the XZ plane is: in, E i ( s ) is a rectangle R 1st i The parametric equation of the edge ( s ∈ [0, 1]), F j ( t ) is a rectangle R 2nd j The parametric equation of the edge ( t ∈ [0, 1]), rectangle R 1 and rectangle R 2 are the projections of the bounding volume of the hoisted component and the bounding volume of the obstacle on the XZ plane, s Indicates the i The point on the edge i The position ratio on the edge of the strip, t Indicates the j The point on the edge j The position ratio on the edge of the strip.

2. The tower crane lifting intelligent control method based on reinforcement learning according to claim 1 is characterized in that: The configuration space is: in, θ Refers to the angle between the tower arm and the X-axis of the local coordinate system, with counterclockwise being positive; r Refers to the distance from the luffing trolley to the tower arm's rotation center; h is the height difference between the hook and the tower arm rotation center; the subscripts 1 and 2 represent tower crane No. 1 and tower crane No. 2 respectively. The tower arm rotation center is the intersection of the rotation center axis of the tower body and the tower arm rotation plane. The tower arm rotation plane represents the plane swept by the center line of the tower arm when it rotates; The local coordinate system satisfies: Origin: The intersection of the tower body's central axis of rotation and the tower arm's rotation plane, that is, the center point where the tower arm rotates around the tower body; Y-axis: vertical direction, coincides with the central axis of the tower, with upward as the positive direction, pointing to the sky; X-axis: horizontal direction, a straight line passing through the origin and parallel to the main support direction of the tower crane base. The tower crane base is a flat plate base, which is a rectangular parallelepiped. The main support direction of the tower crane base is the direction of the long side of the contact surface between the rectangular parallelepiped and the ground. Z axis: Horizontal direction, parallel to the short side of the cuboid's contact surface with the ground, and a straight line passing through the origin.

3. The tower crane lifting intelligent control method based on reinforcement learning according to claim 2 is characterized in that: The scope of the configuration space is: in, C is the configuration space, C= , the minimum limit distance and the maximum limit distance are r min and r max , the height difference from the tower arm rotation center to the ground is h c , subscripts 1 and 2 represent tower crane No. 1 and tower crane No. 2 respectively.

4. The tower crane lifting intelligent control method based on reinforcement learning according to claim 1 is characterized in that: The action space is: in, represents the action space, They correspond to tower arm rotation, trolley luffing and hook lifting respectively. Among them, tower arm rotation is the lifting arm, that is, the tower arm rotates horizontally around the center of the slewing base, changing the angle between the tower arm and the X-axis of the local coordinate system θ The slewing base connects the tower body and the tower arm; the trolley luffing is the axial translation movement of the luffing trolley along the tower crane boom, changing the distance from the luffing trolley to the tower arm rotation center. r The hook is lifted and lowered by moving the hook up and down along the wire rope of the sling, changing the height difference between the hook and the tower arm rotation center. h c , Angle θ Belongs to the range [-π, π] in the configuration space. For the No. 1 tower crane, the distance from the luffing trolley to the tower arm rotation center satisfies the range [ r 1min， r 1max ]; For the No. 2 tower crane, the distance from the luffing trolley to the tower arm rotation center satisfies the range of the configuration space [ r 2min， r 2max ]; For tower crane No. 1, the height difference from the tower arm rotation center to the ground satisfies the range of the configuration space [ 0 ， h 1c ]; For the No. 2 tower crane, the height difference from the tower arm rotation center to the ground satisfies the range of the configuration space [ 0 ， h 2c ].

5. The tower crane hoisting intelligent control method based on reinforcement learning according to claim 4 is characterized in that: The state space is: in, represents the state space, Represents the position information of the tower crane itself, including the angle between the tower arm and the X-axis of the local coordinate system θ , the distance from the luffing trolley to the tower arm rotation center r The height difference from the hook to the tower arm rotation center h c , Represents the environmental state information perceived by the agent, including the location of the target point P The target points include the hook point and the installation point, the minimum value of the minimum distance between the hoisted component and each obstacle, and the interference information between the tower cranes. I d The interference information is the minimum distance between two rectangles of the operating spaces of the two tower cranes projected on the XZ plane.

6. The tower crane hoisting intelligent control method based on reinforcement learning according to claim 5 is characterized in that: The reward function includes the following: the hook searches for the task component, arrives above the material yard, loads the task component onto the hook, and then hooks the main line reward. R D1 , Installation mainline reward for transporting task components to the installation point R D2 , collision reward R D3 and a shaping reward function based on the minimum distance collision detection algorithm R c , where the task component represents the designated hoisted component that needs to be moved by the tower crane; When the distance between the hook point and the center point of the task component is less than the safety threshold, ,on the contrary R D1 =0; When the distance between the center point of the task component and the installation point is less than the installation distance threshold, ,on the contrary R D2 =0; When the minimum value of the minimum distance between the bounding volume of the task component and the bounding volume of each obstacle is less than the safety threshold, ,on the contrary ; in c 1 c 2 c 3 is a constant; The reward function R for: .

7. The tower crane hoisting intelligent control method based on reinforcement learning according to claim 6 is characterized in that: Shaping the Reward Function R c for: in, Indicates the minimum value of the minimum distance between the bounding box of the hoisted component and the bounding box of each obstacle. Indicates the safety threshold of the lifting process.

Images