A tower crane real-time path planning system and method based on deep learning

Abstract

The application discloses a tower crane real-time path planning system and method based on deep learning, which comprises a construction site environment real-time sensing module, a tower crane data acquisition module and a real-time path planning module.The construction site environment real-time sensing module is used for collecting and processing the spatial information of objects in the construction site environment, constructing a real-time construction site spatial model, and performing real-time three-dimensional collision prediction based on the model.The tower crane data acquisition module is used for collecting the operation actions of a tower crane driver and the movement state of the tower crane, forming the operation sequence of the tower crane driver and the movement state sequence of the tower crane according to time records.The real-time path planning module is used for training the neural network of a deep learning algorithm, learning the planning characteristics of the real tower crane driver for the object path, and simultaneously performing real-time dynamic adjustment on the planned path to avoid obstacles dynamically appearing in the construction site.The application guarantees the overall rationality of path planning, updates the path planning in real time according to the real-time information of the tower crane working scene, and avoids the interference caused by newly appearing obstacles in the operation process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of tower crane intelligent control, and particularly relates to a tower crane real-time path planning system and method based on deep learning. BACKGROUND

[0002] In a construction site, efficient and safe use of a tower crane is related to the efficiency of engineering construction and the safety of personnel and equipment. Path planning of a hoisted object of the tower crane is a core problem for realizing efficient and automatic operation of the tower crane.

[0003] A path planning technology of the existing tower crane adopts a heuristic space search algorithm, such as an A* algorithm, to plan a path, which can realize global optimal path planning under a condition of a set cost estimation function. Rationality of a solution of the heuristic algorithm is determined by a heuristic function or a cost estimation function used. In an actual hoisting process of the tower crane, the hoisted wire rope, hook and hoisted object of the tower crane are affected by motion inertia, structural deflection of a tower body, wind load, lifting arm torque and the like, which causes deflection and shaking. A tower crane operator will plan a specific hoisted object path and perform a series of tower crane operations according to experience, so that work is safely and stably performed. However, the heuristic function or the cost estimation function of the heuristic space search algorithm is difficult to consider complicated and implicit prior information. Therefore, the path planned by the algorithm has deficiencies in safety in an actual environment of a construction site, operability of the tower crane operator and stability of the tower crane control. SUMMARY

[0004] The present application aims to solve the technical problems in the prior art, and provides a tower crane real-time path planning system and method based on deep learning. When planning a path of the tower crane, global rationality and safety of the planned path under constraints of a space obstacle and a path length are considered, and real hoisted object path data under operation of the tower crane operator are used to learn and optimize path planning, so as to improve safety and operability of the hoisted object path.

[0005] The technical solution adopted by the present application to solve the technical problems is as follows:

[0006] The present application provides a tower crane real-time path planning system based on deep learning, which comprises the following modules:

[0007] A construction site environment real-time sensing module is used to collect and process spatial information of objects in a construction site environment in real time, to construct a real-time construction site space model, and to perform real-time three-dimensional collision prediction based on the model;

[0008] A tower crane data acquisition module is used to collect operation actions of a tower crane operator and motion states of the tower crane, to form a tower crane operator operation sequence and a tower crane motion state sequence according to time records;

[0009] The real-time path planning module is based on a heuristic space search algorithm and a deep learning algorithm, and learns the planning characteristics of a real crane driver for a lifting object path by training a neural network of the deep learning algorithm based on a sequence of crane driver operation recorded by the crane data acquisition module and a sequence of crane motion states, and performs path planning; meanwhile, the planned path is dynamically adjusted in real time to avoid obstacles dynamically appearing in the construction site by using spatial information of objects in the site environment and three-dimensional collision prediction information provided by the construction site environment real-time sensing module.

[0010] Further, the construction site environment real-time sensing module of the present application simplifies the generation of a spatial bounding box using spatial information of buildings and static structures contained in a BIM model as the basis for the spatial environment of the construction site; at the same time, a laser radar is arranged at the construction site to collect real-time spatial point clouds, and by segmenting the point cloud data, the original spatial bounding box is updated or deleted, and a new spatial bounding box is added; real-time sensing of the spatial information of the construction site environment is achieved, and based on the spatial bounding box model, a three-dimensional collision warning is performed.

[0011] Further, the crane data acquisition module of the present application collects all operation actions of the crane driver and their specific parameter values, collects the corresponding crane motion state at the same time, including the position of the crane trolley, the angle of the jib, and the height of the hook, and collects data at a certain time interval, starting with lifting and ending with unloading, and processes and records the sequence with respect to time.

[0012] Further, the real-time path planning module of the present application uses the sequence of crane driver operation and the sequence of crane motion state recorded by the crane data acquisition module to process and obtain a sequence of discrete points of the real crane lifting object motion path as output samples; based on the heuristic space search algorithm, the sequence of discrete points of the global optimal path with the shortest lifting path length is obtained as input samples by performing path planning from the starting point and the end point of the real lifting object motion path; a deep learning network model is trained to learn the real lifting object path of the manually operated crane to generate a path planning that meets the lifting requirements of the construction site and the operation habits of the driver; at the same time, the real-time path planning module uses the real-time environmental spatial information provided by the construction site environment real-time sensing module to update the planned path in real time.

[0013] The present application provides a crane real-time path planning method based on deep learning, which comprises the following steps:

[0014] Step 1, establish the tower crane hoisting object motion coordinate system O, take the projection point of the center of the tower crane rotation motion on the positive and negative zero horizontal plane as the origin of the coordinate system, and mark it as o; take the north direction as the positive direction of the y-axis of the coordinate system, and take the east direction as the positive direction of the x-axis of the coordinate system to establish a Cartesian coordinate system;

[0015] Step 2, for the construction site environment real-time sensing module, the laser radar laid in the field collects point cloud data in real time, and all point cloud data coordinates are converted to the tower crane hoisting object motion coordinate system O; the point cloud data D () collected at any time t is clustered and segmented into a plurality of point cloud sets by using an octree-based point cloud Euclidean clustering algorithm An axis-aligned bounding box is constructed for each point cloud set, and is marked as The space under the tower crane hoisting object motion coordinate system is divided into orthogonal grids, and any intersection point of the grids is p i , the coordinates are (x i , y i , z i ), and a point set P is formed; the starting point is u; and the target point is e;

[0016] Step 3, for a tower crane operation recorded by the tower crane data collection module, the hoisting object starting point u * and the target point e * , in the real situation, the tower crane driver performs a series of tower crane operation actions {a0, a1,..., a n}, so that the tower crane completes a series of movements, and finally forms a real motion trajectory l * in space; according to the sequence of the operation actions of the tower crane driver, the spatial point sequence formed after discrete processing at a certain time interval is used as the output sample for model training; the input sample of the deep learning model is: given the real hoisting object starting point u * and the target point e * , the hoisting object path spatial point sequence calculated by the heuristic space search algorithm;

[0017] Step 4, before model training, the output sample used for training is corrected;

[0018] Step 5, given any hoisting object starting point and target point, the real-time path planning module first calculates an initial hoisting object path using a heuristic space search algorithm, normalizes the input, uses the trained LSTM network model, and outputs a hoisting object path, and after inverse normalization, a locally optimized path planning result is obtained.

[0019] Further, the method for calculating an initial hoisting object path in step 5 of the present application is:

[0020] The global shortest path planning using the A* algorithm is implemented as follows:

[0021] The minimum cost function from the current point p to the starting point is g(p);

[0022] The heuristic function from the current point p to the target point is h(p);

[0023] Heuristic function weight α;

[0024] The minimum cost estimation function from the current point p to the target point is f(p) = g(p) + αh(p).

[0025] Define the distance function d(p) i p j );

[0026] any point p on the grid i The set of adjacent points is defined as A i ={p j |d(p i p j )=1}

[0027] Define a spatial point sequence: s 0m = [p0, p1, ..., p n-1 p n ]

[0028] Where, p i+1 ∈A i , 0≤i<n;

[0029] The sequence s 0，n Also known as from point p0 to point p n A path; denoted as s i，j From point p i Point P j Let S be a path to a given path. i，j ;

[0030] At the same time, define path s i，j The length is:

[0031] Based on the tower crane's range of motion and the construction site environment, the set of points reachable by the tower crane is defined as Q. It can also be seen from this definition that the region formed by the points within the set is a connected region.

[0032] The distance function is: d(p) i p j )=max(|x i -x j |,|y i -y j |,|zi -z j |)

[0033] The heuristic function takes: h(p) = d(p, e)

[0034] The minimum cost function takes:

[0035] The A* algorithm eventually solves to get the shortest path from the start point u to the target point e

[0036] Further, in order to ensure the real-time and safety of the path planning of the hoisted object, the planning path result output by the deep learning model is subjected to real-time collision detection and prediction in the space bounding box model; if the path exists collision, new space information is used to take the current hoisted object position as the starting point to re-plan the path.

[0037] The beneficial effects of the present application are: when planning the tower crane path, the traditional heuristic space search algorithm and the deep learning method are combined, the safety and operability of the path are optimized according to the actual operation experience on the basis of generating the distance optimal path, the overall rationality of the final path planning is ensured, and the path planning is updated in real time according to the real-time information of the tower crane working scene, so as to avoid the interference caused by new obstacles in the running process.

[0038] The present application is based on the deep learning algorithm, uses the actual tower crane hoisted object motion path data of the construction site, trains the tower crane path planning model, and performs real-time dynamic tower crane path planning based on the real-time space information of the construction site environment. BRIEF DESCRIPTION OF DRAWINGS

[0039] The present application will be further described below in combination with the drawings and examples, and the drawings are as follows:

[0040] Figure 1 is the comparison of the hoisted object trajectory data output by the A* algorithm of the embodiment of the present application and the actual hoisted object trajectory data;

[0041] Figure 2 is the system architecture diagram of the embodiment of the present application;

[0042] Figure 3 is the real-time path planning flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0044] Example 1

[0045] The tower crane real-time path planning system based on deep learning according to embodiments of the present invention, such as Figure 1 As shown, the system includes a real-time construction site environment sensing module, a tower crane data acquisition module, and a real-time path planning module. Specifically:

[0046] The real-time construction site environment sensing module is used to collect and process spatial information of objects in the construction site environment in real time, and build a real-time construction site spatial model.

[0047] The tower crane data acquisition module is used to collect and record the spatial position of the starting point and target point of the hoisted object, the operation actions of the tower crane operator, and the movement status of the tower crane, including the trolley travel position, boom slewing angle, hook height, etc., and record them according to time to form the tower crane operator operation sequence and the tower crane movement status sequence.

[0048] The real-time path planning module is used to plan the path of tower crane lifting objects based on the A* algorithm and deep learning algorithm. By using the real-time spatial information provided by the real-time construction site environment perception module, the planned path is adjusted in real time to avoid obstacles that dynamically appear on the construction site.

[0049] In a preferred embodiment of the present invention, the real-time construction site environment perception module uses the spatial information of buildings and static structures contained in the BIM model to simplify and generate spatial bounding boxes as the basis for the construction site environment space; at the same time, LiDAR is deployed at the construction site to collect spatial point clouds in real time, and by segmenting the point cloud data, the original spatial bounding boxes are updated or deleted and new spatial bounding boxes are added; thus realizing real-time perception of the construction site environment spatial information, and based on the spatial bounding box model, three-dimensional collision warning is performed.

[0050] In a preferred embodiment of the present invention, the tower crane data acquisition module collects all the operating actions of the tower crane operator and their specific parameter values, and collects the corresponding tower crane movement status at the same moment, including: the tower crane trolley travel position, the boom slewing angle, and the hook height; and collects data at certain time intervals, starting with lifting and ending with unloading, and processes and records the data into a time sequence.

[0051] In the preferred embodiment of the present application, the real-time path planning module uses the tower crane data acquisition module to record the tower crane operator sequence and the tower crane motion state sequence, processes to obtain the real tower crane lifting object motion path discrete point sequence as the output sample; based on the heuristic space search algorithm, the path planning based on the starting point and the end point of the real lifting object motion path obtains the discrete point sequence of the global optimal path with the shortest lifting path length as the target, as the input sample; a deep learning network model is used for training to realize the learning of the real manual operation tower crane lifting object path to generate a path planning meeting the construction site lifting requirements and the driver operation habit; at the same time, the path planning module uses the real-time environment space information provided by the construction site environment real-time perception module to update the planning path in real time.

[0052] Embodiment two

[0053] The embodiment of the present application gives part of the implementation method of each module of the tower crane real-time path planning system based on deep learning meeting the requirements of the present application.

[0054] In the embodiment of the present application, the tower crane lifting object motion coordinate system O is a Cartesian coordinate system established with the projection point of the center of the tower crane rotary motion on the project zero level plane as the coordinate system origin, denoted as o; with the north direction as the positive direction of the coordinate system y axis, and with the east direction as the positive direction of the coordinate system x axis.

[0055] In the embodiment of the present application, the construction site environment real-time perception module collects point cloud data in real time by the laser radar arranged on site, and converts all point cloud data coordinates to the tower crane lifting object motion coordinate system O. The point cloud data D (t) collected at any time t is clustered and segmented into a plurality of point cloud sets by using the octree-based point cloud Euclidean clustering algorithm. A bounding box is constructed for each point cloud set, and the bounding box is an axis-aligned bounding box (axis-aligned bounding box), denoted as In particular, the real-time point cloud data of the lifting object is collected by arranging the laser radar on the tower crane boom and aiming at the position of the hook, and a bounding sphere (bounding sphere) is constructed for the lifting object considering the possible self-rotation of the lifting object in the lifting process.

[0056] In the embodiment of the present application, the discretization of the space is to divide the space under the tower crane lifting object motion coordinate system into orthogonal grids with 1m as the unit, and the arbitrary intersection point of the grid is p i , the coordinates are (x i , y i , z i ), and a point set P is formed. In particular, the starting point is denoted as u; and the target point is denoted as e;

[0057] In the embodiment of the present application, the deep learning network model adopts a long short-term memory recurrent neural network (LSTM), which can analyze the input by using a time sequence and learn a function mapping from the input sequence to the output sequence. That is, the path planned by the A* algorithm is mapped to the ideal path of the hoisted object under the real tower crane driver operation. For a certain tower crane operation recorded by the tower crane data acquisition module, the hoisted object has a starting point u * and a target point e * . In a real situation, the tower crane driver performs a series of tower crane operation actions {a0, a1,..., a n}, so that the tower crane completes a series of movements, and finally forms a real movement trajectory l * of the hoisted object in space. According to the operation sequence of the tower crane driver, without considering the actual movement characteristics of the tower crane and the influence of the environment on the movement of the hoisted object, an ideal hoisted object movement trajectory l corresponding to the operation sequence can be calculated. In order to enable the model to better learn the real operation of the tower crane driver, the ideal hoisted object movement path under the real manual operation is discretized at a time interval of 1s to form a spatial point sequence, which is used as an output sample for model training. The input sample of the deep learning model is a given real hoisted object starting point u * and a target point e * , and the spatial point sequence of the hoisted object path calculated by the A* algorithm.

[0058] Before model training, the output sample used for training is corrected, and the correction method adopts a weighted sum of the output sample and the input sample, with weights of 0.8 and 0.2, respectively. Then, the input sample and the output sample data are normalized. The training of the LSTM network adopts the BPTT (back-propagation through time) algorithm.

[0059] In the embodiment of the present application, given any hoisted object starting point and target point, the real-time path planning module first calculates an initial hoisted object path using the A* algorithm, normalizes the initial hoisted object path, and then uses the trained LSTM network model to output a hoisted object path, and after inverse normalization, a locally optimized path planning result is obtained.

[0060] In the embodiment of the present application, the input data in the data set used to train the deep learning network model adopts the global shortest path planning of the A* algorithm, and the specific implementation is as follows:

[0061] The specific implementation of the global shortest path planning of the A* algorithm is as follows:

[0062] The minimum cost function of the current point p to the starting point is g(p);

[0063] The heuristic function of the current point p to the target point is h(p);

[0064] Heuristic function weight α;

[0065] The minimum cost estimation function from the current point p to the target point is f(p) = g(p) + αh(p).

[0066] Define the distance function d(p) i p j );

[0067] any point p on the grid i The set of adjacent points is defined as A i ={p j |d(p i p j )=1}

[0068] Define a spatial point sequence: s 0，n = [p0, p1, ..., p n-1 p n ]

[0069] Where, p i+1 ∈A i , 0≤i<n;

[0070] The sequence s 0，n Also known as from point p0 to point p n A path; denoted as s i，j From point p i Point P j Let S be a path to a given path. i，j ;

[0071] At the same time, define path s i，j The length is:

[0072] Based on the tower crane's range of motion and the construction site environment, the set of points reachable by the tower crane is defined as Q. It can also be seen from this definition that the region formed by the points within the set is a connected region.

[0073] The distance function is: d(p) i p j )=max(|x i -x j |,|y i -y j |,|z i -z j |)

[0074] The heuristic function is: h(p) = d(p, e)

[0075] The minimum cost function is:

[0076] A* algorithm finally solves to obtain the shortest path from the starting point u to the target point e.

[0077] In the preferred embodiment of the present application, in order to ensure the real-time and safety of the hoisted object path planning, the planning path result output by the deep learning model is subjected to real-time collision detection and prediction in the space bounding box model. If the path exists collision, new space information is used to take the current hoisted object position as the starting point to re-plan the path.

[0078] It should be noted that, according to the needs of implementation, each step / component described in the present application can be split into more steps / components, or two or more steps / components or part of the operation of the steps / components can be combined into a new step / component, to achieve the purpose of the present application.

[0079] It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A deep learning-based real-time path planning method for a tower crane, characterized in that, The method comprises the following steps: Step 1, establish tower crane hoisting object motion coordinate system The center of the tower crane rotation motion is projected on the positive and negative zero horizontal plane, and the origin of the coordinate system is recorded as ; The positive direction of the north direction is the positive direction of the y-axis of the coordinate system, and the positive direction of the east direction is the positive direction of the x-axis of the coordinate system; Step 2, for the construction site environment real-time sensing module, the laser radar laid in real-time acquisition of point cloud data, all point cloud data coordinates are converted to tower crane load movement coordinate system ; for any time acquired point cloud data , using octree-based point cloud Euclidean clustering algorithm clustering segmentation into several point cloud sets , for each point cloud set, build a bounding box, the bounding box uses the coordinate axis alignment bounding box, denoted as ; the space under the tower crane load movement coordinate system is divided into orthogonal grids, and the arbitrary intersection point of the grid is , the coordinates are , and the point set is ; the starting point is denoted as ; the target point is denoted as ; Step 3, for a certain tower crane operation recorded by the tower crane data acquisition module, the lifting object starting point and target point , in the real situation, the tower crane driver performs a series of tower crane operation actions , so that the tower crane completes a series of movements, and finally makes the lifting object form a real movement track in space ; according to the sequence of the operation actions of the tower crane driver, the spatial point sequence formed after discrete processing at a certain time interval is taken as the output sample of model training; the input sample of the deep learning model is: given the real lifting object starting point and target point , the lifting object path spatial point sequence calculated by the heuristic space search algorithm; Step 4, before model training, the output sample used for training is corrected, and the correction method adopts weighted summation of the output sample and the input sample, and then the input sample and the output sample data are normalized; Step 5, given any hoisting object starting point and target point, the real-time path planning module first calculates an initial hoisting object path using a heuristic space search algorithm, and after normalization, the trained neural network model is used to output a hoisting object path, and after reverse normalization, a locally optimized path planning result is obtained; The method for calculating an initial hoisting object path in step 5 using a heuristic space search algorithm is as follows: The global shortest path planning using A* algorithm is implemented as follows: current point The minimum cost function to the start point is ; Current point The heuristic function to the goal point is ; Heuristic function weight ; Current point The minimum cost estimate function to the target point is Defining a distance function ; any point on the grid a set of adjacent points is defined as Defining a sequence of spatial points: wherein ; The sequence Also called a path from point to point ; denoted by A path from point to point , the set of all paths is denoted by ; At the same time, the length of the path is defined as: ; According to the tower crane movement range and the construction site environment, define the tower crane hoisting object reachable point set as At the same time, it is known from the definition that the region composed of points in the set is a connected region; The distance function takes: The heuristic function takes: The minimum cost function takes: The A* algorithm ultimately solves the problem, obtaining the solution from the starting point. To the target point shortest path .

2. The deep learning-based real-time path planning method for tower cranes according to claim 1, characterized in that, In order to ensure the real-time performance and safety of the hoisting object path planning, the planning path result output by the deep learning model is subjected to real-time collision detection and prediction in the space bounding box model; if there is a collision in the path, new space information is used, and the current hoisting object position is taken as the starting point to re-plan the path.

3. A deep learning-based tower crane real-time path planning system for implementing the deep learning-based tower crane real-time path planning method of any one of claims 1 or 2, characterized in that, The system comprises the following modules: A construction site environment real-time sensing module for real-time collection and processing of the spatial information of objects in the construction site environment, constructing a real-time construction site space model, and based on the model, performing real-time three-dimensional collision prediction; A tower crane data acquisition module for acquiring the operation actions of the tower crane driver and the tower crane motion state, forming a tower crane driver operation sequence and a tower crane motion state sequence according to the time record; A real-time path planning module that, based on a heuristic space search algorithm and a deep learning algorithm, trains the neural network of the deep learning algorithm using the tower crane driver operation sequence and the tower crane motion state sequence recorded by the tower crane data acquisition module, learns the planning characteristics of the real tower crane driver for the hoisting object path, and plans the path; at the same time, the spatial information of objects in the construction site environment and the three-dimensional collision prediction information provided by the construction site environment real-time sensing module are used to dynamically adjust the planning path in real time to avoid obstacles dynamically appearing in the construction site; The real-time path planning module uses the tower crane driver operation sequence and the tower crane motion state sequence recorded by the tower crane data acquisition module to process the real tower crane hoisting object motion path discrete point sequence as output samples; based on the heuristic space search algorithm, the starting point and the endpoint of the real hoisting object motion path are used for path planning to obtain the discrete point sequence of the globally optimal path with the shortest hoisting path length as the target, as input samples; The deep learning network model is trained to learn the real manual operation of the tower crane hoisting object path to generate a path planning that meets the hoisting requirements of the construction site and the operation habits of the driver; at the same time, the real-time path planning module uses the real-time environment spatial information provided by the construction site environment real-time sensing module to update the planning path in real time.

4. The deep learning-based tower crane real-time path planning system according to claim 3, characterized in that, The construction site environment real-time sensing module simplifies a space bounding box generated by using space information of buildings and static structures contained in a BIM model as a basis for a construction site environment space; meanwhile, a laser radar is arranged at the construction site to collect real-time space point clouds, and by segmenting the point cloud data, the original space bounding box is updated or deleted, and a new space bounding box is added; real-time sensing of construction site environment space information is realized, and based on the space bounding box model, three-dimensional collision early warning is performed.

5. The deep learning-based tower crane real-time path planning system according to claim 3, wherein, The tower crane data acquisition module acquires all operation actions of a tower crane driver and specific parameter values thereof, acquires corresponding tower crane motion states at the same time, including a tower crane trolley running position, a large arm rotation angle, and a hook height, and acquires data at a certain time interval, starting from hoisting and ending with unloading, to process and record a sequence about time.

Images