Tower crane installation path planning method and system, terminal and medium

Abstract

The invention discloses a tower crane installation path planning method and system, a terminal and a medium, and the method comprises the following steps: collecting and preprocessing three-dimensional point cloud data and visual information in a tower crane installation area, fusing the point cloud data with image data, and generating a fused three-dimensional point cloud image which comprises distance and color information; a deep learning model is adopted to carry out obstacle detection on the fused three-dimensional point cloud image; an RRT algorithm is adopted, and a safety path is planned according to the obstacle detection result and the requirement of the hoisting task; and controlling the movement of the hoisting equipment according to the obtained safe path, and executing ascending, descending, horizontal moving and rotating actions of a lifting hook according to a path planning result. The method has the advantages that the safety and the efficiency in the hoisting operation of the power transmission tower are improved, and the quality and the reliability of power transmission line construction are improved.

Description

Technical Field

[0001] The present application relates to the technical field of tower crane installation, and in particular to a tower crane installation path planning method, system, terminal and medium. Background Art

[0002] As a basic industry of the national economy, the electric power industry plays a critical role in the entire national economic system. It has a profound impact on many fields such as social politics, economy, national life, military and national defense. The widespread application of electricity has penetrated into all aspects of social life. Today, the connection between modern society and electric energy has been closely integrated.

[0003] The interconnected system between power plants, substations and power equipment is usually called the power system. The power system includes components such as power plants and power equipment, as well as power transmission and transformation equipment and power lines of various voltage levels, which constitute the elements of the power grid. The function of power transmission in the power system includes transmitting the power generated by the power plant to the area where the power is consumed, or realizing the mutual transmission of power between adjacent power grids, thereby forming an interconnected power grid or a unified power grid, maintaining the balance of power generation, power consumption and supply and demand between the two power grids. Therefore, the responsibility of the transmission line is to transmit electric energy and connect various power plants and substations to ensure their normal operation, thereby realizing the interconnection of the power system. The transmission tower line is regarded as the main blood vessel of the power industry and a key component of the power system. When realizing the interconnection between major cities across the country, the construction of the power grid has become a major project. In the construction of the power grid, the installation of the transmission tower is a crucial link.

[0004] For a long time, transmission tower assembly projects usually require a lot of manpower, and heavy manual operations are required from carrying materials to assembly. This may lead to high consumption and fatigue of human resources. At the same time, transmission tower assembly is a highly dangerous operation. Manual operators may need to work at heights and face safety risks such as falling and falling objects. The assembly of transmission towers also requires a high degree of accuracy and stability to ensure the structure and stability of the tower. Manual operation may be affected by factors such as skill level and fatigue, which may affect the accuracy and stability of assembly. Transmission tower construction usually needs to be carried out under different environmental conditions, such as bad weather, high altitude areas, etc. These environmental factors increase the difficulty and risk of manual operation.

[0005] At present, the construction of power transmission tower assembly is still mainly in the stage of relying on manpower, and the main research is on the construction technology of high-voltage and ultra-high voltage tower assembly. Xiao et al. proposed a special tower assembly crane in the research and application of ultra-high voltage transmission line tower assembly technology to solve the problems existing in the construction of traditional tower assembly lifting equipment, but it still relies on professional personnel to operate, which poses safety risks. Some scholars have also put forward their own views on the combination of power transmission tower construction and computers. Xing et al. proposed an intelligent tower assembly system in the development and research of the intelligent design system for high-voltage overhead transmission tower assembly. The system can output a set of corresponding construction plans based on the basic construction parameters of the tower input by the user. However, no specific design plan can be proposed at present, and it still needs to be explored. Summary of the invention

[0006] The present application provides a tower crane installation path planning method, system, terminal and medium, which have the advantages of improving the safety and efficiency in transmission tower crane installation operations and improving the quality and reliability of transmission line construction.

[0007] On the one hand, the present application provides a tower crane installation path planning method, comprising the following steps:

[0008] S1: Collect 3D point cloud data and visual information in the tower crane installation area and perform preprocessing, fuse the point cloud data with the image data, and generate a fused 3D point cloud image, which contains distance and color information;

[0009] S2: Use deep learning model to detect obstacles on the fused 3D point cloud image;

[0010] S3: Use the RRT algorithm to plan a safe path based on obstacle detection results and lifting task requirements;

[0011] S4: Control the movement of the lifting equipment according to the safe path obtained in step S3, and perform the lifting, lowering, horizontal movement and rotation of the hook according to the path planning result.

[0012] Further, step S1 includes the following steps:

[0013] S11: Install a laser radar and a camera on the hoisting equipment to collect 3D point cloud data and visual image data of the environment. The laser radar scans the surrounding environment by rotation and measures distance data in different directions. These data are presented in the form of 3D point cloud images. Each point includes 3D coordinates and distance information. The camera captures real-time images, including color images and infrared images, which contain visual information.

[0014] S12: Process the image data collected in step S11 by first performing Gaussian filtering on the image data to remove noise. The specific algorithm is as follows:

[0015]

[0016] Where I′(x,y) is the filtered image, I(x,y) is the original image, x,y are the positions of the current pixel, and σ is the standard deviation of the Gaussian kernel, which controls the smoothness of the filter.

[0017] S13: After Gaussian filtering of the point cloud image, color image and infrared image, they are input into the data fusion network to realize data fusion. The specific algorithm is as follows:

[0018] S131: Ensure that the lidar data and camera data have the same coordinate system and reference frame; if their coordinate systems are different, coordinate transformation is required to align them in the same coordinate system; the coordinate transformation matrix is ​​calculated as follows:

[0019]

[0020] In the above formula, R i,j Represents the elements in a matrix; this matrix represents the transformation from one coordinate system to another;

[0021] S132: After the coordinates are aligned, data fusion is required, including color information fusion: extracting color information from the camera image, that is, the RGB value of each pixel; matching the color information with the corresponding point cloud data, and assigning a color value to each point cloud point; associating the points in the point cloud data with their projected positions in the camera image; distance information fusion: the laser radar provides distance information for generating the three-dimensional coordinates of the point cloud; combining the distance information with the color information to generate a fused three-dimensional point cloud image.

[0022] Further, step 2 includes the following steps:

[0023] S21: Obtain the fused 3D point cloud image data, ensure that the data has been aligned and fused in the same global coordinate system for accurate obstacle detection, and randomly arrange all the fused data into training set and training set according to the composition of 8:2;

[0024] S22: The Mask R-CNN deep learning model is used for obstacle detection tasks. First, the acquired 3D point cloud image data is converted into a format suitable for model input to normalize the image to meet the model input requirements. The activation function selected in normalization is the Sigmoid function.

[0025] S23: Input the data into the network model for training, and the training set and test set are input into the network at a ratio of 8:2; the loss function used is the cross entropy loss function, where y represents the actual category label and p represents the category prediction probability distribution of the model;

[0026]

[0027] Among them, y i Represents the unique hot encoding of the actual category label, indicating that the probability of the sample belonging to the i-th category is 1, and the probability of other categories is 0; p i Represents the predicted probability distribution of the model, which represents the predicted probability that the sample belongs to the i-th category; the goal of the cross entropy loss function is to minimize the difference between the actual label and the model prediction so that the training model can classify samples more accurately;

[0028] S24: Train the model for multiple iterations using the training dataset; in each iteration, feed batches of data into the model, calculate the loss, and then backpropagate the error to update the model parameters; regularly evaluate model performance on the validation set to monitor the progress of training;

[0029] S25: Use the trained deep learning model to perform obstacle detection on the fused 3D point cloud image; the model will identify the location of obstacles in the image and generate a segmentation mask for each obstacle to identify the shape and boundary of the obstacle; output the result of obstacle detection, and the output information includes the location, shape, and category attributes of the obstacle.

[0030] Further, step S3 includes the following steps:

[0031] S31: Use the fused 3D point cloud image and the output of obstacle detection to perceive the surrounding environment in real time; determine the location, shape, and category of obstacles in the current environment;

[0032] S32: Determine the specific requirements of the lifting task, including the target location, weight, size and shape of the lifting object; determine the starting point and end point of the task, and the target location of the lifting object;

[0033] S33: Use the path planning algorithm RRT to search for a safe path from the starting point to the target point; consider the obstacle information in the environment to ensure that the path avoids obstacles and maintains a sufficient safety distance; the RRT algorithm is as follows:

[0034] S331: Select a starting point x on the map start And create a tree T containing the point;

[0035] S332: loop the following steps:

[0036] (1) Generate a random point x rand , randomly selected in the map, but with a certain probability of selecting the target point x goal ;

[0037] (2) Find the distance x in the tree Trand The nearest node x nearest ;

[0038] (3) From x nearest Move a certain distance along the direction to get a new node x new ;

[0039] (4) If x new does not intersect with the obstacle, add it to the tree T, and change x nearest Connect to x new ;

[0040] (5) If x new When approaching the target point, check whether there is a feasible path to the target; if so, the algorithm ends and the path is found;

[0041] S333: When the maximum number of iterations is reached or a feasible path is found, the algorithm ends and the path is returned.

[0042] Further, step S4 includes the following steps:

[0043] S41: Collect voltage and vibration data of lifting equipment.

[0044] Further, step S4 includes the following steps:

[0045] S41: collecting voltage and vibration data of the hoisting equipment, and issuing an alarm message and controlling the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed a preset range;

[0046] S42: Save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

[0047] On the other hand, the present application provides a tower crane installation path planning system based on laser radar and visual perception, comprising:

[0048] LiDAR, deployed at the tower crane installation site, scans the surrounding environment in a rotating manner and collects three-dimensional point cloud data;

[0049] A camera is deployed at the tower crane installation site to collect image information of the tower crane installation site;

[0050] Data acquisition module, used to collect data obtained by laser radar and camera;

[0051] The data processing module processes and fuses the 3D point cloud data acquired by the LiDAR and the visual data of the camera. The LiDAR data processing includes data filtering, coordinate conversion, downsampling and data feature extraction; the visual data processing includes image correction, feature extraction and image registration;

[0052] The obstacle detection module deploys a deep neural network to learn the characteristics and locations of obstacles from lidar and vision data;

[0053] The path planning module deploys the RRT algorithm to generate feasible paths;

[0054] and a control module for controlling the hoisting equipment according to the generated path.

[0055] Furthermore, it also includes:

[0056] The safety and emergency module collects the voltage and vibration data of the hoisting equipment, issues an alarm message and controls the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed the preset range;

[0057] and a data storage and analysis module, which is used to save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

[0058] On the other hand, the present application provides a tower crane installation path planning terminal, including a memory and a processor, the memory stores a computer program, and when the computer program is called and executed by the processor, the tower crane installation path planning method as described above is implemented.

[0059] On the other hand, the present application provides a computer-readable medium, wherein the computer-readable medium stores a computer program, and when the computer program is called and executed by a computer, the tower crane installation path planning method as described above is implemented.

[0060] In summary, the beneficial effects of this application are:

[0061] 1. Through lidar and machine vision perception, the system is able to detect obstacles in the surrounding environment, such as buildings, trees, and equipment, in real time; this helps avoid collisions, reduces potential risks in lifting operations, and improves workplace safety;

[0062] 2. Path planning algorithms can help tower cranes quickly find alternative paths when encountering obstacles without having to wait for human intervention; this improves the execution efficiency of construction tasks and reduces the duration of engineering projects;

[0063] 3. The entire system is automated and can sense the environment in real time and make intelligent decisions without human intervention; this reduces reliance on the crane operator and reduces the possibility of human error;

[0064] 4. The system takes into account the complex environmental conditions that may be encountered in the construction of transmission lines, such as high altitude, bad weather, etc. Machine vision technology enables the system to operate under various conditions, thereby enhancing the adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is an overall architecture diagram of a tower crane installation path planning system based on laser radar and visual perception in one embodiment of the present application;

[0066] Figure 2 This is an overall flow chart of a tower crane installation path planning system based on laser radar and visual perception in one embodiment of the present application;

[0067] Figure 3 This is a structural diagram of the Mask R-CNN deep learning model used for obstacle detection in one embodiment of the present application. DETAILED DESCRIPTION

[0068] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0069] A specific embodiment of the present application provides a tower crane installation path planning method, comprising the following steps:

[0070] S1: Collect 3D point cloud data and visual information in the tower crane installation area and perform preprocessing, fuse the point cloud data with the image data, and generate a fused 3D point cloud image, which contains distance and color information;

[0071] S2: Use deep learning model to detect obstacles on the fused 3D point cloud image;

[0072] S3: Use the RRT algorithm to plan a safe path based on obstacle detection results and lifting task requirements;

[0073] S4: Control the movement of the lifting equipment according to the safe path obtained in step S3, and perform the lifting, lowering, horizontal movement and rotation of the hook according to the path planning result.

[0074] Step S1 includes the following steps:

[0075] S11: Install a laser radar and a camera on the hoisting equipment to collect 3D point cloud data and visual image data of the environment. The laser radar scans the surrounding environment by rotation and measures distance data in different directions. These data are presented in the form of 3D point cloud images. Each point includes 3D coordinates and distance information. The camera captures real-time images, including color images and infrared images, which contain visual information.

[0076] S12: Process the image data collected in step S11 by first performing Gaussian filtering on the image data to remove noise. The specific algorithm is as follows:

[0077]

[0078] Where I′(x,y) is the filtered image, I(x,y) is the original image, x,y are the positions of the current pixel, and σ is the standard deviation of the Gaussian kernel, which controls the smoothness of the filter.

[0079] S13: After Gaussian filtering of the point cloud image, color image and infrared image, they are input into the data fusion network to realize data fusion. The specific algorithm is as follows:

[0080] S131: Ensure that the lidar data and camera data have the same coordinate system and reference frame; if their coordinate systems are different, coordinate transformation is required to align them in the same coordinate system; the coordinate transformation matrix is ​​calculated as follows:

[0081]

[0082] In the above formula, R i,j Represents the elements in a matrix; this matrix represents the transformation from one coordinate system to another;

[0083] S132: After the coordinates are aligned, data fusion is required, including color information fusion: extracting color information from the camera image, that is, the RGB value of each pixel; matching the color information with the corresponding point cloud data, and assigning a color value to each point cloud point; associating the points in the point cloud data with their projected positions in the camera image; distance information fusion: the laser radar provides distance information for generating the three-dimensional coordinates of the point cloud; combining the distance information with the color information to generate a fused three-dimensional point cloud image.

[0084] Step 2 includes the following steps:

[0085] S21: Obtain the fused 3D point cloud image data, ensure that the data has been aligned and fused in the same global coordinate system for accurate obstacle detection, and randomly arrange all the fused data into training set and training set according to the composition of 8:2;

[0086] S22: The Mask R-CNN deep learning model is used for obstacle detection tasks. First, the acquired 3D point cloud image data is converted into a format suitable for model input to normalize the image to meet the model input requirements. The activation function selected in normalization is the Sigmoid function.

[0087] S23: Input the data into the network model for training, and the training set and test set are input into the network at a ratio of 8:2; the loss function used is the cross entropy loss function, where y represents the actual category label and p represents the category prediction probability distribution of the model;

[0088]

[0089] Among them, y i Represents the unique hot encoding of the actual category label, indicating that the probability of the sample belonging to the i-th category is 1, and the probability of other categories is 0; p i Represents the predicted probability distribution of the model, which represents the predicted probability that the sample belongs to the i-th category; the goal of the cross entropy loss function is to minimize the difference between the actual label and the model prediction so that the training model can classify samples more accurately;

[0090] S24: Train the model for multiple iterations using the training dataset; in each iteration, feed batches of data into the model, calculate the loss, and then backpropagate the error to update the model parameters; regularly evaluate model performance on the validation set to monitor the progress of training;

[0091] S25: Use the trained deep learning model to perform obstacle detection on the fused 3D point cloud image; the model will identify the location of obstacles in the image and generate a segmentation mask for each obstacle to identify the shape and boundary of the obstacle; output the result of obstacle detection, and the output information includes the location, shape, and category attributes of the obstacle.

[0092] Step S3 includes the following steps:

[0093] S31: Use the fused 3D point cloud image and the output of obstacle detection to perceive the surrounding environment in real time; determine the location, shape, and category of obstacles in the current environment;

[0094] S32: Determine the specific requirements of the lifting task, including the target location, weight, size and shape of the lifting object; determine the starting point and end point of the task, and the target location of the lifting object;

[0095] S33: Use the path planning algorithm RRT to search for a safe path from the starting point to the target point; consider the obstacle information in the environment to ensure that the path avoids obstacles and maintains a sufficient safety distance; the RRT algorithm is as follows:

[0096] S331: Select a starting point x on the map start And create a tree T containing the point;

[0097] S332: loop the following steps:

[0098] (1) Generate a random point x rand , randomly selected in the map, but with a certain probability of selecting the target point x goal ;

[0099] (2) Find the distance x in the tree T rand The nearest node x nearest ;

[0100] (3) From xnearest Move a certain distance along the direction to get a new node x new ;

[0101] (4) If x new does not intersect with the obstacle, add it to the tree T, and change x nearest Connect to x new ;

[0102] (5) If x new When approaching the target point, check whether there is a feasible path to the target; if so, the algorithm ends and the path is found;

[0103] S333: When the maximum number of iterations is reached or a feasible path is found, the algorithm ends and the path is returned.

[0104] Further, step S4 includes the following steps:

[0105] S41: Collect voltage and vibration data of lifting equipment.

[0106] Further, step S4 includes the following steps:

[0107] S41: collecting voltage and vibration data of the hoisting equipment, and issuing an alarm message and controlling the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed a preset range;

[0108] S42: Save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

[0109] Another specific embodiment of the present application provides a tower crane installation path planning system based on laser radar and visual perception, including:

[0110] LiDAR, deployed at the tower crane installation site, scans the surrounding environment in a rotating manner and collects three-dimensional point cloud data;

[0111] A camera is deployed at the tower crane installation site to collect image information of the tower crane installation site;

[0112] Data acquisition module, used to collect data obtained by laser radar and camera;

[0113] The data processing module processes and fuses the 3D point cloud data acquired by the LiDAR and the visual data of the camera. The LiDAR data processing includes data filtering, coordinate conversion, downsampling and data feature extraction; the visual data processing includes image correction, feature extraction and image registration;

[0114] The obstacle detection module deploys a deep neural network to learn the characteristics and locations of obstacles from lidar and vision data;

[0115] The path planning module deploys the RRT algorithm to generate feasible paths;

[0116] A control module, used for controlling the hoisting equipment according to the generated path;

[0117] The safety and emergency module collects the voltage and vibration data of the hoisting equipment, issues an alarm message and controls the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed the preset range;

[0118] and a data storage and analysis module, which is used to save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

[0119] The hierarchical architecture of the tower crane installation path planning system based on LiDAR and visual perception is as follows: Figure 1 shown.

[0120] Figure 1 The sensor module includes lidar and camera.

[0121] Figure 1 The data acquisition module is responsible for acquiring data from the lidar and camera, providing the necessary input for subsequent obstacle detection and path planning. It includes two parts: the lidar sensor and the camera sensor. Among them, the lidar is a sensor used to measure the distance and shape of the surrounding environment. It usually includes a laser transmitter and a receiver, and calculates the distance of the object by measuring the reflection time of the laser beam. In this module, the lidar is used to collect three-dimensional point cloud data. The camera sensor is used to capture image data. The camera can be an ordinary visible light camera or other types of cameras. In this example, an infrared camera and an ordinary camera are used to provide more information.

[0122] Figure 1 In the data processing module, the data processing module is responsible for processing the lidar and visual data from the sensor module to prepare data for obstacle detection and path planning, mainly including the processing and data fusion of the 3D point cloud data obtained by the lidar and the visual data of the camera. Lidar data processing includes data filtering, coordinate conversion, downsampling and data feature extraction. Visual data processing includes image correction, feature extraction, image registration, etc.

[0123] Figure 1The obstacle detection module is responsible for detecting and identifying obstacles in the environment to support path planning and the safety of lifting operations. The obstacle detection module deploys the Mask R-CNN deep neural network to learn the characteristics and positions of obstacles from the data collected by the sensors. The sensor data interface module communicates with sensors such as lidar and cameras to receive three-dimensional point cloud data and visual image data.

[0124] Figure 1 The path planning module is a key component of the system. It is responsible for determining the installation path of the tower crane to ensure its safe operation and avoid obstacles in the environment. The RRT (Rapidly Exploring Random Trees) algorithm is deployed to generate feasible paths. It is the core algorithm of path planning.

[0125] Figure 1 The central control module is the core part of the system, responsible for actually controlling the movement of the tower crane to ensure that it operates safely along the planned path. It mainly includes the lifting equipment controller, which is a key component for controlling the movement of the tower crane, usually including hooks, motors and sensors.

[0126] Figure 1 The safety and emergency module plays a key role in the system. Its main task is to ensure the safety of lifting operations and take necessary measures to reduce potential risks in case of emergency. It mainly includes safety measures implementation devices: This includes various safety devices and equipment, such as emergency stop buttons, safety sensors, sound and light alarms, etc.

[0127] Figure 1 The main task of the data storage and analysis module is to store various data and analyze these data to provide performance evaluation and decision support. It mainly includes data storage and database management system, which are used for long-term storage and management of various data, including sensor data, lifting operation data, path planning data, etc.

[0128] The system has a clear hierarchical architecture and division of labor, modular functions, coordinated operation of each layer and module, and step-by-step calling, which ensures the security, integrity and accuracy of data resources in the entire method. It combines multiple key technologies such as lidar, deep learning, path planning and real-time control, which can significantly improve the safety and automation level of tower cranes, especially in complex and changing construction environments.

[0129] refer to Figure 2 , the tower crane installation path planning method described above is implemented in the tower crane installation path planning system based on laser radar and visual perception, comprising the following steps:

[0130] S1: In the data acquisition and data processing module, the laser radar device is first installed. It scans the surrounding environment in a rotating manner and collects three-dimensional point cloud data. The point cloud data includes the coordinates and distance information of each point. At the same time, the camera captures real-time images and provides visual information. Then the laser radar data and visual images are preprocessed, and finally the laser radar point cloud data is fused with the image data to generate a fused three-dimensional point cloud image, which contains distance and color information.

[0131] S2: In the obstacle detection module, a deep learning model (such as Mask R-CNN) is used to perform obstacle detection on the fused 3D point cloud image.

[0132] S3: In the path planning module, the RRT algorithm (Rapidly Exploring Random Trees) is used to plan a safe path based on the obstacle detection results and the requirements of the lifting task.

[0133] S4: In the control module, the system controls the movement of the hoisting equipment and performs actions such as the rise, fall, horizontal movement and rotation of the hook according to the path planning results. When a warning occurs, the safety and emergency module immediately performs operations such as emergency stop to ensure the safety of the hoisting operation. All system data is stored in the data storage and analysis module for easy storage and management.

[0134] The S1 mainly includes 3 steps:

[0135] S11: Install a laser radar and camera on the hoisting equipment to collect 3D point cloud data and visual image data of the environment. The laser radar scans the surrounding environment by rotating and measures distance data in different directions. These data are presented in the form of 3D point cloud images, and each point includes 3D coordinates and distance information. The camera captures real-time images, which are color images and infrared images, containing visual information.

[0136] S12: The image data collected in the previous step is input into the data preprocessing module for data processing. First, the image data is subjected to Gaussian filtering to remove noise. The specific algorithm is as follows:

[0137]

[0138] Where I′(x,y) is the filtered image, I(x,y) is the original image, x,y are the positions of the current pixel, and σ is the standard deviation of the Gaussian kernel, which controls the smoothness of the filter.

[0139] S13: After Gaussian filtering of the point cloud image, color image and infrared image, they are input into the data fusion network to realize data fusion. The specific algorithm is as follows:

[0140] (1) Ensure that the lidar data and camera data have the same coordinate system and reference frame. If their coordinate systems are different, coordinate transformation is required to align them in the same coordinate system. The coordinate transformation matrix is ​​calculated as follows:

[0141]

[0142] Here, R i,j Represents the elements of a matrix that represents a transformation from one coordinate system to another.

[0143] (2) After the coordinates are aligned, data fusion is required, including: Color information fusion: Extract color information from the camera image, the RGB value of each pixel. Match the color information with the corresponding point cloud data and assign a color value to each point cloud point. This is done by associating the points in the point cloud data with their projected positions in the camera image. Distance information fusion: LiDAR provides distance information, which can be used to generate the three-dimensional coordinates of the point cloud. Combine the distance information with the color information to generate a fused three-dimensional point cloud image.

[0144] (3) The data fusion module usually passes the fused data to the obstacle detection module for obstacle identification and positioning.

[0145] refer to Figure 3 , the S2 mainly includes 5 steps:

[0146] S21: Receive the fused 3D point cloud image data from the data fusion module, and ensure that the data has been aligned and fused in the same global coordinate system for accurate obstacle detection. All fused data are randomly arranged and divided into training set and training set in an 8:2 ratio.

[0147] S22: This system uses the Mask R-CNN deep learning model for obstacle detection tasks. First, the received 3D point cloud image data is converted into a format suitable for model input to normalize the image to meet the model input requirements. The activation function used here in normalization is the Sigmoid function, which is relatively simple and suitable as the early gated convolution layer. Because the depth of the constructed network is relatively deep, it is easy to have gradient vanishing and gradient explosion problems, which affects the accuracy of the network.

[0148] S23: Then input the data into the network model for training, and the training set and the test set are input into the network at a ratio of 8:2. The loss function used here is the cross-entropy loss function, which is usually used for classification tasks, where y represents the actual category label and p represents the category prediction probability distribution of the model.

[0149]

[0150] Among them, y i Represents the one-hot encoding of the actual category label, indicating that the probability of the sample belonging to the i-th category is 1, and the probability of other categories is 0. i Represents the predicted probability distribution of the model, which represents the predicted probability that the sample belongs to the i-th category. The goal of the cross entropy loss function is to minimize the difference between the actual label and the model prediction so that the training model can classify samples more accurately.

[0151] The specific parts of the model are: input image, feature extraction, RPN stage, RoI Align, classification head, bounding box regression head, Mask head. The functions and input and output of each part are as follows:

[0152] 1. Input image: This is the starting point of the solution, representing the input color image, which is the image after data fusion and will be fed into the model for processing.

[0153] 2. Feature Extraction: The feature extraction module is a ResNet convolutional neural network that is used to extract high-level features from the input image. These features are critical for object detection and segmentation because they contain the semantic information of the image and output feature maps.

[0154] 3.RPN stage: RPN (Region Proposal Network) is responsible for generating candidate target regions on the feature map. It proposes a series of candidate regions that contain potential target objects. The role of RPN is to screen out regions that are worthy of further detection.

[0155] 4.RoI Align: The RoI Align module is used to accurately crop the candidate regions generated by RPN in preparation for object classification, bounding box regression, and segmentation operations. RoI Align retains spatial information and helps in more accurate object segmentation.

[0156] 5. Classification head: The classification head is responsible for classifying the cropped candidate regions and determining whether each candidate region contains the target object and its category. This is a key step in target detection.

[0157] 6. Bounding Box Regression Head: The bounding box regression head is used to accurately locate the bounding box of the detected target object. This helps to determine the exact location of the object.

[0158] 7. Mask Head: Mask head is a unique part of Mask R-CNN, which is used to generate segmentation masks for each object. These segmentation masks accurately describe the shape and contour of the object and are used for instance segmentation.

[0159] S24: Train the model for multiple iterations using the training dataset. In each iteration, batches of data are fed into the model, the loss is calculated, and the error is back-propagated to update the model parameters. Model performance is regularly evaluated on the validation set to monitor the progress of training.

[0160] S25: Use the trained deep learning model to perform obstacle detection on the fused 3D point cloud image. The model will identify the location of obstacles in the image and generate a segmentation mask for each obstacle to identify the shape and boundary of the obstacle. The results of obstacle detection are output to the path planning module to support the path planning process. The output information includes the location, shape, category and other relevant attributes of the obstacle.

[0161] The S3 mainly includes 3 steps:

[0162] To achieve path planning, three basic problems need to be solved: (1) the crane arm reaches the desired position; (2) the crane arm avoids obstacles and completes strategic tasks during movement; and (3) the optimal path is achieved. Here, a global path planning method is used.

[0163] S31: Using the fused 3D point cloud image provided by the data fusion module and the output of the obstacle detection module, the surrounding environment is perceived in real time, and the location, shape, category and other relevant information of obstacles in the current environment are determined.

[0164] S32: Determine the specific requirements of the lifting task, including the target location, weight, size and shape of the lifting object, etc. Determine the starting point and end point of the task, and the target location of the lifting object.

[0165] S33: Use the path planning algorithm, RRT (Rapidly Exploring Random Trees) to search for a safe path from the starting point to the target point. Consider the obstacle information in the environment to ensure that the path avoids obstacles and maintains a sufficient safety distance. The RRT algorithm is as follows:

[0166] 1. Select a starting point x on the map start And create a tree T that contains the point.

[0167] 2. Cycle the following steps:

[0168] (1) Generate a random point x rand , usually randomly selected within the map, but with a certain probability (such as ε-greedy) to select the target point x goal .

[0169] (2) Find the distance x in the tree T rand The nearest node x nearest.

[0170] (3) From x nearest Move a certain distance along the direction to get a new node x new .

[0171] (4) If x new does not intersect with the obstacle, add it to the tree T, and change x nearest Connect to x new .

[0172] (5) If x new When approaching the goal point, it checks whether there is a feasible path to the goal. If so, the algorithm ends and the path is found.

[0173] 3. When the maximum number of iterations is reached or a feasible path is found, the algorithm ends and the path is returned.

[0174] The formula involved is as follows:

[0175] Random point generation:

[0176]

[0177] Node search and expansion:

[0178] Among them, dist(x i ,x rand ) represents the distance function between two points, Steer is the function used to generate new nodes, and step_size is the step size of each expansion.

[0179] The S4 mainly includes 2 steps:

[0180] S41: The safety and emergency module monitors the safety of the lifting operation by connecting to the voltage and vibration sensors of the equipment. If potential dangers are detected or abnormal data is received, such as overload of the hook or abnormal operation of the lifting equipment, safety measures will be immediately activated to trigger the emergency stop button to quickly stop the movement of the lifting equipment to prevent accidents. The module monitors data from various sensors and equipment, as well as performance data of the lifting operation. If any abnormality is found, it will generate a report and transmit it to relevant personnel for further analysis and processing.

[0181] S42: The data storage and analysis module will manage and maintain the data stored in the database for a long time, including sensor data, lifting operation data, path planning data, etc. The data is recorded in a time series to ensure the integrity and availability of the data. The collected data will be sorted and stored in an appropriate format, using the database format. Every three days, the module will sort and classify the data, evaluate the accuracy of obstacle detection and the complexity of path planning, and facilitate the optimization of the model later.

[0182] Another specific embodiment of the present application provides a tower crane installation path planning terminal, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is called and executed by the processor, the tower crane installation path planning method as described above is implemented.

[0183] Another specific embodiment of the present application provides a computer-readable medium, wherein the computer-readable medium stores a computer program, and when the computer program is called and executed by a computer, the tower crane installation path planning method as described above is implemented.

[0184] The above is only a preferred implementation of the present application. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present application, and these all fall within the scope of protection of the present application.

Claims

1. A tower crane installation path planning method, characterized in that: The following steps are involved: S1: Collect 3D point cloud data and visual information in the tower crane installation area and perform preprocessing, fuse the point cloud data with the image data, and generate a fused 3D point cloud image, which contains distance and color information; S2: Use deep learning model to detect obstacles on the fused 3D point cloud image; S3: Use the RRT algorithm to plan a safe path based on obstacle detection results and lifting task requirements; S4: Control the movement of the lifting equipment according to the safe path obtained in step S3, and perform the lifting, lowering, horizontal movement and rotation of the hook according to the path planning result.

2. The tower crane installation path planning method according to claim 1, characterized in that: Step S1 includes the following steps: S11: Install a laser radar and a camera on the hoisting equipment to collect 3D point cloud data and visual image data of the environment. The laser radar scans the surrounding environment by rotation and measures distance data in different directions. These data are presented in the form of 3D point cloud images. Each point includes 3D coordinates and distance information. The camera captures real-time images, including color images and infrared images, which contain visual information. S12: Process the image data collected in step S11 by first performing Gaussian filtering on the image data to remove noise. The specific algorithm is as follows: Where I′(x,y) is the filtered image, I(x,y) is the original image, x,y are the positions of the current pixel, and σ is the standard deviation of the Gaussian kernel, which controls the smoothness of the filter. S13: After Gaussian filtering of the point cloud image, color image and infrared image, they are input into the data fusion network to realize data fusion. The specific algorithm is as follows: S131: Ensure that the lidar data and camera data have the same coordinate system and reference frame; if their coordinate systems are different, coordinate transformation is required to align them in the same coordinate system; the coordinate transformation matrix is ​​calculated as follows: In the above formula, R i,j Represents the elements in a matrix; this matrix represents the transformation from one coordinate system to another; S132: After the coordinates are aligned, data fusion is required, including color information fusion: extracting color information from the camera image, that is, the RGB value of each pixel; matching the color information with the corresponding point cloud data, and assigning a color value to each point cloud point; associating the points in the point cloud data with their projected positions in the camera image; distance information fusion: the laser radar provides distance information for generating the three-dimensional coordinates of the point cloud; combining the distance information with the color information to generate a fused three-dimensional point cloud image.

3. The tower crane installation path planning method according to claim 1, characterized in that: Step 2 includes the following steps: S21: Obtain the fused 3D point cloud image data, ensure that the data has been aligned and fused in the same global coordinate system for accurate obstacle detection, and randomly arrange all the fused data into training set and training set according to the composition of 8:2; S22: The Mask R-CNN deep learning model is used for obstacle detection tasks. First, the acquired 3D point cloud image data is converted into a format suitable for model input to normalize the image to meet the model input requirements. The activation function selected in normalization is the Sigmoid function. S23: Input the data into the network model for training, and the training set and test set are input into the network at a ratio of 8:2; the loss function used is the cross entropy loss function, where y represents the actual category label and p represents the category prediction probability distribution of the model; Among them, y i Represents the unique hot encoding of the actual category label, indicating that the probability of the sample belonging to the i-th category is 1, and the probability of other categories is 0; p i Represents the predicted probability distribution of the model, which represents the predicted probability that the sample belongs to the i-th category; the goal of the cross entropy loss function is to minimize the difference between the actual label and the model prediction so that the training model can classify samples more accurately; S24: Train the model for multiple iterations using the training dataset; in each iteration, feed batches of data into the model, calculate the loss, and then backpropagate the error to update the model parameters; regularly evaluate model performance on the validation set to monitor the progress of training; S25: Use the trained deep learning model to perform obstacle detection on the fused 3D point cloud image; the model will identify the location of obstacles in the image and generate a segmentation mask for each obstacle to identify the shape and boundary of the obstacle; output the result of obstacle detection, and the output information includes the location, shape, and category attributes of the obstacle.

4. The tower crane installation path planning method according to claim 1, characterized in that: Step S3 includes the following steps: S31: Use the fused 3D point cloud image and the output of obstacle detection to perceive the surrounding environment in real time; determine the location, shape, and category of obstacles in the current environment; S32: Determine the specific requirements of the lifting task, including the target location, weight, size and shape of the lifting object; determine the starting point and end point of the task, and the target location of the lifting object; S33: Use the path planning algorithm RRT to search for a safe path from the starting point to the target point; consider the obstacle information in the environment to ensure that the path avoids obstacles and maintains a sufficient safety distance; the RRT algorithm is as follows: S331: Select a starting point x on the map start And create a tree T containing the point; S332: loop the following steps: (1) Generate a random point x rand , randomly selected in the map, but with a certain probability of selecting the target point x goal ; (2) Find the distance x in the tree T rand The nearest node x nearest ; (3) From x nearest Move a certain distance along the direction to get a new node x new ; (4) If x new does not intersect with the obstacle, add it to the tree T, and change x nearest Connect to x new ; (5) If x new When approaching the target point, check whether there is a feasible path to the target; if so, the algorithm ends and the path is found; S333: When the maximum number of iterations is reached or a feasible path is found, the algorithm ends and the path is returned.

5. The tower crane installation path planning method according to claim 1, characterized in that: Step S4 includes the following steps: S41: Collect voltage and vibration data of lifting equipment.

6. The tower crane installation path planning method according to claim 1, characterized in that: Step S4 includes the following steps: S41: collecting voltage and vibration data of the hoisting equipment, and issuing an alarm message and controlling the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed a preset range; S42: Save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

7. A tower crane installation path planning system based on laser radar and visual perception, characterized in that: include: LiDAR, deployed at the tower crane installation site, scans the surrounding environment in a rotating manner and collects three-dimensional point cloud data; A camera is deployed at the tower crane installation site to collect image information of the tower crane installation site; Data acquisition module, used to collect data obtained by laser radar and camera; The data processing module processes and fuses the 3D point cloud data acquired by the LiDAR and the visual data of the camera. The LiDAR data processing includes data filtering, coordinate conversion, downsampling and data feature extraction; the visual data processing includes image correction, feature extraction and image registration; The obstacle detection module deploys a deep neural network to learn the characteristics and locations of obstacles from lidar and vision data; The path planning module deploys the RRT algorithm to generate feasible paths; and a control module for controlling the hoisting equipment according to the generated path.

8. The tower crane installation path planning system based on laser radar and visual perception according to claim 7 is characterized in that: Also includes: The safety and emergency module collects the voltage and vibration data of the hoisting equipment, issues an alarm message and controls the hoisting equipment to stop when the voltage and vibration data of the hoisting equipment exceed the preset range; and a data storage and analysis module, which is used to save the data of the hoisting equipment in operation, including sensor data in the hoisting equipment, hoisting operation data and path planning data.

9. A tower crane installation path planning terminal, characterized in that: It includes a memory and a processor, the memory stores a computer program, and when the computer program is called and executed by the processor, the tower crane installation path planning method as described in any one of claims 1-6 is implemented.

10. A computer-readable medium, characterized in that The computer-readable medium stores a computer program, and when the computer program is called and executed by a computer, the tower crane installation path planning method according to any one of claims 1 to 6 is implemented.

Images