Cantilever crane automatic returning and obstacle avoidance method based on image processing

Abstract

The application provides a kind of based on image processing's cantilever crane automatic put back and obstacle avoidance method, belongs to hoisting equipment automation control technical field.Through image data acquisition, preprocessing, accurate target detection and identification, and combining clustering analysis to solve the problem of occlusion, the 3D structure and size information of the obstacle are constructed.Based on the minimum boundary rectangle algorithm, sub-pixel positioning and coordinate conversion, the accurate pose estimation of the suspended object is realized.Combined with the motion characteristics of the cantilever crane, the segmented path planning is carried out, and the obstacle avoidance and accurate hoisting are realized.Through motion control and visual verification to form a closed loop, the whole process of picking up, transporting and stably putting back the object is finally completed.The high-precision, automatic identification and positioning of the suspended object in a complex occlusion environment are realized, the operation safety and running stability are guaranteed through intelligent obstacle avoidance path planning, the perception-planning-control closed loop is formed, and the intelligent level and operation precision of the hoisting system are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of automated control technology for lifting equipment, specifically relating to an image processing-based method for automatic return and obstacle avoidance of cantilever cranes. It is applicable to cantilever cranes with inflexible booms in industrial settings, such as single-beam cantilever cranes and bridge cantilever cranes. It enables the automatic return of items to their original lifting position after manual handling and allows for autonomous obstacle avoidance during movement. It is particularly suitable for the batch lifting, handling, and return of heavy items such as electrical workpieces and prefabricated components in cyclical operations. Background Technology

[0002] In recent years, cantilever cranes (such as single-girder cantilever cranes and bridge cantilever cranes) have become core equipment for heavy goods transportation in industrial production, and their application scenarios have continued to expand. However, the existing technology system has significant limitations in automated return and obstacle avoidance, making it difficult to meet the needs of efficient, accurate, and safe operation and maintenance. In particular, in terms of automation, with the acceleration of industrial automation transformation, the bottleneck of manual reliance on traditional lifting, handling, and return processes has become increasingly prominent, and the deficiencies of existing technologies in automation and accuracy are insufficient to meet the needs of efficient operation and maintenance.

[0003] Traditional cantilever cranes typically rely on manual control by one or two operators during the placement process. During operation, workers must continuously observe the relative position of the cantilever crane with respect to items and obstacles, manually controlling the horizontal arm's translation, hook lifting, and directional adjustments. This method not only incurs high labor costs but also significantly impacts placement accuracy due to operator experience, easily leading to item misalignment or even collisions with surrounding equipment. Even semi-automated placement solutions are only suitable for neatly arranged items or those relying on fixed tracks, unable to dynamically adapt to original positional shifts, easily causing process delays. The inefficiency and safety hazards of manual operation have become key constraints on production efficiency.

[0004] Existing cantilever crane positioning and obstacle avoidance technologies suffer from significant technical bottlenecks, making it difficult to balance accuracy and scene adaptability. In terms of positioning, 2D vision solutions lack depth information, resulting in low recognition success rates and large positioning deviations under varying lighting conditions. While traditional point cloud camera-based 3D vision solutions can acquire three-dimensional information, data processing cannot meet real-time control requirements, and matching accuracy decreases for smooth metal workpieces. Positioning markers are often simple black-and-white blocks or QR codes; under dust, oil contamination, and strong workshop lighting, recognition rates plummet, and blurred edges further amplify errors. Regarding obstacle avoidance, most low- to mid-range cantilever cranes lack active detection capabilities, resulting in high miss rates for small obstacles and failing to meet real-time control requirements. High-end LiDAR solutions are costly and lack data interaction with the positioning system, creating information silos. Path planning fails when facing occlusions or temporary obstacles, leading to high collision risks and severely inadequate technical adaptability and safety.

[0005] The complex industrial environment and lack of data collaboration further exacerbate the shortcomings of existing technologies. Vibration interference, such as the vibration of horizontal arms and low-frequency vibrations of ground equipment, causes slight camera jitter, resulting in large positioning deviations without compensation mechanisms. Dust and oil contamination reduce lens clarity and increase equipment failure rates. Light fluctuations, such as the switching between natural and artificial lighting and welding sparks, cause over-segmentation or under-segmentation of fixed-threshold algorithms, leading to contour extraction failure. Regarding data collaboration, the original location of objects largely relies on manual recording, resulting in excessively long backtracking times and high error rates; the lack of historical data traceability leads to the recurrence of the same problems. Existing technologies are fragmented, failing to form an integrated solution encompassing automated placement, precise positioning, dynamic obstacle avoidance, environmental interference resistance, and data collaboration, necessitating technological innovation to overcome these bottlenecks. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies, such as low positioning accuracy, poor environmental adaptability, weak obstacle avoidance ability, complex motion control, lack of data collaboration, and insufficient long-term stability, and to provide an image processing-based method for automatic return and obstacle avoidance of cantilever cranes.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing, comprising the following steps: Acquire image data of the target area and preprocess the acquired image data; Target detection is performed on the preprocessed image data to locate and identify the type and location of the object to be suspended in the image data; cluster analysis is combined to solve the occlusion problem between obstacles, determine the spatial location of the target and the obstacle, and obtain the 3D structure and size of the obstacle; Accurate pose estimation of the suspended object is achieved by using the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation. Based on the accurate pose estimation of the object to be suspended and the linear motion characteristics of the cantilever crane, combined with the original lifting position of the object to be suspended and the 3D structure and size of the obstacle, segmented paths of lifting, translation and descent are planned to realize path planning and obstacle avoidance from the original lifting position to the target placement position. Through motion control and visual verification, the cantilever crane is driven to grab items along the planned segmented path and transport them to the target placement location, so as to smoothly return the items from the end of the path to the original position.

[0008] The method for acquiring image data of the target area and preprocessing the acquired image data is as follows: Anti-interference marks are set on the surface of the object to be suspended. The anti-interference marks are circular surface light sources that can switch light sources in conjunction with the camera. A camera is installed at the end of the cantilever to collect image data of the target area, including pixel coordinates, depth values ​​and RGB color information. Anti-interference extraction and data preprocessing are performed on the region where anti-interference markers are located, including: Calculate the difference image between the anti-interference marker light source turned-on image and the light source turned-off image; Gaussian filtering is applied to the difference image, and pixels that meet the conditions are extracted based on a preset contrast threshold to form a marked region. To compensate for camera shake, two fixed reference markers are deployed on both sides of the working area of ​​the cantilever crane. The translation and rotation errors of the current frame relative to the initial calibration frame are calculated based on the coordinate changes of the two reference markers. The pixel coordinates of the object marker area are then transformed in reverse to obtain the corrected pixel coordinates. The depth values ​​and the corrected pixel coordinates are standardized to eliminate dimensional differences and generate a standardized data matrix, resulting in preprocessed image data.

[0009] The method for performing target detection on preprocessed image data, locating and identifying the category and location of the object to be suspended in the image data; and combining cluster analysis to solve the occlusion problem between obstacles, determining the spatial location of the target and obstacles, and obtaining the 3D structure and size of the obstacles is as follows: The Faster R-CNN model is used to perform target detection on the preprocessed image data, and to locate and identify the type and location of the objects to be suspended in the image data. The edge of the target region image data is extracted by the Canny operator to obtain the edge detection result. The Hough transform is applied to the edge detection result to convert the edge points into peak values ​​in the parameter space. The obstacle edge line segments and corresponding depth values ​​are extracted to initially outline the obstacle contour. Based on the preliminary outline of the obstacle, the endpoints of the obstacle line segments that are close to each other are analyzed and merged to obtain the complete obstacle outline; Based on the complete obstacle outline and the spatial vertical relationship of the obstacle line segments, the 3D structure and size of the obstacle are confirmed.

[0010] The method for obtaining a complete obstacle outline by analyzing and merging the endpoints of closely spaced obstacle line segments based on the initially sketched obstacle outline is as follows: To calculate the endpoint distance, iterate through the endpoints of all segments in the obstacle segment set and calculate the distance between any two endpoints. and Euclidean distance:

[0011] Among them, if Then and The nodes are grouped into the same cluster, with D being the distance threshold. Then, for each cluster, the average coordinates of all endpoints within the group are calculated and used as the coordinates of the merged cluster center. Replace the endpoints of the original line segments with cluster center points, reconnect the line segments, and form a complete obstacle outline.

[0012] The method for accurately estimating the pose of a suspended object using the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation is as follows: Extract the contour point set of the anti-interference marked area image, obtain the minimum bounding rectangle of the contour through the minimum bounding rectangle algorithm, and calculate the initial pose of the object to be suspended through the minimum bounding rectangle; A moment basis edge operator and an ellipse fitting algorithm are used to perform sub-pixel center localization of the suspended object; Based on the initial pose and sub-pixel center localization of the object to be suspended, the object coordinates in the camera's own coordinate system are converted into the base coordinate system for the motion control of the cantilever crane, thus completing the backtracking of the original position and obtaining a precise pose estimate of the object to be suspended.

[0013] The step of extracting the contour point set of the anti-interference marked region image, obtaining the minimum bounding rectangle of the contour using the minimum bounding rectangle algorithm, and calculating the initial pose of the object to be suspended using the minimum bounding rectangle is as follows: Scan each column of the smallest bounding rectangle region, calculate the start and end coordinates of the target in each column, and solve for the column centroid; Based on the set of contour points, the vertices of the minimum enclosing rectangle are calculated by least-squares fitting of the horizontal and vertical principal axes. Compare the lengths of adjacent sides of the rectangle, take the smaller value as the minor axis, output the center, width, height of the rectangle and the angle between the minor side and the horizontal axis of the image, and calculate the pose angle accordingly to obtain the initial pose of the object to be suspended. The method for sub-pixel center localization of suspended objects is as follows: Using the center of the item marker as the origin, a 5×5 window is used to calculate the two-dimensional spatial moments, and the edge parameters are corrected to obtain sub-pixel level edge points. Least-squares elliptic fitting is used for the edge points:

[0014] Solving for the sub-pixel center of the item marker .

[0015] The step of converting the object's coordinates in the camera's own coordinate system to the base coordinate system of the cantilever crane's motion control based on the initial pose and sub-pixel center localization of the object to be suspended, thus completing the backtracking of the original position and obtaining the accurate pose estimation of the object to be suspended, involves obtaining the object's coordinates in the camera coordinate system. Convert to cantilever base coordinates The formula is as follows:

[0016] Among them, the transformation matrix from the camera coordinate system to the cantilever base coordinate system , where R is the rotation matrix and t is the translation vector; The coordinates of the object in the coordinate system of the cantilever crane base. The coordinates of the object in the camera coordinate system.

[0017] The process involves accurately estimating the pose of the object to be suspended and the linear motion characteristics of the cantilever crane. Combining the original lifting position of the object with the 3D structure and dimensions of the obstacle, a segmented path is planned for lifting, translation, and lowering. This process enables path planning and obstacle avoidance from the original lifting position to the target placement position. In this step, the path planning must meet the mechanical constraints of the cantilever crane. The lifting segment of the segmented path must ensure that the workpiece is vertically removed from the obstacle by a safe height of not less than 0.2m. The translation segment must be within the horizontal stroke range of the boom. The lowering segment must be precisely aligned with the target position.

[0018] In the step of driving the cantilever crane to grab and transport items along a planned segmented path to the target placement location through motion control and visual verification, and smoothly returning the items from the end of the path to the original position, the motion control method is as follows: The deviation between the target placement position and the actual position is obtained and verified using the following formula:

[0019] in, For the target location, For actual location, , These are the first control parameter and the second control parameter, respectively.

[0020] It includes an edge computing terminal deployed locally on the cantilever crane and a remote cloud platform, used to perform real-time image processing and anomaly response, as well as receive data uploaded by the edge computing terminal and perform incremental learning optimization on the recognition and localization model.

[0021] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an image processing-based method for automatic return and obstacle avoidance of cantilever cranes. Through image data acquisition, preprocessing, precise target detection and recognition, and combined with cluster analysis to solve occlusion problems, a 3D structure and size information of obstacles are constructed. Based on the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation, accurate pose estimation of the suspended object is achieved. Based on this, and combined with the motion characteristics of the cantilever crane, segmented path planning (lifting-translation-descent) is performed, effectively achieving obstacle avoidance and precise lifting. A closed loop is formed through motion control and visual verification, ultimately completing the entire process from object grabbing, transportation to smooth return. This method achieves high-precision, automated identification and positioning of the suspended object in complex occlusion environments. Intelligent obstacle avoidance path planning ensures operational safety and stability, forming a perception-planning-control closed loop, significantly improving the intelligence level, operational accuracy, and overall operational efficiency of the lifting system.

[0022] Furthermore, anti-interference markers are applied to the surface of the object to be suspended, providing a stable and identifiable feature reference for the vision system, greatly enhancing its robustness and data reliability in complex environments. Based on this, by combining the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation, extremely high-precision pose estimation of the marker and its attached object is achieved. This technical approach not only ensures millimeter-level and even sub-pixel-level measurement accuracy of the object's position and orientation but also reduces the impact of environmental noise on the measurement results through active anti-interference design. This allows subsequent path planning, obstacle avoidance, and motion control to be based on more accurate and stable perception data, thereby improving the overall positioning accuracy, operational stability, and task execution reliability of the system.

[0023] Furthermore, by introducing cluster analysis to address the occlusion problem between obstacles, the scene structure can be analyzed more accurately, the spatial relationship between the target and the obstacle can be determined, the 3D structure and size of the obstacle can be restored, and the perception robustness of the system in unstructured, multi-object stacked scenes can be enhanced. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a comparison chart of positioning errors under different interference environments in Embodiment 2 of the present invention; Figure 3 These are obstacle avoidance effect diagrams under different occlusion scenarios in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram illustrating the accuracy decay trend of the cantilever crane during 30 days of long-term operation in Embodiment 2 of the present invention. Figure 5 This is a schematic diagram comparing the operation and maintenance results in Embodiment 2 of the present invention. Detailed Implementation

[0025] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0026] Example 1 like Figure 1 As shown, an image processing-based method for automatic cantilever crane repositioning and obstacle avoidance includes the following steps: S1: Acquire image data of the target area and preprocess the acquired image data; S2: Perform target detection on the preprocessed image data, locate and identify the category and location of the object to be suspended in the image data; combine cluster analysis to solve the occlusion problem between obstacles, determine the spatial location of the target and the obstacle, and obtain the 3D structure and size of the obstacle; S3: Accurate pose estimation of the suspended object is achieved through the minimum bounding rectangle algorithm, sub-pixel positioning, and coordinate transformation. S4: Based on the accurate pose estimation of the object to be suspended and the linear motion characteristics of the cantilever crane, combined with the original lifting position of the object to be suspended and the 3D structure and size of the obstacle, plan the segmented path of lifting, translation and descent to realize path planning and obstacle avoidance from the original lifting position to the target placement position. S5: Through motion control and visual verification, the cantilever crane is driven to grab items along the planned segmented path and transport them to the target placement location, so as to smoothly return the items from the end of the path to the original position.

[0027] Specifically, in S1, image data of the target area is acquired and preprocessed. High-quality data acquisition is achieved through anti-interference markers and an RGB-D camera, and environmental interference is eliminated through image preprocessing.

[0028] First, hardware deployment and data acquisition are carried out. Anti-interference markers—circular surface light sources—are affixed to the surface of the object to be processed (suspended), covered with a special patterned film. These markers integrate a wireless synchronization module, allowing for synchronized switching of the light source with the camera. A RealSense D435i RGB-D camera is used, mounted at the end of the cantilever boom, covering a 1.5m x 1.5m processing area. Two fixed reference markers, SRM1 and SRM2, are simultaneously deployed on either side of the processing area, spaced 5m apart and 3m high, for camera shake compensation. After processing, the operator presses the trigger button, and the camera simultaneously acquires images of the light source at 10fps. Image with light source off Synchronously output pixel coordinates Depth value and RGB color information The resolution of a single frame image is set to 1280×720 pixels.

[0029] Next, anti-interference region extraction data preprocessing is performed. By utilizing the image differences under light source on / off states, background interference caused by ambient light and dust is suppressed, increasing the grayscale contrast of the marked region to over 30:1 to ensure the accuracy of subsequent edge extraction. The difference image is then calculated and the marked region is extracted.

[0030] For difference images Gaussian filtering is used to eliminate image noise. The region of interest (ROI) is defined by pixels that meet the following conditions. constitute: ,in A preset contrast threshold is used, and then bilateral filtering is applied to preserve the details of the marked edges. Camera shake compensation is then performed based on the coordinate changes of SRM1 and SRM2. The translation and rotation changes of the current frame relative to the initial calibration frame are calculated using the coordinates of two fixed reference points.

[0031] The translation error compensation uses the average value of the coordinate changes of the two reference points as the image plane translation estimate:

[0032] in , The current coordinates of SRM1 , These are the initial coordinates; , The current coordinates of SRM2 , These are the initial coordinates.

[0033] The rotation error around the Z-axis is calculated by changing the angle of a vector formed by two reference points between consecutive frames, which is taken as the camera's rotation angle around the optical axis.

[0034] pixel coordinates of item markers The translation obtained by applying the above calculations and rotation A reverse transformation is performed to obtain the corrected coordinates, ensuring that the positioning deviation caused by image jitter does not exceed 0.02cm. , Mark the pixel coordinates of the object; after correction, the positioning deviation caused by image jitter should not exceed 0.02cm; finally, the depth value... With pixel coordinates Perform Min-Max normalization: This eliminates dimensional differences and generates a standardized data matrix. Where n is the number of samples, and 3 represents u, v, ... The dimensions are used to obtain a preprocessed binary image of the labeled region.

[0035] Specifically, in S2, target detection is performed on the preprocessed image data to locate and identify the category and location of the objects to be suspended in the image data. Cluster analysis is used to solve the occlusion problem between obstacles, determining the spatial location of the target and obstacles, and obtaining the 3D structure and dimensions of the obstacles. A deep learning model is used to achieve accurate object classification, and traditional image processing algorithms are used to extract obstacle contours. Cluster analysis is combined to solve the occlusion problem, ultimately completing the spatial location of the target and obstacles, achieving accurate object and obstacle classification and contour extraction, and providing a basis for subsequent path planning.

[0036] S21: First, the Faster R-CNN (Faster Region-based Convolutional Neural Network) model is used to perform object detection on the preprocessed RGB image data. Introducing a Region Proposal Network (RPN) significantly improves detection speed and accuracy compared to traditional object detection methods.

[0037] The model was trained on a dataset containing 10 categories of industrial items, using ResNet-50 as the backbone network and optimizing its parameters through multiple iterations. Deployed in the cantilever crane control module, after inputting an image, the model generates region proposals that may contain the target item via the RPN. Subsequent network layers then classify these proposed regions and perform bounding box regression, outputting the item category, such as wedge clamps, C-shaped clamps, etc. The model then locates the coordinates and assigns a confidence score, setting a confidence threshold of 0.85 to ensure the reliability of the detection results. This effectively solves the problem of accurate classification and localization of items in complex industrial scenarios, providing crucial information for the cantilever crane to accurately grasp items.

[0038] S22: For the entire processing area, i.e., the image data of the target area acquired in S1, the unprocessed image data is first subjected to Gaussian filtering to reduce image noise interference. Then, the Canny operator is applied to extract edges (these edges usually correspond to structural obstacles in the scene). The threshold is iteratively adjusted until the edge extraction completeness meets the requirements. This operator is designed based on low error rate, good localization performance and single response constraint, which can effectively identify real edges in the image and suppress false edges caused by noise, transforming areas of abrupt intensity changes in the image into sets of edge pixels. Next, Hough transform is applied to the Canny edge detection results, with a distance resolution of 1 pixel, an angle resolution of π / 180 radians, and a voting threshold of 100, converting the edge points into peak values ​​in parameter space, extracting obstacle edge segments, and outputting the coordinates of the endpoints of the segments. and the corresponding depth value The initial outline of the obstacle has been sketched.

[0039] S23: Based on the initially outlined obstacle contour, analyze and merge the endpoints of obstacle segments with relatively close distances to solve the problem of segment breakage caused by obstacle occlusion, effectively handle occlusion, and achieve complete obstacle contour reconstruction. First, calculate the endpoint distances, then traverse the set of obstacle segments. Find the endpoints of all line segments in the given equation, and calculate any two endpoints. and Euclidean distance:

[0040] The distance threshold is set based on the accuracy requirements for obstacle recognition in industrial scenarios. ,like Then and Group them into the same cluster. Then, for each cluster, calculate the average coordinates of all endpoints within the group, and use this average as the coordinates of the merged cluster center. Replace the endpoints of the original line segments with cluster center points, reconnect the line segments, and form a complete obstacle outline.

[0041] S24: Confirm the 3D structure and dimensions of the obstacle by analyzing the spatial vertical relationships of the obstacle segments. For the reconstructed obstacle segments, determine the endpoint coordinates... and Calculate the direction vector of the line segment Select three line segments from the obstacle outline that intersect at the same point, and calculate the dot product of the direction vectors of any two line segments:

[0042] , , Let be the direction vector of the line segment, and set the perpendicularity threshold to 0.05, i.e. If the three line segments are approximately perpendicular to each other, the contour is confirmed as a valid three-dimensional obstacle.

[0043] Specifically, in S3, the MinBRect algorithm, subpixel localization, and coordinate transformation are used to achieve accurate pose estimation of the object to be suspended and obtain the original lifting position.

[0044] S31: Extract the contour point set from the binary image of the marked region extracted in S1. Apply the MinBRect algorithm to directly obtain the minimum bounding rectangle of the contour. Scan each column of the minimum bounding rectangle region and calculate the starting point of the target in each column. and the finish line Coordinates, solve for column centroid:

[0045] Based on the set of contour points, the horizontal principal axis is fitted using least squares. with vertical principal axis Calculate the vertices of the minimum bounding rectangle. ; Compare the lengths of adjacent sides of a rectangle: The smaller value is used as the minor axis. The center, width, height, and angle between the short side and the horizontal axis of the image of the output rectangle are used to calculate the pose angle. The initial position of the object to be suspended:

[0046] in , is the angle between the minor axis of the item and the horizontal axis. When the width is less than the height, The angle is along the width direction (minor axis); when the height is less than the width, The angle is in the height direction (minor axis). The measured pose angle error does not exceed ±0.1°.

[0047] S32: Sub-pixel localization is performed using a moment-based edge operator and an ellipse fitting algorithm. A 5×5 window is used with the object marker center as the origin to calculate the two-dimensional spatial moments, correcting edge parameters to obtain sub-pixel level edge points. Least-squares elliptic fitting is used for the edge points:

[0048] Solve for the coordinates of the center of the ellipse That is, the sub-pixel center of the item marker has a positioning error of no more than 0.05 pixels, which translates to a physical error of no more than 0.5 cm.

[0049] S33: Based on the initial pose and sub-pixel center localization of the object to be suspended, the object coordinates in the camera's own coordinate system are transformed into the base coordinate system of the cantilever crane's motion control, completing the backtracking of the original position and obtaining a precise pose estimate of the object to be suspended. A transfer matrix from the camera to the cantilever crane's base coordinate system is established using coordinates, relying on external camera inputs, to transform the object coordinates in the camera's own coordinate system into the base coordinate system of the cantilever crane's motion control.

[0050] Item coordinates in camera coordinate system Convert to cantilever base coordinates The formula is as follows:

[0051] Among them, the transformation matrix from the camera coordinate system to the cantilever base coordinate system , where R is the rotation matrix and t is the translation vector. Let be the coordinates of the object in the cantilever crane coordinate system, where The value is fixed at 0, which conforms to the linear motion characteristics of a cantilever crane. Let be the coordinates of the object in the camera coordinate system, where Calculated from RGB image pixel coordinates and camera intrinsic parameters. Depth values ​​are obtained directly from the RGB-D camera.

[0052] Since the coordinate data output by the camera has a mapping relationship between pixels and physical dimensions, it is necessary to convert the pixel-level coordinates into actual physical coordinates using a scaling factor. The scaling factor is calculated using the known physical dimensions of the anti-interference markers and the pixel dimensions of the markers in the image, as shown in the following formula:

[0053] in , , which are the scale factors in the X and Y axis directions; Describe the known physical diameter of the anti-interference marker; , This represents the number of pixels along the major and minor axes of the anti-interference marker ellipse in the image.

[0054] To obtain the original lifting position of an item, based on the item category identified in step S2, a collaborative data mechanism based on BIM-IFC and MLP (Multilayer Perceptron) is designed. The GlobalID of the item is retrieved from the BIM database based on IFC format, and the corresponding original lifting position coordinates (X0, Z0) are obtained by calling the database interface. Here, X0 is the X-axis coordinate of the original lifting position in the cantilever base coordinate system, and Z0 is the vertical height coordinate of the original lifting position. This process requires no manual intervention, and the backtracking time is ≤0.5s, ensuring the rapid and accurate acquisition of original position data. An MLP neural network is used to input normalized features of the item, such as area and main axis length, to perform secondary verification of the item type and avoid misplacement. After placement, the latest placement time and position deviation of the item in the database are automatically updated, providing traceability data for operation and maintenance.

[0055] Specifically, in S4, based on the accurate pose estimation of the object to be suspended and the linear motion characteristics of the cantilever crane, combined with the original lifting position of the object and the 3D structure and dimensions of the obstacle, segmented paths for lifting, translation, and descent are planned to achieve path planning and obstacle avoidance from the original lifting position to the target placement position. The path planning must meet the mechanical constraints of the cantilever crane: a maximum horizontal travel of 4m for the boom and a maximum vertical lifting height of 5m for the hook; and the lifting segment of the segmented path must ensure that the workpiece is vertically removed from the obstacle by a safe height of at least 0.2m, the translation segment must be within the horizontal travel range of the boom, and the descent segment must be precisely aligned with the target position.

[0056] Specifically, first control the hook from its current height. Lift vertically along the Z-axis to a safe height The motion parameters are set to , The motor power and load are adjusted to prevent the object from swaying; then the horizontal arm moves along the X-axis from its current position. Follow the broken line path (3m→1.7m→2.3m→original position) The movement involves translating around the horizontal obstacle by passing through inflection points at 1.7m (0.1m safety boundary to the left of the obstacle) and 2.3m (0.1m safety boundary to the right of the obstacle). The translation parameters are as follows. , It is compatible with the transmission precision of a 4m horizontal boom track; finally, the hook descends vertically from 2.5m to its original height. The parameters of the lifting section are used to ensure accuracy, and the total time for the three paths does not exceed 20 seconds, which is in line with the operation rhythm of the short horizontal boom scenario.

[0057] After path planning, path points are sampled at 0.01m intervals. If a sample point falls into an obstacle area, a collision risk is determined, and the path is adjusted upwards in 0.5m increments. And re-inspect until there is no collision; if If a collision still occurs even after reaching the 5m hanging height limit, an audible and visual alarm is triggered, and movement is paused. After the path verification is successful, the vision module reconfirms the deviation between the current pose of the object and its original position. If the deviation is less than 0.5cm, the movement proceeds directly; if it exceeds 0.5cm, the hook position is finely adjusted via the PI controller. Once the deviation is met, the path movement is initiated.

[0058] S5: Through precise motion control and visual verification, the cantilever crane is driven to grab items along the planned segmented path and transport them to the target placement position, realizing the smooth return of items from the end of the path to the original position. The core solution is to solve the problem of linear motion precision control and attitude matching, ensuring that the return error meets the requirements of industrial operation.

[0059] First, calculate the motion parameters: Considering the motion characteristics of the cantilever crane, first calculate the object pose angle obtained in step S3. Adjust the hook rotation angle to Ensure the item is returned to the same orientation as when it was initially lifted, and set the rotation speed to [value missing]. To prevent items from swaying due to excessive rotation, a PI controller is used for closed-loop control of the boom translation and hook lifting—the input is the deviation between the target position and the actual position, where... Acquired by the transverse arm track encoder, The output control quantity is obtained from the hook lifting encoder, and the formula is as follows:

[0060] in, For the target location, For actual location, , The first and second control parameters of the controller are used to balance response speed and stability, ultimately achieving a horizontal arm translation error of no more than 0.3cm and a hook lifting error of no more than 0.2cm, meeting the precise positioning requirements of short horizontal arm scenarios.

[0061] Then, the return confirmation and release are performed: During the hook descent, the RGB-D camera mounted on the end of the boom acquires images of the anti-interference mark on the surface of the object in real time. Using sub-pixel positioning technology, the deviation between the mark center and the original position is calculated. When the position deviation does not exceed 0.5cm and the pose angle deviation does not exceed 0.1°, the control module triggers the hook electromagnetic release signal. The release response time does not exceed 100ms to avoid delays that could cause positional shifts. If the deviation exceeds the threshold, the PI controller immediately fine-tunes the position of the boom or hook until the deviation meets the standard before releasing the object, ensuring that the object falls accurately to the original lifting position. The measured return success rate is no less than 99.9%.

[0062] Furthermore, by combining real-time edge processing with cloud-based model iteration, the system's long-term stability is ensured. Addressing the short-path, high-response requirements of cantilever cranes, an operable process combining real-time edge processing with continuous cloud optimization and data traceability for maintenance resolves issues such as long-term accuracy degradation, scene adaptation changes, and missing maintenance data that cannot be covered by the preceding steps S1 to S5. At the edge, an NVIDIA Jetson Nano 2GB edge terminal is embedded in the cantilever crane control box, communicating with the encoder via RS485 and the RGB-D camera via Ethernet. A cropped Faster R-CNN optimized for small target recognition and a simplified MinBRect algorithm are deployed, detecting deviations with replacement every 100ms, responding to anomalies within 30ms, and pausing the alarm when the deviation exceeds 0.5cm, adapting to the rapid error correction requirements of short boom scenarios. Only 11-dimensional object feature vectors are uploaded, reducing daily data transmission by 800GB. The cloud receives edge data daily at set intervals, uses SGD incremental learning to optimize the model, and then distributes update packages via OTA to ensure a stable recognition accuracy of no less than 99.5%. Meanwhile, the edge terminal writes data such as return time and deviation into the BIM database in real time, and the cloud-based MLP performs secondary verification of the item type. During operation and maintenance, data can be traced to locate faults, shortening the processing time and forming a complete closed loop for the implementation of this technology.

[0063] This invention addresses the core pain points of cantilever cranes, such as high reliance on manual labor, low return accuracy, weak obstacle avoidance, and poor long-term stability. It utilizes a full-process approach including multi-source image anti-interference preprocessing, target obstacle recognition, pose localization and backtracking, segmented path obstacle avoidance, linear motion control, and edge cloud collaborative operation and maintenance. Relying on core technologies such as anti-interference marking, PI closed-loop control, and incremental learning, and combining the advantages of deep learning and traditional image processing, it ultimately achieves a return error of no more than 0.5cm, a collision risk of no more than 1%, and a return error rate of no more than 0.1%, forming an integrated technology system for automatic cantilever crane return and obstacle avoidance with a closed loop of perception, decision-making, execution, and optimization.

[0064] This system deeply integrates optical control contrast anti-interference technology, short boom adaptable segmented path planning, edge cloud collaborative iteration mechanism with the structural characteristics of cantilever cranes. It breaks through the limitations of traditional single-link optimization and poor scenario adaptability, significantly improving the automation level, positioning accuracy and long-term stability of the equipment, and providing a highly adaptable and feasible technical paradigm for the automation upgrade of industrial-grade small and medium-sized cantilever cranes.

[0065] Example 2 To verify the effectiveness of the intelligent control method for cantilever cranes described in this invention, a small-to-medium-sized cantilever crane with a lifting height of 5m and a horizontal boom length of 4m was selected from a power equipment factory as the test platform. The test environment was set up in a 10kV workpiece processing workshop with a dust concentration of approximately 5mg / m³, intermittent light intensity fluctuations of 500–2000 lux, and typical obstacles such as tool racks and stacked workpieces. The tests covered 10 types of industrial items, including power-specific workpieces such as wedge clamps and C-shaped clamps, and a total of 100 sets of lifting, processing, and return cycle tests were conducted. The technical effectiveness was verified based on four core indicators: positioning accuracy, obstacle avoidance capability, long-term operational stability, and maintenance efficiency. A detailed description is provided below with reference to the accompanying drawings.

[0066] like Figure 2 As shown, the comparative results of positioning errors under different interference environments are demonstrated. In interference scenarios with dust and light levels fluctuating between 500 and 2000 lux, the positioning method of this invention, which combines anti-interference markers with sub-pixel positioning, stabilizes the error within the range of 0.2–0.5 cm, with an average of only 0.32 cm and no significant fluctuations. In contrast, traditional visual positioning methods, lacking anti-interference markers, exhibit drastic error fluctuations of 1.5–3 cm. While manual-assisted positioning relies on experience to control the error within 0.8–1.2 cm, it requires repeated manual calibration. This indicates that the present invention, by using anti-interference markers combining a light-controlled contrast film and a wirelessly synchronized light source, blocks environmental interference. Combined with sub-pixel positioning and dual-reference marker jitter compensation, it breaks through the pixel-level accuracy bottleneck, achieving high-precision and stable positioning with anti-interference capabilities. This solves the positioning pain points of small and medium-sized cantilever cranes, which are sensitive to environmental conditions and heavily reliant on manual labor.

[0067] like Figure 3As shown, the comparison of obstacle avoidance performance under different occlusion scenarios is clearly demonstrated. The segmented path technology of lifting, translation, and descent provided by this invention is particularly outstanding in its precise adaptability to the 4m short horizontal arm stroke and its superior obstacle avoidance. In three scenarios—unobstructed, partially obstructed, and fully obstructed—the obstacle outline can be accurately identified, and then obstacle avoidance is performed based on the three-segmented path of lifting, translation, and descent: In the fully obstructed scenario, the workpiece is first vertically lifted to a safe height above the top of the tool rack, then horizontally translated to the unobstructed area directly above the target, and finally vertically descended to complete the grabbing, thereby achieving an obstacle avoidance success rate of no less than 92% and no collision. In contrast, the straight path used by the traditional method cannot avoid obstacles in the fully obstructed scenario because it does not match the range of motion of the short horizontal arm, resulting in an obstacle avoidance success rate of only 45% and a collision rate of 18%. This indicates that the present invention, through deep coupling of segmented paths and short boom travel, overcomes the limitation of insufficient obstacle avoidance space in traditional straight-path short boom scenarios, achieves precise adaptation of segmented paths and low collision and high avoidance rate in all scenarios, and effectively solves the drawback of difficulty in obstacle avoidance in complex occlusion scenarios under the short boom of small and medium cantilever cranes.

[0068] like Figure 4 As shown, the comparison of the accuracy degradation trend over 30 days of long-term operation is clearly demonstrated. The support provided by the edge-cloud collaborative optimization and segmented path parameter dynamic calibration mechanism for the long-term stability of the scene is particularly crucial. In the test, the method of this invention collects segmented path operation data in real time at the edge and uses a daily incremental learning method in the cloud. To address the issues of depth value fluctuation within ±2cm and cumulative offset of the 4m horizontal arm lens, the method dynamically adjusts the safety height compensation coefficient and translation inflection point coordinate correction value of the segmented path. Ultimately, the readback error gradually increased from 0.3cm to 0.48cm within 30 days, with an accuracy degradation of only 0.05cm. In contrast, the traditional method, lacking a parameter iteration mechanism, always uses the initial parameters for the segmented path. As the equipment ages and environmental dust accumulates, the error continues to expand, surging from 0.3cm to 1.05cm within 30 days, a degradation of 119%, and exceeding the 0.8cm error threshold allowed for industrial operations from the 20th day onwards. This indicates that the present invention, through edge cloud collaboration and dynamic optimization of segmented path parameters, overcomes the limitations of traditional fixed path long-term operation parameter mismatch and rapid accuracy decay, and achieves full-cycle parameter self-adaptation and high-precision stable operation.

[0069] like Figure 5As shown, the comparison of time consumption and effectiveness of the fault maintenance process is clearly demonstrated, with a particularly significant improvement in maintenance efficiency. The traditional maintenance process on the left requires four steps: manual inspection, equipment shutdown, experience-based judgment, and repair and debugging. Due to a lack of data support, maintenance personnel must check the boom rail, hook motor, and vision module one by one, often resulting in repeated trial and error due to experience-based misjudgments, leading to long processing times. In contrast, the process of this invention on the right, through BIM database data tracing, fault location, and precise repair, is significantly faster. Its core lies in: relying on the BIM database to retrieve the segmented path operation records and historical positioning error data associated with the GlobalID of the item in real time, quickly locating the fault source. When an increase in positioning deviation is detected synchronized with the lens cleaning cycle, lens dust contamination is directly identified, eliminating the need to check the boom mechanical structure. Simultaneously, the PI controller parameter log from step S5 is linked; if a translation error exceeds 0.3cm, it can be traced with one click to determine if it is due to track encoder parameter mismatch, avoiding the blindness of traditional manual disassembly.

[0070] This indicates that the present invention, by replacing manual investigation with data traceability and linking operation and maintenance data with segmented paths and positioning accuracy data, breaks through the limitations of traditional operation and maintenance that rely on experience-based trial and error and have long downtime. It achieves a leap from blind operation and maintenance in traditional short crossarm or low-hanging high scenarios to precise operation and maintenance of the present invention, which greatly improves efficiency and avoids secondary failures caused by misjudgment.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing, characterized in that, Includes the following steps: Acquire image data of the target area and preprocess the acquired image data; Target detection is performed on the preprocessed image data to locate and identify the type and location of the object to be suspended in the image data; cluster analysis is combined to solve the occlusion problem between obstacles, determine the spatial location of the target and the obstacle, and obtain the 3D structure and size of the obstacle; Accurate pose estimation of the suspended object is achieved by using the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation. Based on the accurate pose estimation of the object to be suspended and the linear motion characteristics of the cantilever crane, combined with the original lifting position of the object to be suspended and the 3D structure and size of the obstacle, segmented paths of lifting, translation and descent are planned to realize path planning and obstacle avoidance from the original lifting position to the target placement position. Through motion control and visual verification, the cantilever crane is driven to grab items along the planned segmented path and transport them to the target placement location, so as to smoothly return the items from the end of the path to the original position.

2. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 1, characterized in that, The method for acquiring image data of the target area and preprocessing the acquired image data is as follows: Anti-interference marks are set on the surface of the object to be suspended. The anti-interference marks are circular surface light sources that can switch light sources in conjunction with the camera. A camera is installed at the end of the cantilever to collect image data of the target area, including pixel coordinates, depth values ​​and RGB color information. Anti-interference extraction and data preprocessing are performed on the region where anti-interference markers are located, including: Calculate the difference image between the anti-interference marker light source turned-on image and the light source turned-off image; Gaussian filtering is applied to the difference image, and pixels that meet the conditions are extracted based on a preset contrast threshold to form a marked region. To compensate for camera shake, two fixed reference markers are deployed on both sides of the working area of ​​the cantilever crane. The translation and rotation errors of the current frame relative to the initial calibration frame are calculated based on the coordinate changes of the two reference markers. The pixel coordinates of the object marker area are then transformed in reverse to obtain the corrected pixel coordinates. The depth values ​​and the corrected pixel coordinates are standardized to eliminate dimensional differences and generate a standardized data matrix, resulting in preprocessed image data.

3. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 2, characterized in that, The method for performing target detection on preprocessed image data, locating and identifying the category and location of the object to be suspended in the image data; and combining cluster analysis to solve the occlusion problem between obstacles, determining the spatial location of the target and obstacles, and obtaining the 3D structure and size of the obstacles is as follows: The Faster R-CNN model is used to perform target detection on the preprocessed image data, and to locate and identify the type and location of the objects to be suspended in the image data. The edge of the target region image data is extracted by the Canny operator to obtain the edge detection result. The Hough transform is applied to the edge detection result to convert the edge points into peak values ​​in the parameter space. The obstacle edge line segments and corresponding depth values ​​are extracted to initially outline the obstacle contour. Based on the preliminary outline of the obstacle, the endpoints of the obstacle line segments that are close to each other are analyzed and merged to obtain the complete obstacle outline; Based on the complete obstacle outline and the spatial vertical relationship of the obstacle line segments, the 3D structure and size of the obstacle are confirmed.

4. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 3, characterized in that, The method for obtaining a complete obstacle outline by analyzing and merging the endpoints of closely spaced obstacle line segments based on the initially sketched obstacle outline is as follows: To calculate the endpoint distance, iterate through the endpoints of all segments in the obstacle segment set and calculate the distance between any two endpoints. and Euclidean distance: Among them, if Then and The nodes are grouped into the same cluster, with D being the distance threshold. Then, for each cluster, the average coordinates of all endpoints within the group are calculated and used as the coordinates of the merged cluster center. Replace the endpoints of the original line segments with cluster center points, reconnect the line segments, and form a complete obstacle outline.

5. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 3, characterized in that, The method for accurately estimating the pose of a suspended object using the minimum bounding rectangle algorithm, sub-pixel localization, and coordinate transformation is as follows: Extract the contour point set of the anti-interference marked area image, obtain the minimum bounding rectangle of the contour through the minimum bounding rectangle algorithm, and calculate the initial pose of the object to be suspended through the minimum bounding rectangle; A moment basis edge operator and an ellipse fitting algorithm are used to perform sub-pixel center localization of the suspended object; Based on the initial pose and sub-pixel center localization of the object to be suspended, the object coordinates in the camera's own coordinate system are converted into the base coordinate system for the motion control of the cantilever crane, thus completing the backtracking of the original position and obtaining a precise pose estimate of the object to be suspended.

6. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 5, characterized in that, The step of extracting the contour point set of the anti-interference marked region image, obtaining the minimum bounding rectangle of the contour using the minimum bounding rectangle algorithm, and calculating the initial pose of the object to be suspended using the minimum bounding rectangle is as follows: Scan each column of the smallest bounding rectangle region, calculate the start and end coordinates of the target in each column, and solve for the column centroid; Based on the set of contour points, the vertices of the minimum enclosing rectangle are calculated by least-squares fitting of the horizontal and vertical principal axes. Compare the lengths of adjacent sides of the rectangle, take the smaller value as the minor axis, output the center, width, height of the rectangle and the angle between the minor side and the horizontal axis of the image, and calculate the pose angle accordingly to obtain the initial pose of the object to be suspended. The method for sub-pixel center localization of suspended objects is as follows: Using the center of the item marker as the origin, a 5×5 window is used to calculate the two-dimensional spatial moments, and the edge parameters are corrected to obtain sub-pixel level edge points. Least-squares elliptic fitting is used for the edge points: Solving for the sub-pixel center of the item marker .

7. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 6, characterized in that, The step of converting the object's coordinates in the camera's own coordinate system to the base coordinate system of the cantilever crane's motion control based on the initial pose and sub-pixel center localization of the object to be suspended, thus completing the backtracking of the original position and obtaining the accurate pose estimation of the object to be suspended, involves obtaining the object's coordinates in the camera coordinate system. Convert to cantilever base coordinates The formula is as follows: Among them, the transformation matrix from the camera coordinate system to the cantilever base coordinate system , where R is the rotation matrix and t is the translation vector; The coordinates of the object in the coordinate system of the cantilever crane base. The coordinates of the object in the camera coordinate system.

8. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 7, characterized in that, The process involves accurately estimating the pose of the object to be suspended and the linear motion characteristics of the cantilever crane. Combining the original lifting position of the object with the 3D structure and dimensions of the obstacle, a segmented path is planned for lifting, translation, and lowering. This process enables path planning and obstacle avoidance from the original lifting position to the target placement position. In this step, the path planning must meet the mechanical constraints of the cantilever crane. The lifting segment of the segmented path must ensure that the workpiece is vertically removed from the obstacle by a safe height of not less than 0.2m. The translation segment must be within the horizontal stroke range of the boom. The lowering segment must be precisely aligned with the target position.

9. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 8, characterized in that, In the step of driving the cantilever crane to grab and transport items along a planned segmented path to the target placement location through motion control and visual verification, and smoothly returning the items from the end of the path to the original position, the motion control method is as follows: The deviation between the target placement position and the actual position is obtained and verified using the following formula: in, For the target location, For actual location, , These are the first control parameter and the second control parameter, respectively.

10. The method for automatic retraction and obstacle avoidance of a cantilever crane based on image processing according to claim 1, characterized in that, It includes an edge computing terminal deployed locally on the cantilever crane and a remote cloud platform, used to perform real-time image processing and anomaly response, as well as receive data uploaded by the edge computing terminal and perform incremental learning optimization on the recognition and localization model.

Images