Crane grabbing operation method and equipment based on assistance of unmanned aerial vehicle and medium

Abstract

The invention discloses a crane grabbing operation method and device based on unmanned aerial vehicle assistance and a medium, and belongs to the technical field of engineering machine.The method comprises the steps that two preset target positions corresponding to a container lock hole and a container truck hole respectively are sent to an unmanned aerial vehicle according to the size and position of a container to be grabbed, two pictures including a container lock hole and a container truck hole are obtained; image recognition is carried out on the two pictures, and recognition results are fed back to a cab; guiding the unmanned aerial vehicle to reach the observation position of the sling; the unmanned aerial vehicle is controlled to move along with the lifting appliance, and environment recognition is conducted on the lifting appliance moving path through a laser radar in the moving process; and an environment recognition result is fed back to a cab so as to assist a driver to complete a container grabbing task through the crane sling. The unmanned aerial vehicle can assist the crane in grabbing the container without arranging too many cameras, so that the observation intensity of a driver is reduced, and the operation efficiency and the operation safety are improved.

Description

Technical Field

[0001] This invention belongs to the field of engineering machinery technology, specifically relating to a crane grabbing operation method, equipment, and medium based on unmanned aerial vehicle (UAV) assistance. Background Technology

[0002] Conventional port loading and unloading equipment typically employs multi-point camera installation for spreader positioning and safety monitoring. Specifically, multiple cameras and laser positioning devices are placed near the spreader, under the gantry beam, and other key locations. Video feeds assist the driver in observing the relative position of the spreader and container, the activities of surrounding personnel, and the anti-lifting status of the truck, thus enabling the successful handling of the load.

[0003] However, this solution has significant shortcomings: First, the fixed camera positions necessitate increasing the density of camera deployment to achieve comprehensive, blind-spot-free coverage, leading to a significant increase in hardware costs. Second, the large number of camera feeds displayed on the driver's cab screen, coupled with limited screen size, not only easily results in missing information or excessively small displays but also makes it difficult for the driver to quickly and accurately obtain key information, increasing the complexity of observation and judgment. Furthermore, the frequent switching and differentiation of multiple feeds severely distracts the driver, directly impacting operational efficiency and safety, and increasing the difficulty of operation. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a method, equipment, and medium for crane grabbing operations based on drone assistance. It enables cranes to grab containers with the help of drones, eliminating the need for excessive cameras, thereby reducing the driver's observation workload and improving operational efficiency and safety.

[0005] This invention provides the following technical solution:

[0006] Firstly, a method for crane grabbing operations based on drone assistance is provided, which utilizes a drone equipped with lidar and a first camera to assist a crane spreader in completing container grabbing tasks. The method includes: Step S1: Based on the size and location of the container to be grabbed, send two preset target locations corresponding to the container lock hole and the truck hole to the drone respectively, and after the drone reaches the preset target location, take pictures through the first camera to obtain a first picture containing the container lock hole and a second picture containing the truck hole. Step S2: Perform image recognition on the first and second screens respectively, and send the recognition results back to the cab to start the crane's grabbing operation; Step S3: Obtain the real-time geographic coordinates of the spreader, guide the drone to the observation position of the spreader, and lock the spreader as the tracking target; Step S4: Collect the command information of the operating handle during the crane grabbing operation and the information of the lifting device fed back by the encoder of the lifting device drive motor, control the drone to follow the movement of the lifting device, and use LiDAR to identify the environment of the lifting device movement path during the movement to determine whether there is a risk of collision. Step S5: Feed back the collision risk assessment results and environmental identification results to the cab to assist the driver in completing the container grabbing task using the crane spreader.

[0007] Optionally, step S1 specifically includes: Step S1.1: Obtain the geographic coordinates, dimensions, and orientation of the container to be grabbed; Step S1.2: Calculate the two target positions based on the dimensions of the container to be grabbed, the preset relative positions of the lock hole and the truck hole; Step S1.3: Send the two target locations to the drone via a wireless link. After the drone reaches the first target location, it takes a first picture of the container lock hole. After taking the first picture, the drone reaches the second target location and takes a second picture of the container lock hole.

[0008] Optionally, step S2 specifically includes: Step S2.1: Preprocess the captured first and second images; Step S2.2: Use a deep learning model to identify the keyhole in the first screen and the container door in the second screen, and determine whether the keyhole or the container door is in a lifting-ready state. Step S2.3: Transmit the recognition result to the display screen in the cab and prompt the driver in a set manner so that the driver can respond to the operating handle to perform the crane's grabbing operation or process the status of the container lock hole and truck lock hole.

[0009] Optionally, step S3 specifically includes: Step S3.1: Obtain the real-time geographic coordinates of the spreading gear; Step S3.2: Calculate the observation position of the UAV based on the position of the lifting device and the preset safe distance; Step S3.3: The UAV automatically navigates to the observation position and uses the preset feature mark of the spreader to lock the spreader as the tracking target.

[0010] Optionally, step S3 further includes: Step S3.4: After the drone reaches the observation position, use the second camera located in front of the drone to capture the front view of the drone and transmit the captured image back to the cockpit.

[0011] Optionally, step S4 specifically includes: Step S4.1: Obtain the gear position information of the crane operating handle, as well as the speed and direction of the spreader fed back by the encoder of the spreader drive motor, and predict the short-term motion trajectory of the spreader; Step S4.2: Based on the predicted short-term motion trajectory of the spreader and the real-time deviation of the tracking by the first camera, adjust the speed and direction of the drone to keep the relative position of the drone and the spreader fixed. Step S4.3: Use LiDAR to monitor the surrounding environment and generate real-time point cloud data of the environment; use clustering algorithm to extract features from the point cloud data to identify obstacles, compare the obstacle information with the predicted motion path of the lifting device, and determine whether there is a risk of collision.

[0012] Optionally, step S4.3 specifically includes: Step S4.3.1: Preprocess the point cloud data; Step S4.3.2: Separate the ground point cloud and non-ground point cloud using the RANSAC algorithm, cluster the non-ground point cloud to form obstacles, and output the movement speed, center position and bounding box size of the obstacles; Step S4.3.3: Track obstacles and predict their trajectories using a multi-target tracking algorithm; Step S4.3.4: Based on the dimensions of the spreader and the container and the predicted short-term trajectory of the spreader, generate the three-dimensional spatial envelope of the spreader in the future time window, and determine whether the predicted position of the obstacle has a spatial intersection with the three-dimensional spatial envelope of the spreader. If so, there is a risk of collision. Step S4.3.5: Based on the depth of the obstacle intruding into the three-dimensional spatial envelope of the lifting device and the relative speed between the obstacle and the lifting device, output the collision risk into preset levels.

[0013] In a second aspect, a computer device is provided, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the unmanned aerial vehicle-assisted crane grasping operation method described in any one of the first aspects.

[0014] Thirdly, a computer-readable storage medium is provided for storing a computer program; when executed by a processor, the computer program implements the steps of the unmanned aerial vehicle-assisted crane grasping operation method described in any one of the first aspects.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves dynamic and intelligent operational assistance by configuring a drone equipped with LiDAR and a high-definition camera. First, during the preparation phase, the drone can autonomously fly to the position directly opposite the container lock hole and truck port according to instructions to take detailed pictures. It then automatically analyzes the status of the lock hole and truck port through image recognition, providing intuitive feedback to the driver, greatly reducing the difficulty of observation. Second, during the movement of the spreader, the drone can follow in real time and use LiDAR to scan the operating path in advance, automatically identifying the size and distance of obstacles and providing risk warnings. This invention transforms the passive observation of the human eye in existing technologies into an active perception method, not only eliminating blind spots in monitoring but also reducing the cognitive load on the driver through information fusion. It provides the driver with intuitive and timely decision support, significantly improving the safety, efficiency, and intelligence level of container loading and unloading operations. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps of the crane grasping operation method based on UAV assistance of the present invention; Figure 2 This is a schematic diagram of the crane grasping operation method based on unmanned aerial vehicle (UAV) assistance of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variations thereof in the specification, claims and the above-mentioned drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0018] Example 1: like Figure 1 and Figure 2 As shown, a method for crane grasping operations based on drone assistance is provided, which specifically includes the following steps: Step S1: Based on the size and location of the container to be grabbed, send two preset target locations corresponding to the container lock hole and the truck hole to the drone respectively, and after the drone reaches the preset target location, take pictures through the first camera to obtain a first picture containing the container lock hole and a second picture containing the truck hole. Step S2: Perform image recognition on the first and second screens respectively, and send the recognition results back to the cab to start the crane's grabbing operation; Step S3: Obtain the real-time geographic coordinates of the spreader, guide the drone to the observation position of the spreader, and lock the spreader as the tracking target; Step S4: Collect the command information of the operating handle during the crane grabbing operation and the information of the lifting device fed back by the encoder of the lifting device drive motor, control the drone to follow the movement of the lifting device, and use LiDAR to identify the environment of the lifting device movement path during the movement to determine whether there is a risk of collision. Step S5: Feed back the collision risk assessment results and environmental identification results to the cab to assist the driver in completing the container grabbing task using the crane spreader.

[0019] In this embodiment, step S1 obtains the precise geographic coordinates (such as RTK-GPS data), dimensions (20 / 40 feet, etc.), and orientation of the container to be grabbed. Specifically, this information can be obtained through manual input or by reading from the work plan.

[0020] Based on the container dimensions and the preset relative positions of the lock hole / truck opening, the three-dimensional coordinates of two preset target positions, including altitude and horizontal offset, are calculated. These two target positions are then transmitted to the drone via a wireless link, with the task sequence set (first target position, then second target position). The first target position is directly opposite the container lock hole, and the second target position is directly opposite the truck opening. The drone takes off and navigates to the first target position using its own perspective. It adjusts the gimbal angle of its first camera, aligning it directly with the container lock hole area. At this point, the drone captures high-definition video or images of the lock hole according to a preset trajectory (such as hovering or small-range circling), ensuring a clear and unobstructed image. After completing the first image capture, the drone automatically flies to the second target position and similarly adjusts its attitude to capture images of the truck opening area.

[0021] In this embodiment, step S2 specifically includes: Step S2.1: Preprocess the first and second images acquired, including noise reduction, distortion correction, and illumination equalization, to improve recognition accuracy.

[0022] Step S2.2: Use a deep learning model to identify the keyhole in the first screen and the container gate in the second screen, and determine whether the keyhole or the container gate is in a lifting-ready state.

[0023] The system determines whether the lock hole or the truck opening is in a lifting-ready state. Specifically, it checks whether the lock hole is closed and whether the truck opening is unlocked and detached from the container. If the container lock hole is closed, it is in a lifting-ready state; if the truck opening is unlocked, it is in a lifting-ready state. The deep learning model can utilize existing technologies, such as the YOLO series algorithms or convolutional neural networks.

[0024] Step S2.3: Transmit the recognition results to the driver's cab.

[0025] The identified information (such as keyhole coordinates and abnormal markings) is superimposed on the original image of the first camera and a brief text prompt is generated, such as "Left keyhole not aligned" or "Truck hole disengaged". Of course, it can also be indicated directly by status lights (green when lifting is possible, red otherwise). That is, if the four keyholes are not locked, the driver is reminded to lock them; if the four truck holes are not unlocked, the driver is reminded to unlock the truck.

[0026] In this embodiment, step S3 specifically includes the following sub-steps: Step S3.1: Obtain the real-time geographic coordinates of the spreading device.

[0027] Geographic coordinates obtained from the RTK label on the spreader or by clicking the encoder.

[0028] Step S3.2: Calculate the observation position of the UAV based on the position of the lifting device and the preset safe distance.

[0029] Based on the preset safe distance, i.e. the preset relative position, the coordinates of the observation position that the drone should reach are calculated. The preset relative position is usually that the drone is 2 meters directly above the hoist and the horizontal distance between the drone and the hoist is 2 meters.

[0030] Step S3.3: The UAV automatically navigates to the observation position and uses the characteristic markings of the spreader to lock onto the spreader as the tracking target.

[0031] The drone automatically flies from its current location to the calculated observation point, adjusting its flight attitude to ensure the first camera remains aligned with the center of the spreader. Once at the observation position, the drone automatically searches for preset feature markers on the spreader (e.g., colored markings, crosses, or other patterns composed of several feature points). Upon successful identification, it locks onto the feature marker, ensuring it remains centered in the drone's first camera's view. It's important to note that during the drone's tracking of the spreader, the tracking confidence level needs to be monitored in real-time, such as the number of feature points on the marker. If tracking is lost, a re-search is automatically triggered.

[0032] Step S3.4: After the drone reaches the observation position, use the second camera located in front of the drone to capture the front view of the drone and transmit the captured image back to the cockpit.

[0033] The second camera's view is designed to provide the driver with an intuitive first-person perspective, making it easier for the driver to operate the handle to perform the grasping operation. Typically, the second camera is a wide-angle camera.

[0034] In this embodiment, step S4 specifically includes the following sub-steps: Step S4.1: Obtain the gear position information of the crane operating handle, as well as the speed and direction of the spreader fed back by the encoder of the spreader drive motor, and predict the short-term motion trajectory of the spreader.

[0035] Specifically, the current gear and direction signals, such as hoisting / lowering, crane trolley / gantry movement direction, and speed gear, are read from the PLC of the crane's operating handle. Feedback data from the encoder of the spreader drive motor includes: hoisting motor speed / position, which is used to calculate the spreader height change; and trolley / gantry drive motor speed / position, which is used to calculate the horizontal movement speed and direction.

[0036] The lever position information and the spreader information are fused, and noise is removed using algorithms such as Kalman filtering to obtain the current speed and acceleration of the spreader. Based on this, trajectory prediction is performed using existing trajectory prediction algorithms, including constant speed / acceleration models and Kalman prediction models.

[0037] Step S4.2: Based on the predicted short-term motion trajectory of the spreader and the real-time deviation tracked by the first camera, adjust the speed and direction of the drone to keep the relative position of the drone and the spreader fixed.

[0038] The predicted position of the spreader is added to a preset safety distance to obtain the expected position that the drone should reach. Based on the image captured by the first camera, the pixel deviation of the feature marker of the spreader in the image is obtained and converted into coordinates to obtain the actual spatial deviation of the drone. Using a PID algorithm, the speed command of the drone on the X / Y / Z axes is calculated according to the expected position and the actual spatial deviation of the drone.

[0039] Step S4.3: Use LiDAR to monitor the surrounding environment and generate real-time point cloud data of the environment; use clustering algorithm to extract features from the point cloud data to identify obstacles, compare the obstacle information with the predicted motion path of the lifting device, and determine whether there is a risk of collision.

[0040] LiDAR rotates or scans in a fixed frequency (such as 10Hz or 20Hz), emitting lasers and receiving reflected signals, generating one frame of raw environmental point cloud data per scanning cycle.

[0041] In this embodiment, step S4.3 includes: Step S4.3.1: Preprocess the point cloud data.

[0042] Preprocessing includes denoising, distortion correction, and filtering downsampling, and the preprocessed point cloud data is transformed from LiDAR coordinates to the global coordinate system.

[0043] Step S4.3.2: Separate the ground point cloud and non-ground point cloud using the RANSAC algorithm, cluster the non-ground point cloud to form obstacles, and output the movement speed, center position and bounding box size of the obstacles.

[0044] Clustering can use existing clustering algorithms, such as density-based clustering algorithms, to group points belonging to the same object into a cluster, forming obstacles.

[0045] Step S4.3.3: Track obstacles and predict their trajectories using a multi-target tracking algorithm.

[0046] For multi-target tracking algorithms such as Kalman filtering, please refer to existing technologies.

[0047] Step S4.3.4: Based on the dimensions of the spreader and the container and the predicted short-term trajectory of the spreader, generate the three-dimensional spatial envelope of the spreader in the future time window, and determine whether the predicted position of the obstacle has a spatial intersection with the three-dimensional spatial envelope of the spreader. If so, there is a risk of collision.

[0048] When generating a three-dimensional spatial envelope, it is necessary to increase the safety range for the swing or rotation of the lifting device.

[0049] Step S4.3.5: Based on the depth of the obstacle's intrusion into the three-dimensional spatial envelope of the lifting device and the relative speed between the obstacle and the lifting device, the collision risk is divided into preset levels.

[0050] Collision risk levels can be categorized into no risk, warning risk, alert risk, and emergency braking risk.

[0051] The drone and crane control system are connected via wireless communication to ensure low-latency, highly reliable data transmission.

[0052] Step S5: Feed back the collision risk assessment results and environmental identification results to the cab to assist the driver in completing the container grabbing task using the crane spreader.

[0053] In this embodiment, the footage captured by the second camera also needs to be synchronously fed back to the cab. The footage captured by the second camera, the position and size of the obstacles can usually be fused using existing fusion software to generate an environmental situation map, which can better assist the driver in completing the container grabbing task using the crane spreader. In this embodiment, the footage of the drone following the spreader using the first camera can be not displayed directly, or displayed at the edge of the display screen, so as to avoid obstructing the footage of the second camera and the recognition results of obstacles.

[0054] Example 2: The present invention provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the above-described method for crane grasping operation based on unmanned aerial vehicle (UAV) assistance.

[0055] For a more detailed explanation of the above method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0056] Example 3: The present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the above-described method for crane grasping operations based on unmanned aerial vehicle (UAV) assistance.

[0057] For a more detailed explanation of the above method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0058] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the devices and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section. Those skilled in the art will clearly understand that the technologies in the embodiments of this invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this invention, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments of this invention.

[0059] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A crane grasping operation method based on unmanned aerial vehicle (UAV) assistance, characterized in that, The method of using a drone equipped with lidar and a first camera to assist a crane spreader in completing a container grabbing task includes: Step S1: Based on the size and location of the container to be grabbed, send two preset target locations corresponding to the container lock hole and the truck hole to the drone respectively, and after the drone reaches the preset target location, take pictures through the first camera to obtain a first picture containing the container lock hole and a second picture containing the truck hole. Step S2: Perform image recognition on the first and second screens respectively, and send the recognition results back to the cab to start the crane's grabbing operation; Step S3: Obtain the real-time geographic coordinates of the spreader, guide the drone to the observation position of the spreader, and lock the spreader as the tracking target; Step S4: Collect the command information of the operating handle during the crane grabbing operation and the information of the lifting device fed back by the encoder of the lifting device drive motor, control the drone to follow the movement of the lifting device, and use LiDAR to identify the environment of the lifting device movement path during the movement to determine whether there is a risk of collision. Step S5: Feed back the collision risk assessment results and environmental identification results to the cab to assist the driver in completing the container grabbing task using the crane spreader.

2. The method for crane grasping operation based on UAV assistance according to claim 1, characterized in that, Step S1 specifically includes: Step S1.1: Obtain the geographic coordinates, dimensions, and orientation of the container to be grabbed; Step S1.2: Calculate the two target positions based on the dimensions of the container to be grabbed, the preset relative positions of the lock hole and the truck hole; Step S1.3: Send the two target locations to the drone via a wireless link. After the drone reaches the first target location, it takes a first picture of the container lock hole. After taking the first picture, the drone reaches the second target location and takes a second picture of the container lock hole.

3. The method for crane grasping operation based on UAV assistance according to claim 1, characterized in that, Step S2 specifically includes: Step S2.1: Preprocess the captured first and second images; Step S2.2: Use a deep learning model to identify the keyhole in the first screen and the container door in the second screen, and determine whether the keyhole or the container door is in a lifting-ready state. Step S2.3: Transmit the recognition result to the display screen in the cab and prompt the driver in a set manner so that the driver can respond to the operating handle to perform the crane's grabbing operation or process the status of the container lock hole and truck lock hole.

4. The crane grasping operation method based on UAV assistance according to claim 1, characterized in that, Step S3 specifically includes: Step S3.1: Obtain the real-time geographic coordinates of the spreading gear; Step S3.2: Calculate the observation position of the UAV based on the position of the lifting device and the preset safe distance; Step S3.3: The UAV automatically navigates to the observation position and uses the preset feature mark of the spreader to lock the spreader as the tracking target.

5. The method for crane grasping operation based on UAV assistance according to claim 1, characterized in that, Step S3 further includes: Step S3.4: After the drone reaches the observation position, use the second camera located in front of the drone to capture the front view of the drone and transmit the captured image back to the cockpit.

6. The method for crane grasping operation based on UAV assistance according to claim 1, characterized in that, Step S4 specifically includes: Step S4.1: Obtain the gear position information of the crane operating handle, as well as the speed and direction of the spreader fed back by the encoder of the spreader drive motor, and predict the short-term motion trajectory of the spreader; Step S4.2: Based on the predicted short-term motion trajectory of the spreader and the real-time deviation of the tracking by the first camera, adjust the speed and direction of the drone to keep the relative position of the drone and the spreader fixed. Step S4.3: Use LiDAR to monitor the surrounding environment and generate real-time point cloud data of the environment; use clustering algorithm to extract features from the point cloud data to identify obstacles, compare the obstacle information with the predicted motion path of the lifting device, and determine whether there is a risk of collision.

7. The method for crane grasping operation based on UAV assistance according to claim 6, characterized in that, Step S4.3 specifically includes: Step S4.3.1: Preprocess the point cloud data; Step S4.3.2: Separate the ground point cloud and non-ground point cloud using the RANSAC algorithm, cluster the non-ground point cloud to form obstacles, and output the movement speed, center position and bounding box size of the obstacles; Step S4.3.3: Track obstacles and predict their trajectories using a multi-target tracking algorithm; Step S4.3.4: Based on the dimensions of the spreader and the container and the predicted short-term trajectory of the spreader, generate the three-dimensional spatial envelope of the spreader in the future time window, and determine whether the predicted position of the obstacle has a spatial intersection with the three-dimensional spatial envelope of the spreader. If so, there is a risk of collision. Step S4.3.5: Based on the depth of the obstacle intruding into the three-dimensional spatial envelope of the lifting device and the relative speed between the obstacle and the lifting device, output the collision risk into preset levels.

8. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the unmanned aerial vehicle-assisted crane grasping operation method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; when executed by a processor, the computer programs implement the steps of the unmanned aerial vehicle-assisted crane grasping operation method according to any one of claims 1-7.

Images