Method for autonomous operation of a crane, autonomously operated crane and retrofit kit for producing an autonomously working crane

Abstract

The invention enables fully automatic and autonomous crane control, on the basis of an optical sensor system (5) that is evaluated by artificial intelligence. A control system (4) of the crane (1), which is based on artificial intelligence, thereby takes into account, in particular in addition to passively determined state information of the crane (1), measurement data which are actively acquired by means of the optical sensor system (5). From this, the control system (4) can determine precise information concerning a current position of a crane gripper (6), but also concerning objects (7) to be picked up with the gripper (6). This open-loop and closed-loop approach also makes it possible to position the gripper (6) without oscillation, as well as to avoid collisions between the gripper (6) and stationary objects, the position of which may change depending on the situation, or also to avoid an unwanted rotational oscillation of the gripper (6).

Description

[0001] The invention relates to the technical field of cranes for lifting loads, more precisely to the field of controlled cranes that can autonomously perform tasks. The invention builds upon insights from the field of computer vision, i.e., the computer-implemented recognition of people and objects based on visual information from images or videos, sometimes also described as machine vision. Furthermore, the invention is based on the application of artificial intelligence (AI).

[0002] In detail, the invention relates to a method for the autonomous control / operation of a crane and to an associated crane that can be operated autonomously using this method. Furthermore, the invention relates to a retrofit kit with which an intelligent control system according to the invention can be easily retrofitted to an existing crane. With this latter approach, a conventional crane, which lacks its own intelligence and can only be controlled by a human operator via a manual control unit, can be transformed into an intelligent crane as defined by the invention, enabling the crane to perform tasks autonomously after retrofitting.

[0003] A crane, in this context, can be understood as a mobile or stationary device with a movable boom and a lifting mechanism that can lift and move loads or bulky objects, such as tree trunks. These cranes are used on construction sites, but primarily for loading and unloading trucks, ships, or freight cars. The lifting mechanism is often implemented using a cable system (frequently employing block and tackle), whereby bulky goods like tree trunks can be gripped and lifted with a grab attached to the lower end of the cable. The crane is often powered by motors or hydraulics.Grippers are also known that can be rotated around a rotational axis relative to the boom, so that the gripper can grasp objects in different orientations.

[0004] In the prior art, for example, non-intelligent cranes based on a lightweight, scaffold-like support structure with a driver's cab are known. In these cranes, a human operator controls the boom and, if applicable, the grab, either from the cab or via remote control, monitoring and controlling the lifting and moving of loads using the crane's lifting mechanism. For instance, in the timber loading area of ​​pulp mills, the 40-meter-high cranes used, which move loads of 30 tons, are still largely manually operated. Apart from simple anti-sway systems and remote controls, no extensive automation is available. This is also due to the sensory uncertainties associated with the irregularly shaped material "logs," the highly variable delivery trucks, and the often unstructured storage and production processes.

[0005] However, there is a growing desire to support or even completely replace human operators with intelligent control systems that enable the crane to perform a specific task without external human intervention, such as unloading cargo from a train container or unloading logs from a truck bed. Depending on the application, for example, if a risk to people in the crane's working area can be ruled out, the goal may even be to allow the crane's autonomous and intelligent control to operate without any human supervision, thereby requiring a correspondingly high level of autonomy (e.g., Level 3 or Level 4 autonomy).

[0006] The following publications provide important technical background information regarding autonomously operating cranes: "KAI - The German crane AI | PSIORI", August 13, 2022, pages 1-10, XP093176612, (URL: https: / / web.archive.org / web / 202208131 12218 / https: / / crane.psiori.com / ); " PSIORI - AI CRANE - LONG - V3", Intelligentmobiles, March 7, 2022, XP93176820, (URL: https: / / vimeo.com / 685385572); " [bauma 2022] PSIORI - Autonomous Woodyard crane", November 20, 2022, pages 1-7, XP93176822, (URL: https: / / usa.worldtradeshow.tv / contents / 43136 / ); and " [bauma 2022 Munich] PSIORI - Autonomous Woodyard crane", World Trade Show TV, Germany, November 4, 2022, XP93176827, URL: https: / / www.youtube.com / watch?v=aDIkwkPd LKA&t=31s.

[0007] Further approaches for autonomously operating cranes are described in CN 110 271 965 B and in WO 2021 / 062314 A1. Furthermore, WO 2022 / 221311 A1 describes an intelligent assistance system that can support a human crane operator.

[0008] Against this background, the invention aims to provide an intelligent crane control system that enables autonomous crane operation, thereby increasing operational efficiency and safety compared to previously known approaches. Through digitalization and the implementation of processes and procedures, human error and mistakes are reduced or completely eliminated. The intelligent control system and the resulting increase in safety are intended to support a human operator monitoring the crane's operation. In application situations where safety requirements permit, the ultimate goal is for the control system to even eliminate the need for human monitoring of the crane's autonomous operation.

[0009] To solve this problem, the features of claim 1 are provided according to the invention in a method for controlling a crane. In particular, it is thus proposed according to the invention to solve the problem in a method for the autonomous operation of a crane that the control system is based on artificial intelligence (AI) and that the AI ​​processes real-time measurement data from an optical sensor system attached to the crane and converts it into corresponding control signals with which the crane is controlled autonomously, i.e., without human intervention.

[0010] The AI ​​can be implemented using machine learning methods, particularly based on deep learning and / or a neural network. This allows uncertainties regarding the task to be solved by the crane and its environment to be handled flexibly and robustly, which is particularly advantageous for slewing and gantry cranes.

[0011] Loading and unloading operations can thus be handled autonomously by the crane, without the need for the vehicle being loaded / unloaded, the load itself, or the storage location to have known dimensions, markings, or other modifications. The achievable speed of crane control is comparable to that of a human crane operator, although the autonomous control system delivers more consistent results than a human operator.

[0012] The control system can be designed to access the crane's drives and sensors either directly or indirectly via a (possibly already existing) crane control system, in a controlling and / or reading capacity.

[0013] The artificial intelligence-based control system according to the invention can, for example, be used to autonomously control a crane that loads logs in a paper mill without requiring human intervention. However, to increase safety, operation can still be monitored by a person who, if necessary, intervenes in the crane's operation via a control unit (joystick, emergency button, touch display, etc.).

[0014] For example, when loading a waste incineration plant with a loading crane within an area inaccessible to people from the outside (for example, blocked off by appropriate gates), the control system can even be used to autonomously control the loading crane of the waste incineration plant within the protected area, without any human monitoring.

[0015] Preferably, components of the optical sensor system, in particular a (first or) primary camera (e.g., based on a 2D image sensor), are mounted on the movable boom of the crane, so that at least part of the sensor system moves in space together with the boom. The cameras used in the sensor system may also optionally have zoom optics, so that a specific optical magnification can be set by the control system as needed.

[0016] The cognitive architecture of the control system can combine a belief-desire-intention (BDI) architecture with a reactive approach (behavior-based architecture).

[0017] Furthermore, the control system can be designed to implement crane control, which is structured as a finite automaton (simplified as the "core"). Here, individual states of the automaton can correspond to specific typical process steps that the crane is intended to execute in the respective application (e.g., assuming a waiting position; approaching the truck's loading platform; unloading the truck; etc.). The transitions between the automaton's states can be implemented rule-based (especially based on predicate logic rules) and can optionally be dynamically supplemented and / or modified by the control system. For this purpose, the control system can access information from a predictive world model, in which knowledge ("belief") about the crane's own state and its environment is collected, filtered, integrated, and propagated (predicted) in a coherent form. This world model is explained in more detail below.External information (e.g., provided by a person or an external system) can also be processed by the control system as additional "external requirements".

[0018] The crane control system, which implements the intelligent control system, can be structured so that each process state of the state machine pursues a specific, defined goal. Within the control system, the respective goal can then be subdivided into intermediate goals (e.g., goal: "Unload"; intermediate goals: "Approach load with gripper"; "Open gripper without collision"; "Securely enclose wood with gripper"; "Lift load"). When determining the path, i.e., defining a route for the crane to follow, and when subdividing this path into intermediate goals, the AI ​​of the control system can also independently define intermediate goals (waypoints). Once these intermediate goals have been reached, they can be deleted.

[0019] The autonomous behavior of the crane during a process step can be determined by a hierarchical, behavior-based architecture. In the intelligent control system, the desired behavior of the crane in each process step can be nested into a complex strategy using, for example, just two arbitration methods (priority list and sequence), which the crane then executes sequentially. The control system can select the current individual behavior (intention) by considering a current objective, a specific state in the world model, preconditions, and boundary conditions. Based on the selected behavior, the control system then calculates the control signals (corresponding to specific control commands) and uses them to control the crane, particularly by transmitting them to the crane's control unit.The control system continuously and sequentially generates control signals or specific control commands to process the intermediate steps and thus achieve one intermediate goal after another. In a further development, it is possible for the control system to create and / or modify (especially depending on the situation) sub-strategies that are pursued when processing the intermediate goals.

[0020] The aforementioned reactive approach, on the other hand, can be used during a process step to adapt the crane behavior to the situation, for example to avoid collisions or vibrations, as will be explained in more detail later.

[0021] According to the invention, the problem can also be solved by further advantageous embodiments according to the dependent claims, which are described below: As mentioned, the crane can, for example, have a gripper mounted on a movable boom of the crane. In this case, the method can be provided that an object to be picked up, which is located within a picking area, is grasped with the gripper and then moved by the crane into a deposit area and placed there (with the aid of the gripper). The gripper can be suspended from the movable boom and thus be capable of oscillation, for example by means of at least one cable pulley.In such configurations, the distance between the gripper and the boom, particularly the vertical distance, can be changed using the control system, for example, by controlling the drive of the cable pull in such a way that a desired distance between the gripper and the boom is set. This allows the control system not only to control the gripping of loads with the gripper, but also to influence the pendulum characteristics of the gripper by actively controlling the cable pull, i.e., by changing the cable length.

[0022] It goes without saying that this method can be used for both loading and unloading goods. In the case of loading a truck, the storage location of the objects would be the receiving area and the truck's loading platform the unloading area; in the case of unloading, however, it could be the other way around.

[0023] As will be explained in more detail below, a control system according to the invention, including its associated optical sensor system, can be designed, in particular, as a retrofit kit that can be easily retrofitted to an existing crane that has its own non-intelligent (i.e., without artificial intelligence) crane control system, without requiring major modifications to the existing crane control system. For this purpose, it can be provided that the intelligent control system controls the crane indirectly through the separate crane control system, taking into account the control behavior and / or time latency of the crane control system. This consideration can, in particular, consist of the control system modeling the crane control system and adapting the control signals transmitted to the crane control system based on this model.If, for example, the crane control is designed as a programmable logic controller (which can implement a speed controller, for example), it may be sufficient for the intelligent control system to model the crane control using a constrained acceleration model or a PT1 model.

[0024] An advantage of this approach is that the existing crane control system can be retained, which significantly simplifies the retrofitting of the intelligent control system. A preferred embodiment provides that the control system transmits the control signals generated by the AI ​​to the crane control system via a user interface designed for use with a manual operating unit. For example, the control signals can be transmitted via a joystick interface already present on the crane, through which control signals are normally input using a joystick. It is particularly preferred that an existing manual operating unit is retained, allowing a crane operator to continue to access and control the crane's control system via the manual unit.This approach allows the operator to monitor the autonomous operation of the crane, but still intervene manually via the control unit if necessary. Preferably, the crane's control system can be designed so that control signals entered via the control unit take precedence over control signals generated by the intelligent control system. This allows the human operator to override the control system at any time, thus increasing safety.

[0025] To increase the speed of autonomous crane control, one implementation of the method proposes that, during crane positioning maneuvers—that is, when the crane moves to a predetermined position—the intelligent control system should control at least two of the crane's axes separately / independently. This is because certain deviations from the pre-planned "Cartesian" path are acceptable during such rough positioning maneuvers, as these can be compensated for later by the control system. To increase the accuracy of controlling the crane at the target location (loading or unloading point), it can be additionally or alternatively provided that, when objects are grasped or placed using a crane's grabber, the control system controls at least two of the crane's axes together, i.e., interdependently.In this joint control system, current control values ​​for one of the crane's axes to be controlled are incorporated into the control of at least one other axis of the crane; in separate control, the affected axis is controlled independently of the other axes. An axis to be controlled can be understood here as any joint or other moving part of the crane that can change the crane's movement / kinematics. Examples include a driven slewing ring, a driven trolley, or a driven hoist. Preferably, in all these cases, the control system always (i.e., in every control cycle) controls all axes of the crane simultaneously.This approach often allows the maximum speed of the crane to be achieved along the individual axes, avoids unnecessary corrections compared to an originally planned path, and simultaneously facilitates the development of the individual controllers (for the respective axis) within the control system.

[0026] The control system can preferably include a Gigabit Ethernet network. This allows signals from the optical sensor system to be evaluated in real time and enables control with a latency in the range of a few milliseconds. This network can be directly connected to the crane's existing control system.

[0027] According to a preferred embodiment, the control system can include a world model that contains status information about the crane, in particular current sensor information about the positions and / or movements of the crane's joints. Preferably, the control system acquires all information available in the crane about the joint positions and movements and continuously transmits this information to the world model. Such information can be acquired, for example, by incremental encoders and encoders in the crane.

[0028] The real-time measurement data additionally acquired with the optical sensor system (especially measurement data from at least one camera and one LIDAR sensor) can be used by the control system to close gaps in the accuracy of the sensors already present on the crane and / or to measure additional information about the "pose" (= position and orientation in relation to the spatial axes) of a crane gripper swinging freely on ropes and / or of objects to be picked up (e.g. truck load) and also integrate this information into the world model.

[0029] The control system can also evaluate data from an inertial measurement unit (IMU), for example, a 6-axis position sensor (3 accelerometers and 3 gyroscopes arranged along the three spatial axes), mounted on a moving part of the crane, particularly a crane gripper. This IMU can also include a compass or GPS sensor to compensate for sensor drift and determine the absolute position relative to the environment. This allows for the acquisition of additional status information that can be integrated into the world model and / or considered by the control system when generating control signals.This approach can improve the accuracy of the captured crane kinematics, particularly for cranes that exhibit variable twisting of their supporting structure due to high loads during lifting and / or large lever arms, as is often the case with construction cranes. The IMU can therefore be configured and mounted on the crane in such a way that instantaneous twisting of the crane's supporting structure and / or a sensor mount (which, for example, carries an optical sensor) can be detected by the IMU.

[0030] As mentioned, the world model preferably also includes at least one piece of information about the current pose (technical term for "position and orientation" with respect to three different spatial axes, in particular three Cartesian axes x, y, z; a "pose" thus describes six dimensions, for example, x / y / z position plus the respective rotation around the respective axis) of a crane's grabber and / or an object to be picked up. Put simply, the world model can, for example, know the current "posture" that the grabber is assuming in space.

[0031] To improve accuracy and increase safety, the control system can query information from the world model and incorporate it into the generation of control signals. The world model serves to coherently represent the crane's current state, particularly in the sense of a "digital twin." Based on the world model, the crane's current state can also be visualized for a user using a CAD model. The world model can thus include parameters relating to the crane's current state (kinematics) as well as environmental factors (position and / or movement of obstacles; fixed target areas and approach positions, etc.). Furthermore, the control system can be configured to continuously collect, filter, and, if necessary, predict information in order to continuously update the world model.

[0032] The control system can also employ at least two (but potentially dozens) different coordinate systems (each defining a vector space) to process sensor information (regarding the kinematics of the crane and / or the optical sensor system) and to consider it when generating the control signals. Polar coordinate systems, Cartesian coordinate systems, and so-called joint spaces can be used, the latter being suitable for representing the angles (and angular velocities) of the crane's joints. Furthermore, at least one of these coordinate systems can represent a reference frame for the crane.

[0033] Furthermore, the control system can transform data between at least two different coordinate systems, for example using a transformation tree, in particular by converting information in a first coordinate system into a second coordinate system. For example, state information of the crane, which is typically stored in a joint space, can be converted by the control system into Cartesian position data, which can then be processed more easily by the control system.

[0034] In particular, the described world model can include such a "transformation tree." Using this kinematic method, borrowed from robotics, the relationship between all joints and parts of the crane, as well as the attached sensors, can be described (forward and inversely) and updated according to the currently sensor-detected movements. The transformation tree thus makes it possible to transform coordinates from the reference frame of each sensor and joint of the crane into any other reference frame. For example, measured joint angles can be transformed into a position in Cartesian coordinates, or a target position in Cartesian coordinates can be transformed into a position and movement within the joint space.

[0035] In addition to camera data from the optical sensor system, the control system can also evaluate data from an optical distance sensor, particularly a LiDAR sensor, to create at least one elevation map. Preferably, the control system can be configured to automatically create elevation maps from incoming measurements of the distance sensor (e.g., a 3D point cloud from a single measurement) and to continuously update them. These elevation maps can also be part of the world model. The elevation maps can, in particular, cover an environment that the grabber must traverse, especially in the picking or drop-off area. The control system can use the elevation maps, for example, for collision avoidance, path planning (the path along which the crane should move in space), and for determining a drop-off and pick-up location.The control system can first plan the route to be followed by the crane, taking the elevation map into account, and then, independently of the plan, check whether the grab is currently on a collision course (for example, due to a deviation from the plan, an inaccuracy, or a defect); if this is the case, the control system can adjust the grab's current path and / or slow it down.

[0036] Furthermore, the control system can autonomously recognize the recording area within a live image captured by the optical sensor system. This can be done using optical markers (which is suitable, for example, indoors, but less so outdoors) or without markers.

[0037] The artificial intelligence (AI) can then segment the live image into multiple image areas, which can be delineated by a contour line. Such contours are helpful for efficient control. The AI ​​then assigns a specific, predefined object class to each of these segmented image areas; in particular, the AI ​​can distinguish between objects of interest (loading area, gripper, cargo) and non-interesting objects (items outside the loading area, other parts of the truck).When unloading wood from a truck bed, these object classes can include, for example: the truck bed, the crane's grab, the posts defining the bed's boundaries, the truck's cab, and the logs. Thus, the artificial intelligence can, for instance, recognize and locate the object to be picked up (individual logs or bundles of wood) within one of these image areas, to which, for example, the predefined object class "log stack" has been assigned. Typically, the AI ​​segments 12 objects within the live image. This approach avoids, for example, error-prone edge detection (which can be faulty due to changing lighting conditions, shadows, local occlusions, damage, or varying surface properties and textures), as is often used in other prior art approaches.By limiting the respective image area and robustly identifying the image points (pixels) belonging to the respective object, on the basis of which a previously known object can then be located more precisely and possibly kinematic quantities of the object can be determined, image processing is made significantly easier, faster and more reliable.

[0038] The detection of the recording area within the live image can be achieved, for example, using image recognition. This can also involve the use of stationary markers that define the recording area. For instance, if a truck is to be unloaded with a crane, the truck can first be driven into the recording area defined by several markers and parked there. The control system can then use its optical sensor system to first capture the recording area and then the parked truck within it. The segmentation can then result, for example, in the control system recognizing the truck's loading platform as one image area within the recording area and, within this image area, recognizing and locating tree trunks (as another image area) as objects to be unloaded.The AI ​​is trained to recognize the image area and, through segmentation, precisely define the spatial location of the objects to be recorded. It follows that the AI ​​must therefore recalculate the current position of this image area depending on the situation. The control system can also use other information from the world model, such as the current position of the crane joints and / or the crane kinematics, to define and calculate the respective image areas.

[0039] The segmentation is preferably based on a specific unloading situation within the recording area and / or a specific loading situation within the storage area. Here, the artificial intelligence can perform the segmentation situationally and anew each time using the live image, taking into account typical (image) situations that occur in this specific situation. The AI ​​has learned this knowledge from training image data.

[0040] Unlike previously known systems, this approach does not simply use a consistently identical rectangular search window ("region of interest") with Cartesian coordinates to restrict an area within the live image where the object to be detected is presumably located; instead, it proposes to use intelligent image segmentation based on training data to select a suitable image area within the live image, which can change from situation to situation (for example, the loading area of ​​the truck may vary or the exact parking location of the truck may vary within the recording area), and to identify and locate the object within this image area.Once the object has been located, the control system can then position the gripper precisely above it in the correct orientation using the movable boom and / or the described cable system, and securely grip the object, with the AI ​​also determining the current position and / or orientation of the gripper using image segmentation.

[0041] The detection of the recording area within the captured live image can alternatively (and preferably) be performed without markers, for example, by the control system dynamically placing a corresponding search window in the live image and identifying the recording area using object recognition. The position of the search window can be determined by the control system based on knowledge of the crane's current kinematics (in particular, based on sensor-detected joint angles and / or the crane's geometry). A projection matrix from a (preferably calibrated) camera of the optical sensor system can be used to assist in determining the search window's position. Furthermore, the control system can also consider information from the described world model; for example, the AI ​​can derive the precise position of a road on which the goods to be unloaded are located from the world model as "world coordinates."

[0042] To ensure particularly reliable segmentation, it is preferred that the artificial intelligence performs the segmentation based on parameters it has learned from training image data that has undergone human evaluation. This training image data can comprise only a few thousand images, each recorded in a typical loading or unloading situation within the recording or unloading area. A human then performs the segmentation of each image, and the AI ​​subsequently learns the correct image segmentation based on this analysis, provided that a suitable pre-selection of the training image data (before human evaluation) has taken place. This limits the effort required to train the AI.

[0043] If the crane is to be used in a new area, this type of training can be repeated, allowing the control system to be adapted to the new loading or unloading situation. Furthermore, the artificial intelligence training can also be iterated several times under the same loading or unloading conditions to improve the accuracy and reliability of object recognition / image segmentation and thus the intelligent control.

[0044] The optical sensor system can include a (especially 2D) camera (primary sensor) mounted on the crane (for example, on a trolley of the boom) in such a way that it allows a downward view of the gripper and / or objects to be picked up by the crane. This bird's-eye view makes it particularly easy to observe the gripper's vibration behavior (by optically measuring the gripper's angular deflection) and, if necessary, to actively control it. This allows, for example, the implementation of an optical camera-based and markerless anti-sway control system for the gripper in conjunction with the control system. Furthermore, this approach enables particularly simple and robust control of the gripping of objects to be picked up.

[0045] To increase safety against collisions of the grab with obstacles during loading or unloading, it is preferable to mount an additional camera of the optical sensor system on the crane in such a way that this, in particular a second, camera can capture images of side views of the grab (i.e., especially with a horizontal view of the grab). For example, such a secondary sensor can be mounted on a crossbeam of a crane's undercarriage. The secondary sensor / additional camera allows, in particular, finer control of the crane's undercarriage and the grab's height.

[0046] To enable the fastest possible data acquisition from each optical sensor within the optical sensor system, one design provides that each optical sensor (camera, LiDAR, etc.) has its own processing unit with a direct interface to the sensor, which is configured to process that sensor's signals. This allows for the creation of individual "sensor nodes" that process the sensor signals independently with high bandwidth, for example, within a control cycle of 50 ms (corresponding to 20 control cycles per second). The respective sensor processing unit then simply outputs the results of the sensor signal processing to the control system / into the respective control loop. Depending on the sensor used, the processing units can be equipped with a standard CPU or, for example, a "neural inference unit" (e.g., a 50 ms CPU).B: Jetson from NVIDIA) can be implemented so that sensor data can be processed in a bandwidth range of 1 Gbit / sec per sensor; however, with improved network technology, higher bandwidths are also possible depending on requirements, e.g., 10 Gbit / s or more. This approach particularly allows for easy horizontal scaling of the number of sensors in the optical sensor system, depending on the application.

[0047] According to the invention, the live images from at least one camera of the optical sensor system (i.e., in particular, live images from the described primary and / or secondary sensor) are evaluated by the control system using a multi-layered intelligent image processing chain based on artificial intelligence (in particular, the aforementioned artificial intelligence). This approach allows the required information to be extracted from the images particularly reliably and relatively easily. It is especially advantageous if the respective camera is pre-calibrated. Preferably, both intrinsic and extrinsic camera parameters can be calibrated. Thus, for example, it is advantageous if the control system knows the exact position of the camera on the crane (extrinsic parameter) and / or any optical distortion of the camera's optics (intrinsic parameter).

[0048] The first layer of the image processing chain is envisaged as being formed by an (artificial) neural network (NN), specifically in the form of an "All-Convolutional-Net" or a "U-Net"-like network topology, which classifies image areas of a (current) live image into different object classes. The output of the first layer can therefore be simple 2D information, such as the contour line of the respective image area assigned to a specific object class. This classification can be achieved, in particular, using the segmentation described above. Within an object class (e.g., gripper), the neural network can also perform subclassifications (such as "gripper filled" or "gripper empty").This neural network is not a pre-trained network, but is preferably trained based on specific crane data and training image datasets acquired in the real environment using the optical sensor system; these datasets can also be supplemented with machine data from the crane's control system and / or with AI log data. The individual training steps (data collection; curation; annotation; network training; network evaluation) can be repeated iteratively.

[0049] This intelligent approach enables higher accuracy and robustness against lighting, weather, and environmental conditions, as well as other changes, which is not possible with traditional methods alone. For example, an entire image from a camera mounted on a crane trolley can be segmented by the neural network in a first step. This means that for each pixel, it can be determined to which object class (e.g., gripper, gripper mounting, wood in the gripper, wood load on the truck bed, truck cab, background, etc.) the respective pixel belongs. The result is a segmented image (segmentation image) in which the shape and / or orientation of the classified objects can be determined much more easily and robustly in a subsequent step than in the unprocessed original image.

[0050] A second layer of the image processing chain is designed to locate objects within the segmentation image produced by the first layer—that is, within the respective classified and / or segmented image areas—based on their shape and preferably track them spatially in a continuous sequence of live images (optical tracking). With this approach, objects are identified based on their appearance and thus not on markers. The second layer can therefore perform shape tracking of objects classified in the first layer. Markov Random Fields (MRF) and particle filters, for example, can be used for this purpose. In particular, "classic" symbolic methods from the fields of machine vision and probabilistic robotics can be employed to find and, if necessary, track objects within the segmentation image based on their shape.

[0051] A third layer of the image processing chain is designed to extract further kinematic quantities from the objects localized by the second layer. For example, based on a point mass model of an object (in particular, at least one) classified in a live image and located by its shape, angular velocities and / or the pose of the object can be estimated. Furthermore, this approach also makes it possible to estimate the object's center of mass. Such process steps can be implemented and predicted, for example, using a Kalman filter. With such approaches, the control system can, in particular, perform a dynamic estimation of one of the objects (for example, the gripper currently rotating or oscillating along with its load), as provided for in claim 1. This approach is capable of handling occlusions, superimpositions, and only partially visible objects.Based on an estimated uncertainty, the control system can decide whether an object has been detected with sufficient accuracy. Estimates that are not physically plausible, such as those for sudden, abrupt movements, are rejected by the control system in this case.

[0052] To further enhance safety, the control system can include a safety overseer instance that checks the control signals generated by the artificial intelligence for unsafe crane behavior and overwrites the control signals if at least one safety rule is violated. If such a safety rule is violated, the safety overseer instance can bring the crane into a safe state. This allows a control signal calculated by the control system to be checked for detectable unsafe behavior before being sent to the crane's control unit. For this purpose, a simplex architecture, commonly used in safety engineering, can be implemented.For example, the overseer can monitor compliance with limit values ​​for positions and / or speeds and / or check the current movement of the crane, especially the grabber, with regard to a possible collision with obstacles detected in the world model.

[0053] The control system can further include a non-linear model predictive controller, preferably based on a dynamic model and feedback sensor data. Such a controller ensures that the crane moves autonomously and in a time-optimized manner to a specific target location, particularly without significant residual oscillation of the grab. The control system, or the controller in question, can thus take trajectories into account (especially considering the world model) as well as the physical limits of the crane, and can implement collision avoidance by monitoring safety distances.

[0054] A preferred embodiment of the method provides that the real-time measurement data acquired by the crane's optical sensor system is obtained using at least two different optical measurement methods. With this approach, the control system can, for example, reliably detect the current position and / or orientation and / or direction of movement of the gripper. The control system can then precisely adjust / regulate the gripper's movement to enable reliable and accurate control of its motion.

[0055] In previously known crane control systems, the regulation of crane movement is often based on passive state detection, for example via (typically non-optical) sensors on the crane's rotary joints. This approach is also frequently used in industrial robots that are not equipped with optical sensors. However, a prerequisite for precise control in such approaches is typically that the robot arm is as rigid as possible, so that the robot can determine the current position of the robot arm's tip in space via state detection, for example by reading sensors in the robot arm's joints. The actual position of the robot arm's tip is only calculated but not actively detected by sensors. In other words, such previously known control systems operate passively and have no active measuring means to perform an active, real-time position determination.

[0056] In contrast, the optical sensor system of the crane discussed here allows for active, real-time optical detection of the grab's position. This is advantageous, among other reasons, because the accuracy of this active sensor system does not depend on the rigidity of the crane's supporting structure, particularly its movable boom. Furthermore, relative movements between the grab and boom, caused, for example, by gusts of wind or oscillation of the grab, can be reliably detected and taken into account by the control system when operating the crane.

[0057] One of the two measurement methods described above can, for example, be based on a computer vision algorithm that processes 2D image data. Such an algorithm can preferably calculate absolute position coordinates of the object to be measured and / or the gripper from relative image coordinates using a projection matrix. The AI ​​can then process these absolute position coordinates to generate control signals.

[0058] Furthermore, it may be additionally or alternatively provided that one, in particular a second, of the two measurement methods is based on one of the following approaches: a) triangulation using two cameras; b) optical distance measurement, in particular using LIDAR (light detection and ranging); or c) use of structured lighting and a camera to calculate depth information.

[0059] If the first measurement method only provides relative position coordinates, these can be converted into absolute position coordinates using measurements from the second measurement method. For example, the control system can calculate relative position coordinates of objects captured in the 2D image data from a calibrated camera of the optical sensor system, taking into account a projection matrix of this camera. Subsequently, these relative position coordinates can be converted into absolute position coordinates by the control system using a depth map determined with an optical distance sensor (preferably using LiDAR) of the optical sensor system and / or taking into account sensor-acquired and / or calculated status information of the crane.However, an absolute coordinate can also be calculated from the (respective) relative coordinates by means of measurements carried out with at least two spatially separated cameras of the optical sensor system, for example with the aforementioned primary and secondary camera.

[0060] Preferably, the second measurement method can generate redundant information regarding the location and / or position of the gripper and / or the object to be picked up. Since the artificial intelligence processes all this measurement data, the autonomous control of the crane can be designed to be more precise and reliable. In particular, this allows for the implementation of robust optical tracking of the gripper in space. Preferably, the optical tracking of the gripper is carried out using at least one non-Cartesian coordinate system by the control system, in particular a coordinate system that defines a joint space of the crane.

[0061] By using the two independent optical measurement methods in parallel / simultaneously, redundant optical object recognition can be achieved. With this approach, the control system can reliably and safely detect the spatial position and / or orientation of unstructured objects such as tree trunks or other loose goods located on a loading platform.

[0062] In addition, the control system can reliably and safely determine the gripper's current position and / or orientation by using the two independent measurement methods, thus improving the accuracy of the gripper's control. This approach can be particularly helpful for implementing automatic collision prevention with the control system, in which the control system predicts potential collisions of the gripper with objects within the detection area based on the optically captured instantaneous movement of the gripper and then proactively slows the gripper's movement in space to avoid the anticipated collision.

[0063] It is therefore proposed, in particular, to use two redundant optical measurement methods, each implemented independently using different procedures, in parallel and simultaneously to detect the relative movement of the boom, especially the relative movement and / or position of the gripper, with the optical sensor system. This will significantly increase the safety of the autonomous crane's operation.

[0064] The use of two redundant optical measurement methods offers significant advantages, for example, in the described application where logs are to be moved by crane, their position, size, and orientation being unknown beforehand. This is because such applications do not involve unloading known goods that are in a fixed orientation and always the same size, but rather unloading unstructured or arbitrarily arranged goods, which considerably increases the complexity of the control system. The redundancy achieved through the different measurement methods significantly improves the safety of crane operation. Furthermore, the optical sensor system can precisely determine the exact placement location and orientation of the logs.

[0065] Previous approaches, which passively determined the position of the crane boom using sensors, always resulted in considerable uncertainty regarding the actual current position of the gripper, as this was not actively detected by sensors. This is because, firstly, the crane boom can twist depending on the load, leading to inaccuracies in the position of the boom tip, unlike a rigid robot arm; and secondly, because the gripper can, for example, begin to oscillate in the wind, causing its current position to differ from a nominal position without wind. The inventive approach allows all these factors to be actively detected with a high degree of certainty using the optical sensor system, thus ensuring safe crane operation.

[0066] The method can further be designed such that the real-time measurement data includes: a) a 3D point cloud whose points are calculated from distances measured with an optical distance sensor of the sensor system; and b) spatial position and / or spatial orientation data, the latter being obtained from image data captured with a camera of the sensor system using a computer vision algorithm.

[0067] The optical distance meter mentioned can be implemented, for example, in the form of a laser scanner, especially based on LIDAR.

[0068] Preferably, the camera and optical distance sensor can be mounted on the movable boom. This allows the optical sensor system to move with the boom at all times. In this way, the sensor system can continuously monitor a target area within which the gripper can detect objects. The measured distances, as well as position data (e.g., xyz coordinates) and orientation data (e.g., rotations along the x / y / z axes), can relate to the crane's gripper and / or the object being picked up.

[0069] In the method according to the invention, the control system can thus determine the spatial position of the gripper and / or the object to be picked up from the real-time measurement data and take this into account during the autonomous control of the crane. Preferably, the control system can also additionally determine the spatial orientation of the gripper and / or the object to be picked up from the real-time measurement data. This enables the control system to control the movement of the boom and / or the movement of the gripper more precisely.

[0070] In general, for greater robustness of the crane's autonomous operation, it is advantageous if the spatial position and / or orientation of the gripper and / or the object being picked up is detected without markers, i.e., without the use of optical reflectors. This markerless approach offers significant advantages in practice because it makes the crane's control much more robust against strong sunlight or, for example, when the objects being picked up are heavily soiled, which is problematic when using markers on the objects themselves. Further disadvantages of using markers include potential damage to the markers during loading / unloading or obstruction by the cargo.

[0071] The approaches described above thus enable an optimized control method for the crane based on markerless optical tracking of the gripper, whereby the optical tracking is carried out using the optical sensor system.

[0072] One embodiment of the method involves the control system determining the tilt of the object being recorded within the recording area using a gradient method that processes image data of the recording area acquired by the optical sensor system. The gradient method can consider eigenvectors of a Hessian matrix in a pixel neighborhood. These eigenvectors can be interpreted as the distribution of the gradient to be calculated in the neighborhood and are often referred to as structure tensors. Thus, the gradient method can be configured to calculate the structure tensors for individual pixels of a 2D image acquired by the optical sensor system in order to ultimately determine the object's tilt. To integrate the neighborhood information, a box filter can, for example, first be applied to the image.This results in an average direction for each pixel and its neighborhood. For example, to detect bundles of wood within a 2D image, regions with a common direction can be defined and subdivided into groups. The resulting groups can then be interpreted by the control system as individual bundles.

[0073] Another variant of the method involves the control system using an optical sensor system to detect and preferably document the actual final placement location of the object within the designated area. This detection can occur precisely when the gripper releases the object. Preferably, the optical sensor system can also detect the object's actual final spatial orientation within the placement area. The word "actual" here emphasizes that the final placement location does not necessarily have to correspond to the location where the gripper released the object. For example, when placing logs, it is common for the log to continue moving during the process, changing its position and / or orientation until it comes to rest in its final spatial orientation at the designated placement location.Recording the actual final storage location and the final spatial orientation can be helpful, for example, so that when placing another object within the storage area, the control system can take into account the previously placed object (more precisely, its position and location).

[0074] The crane's control system can also be designed to implement active collision prevention. This can be achieved by the control system or artificial intelligence virtually defining a safety zone around the object to be lifted (by the crane) within a live image captured by the optical sensor system. This definition of a virtual safety zone within the captured live image can be based on the previously described and potentially already performed segmentation of the live image and / or on object recognition.For example, the control system can use the optical sensor system to detect the outer boundary of a truck's loading platform and define a safety zone at a certain distance from this boundary. Within this zone, the gripper can be positioned without collision to pick up objects from the truck's loading platform. In such a configuration, it is further preferred that the control system proactively slows down the gripper's movement as soon as it detects, based on real-time measurement data, that the gripper would otherwise move out of the safety zone.

[0075] The described active collision prevention system can utilize measurement data from a LiDAR sensor. Such collision avoidance is of considerable importance in practice because reaction times are typically extremely short. The first step in this process is to determine the gripper's position in space very precisely, which is possible with the optical sensor system. The second step is to virtually define the safety zone within the captured live image to ensure that the gripper moves only within this safety zone without risk of collision.

[0076] The safety zone can therefore be redefined and adjusted by the control system depending on the situation before each loading or unloading operation. The size and / or position of the safety zone in space (especially relative to the crane if it is stationary) can change. This approach addresses the problem, for example, that when unloading a truck, the loading platform can vary in position and orientation depending on the truck's exact parking location. If the objects to be unloaded by the crane are delivered by a different type of truck, the control system can also take this into account by adjusting the safety zone accordingly. This efficiently and reliably prevents the grab from colliding with the truck when unloading the objects from the loading platform.

[0077] In contrast, during crane navigation, the control system can avoid collisions by determining the grab's current position in a live image captured by the optical sensor system and converting it into absolute 3D position data of the grab, particularly taking into account calibration data from the optical sensor system and / or current status information on the crane's kinematics. Subsequently, a collision check / estimation is performed based on the determined 3D position data and a height map acquired by the optical sensor system (which can detect obstacles on the grab's current path). The control system can also compare this data with the height map stored in its memory.This allows a collision with an object outside the current detection range of the sensors to be prevented, even in the case of very long braking distances, provided that this object was detected in a previous pass (made by the crane) and integrated into the elevation map. The elevation map can thus be continuously updated and revised during the crane's operation based on new / ongoing measurements.

[0078] In previously known industrial applications that use a light curtain to implement an automatic shutdown mechanism for collision prevention, the safety zone defined by the light curtain remains constant because the entire system is stationary and the picking or placing surfaces from which objects are to be picked up or placed by a robotic gripper typically do not change or move. The solution proposed here differs from these approaches in that, thanks to precise object recognition—for example, of a truck's loading platform, including the detection of its size and / or orientation—the virtual safety zone within which the gripper can move without collision can be redefined for each situation using the optical sensor system.

[0079] Furthermore, the control system can be designed to implement active vibration damping of the gripper. For this purpose, the control system can be designed to proactively brake the gripper's movement by appropriately controlling a drive device of the crane as soon as the control system detects, based on real-time measurement data, that the gripper would otherwise vibrate.

[0080] This approach takes into account that the grab, especially when suspended from the boom by a cable, represents a freely swinging element. The length of the cable, and thus the length of the pendulum formed by the grab suspended from the cable, can change during crane operation, which can also alter the oscillation behavior. However, by optically detecting the grab's movement in space using the optical sensor system, the control system can assess the grab's current oscillation tendency and intervene accordingly to prevent excessive oscillation. This significantly improves safety during crane operation because the grab's position can be monitored more reliably.

[0081] With this approach, the control system can therefore implement an "anti-sway system".

[0082] A particularly preferred embodiment of the method provides that a drive device of the crane, in particular the one described above, is controlled by the control system by means of a so-called input shaper function, which enables passive vibration compensation. This passive method serves to minimize the impact of vibrations.

[0083] To induce vibrations during crane movements, such as those of the undercarriage and trolley. The input shaper function can be specifically designed and mathematically configured to (passively) prevent or at least minimize vibrations of the grab relative to the crane boom.

[0084] Furthermore, the control system can be configured to estimate the endpoint of a gripper's current movement in space based on a (temporal) integration using the input shaper function. The control system can then decide, based on this estimate, whether the gripper's movement should continue or be decelerated. A key feature of this method is its ability to predict the braking distance, including any extension due to the implemented vibration damping, at each time step. This allows the system to determine the optimal start of the braking process to achieve a specific target with the gripper. Because the remaining braking distance can also be continuously monitored for collisions by the control system, braking can be initiated in a timely manner, preventing the intervention of collision avoidance mechanisms and thus the coupling of vibrations (preventively).The active method used in this approach to vibration damping, employing the crane / grab, serves to actively dampen residual vibrations and externally induced vibrations (wind, contact with obstacles). The control system can operate predictively, utilizing the gripper's pendulum dynamics (in the case of a cable-suspended gripper), and in particular estimate the phase, period, and amplitude of the vibration, for example, using a regression method. Based on this estimate, the control system can then intervene in the crane's current movement to control and correct it.

[0085] With this approach, the control system implements a prediction method in which, through integration via the input-shaper function, an estimate is made of the final position in space of a current gripper movement, which is being detected by the optical sensor system. The movement is triggered by applying the input-shaper method (i.e., by controlling the drive device using the input-shaper function). Such an estimate is particularly valuable for the precise positioning of the gripper, as the gripper's deceleration is controlled by the input-shaper function in such a way that gripper oscillation is (largely) avoided. This approach thus enables vibration-free and target-precise deceleration of the gripper, completely autonomously, without any user input.

[0086] Another practically relevant problem is preventing torsional vibrations that occur in a crane grab suspended by cables when it is rotated around a rotational axis by a drive mechanism (for example, to change the spatial orientation of a load gripped by the grab) or due to gusts of wind. For instance, the crane grab may be suspended by cables from a rotating head block (grab suspension), which itself is mounted on a pivot. When the head block / grab suspension rotates, torsional stress initially builds up in the cables between the head block and the grab, especially if the grab has enclosed a load and thus has a high rotational moment of inertia. This torsional stress or force accelerates the grab, causing it to begin rotating.Due to its inertia, the gripper would continue this rotational movement when the gripper is aligned with the rotating block again, resulting in an oscillating rotational vibration of the gripper.

[0087] To implement torsional vibration compensation of the gripper in such situations, the control system can be configured to optically measure the torsion of the gripper suspension cables using the optical sensor system (in particular, the aforementioned first camera / primary sensor). At the appropriate moment, the control system generates torsion in the cables by rotating the gripper suspension, which counteracts the gripper's momentary rotation. The control system can preferably achieve this by appropriately controlling a rotary drive that rotates the gripper relative to the gripper suspension (thus causing a rotational movement of the gripper suspension relative to the gripper). The gripper's rotational movement can also be detected by the optical sensor system or, for example, by means of an IMU on the gripper.The generated counter-torsion allows the control system to actively decelerate any momentary rotational movement of the gripper. In particular, this enables the gripper to be stopped / brought to a vibration-free halt in a desired end position.

[0088] This active torsional vibration compensation implemented by the control system can thus take into account optical measurements of the current torsion of the ropes as well as the (sensor- or optically detected) rotational speed of the gripper suspension in order to reduce the torsion at the appropriate moment (by rotating the rotating block), taking the dynamics into account, and thus stop the rotation of the load precisely at the desired final rotational position. If a rotation with the gripper around a specific angle is to be performed vibration-free, the control system initially builds up torsion (center) by rotating the rope suspension to initiate the rotation. To decelerate the rotation, torsion is then built up in the opposite direction. The sign of the respective control signals can be reversed in this process.

[0089] It is further proposed that the crane not be operated partially autonomously, as is currently the norm, but essentially still controlled by a user via a control unit such as a joystick, where, for example, a specific direction or speed of travel is specified by the user, but rather that the control system or the crane itself executes abstract tasks completely autonomously. For example, the crane can be instructed via a suitable interface to perform a specific task, such as unloading a particular truck loaded with logs. As explained, the crane has an optical sensor system that allows it to precisely identify the truck's loading platform and the logs on it in a live image and calculate the exact position data of the logs (localization of the objects to be picked up).This situation differs significantly from that of a typical industrial robot, which grasps objects in a defined environment that are always in the same place and typically oriented the same way. When the truck approaches the picking area, the position of the loading platform will vary, and its size or shape can also differ depending on which truck is delivering the logs. The AI ​​must take all of this into account. The technical solution presented here enables full autonomy of the crane even in such situations, allowing it to perform the desired task completely autonomously and safely.

[0090] Accordingly, the procedure may in particular provide that the control system processes a catalog of tasks by autonomously operating the crane, each of which includes at least two (or all) of the following steps a) to f): (a) autonomous detection and localization of a current recording area; (b) autonomous detection and localization of a loading or storage area within the recording area in which objects to be recorded are stored; (c) autonomous detection and localization of a suitable or assigned drop-off area in which the objects can be dropped off; (d) autonomous detection and localization of a loading or storage area within the drop-off area in which the objects can be dropped off safely; (f) autonomous loading of a loading or storage area, in particular another loading or storage area, within the drop-off area;

[0091] In other words, it is proposed that the AI ​​of the control system independently determines pickup positions within the pickup area and / or unloading positions within the drop-off area based on real-time measurement data. The respective detection and localization can preferably be based on the real-time measurement data acquired by the optical sensor system. Of course, other sensor data, such as crane status data, can also be considered by the AI ​​when performing such tasks.

[0092] With this approach of controlling the crane via tasks, it must be taken into account that the travel positions to which the crane's gripper is to move autonomously are not predetermined (as is common with industrial robots, for example), but can change depending on the specific application situation. For example, when a new truck delivers logs, these travel positions can change, which the control system must take into account.

[0093] To solve the aforementioned problem, the features of independent device claim 14 are also provided according to the invention. In particular, an autonomously operable crane is thus proposed according to the invention to solve the problem. This crane comprises a drive device for driving a movement of a movable boom of the crane, an optical sensor system for (at least indirectly) detecting movements of the boom, wherein the optical sensor system can in particular be configured to detect a movement and / or position of a gripper mounted on the boom, and a (according to the invention) control system for controlling autonomous operation of the crane.This crane differs from previously known cranes in that its control system is based on artificial intelligence. This AI is designed to process real-time measurement data acquired by the optical sensor system and convert it into corresponding control signals, enabling autonomous control of the crane. Both the control system and the optical sensor system of such a crane can be configured as described above.

[0094] The drive device can be implemented, for example, by means of electric actuators and / or hydraulic and / or pneumatic actuators.

[0095] Artificial intelligence (AI) can be designed to implement a machine learning method. For example, AI can be implemented using a neural network, which in turn is realized using an electronic circuit.

[0096] This crane can of course be used particularly advantageously when the control system is configured to carry out a method according to the invention as described above or according to one of the claims directed to such a method.

[0097] As previously explained, the crane's optical sensor system may include a camera set up to record live image data, as well as an optical distance meter to determine an instantaneous distance between the crane and the object to be picked up and / or between the crane and the grabber.

[0098] If the crane is used indoors, structured lighting can also be employed. In this case, the optical sensor system can include a lighting unit to generate this structured lighting. The structured lighting can, for example, illuminate a target area in which the object to be picked up is located and / or in which the gripper can be positioned. When using structured (especially invisible) lighting, the sensor system can include an additional camera to detect the illuminated target area. This camera can then use the structured lighting detected by its sensor to obtain additional depth information from the target area, which can improve the accuracy in locating the object to be picked up and / or the gripper.

[0099] The described movable boom of the crane may include a lifting mechanism for raising a load. However, this is not mandatory; for example, it is also possible to raise the boom directly to lift and move loads.

[0100] The boom can also include a grab as described above, which can be used to grasp a load in order to then move the load with the crane.

[0101] Accordingly, the drive device can include a drive for operating the lifting mechanism and / or a drive for operating the gripper. In such configurations, it is naturally advantageous if the control system is configured to control the lifting mechanism and / or the gripper.

[0102] To solve the problem, a retrofit kit according to claim 15 is also proposed, which is designed for integration into an existing crane that has its own (non-intelligent) crane control system. This retrofit kit comprises an optical sensor system, which can be configured as described above and installed on the crane; and an intelligent (inventive) control system, which is based on artificial intelligence and is configured to process real-time measurement data from the optical sensor system and convert it into corresponding control signals, enabling the crane to be controlled autonomously via the existing crane control system. It is understood that the control system can be configured as described above, i.e., the control system can be configured to perform calculations, control, regulation, considerations, evaluations, etc., as described above.to carry out and / or to perform a procedure as described above.

[0103] Furthermore, the control system may have an interface for transmitting control signals to the crane control unit. This interface may be configured for use with the user interface already present on the crane.

[0104] The invention will now be described in more detail with reference to exemplary embodiments, but is not limited to these embodiments. Further embodiments of the invention can be derived from the following description of a preferred embodiment in conjunction with the general description, the claims, and the drawings.

[0105] In the following description of various embodiments of the invention, elements that are identical in function are given identical reference numbers even if they differ in design or shape.

[0106] It shows: Fig. 1 a schematic representation of a control system or a crane according to the invention, Fig. 2 a schematic representation of an unloading situation in which the crane is made of Figure 1 Fig. 3 and 4 show a possible implementation of a control system according to the invention, Fig. 5 shows images captured and generated by the sensor system of the control system, taken from the side view of the crane's gripper, Fig. 6 shows an active rotational vibration compensation executable with a control system according to the invention, and Fig. 7 shows a detailed view of the Figure 8 and Fig. 8 with the sensor system of the control system images captured and generated, taken from above looking down at the gripper of the crane.

[0107] The Figure 1Figure 1 schematically shows the components of a control system 4 according to the invention as part of a crane 1. The control system 4 is based on artificial intelligence (AI) that processes real-time measurement data from an optical sensor system 5, which is mounted on the crane 1, and converts it into corresponding control signals. With the aid of a camera 10, the control system 4 can actively detect the movements of a gripper 6, which is mounted on a movable boom 3 of the crane 1, and then control a drive device 2, which causes the movements of the boom 3 and the gripper 6. Optionally, the movements can also be controlled (as shown in the figure 5). Figure 1 (illustrated) with an additional optical distance sensor 11.

[0108] The Figure 2schematically illustrates a typical work situation in which the crane 1, with the help of its movable gripper 6, is to pick up an object 7 that is located within a picking area 8, whereby only partial components of the crane 1 are made of Fig. 1The control system 4 is shown schematically. Using the optical sensor system 4, the control system 4 autonomously detects the detection area 8 and, by means of a multi-layered image processing chain that processes live images 17 from the camera 10, also the precise spatial position and orientation of the object 7 to be detected, as well as any obstacles within the detection area 8. The control system 4 then moves the gripper 6 into the detection area 8 without collision and grasps the identified and located object 7 there in order to transport it to a storage location within a storage area 9 using the crane 1. The optical sensor system 5 can also detect the actual final storage location 12 where the object 7 ultimately comes to rest after being placed by the crane 1.

[0109] The Figure 3 illustrates a possible system architecture of a control system 4 according to the invention, wherein the Figure 4The corresponding terms are given in German. The central process, referred to as the core, which is executed by the control system 4, performs the individual processing steps (receiving sensor data, processing sensor data, updating the world model, querying information from the world model, decision-making, safety check using a safety overseer instance, generation of control signals) sequentially in each control cycle. A world model, constantly updated by the control system using sensor data, serves as a central element in which the current state of the crane and its environment is coherently represented. The processing of the individual signals supplied by the sensors of the optical sensor system 5 is outsourced to several "sensor nodes," each of which evaluates sensor signals from the respective sensor (e.g., first and / or second camera; LiDAR, etc.) in parallel.Each sensor node can have its own processing unit with a direct interface to the respective sensor. The results of the sensor signal processing provided by the respective processing unit are synchronously integrated into the central processing unit in the core before decision-making within a control cycle begins. The central processing and command / control signal generation within the control system 4 is thus organized sequentially and synchronously, as in a classic control loop. Sensor information from the optical sensor system 5 and other existing sensors (e.g., on the swivel joints of crane 1) is processed and then integrated into the world model. A decision is then made regarding a currently pursued goal, resulting in a specific action to be performed by crane 1, and from this, a concrete control command (transmitted via the generated control signals) is generated.

[0110] As in Figure 3 / 4 As can be seen, the control system 4 can transmit the generated control signals via a crane communication layer to a crane controller 16 of the crane 1, which was already implemented in the crane 1 by means of a programmable logic controller (PLC). It follows that the entire control system 4, including the associated optical sensor system 5, can therefore be easily retrofitted to an existing crane 1 with a non-intelligent crane controller 16, which can be controlled, for example, via an operating unit or another human-machine interface.

[0111] Figure 5The left half illustrates a live image 17, recorded with a second camera 10 (secondary sensor) of the optical sensor system 5. This camera 10 provides a side view of the crane 1's gripper 6 as it unloads a truck 23. Using a neural network, the control system 4 classifies numerous image areas 25 within the live image 17 into different object classes, resulting in a segmentation image 22 as shown in the right half. Figure 5It can be shown that only the relevant image areas 25 within the image are segmented and the corresponding pixels are marked by their respective object class (number). The object classes here are the grabber 6, the load / the objects to be picked up by the crane 1 7 (namely the bundle of logs), the chassis 24 of the truck 23 (from which the position of the loading platform 15 can be determined), and the posts 26 as boundaries of the loading platform 15 of the truck 23, which represent possible obstacles.

[0112] The Figures 7 and 8Figure 17 illustrates a similar live image, as does a segmentation image 22 calculated from it by the AI ​​of the control system 4. This segmentation image again contains several image areas 25, each segmented to identify objects to be located, such as the gripper 6, the logs as cargo, or the chassis 24 of the truck 23, which were detected and classified accordingly. The contour lines around each image area 25, which the AI ​​defined to delineate the image areas 25, are clearly visible in the right-hand segmentation image 22. The AI ​​has hidden objects that are not of interest (but were nevertheless detected), such as the cab at the right end of the truck 23, in the segmentation image 22.

[0113] Figure 6Figure 1 illustrates a possible embodiment of a crane 1 according to the invention, in which the gripper 6 is suspended via a gripper suspension 19 by means of steel cables 18 from a trolley 21, which is mounted on a boom 3 of the crane 1 and can be moved along the longitudinal axis of the boom 3. A camera 10 of the optical sensor system 5 is mounted on the trolley 21 (or, for example, on a boom mounted on the trolley 21), with which the gripper 6 can be observed from above.

[0114] The gripper 6 is mounted on, or supported by, a head block 20 of the gripper suspension 19, the head block 20 being suspended from the trolley 21 via steel cables 18. The head block 20 includes a rotary drive 28, by means of which the gripper 6 can be rotated about a rotation axis 27 defined by the rotary drive 28. The rotary drive 28 can preferably be hydraulically or pneumatically operated, so that only an electrical power supply needs to be routed from the trolley 21 to the head block 20, because the hydraulic / pneumatic rotary drive 28 then has its own electric motor for generating hydraulic / pneumatic operating pressure.

[0115] As indicated by the block arrows, the vertical height of the head section 20, and thus of the gripper 6, can be adjusted by means of a lifting mechanism 14 designed as a cable drive. Furthermore, the control system 4 can precisely adjust the position of the gripper 6 by pivoting a movable boom 3 of the crane 1 (if the crane 1 is designed as a slewing crane) or by moving a crane bridge of the crane 1 (in the case of a gantry or overhead crane). The position of the gripper 6 can also be adjusted by moving the trolley 21.

[0116] When the gripper 6 is set into rotation about the axis of rotation 27 by means of the rotary drive 28, the gripper 6 can transmit a rotational force to the head 20 due to inertial forces (especially when the gripper 6, weighing approximately 6 tons, has gripped a load of, for example, 30 tons), so that the (relatively light) head 20 is twisted about the axis 27 relative to the (non-rotating) trolley 21. In this case, however, a restoring torsional force builds up due to the cable suspension 19, which acts on the head 20 and thus also on the gripper 6 via the steel cables 18.

[0117] The control system 4 optically measures the instantaneous rotation between the head 20 and the trolley 21 using the camera 10 and estimates the direction and approximate magnitude of the acting torsional force transmitted via the cables 18 of the gripper suspension 19, possibly taking into account the instantaneous cable length of the suspension 19 (e.g., as a function of the vertical height of the gripper 6 or its vertical distance to the trolley 21). Furthermore, the control system 4 also uses the camera 10 to detect the instantaneous rotational speed of the gripper 6 about the axis of rotation 27. By appropriately controlling the rotary drive 28, the control system 4 can generate a torsion that acts against the instantaneous rotational movement of the gripper 6, thus selectively decelerating the rotational movement of the gripper 6 when the control system 4 wants to move the gripper 6 into a specific rotational position without vibration.With this approach, the control system 4 thus achieves an active compensation of torsional vibrations that the gripper 6 performs around the rotational axis 27.

[0118] In summary, the invention enables fully automatic and autonomous crane control based on an optical measuring system 5, which is preferably designed with redundancy. The control system 4 of the crane 1, which is based on artificial intelligence, takes into account not only passively determined status information of the crane 1, but also measurement data actively acquired by the optical sensor system 5. From this, the control system 4 can determine precise information about the current position of the crane grab 6, as well as about objects 7 (e.g., tree trunks) to be gripped by the grab 6. This control approach based on artificial intelligence also enables vibration-free positioning of the grab 6 and the avoidance of collisions between the grab 6 and stationary objects whose position may change depending on the situation.In this case, the gripper 6, suspended on ropes 18, represents an important technical boundary condition as a freely swinging link, which the control system 4 takes into account in an intelligent way.

[0119] The proposed control system 4 differs, for example, from previously known industrial robotics systems in that the crane 1 is enabled to autonomously pick up unsorted objects 7 at different picking locations and place them at a desired storage location 12. The proposed optical sensor system 5 enables markerless optical tracking of the gripper 6 and the objects 7 to be grasped, thus ensuring safe operation of the crane 1 even under varying weather conditions.With a crane designed according to the invention, virtually any load, object (such as trucks), unloading areas, grippers, and obstacles can be detected at sufficient speed (typically at a temporal resolution of 20-60 Hz) and with high accuracy and robustness, even under changing light and weather conditions, without requiring any modification, marking, or other restriction of the system environment and relevant objects. For this purpose, it may be sufficient for the artificial intelligence to have learned to recognize objects of a specific object class ("truck in the form of a semi-trailer"). This means that the training does not necessarily have to be limited to the same object (e.g., a very specific type of semi-trailer), so that the AI ​​can also recognize different objects (e.g.,can correctly recognize and classify multiple types of different semi-trailer trucks within the same object class. Reference symbol list

[0120] 1 Crane 2 Drive device (of 1) 3 Boom (movable, of 1) 4 Control system (for controlling the operation of 1) 5 Sensor system (for detecting a movement of 3 / 5) 6 Gripper (mounted on 3) 7 Object (is picked up by 6) 8 Pickup area (where 7 is located) 9 Drop-off area (where 7 is to be moved by 1) 10 Camera 11 Optical distance sensor 12 Final drop-off location (where 7 will be located within 9) 13 Safety zone (around 7, within 8) 14 Lifting mechanism 15 Loading or storage area (on which several 7 are currently stored, which are to be unloaded / picked up by 1 or onto which several 7 are to be moved by 1) 16 Crane control (of 1, for example designed as a PLC = programmable logic controller) 17 Live image (recorded with 5) or10) 18 Rope 19 Gripper suspension 20 Head block (carries 6) 21 Trolley 22 Segmentation image 23 Truck (transports 7, parks in 8) 24 Chassis (of 23) 25 Image area (within 17) 26 Posts (limit 15 of 23) 27 Rotary axis (of 6 / 28) 28 Rotary drive (for rotating 6).

Claims

1. Method for the autonomous operation of a crane (1) using an intelligent control system (4), wherein: - the intelligent control system (4) is based on artificial intelligence, - the artificial intelligence processes real-time measurement data from an optical sensor system (5) mounted on the crane (1) and converts it into corresponding control signals with which the crane (1) is controlled autonomously, without human intervention, characterized in that - the control system (4) evaluates live images from at least one camera (10) of the optical sensor system (5) using a multi-layer intelligent image processing chain based on artificial intelligence, wherein - a first layer of the image processing chain is formed by a neural network, which classifies image regions (25) within a live image (17) into different object classes and thus generates a segmented image, - a second layer of the image processing chain localizes objects within the classified image regions (25) of the segmented image based on their shape, and - a third layer of the image processing chain extracts at least one kinematic parameter from one of the localized objects, based on which parameter the control system (4) performs a dynamic estimation of the object.

2. Method according to one of the preceding claims, wherein the control system (4) comprises a world model that includes state information about the crane (1), in particular current sensory information regarding positions and / or movements of joints of the crane (1), preferably and including at least one pose information item that describes a current spatial position and orientation of a gripper (6) of the crane (1) and / or an object (7) to be picked up, and - wherein the control system (4) retrieves information from the world model and takes this into account when generating the control signals.

3. Method according to one of the preceding claims, wherein a gripper (6) mounted on a movable boom (3) of the crane (1) grasps an object (7) which is to be picked up and is located within a pickup area (8), and the object (7) is subsequently moved by the crane (1) to a placement area (9) and placed there, - in particular wherein the gripper (6) is mounted on the movable boom (3) in a suspended and thus oscillating manner, in particular by means of at least one cable hoist, and / or wherein a distance, in particular a vertical distance, between the gripper (6) and the boom (3) can be varied with the aid of the control system (4).

4. Method according to one of the preceding claims, wherein the control system (4) - controls at least two axes of the crane (1) independently of one another during positioning movements of the crane (1) and / or - when picking up or setting down objects (7) with the aid of a / the gripper (6) of the crane (1), controls at least two axes of the crane (1) jointly and thus interdependently, - preferably and, in this process, controls all axes of the crane (1) simultaneously in each case at all times.

5. Method according to one of claims 3 or 4, wherein the control system (4) autonomously detects the pickup area (8) within a live image (17) captured by the optical sensor system (5), either marker-based or markerless, and - wherein the artificial intelligence subsequently segments the live image (17) into multiple image regions (25), and assigns each of the image regions (25) segmented in this manner to a specific predefined object class, - preferably wherein the segmentation is based on a specific unloading situation within the pickup area (8) and / or is performed anew by the artificial intelligence on a situation-dependent basis using the live image (17), and / or - wherein the artificial intelligence performs the segmentation based on parameters that the artificial intelligence has learned from training image data that has been subjected to human evaluation.

6. Method according to one of the preceding claims, - wherein the kinematic parameter extracted by the third layer is - an angular velocity and / or - pose information and / or - a center of mass, preferably wherein this dynamic estimation is implemented using a Kalman filter.

7. Method according to one of the preceding claims, wherein the control system (4) comprises a safety overseer instance that checks the control signals generated by the artificial intelligence for unsafe behavior of the crane (1) and overrides the control signals if at least one safety rule is violated, - preferably wherein, in the event of a safety rule being violated, the safety overseer instance transitions the crane (1) to a safe state.

8. Method according to one of the preceding claims, wherein the real-time measurement data is obtained using at least two different optical measurement methods with the optical sensor system (4) and - wherein a first of the two measurement methods is based on a computer vision algorithm that processes 2D image data, - in particular wherein the algorithm calculates absolute position coordinates of the object (7) to be picked up and / or the gripper (6) from relative image coordinates and using a projection matrix, and / or - wherein a second of the two measurement methods is based on one of the following approaches: - triangulation using two cameras (10); - optical distance measurement, in particular using LIDAR; - and / or wherein the control system (4) evaluates data from an inertial measurement unit mounted on a movable part of the crane (1), in particular to detect via sensor an instantaneous torsion of a supporting structure of the crane (1) and / or the torsion of a sensor mount.

9. Method according to one of the preceding claims, wherein the real-time measurement data comprises - a 3D point cloud, the points of which are calculated from distances measured by an optical distance sensor (11) of the sensor system (4), as well as - spatial position and / or spatial orientation data obtained using a computer vision algorithm from image data captured by a camera (10) of the sensor system (4).

10. Method according to one of the preceding claims, wherein the control system (4) determines a spatial position, preferably and a spatial orientation of the gripper (6) and / or the object (7) to be picked up from the real-time measurement data and takes into account, in the autonomous control of the crane (1), in particular for controlling a movement of the boom (3) and / or for controlling a movement of the gripper (6), - preferably wherein the detection of the spatial position and / or orientation is performed markerless, i.e., without the use of optical reflectors.

11. Method according to one of the preceding claims, wherein the control system (4) implements active collision avoidance by virtually defining a virtual safety area (13) around the object (7) to be picked up within a live image (17) captured by the optical sensor system (4), in particular based on the performed segmentation of the live image (17) and / or on the basis of object recognition, - preferably wherein the control system (4) preventively brakes the movement of the gripper (6) as soon as the control system (4) detects, based on the real-time measurement data, that the gripper (6) would otherwise veer out of the safety area (13).

12. Method according to one of the preceding claims, wherein the control system (4) implements active vibration damping of the gripper (6) by the control system (4) preventively braking a movement of the gripper (6) by correspondingly controlling a drive device (2) of the crane (1) as soon as the control system (4) detects, based on real-time measurement data, that the gripper (6) would otherwise carry out an oscillation.

13. Method according to one of the preceding claims, wherein the control system (4), in order to implement active rotary oscillation compensation of the gripper (6) of the crane (1), measures the torsion of cables (18) of a gripper suspension (19) of the gripper (6) using the optical sensor system (5) and, at the appropriate moment, by rotating the gripper suspension (19) accordingly, in particular by respectively controlling a rotary drive (28) with which the gripper (6) can be rotated relative to the gripper suspension (19), builds up a torsion that acts against a current rotational movement of the gripper (6), - in particular so that the rotational movement of the gripper (6) is decelerated and / or the gripper (6) stops oscillation-free in a desired final rotational position.

14. Autonomously operable crane (1), comprising - a drive device (2) for driving a movement of a movable boom (3) of the crane (1) - an optical sensor system (4) for detecting movements of the boom (3), in particular for detecting a movement and / or position of a gripper (6) mounted on the boom (3), wherein the optical sensor system (4) comprises at least one camera (10) for recording live image data, and - a control system (4) for controlling autonomous operation of the crane (1), characterized in that, - the control system (4) is based on artificial intelligence configured to process real-time measurement data from the optical sensor system (4) and convert it into corresponding control signals with which the crane (1) can be controlled autonomously, and - wherein the control system (4) is configured to evaluate live images from the at least one camera (10) using a multi-layer intelligent image processing chain based on artificial intelligence, wherein - a first layer of the image processing chain is formed by a neural network, which is configured to classify image regions (25) within a live image (17) into different object classes and thus generate a segmented image, - a second layer of the image processing chain is configured to localize objects within the classified image regions (25) of the segmented image based on their shape, and - a third layer of the image processing chain is configured to extract at least one kinematic parameter from one of the localized objects, thereby enabling a dynamic estimation of the object by the control system (4), - preferably wherein the control system (4) is configured to execute a method according to one of claims 1 to 13.

15. Retrofit kit for integration into an existing crane (1) that has its own crane control system (16), wherein the retrofit kit comprises: - an optical sensor system (5) that can be installed on the crane (1) and comprises at least one camera (10) for recording live image data, and - an intelligent control system (4) based on artificial intelligence, comprising a multi-layer intelligent image processing chain and configured to process real-time measurement data from the optical sensor system (4) and convert it into corresponding control signals with which the crane (1) is autonomously controllable via the existing crane control system (16), - wherein the retrofit kit is designed such that, after retrofitting the existing crane with the retrofit kit, the crane is converted into an intelligent crane (1) according to claim 14, so that the resulting intelligent crane (1) can autonomously perform tasks after retrofitting.

Images