Method for autonomous operation of a crane, autonomously operated crane and retrofit kit for producing an autonomously working crane

Abstract

The invention enables fully automatic and autonomous crane control, on the basis of an optical sensor system (5) that is evaluated by artificial intelligence. A control system (4) of the crane (1), which is based on artificial intelligence, thereby takes into account, in particular in addition to passively determined state information of the crane (1), measurement data which are actively acquired by means of the optical sensor system (5). From this, the control system (4) can determine precise information concerning a current position of a crane gripper (6), but also concerning objects (7) to be picked up with the gripper (6). This open-loop and closed-loop approach also makes it possible to position the gripper (6) without oscillation, as well as to avoid collisions between the gripper (6) and stationary objects, the position of which may change depending on the situation, or also to avoid an unwanted rotational oscillation of the gripper (6).

Description

[0001] Method for autonomous operation of a crane, autonomously operable crane and retrofit kit for realising an autonomously operating crane

[0002] The invention relates to the technical field of cranes for lifting loads, more precisely to the field of controlled cranes that can perform tasks autonomously. The invention is based on findings from the field of computer vision, i.e. the computer-implemented recognition of people and objects based on visual information from images or videos, which is sometimes also described under the term machine vision. Furthermore, the invention is based on the application of artificial intelligence (AI).

[0003] In detail, the invention relates to a method for the autonomous control / autonomous operation of a crane and to an associated crane which can be operated autonomously using this method. Furthermore, the invention relates to a retrofit kit with which an intelligent control system according to the invention can be easily retrofitted to an existing crane. With the latter approach, a conventional crane which has no intelligence of its own and can only be controlled by a human operator via a manual control unit who passes the control commands to a crane control system of the crane can be converted into an intelligent crane within the meaning of the invention, so that the crane can carry out tasks autonomously after the retrofitting.

[0004] A crane can be understood here, for example, as a mobile or stationary device having a movable boom with a lifting mechanism with which loads or bulky objects, such as tree trunks, can be lifted and moved. Such cranes are used on construction sites, but above all for loading and unloading lorries, ships or freight wagons on trains. The lifting mechanism is often implemented by means of a cable pull (frequently using pulleys), whereby bulky goods such as tree trunks can be grasped and lifted with a gripper that is attached to the lower end of the cable pull. The necessary drive for the crane is often implemented using appropriate motors or, for example, hydraulics.Grippers are also known that can be rotated about a rotation axis relative to the boom so that the gripper can grip objects in different orientations.

[0005] The state of the art, for example, includes non-intelligent cranes based on a lightweight scaffold-like support structure with a driver's cab. In this case, a human operator operates the boom and, if necessary, the grab of the crane either from the driver's cab or via remote control, and monitors and controls the lifting and moving of the loads using the crane's lifting mechanism. For example, in the wood loading area of ​​pulp mills, the 40 m high cranes used, which move loads of 30 tons, are still largely manually controlled. Apart from simple anti-sway systems and remote controls, no extensive automation is available. This is also due to the sensory uncertainties associated with the bulky goods "logs", the highly variable delivery trucks, and the often poorly structured storage and production processes.

[0006] However, there is now a growing desire to support or even completely replace the human operator with an intelligent control system that enables the crane to perform a specific task without external human intervention, such as unloading a load of goods from a container on a train or unloading tree trunks from the back of a truck. Depending on the application situation, for example, if a danger to people in the crane's working area can be ruled out, the goal may even be to allow the autonomous and intelligent control of the crane to run without any human supervision, in which case a correspondingly high level of autonomy must be achieved (e.g., Level 3 or Level 4 autonomy).

[0007] Against this background, the object of the invention is to provide an intelligent control system for a crane that enables autonomous operation of the crane, thereby increasing the efficiency and safety of operation compared to previously known approaches, and reducing or completely eliminating human errors and mistakes through digitalization and implementation of processes and procedures. The intelligent control system and the increased safety that can be achieved with it are intended to support a human operator who monitors the operation of the crane. In application situations where safety requirements permit this, the ultimate goal of the control system is to even make human monitoring of the autonomous operation of the crane superfluous.

[0008] To achieve this object, the features of claim 1 are provided according to the invention in a method for controlling a crane. In particular, to achieve the object in a method for the autonomous operation of a crane, the invention proposes that the control system is based on artificial intelligence (AI) and that the AI ​​processes real-time measurement data from an optical sensor system attached to the crane and converts it into corresponding control signals with which the crane is controlled autonomously, i.e. without human intervention. The AI ​​can be implemented using machine learning methods, in particular based on deep learning and / or on a neural network. This allows uncertainties relating to the task to be performed by the crane and its environment to be handled flexibly and robustly, which is particularly advantageous for slewing and gantry cranes.

[0009] Loading and unloading operations can thus be handled autonomously by the crane, without the vehicle being loaded / unloaded, the load, or the storage location requiring known dimensions, markings, or other adjustments. The achievable speed when controlling the crane is comparable to that achieved by a human crane operator, with the autonomous control system delivering more consistent results than humans.

[0010] The control system can be designed in such a way that it has control and / or reading access to the crane's drives and sensors either directly or indirectly via a (possibly already existing) crane control system.

[0011] The inventive control system based on artificial intelligence can, for example, be used to autonomously control a crane loading logs in a paper factory without the need for human intervention. However, to increase safety, the operation can continue to be monitored by a human who can intervene in the crane's operation via a control unit (joystick, emergency button, touch display, etc.) if necessary.

[0012] For example, when loading a waste incineration plant with a loading crane within an area that is not accessible to people from outside (for example, blocked off by appropriate grilles), the control system can even be used to control the loading crane of the waste incineration plant autonomously within the protected area, without any human supervision.

[0013] Preferably, components of the optical sensor system, in particular a (first or) primary camera (e.g., based on a 2D image sensor), are mounted on the movable boom of the crane, so that at least part of the sensor system moves in space together with the boom. For example, cameras used in the sensor system may also have zoom optics, so that a specific optical magnification can be set by the control system depending on the situation.

[0014] The cognitive architecture of the control system can combine a belief-desire-intention (BDI) architecture with a reactive approach (behavior-based architecture).

[0015] Furthermore, the control system can be designed in such a way that it implements a crane control system which is constructed as a finite state machine (simply referred to as a "core"). In this case, individual states of the machine can correspond to certain typical process steps which the crane is to process in the respective application (e.g.: assuming a waiting position; approaching the loading area of ​​the truck, unloading the truck; etc.). The transitions between the states of the machine can be implemented in a rule-based manner (in particular based on predicate logic rules) and can optionally be dynamically supplemented and / or modified by the control system. For this purpose, the control system can access information from a predictive world model in which the knowledge ("belief") about the crane's own state and its

[0016] environment is collected, filtered, integrated and continued (predicted) in a coherent form, whereby the world model is explained in more detail below. External information (for example, introduced by a

[0017] Human or an external system) are processed as additional "external requirements" by the control system.

[0018] The crane control implemented by the intelligent control system can be structured so that each process state of the state machine pursues a specific, defined goal. In the control system, the respective goal can then be subdivided into intermediate step goals (e.g., goal: "Unload"; intermediate step goals: "Approach load with grab"; "Open grab without collision"; "Securely enclose timber with grab"; "Lift load"). During pathfinding, i.e., defining a path the crane should follow, as well as dividing this path into intermediate goals, the control system's AI can also independently define intermediate goals (waypoints). Once these intermediate goals have been reached, they can be deleted.

[0019] The autonomous individual behavior of the crane during a process step can be determined by a hierarchical behavior-based architecture. In the intelligent control system, the desired behavior of the crane in each process step can be nested into a complex strategy, for example, using only two arbitration methods (priority list and sequence), which the crane then executes sequentially. The control system can select the current individual behavior (intention) by taking into account a current individual goal, a specific state in the world model, preconditions, and boundary conditions. Based on the behavior selected in this way, the control system then calculates the control signals (which correspond to specific control commands) and uses these to control the crane, in particular by transmitting them to a crane control system.The control system continuously and sequentially generates control signals or specific control commands to process the intermediate steps and thus execute one intermediate step goal after the next. In a further development of this, the control system can create and / or modify (particularly depending on the situation) sub-strategies that are followed when executing the intermediate step goals.

[0020] The reactive approach mentioned above, however, can be used during a process step to adapt the crane behavior to the situation, for example to avoid collisions or vibrations, as will be explained in more detail.

[0021] According to the invention, the object can also be achieved by further advantageous embodiments according to the subclaims, which are described below:

[0022] For example, as mentioned, the crane can have a gripper mounted on a movable boom of the crane. In this case, the method can provide for an object to be picked up, which is located within a picking area, to be gripped with the gripper and then the object to be moved with the help of the crane to a depositing area and deposited there (with the help of the gripper). The gripper can be mounted or suspended from the movable boom in a suspended manner and thus capable of oscillation, for example by means of at least one cable pull. In such embodiments, a distance, in particular a vertical one, between the gripper and the boom can be changed with the help of the control system, for example by the control system controlling the drive of the cable pull such that a desired distance between the gripper and the boom is set.This means that the control system can not only control the gripping of loads with the gripper, but can also influence the pendulum properties of the gripper by actively controlling the cable pull, i.e. by changing the length of the cable.

[0023] It goes without saying that this method can be used for both loading and unloading goods. In the case of loading a truck, the storage area of ​​the objects would be the receiving area, and the truck's loading area would be the depositing area; in the case of unloading, however, it could be the other way around.

[0024] As will be explained in more detail, a control system according to the invention, including the associated optical sensor system, can be designed in particular as a retrofit kit that can be easily retrofitted to an existing crane that has its own unintelligent crane control system (i.e. one designed without artificial intelligence), without having to make major adjustments to the existing crane control system. For this purpose, it can be provided that the intelligent control system controls the crane via the separate crane control system, and that the control system takes into account the control behavior and / or the time latency of the crane control system. This consideration can consist in particular in the control system mapping the crane control system in a model and adapting the control signals transmitted to the crane control system on the basis of this model.If, for example, the crane control is designed as a programmable logic controller (which can, for example, implement a speed controller), it may be sufficient for the intelligent control system to model the crane control using a restricted acceleration model or a PTI model. The advantage of this approach is that the crane control can be retained, which makes retrofitting the intelligent control system much easier. A preferred embodiment provides that the control system transfers the control signals generated by the AI ​​to the crane control via a user interface that is intended for use with a manual control unit. For example, the control signals can be transferred via a joystick interface on the crane, via which control signals can normally be entered using a joystick as the control unit.Particularly preferably, it can be provided that an existing manual control unit is retained so that a crane operator can continue to have control access to the crane controls via the manual control unit. With this approach, the operator can monitor the autonomous operation of the crane, but can still intervene via the control unit if necessary. Preferably, the crane control system can therefore be designed such that control signals entered via the control unit have priority over control signals generated by the intelligent control system. The human operator can therefore override the control system at any time, which increases safety.

[0025] To increase the speed of autonomous crane control, one embodiment of the method proposes that the intelligent control system controls at least two of the crane's axes separately / independently of each other during positioning movements of the crane, i.e. when the crane approaches a predetermined position. This is because during such rough positioning movements, certain deviations from the pre-planned "Cartesian"

[0026] path are accepted because they can be compensated for later by the control system. In order to increase the accuracy when controlling the crane at the destination (loading or unloading location), it can be provided additionally or alternatively that the control system controls at least two axes of the crane together, i.e. independently of one another, when objects are picked up or put down using a grabber of the crane. With this joint control, current control values ​​for one of the axes of the crane to be controlled are incorporated into the control of at least a second axis of the crane; with separate control / regulation, however, the axis in question is controlled independently of the other axes. An axis to be controlled can in particular be understood as any joint or other movable part of the crane with which a movement / kinematics of the crane can be changed.For example, a drivable swivel joint, a drivable trolley, or a drivable rope hoist. In all of these cases, the control system preferably always controls all axes of the crane simultaneously (i.e., in every control cycle). This approach often allows the crane's maximum speed to be achieved along the individual axes, avoids unnecessary corrections to an originally planned path, and simplifies the development of the individual controllers (for each axis) within the control system.

[0027] The control system can preferably comprise a Gigabit Ethernet network. This allows signals from the optical sensor system to be evaluated in real time, enabling control with a latency of a few milliseconds. Said network can be directly connected to an existing crane control system.

[0028] According to a preferred embodiment, the control system can comprise a world model that contains status information about the crane, in particular current sensory information about the positions and / or movements of the crane's joints. The control system preferably records all information available in the crane about the crane's joint positions and movements and continuously transmits this information to the world model. Such information can be recorded by sensors, for example, using incremental encoders and encoders of the crane.

[0029] The real-time measurement data additionally acquired with the optical sensor system (in particular measurement data from at least one camera and one LIDAR sensor) can be used by the control system to close gaps in the accuracy of the sensors already present on the crane and / or to measure additional information about the "pose" (= position and orientation in relation to the spatial axes) of a grab of the crane swinging freely on ropes and / or of objects being picked up (for example a truck load) and to also integrate this information into the world model.

[0030] The control system can also evaluate data from an inertial measurement unit (IMU for short), for example designed using a 6-axis position sensor (3 acceleration sensors and 3 rotation rate sensors / gyroscopes, each arranged along the three spatial axes), which is mounted on a moving part of the crane, in particular on a grab of the crane. This IMU can also contain a compass or a GPS sensor in order to compensate for sensory drift and to be able to determine the absolute position in relation to the environment / world. In this way, additional status information can be obtained which can be integrated into the world model and / or taken into account when generating the control signals by the control system.This approach can improve the accuracy of the crane's recorded kinematics, particularly for cranes that exhibit variable torsion of their supporting structure due to high load-lifting stresses and / or large levers, as is often the case with construction cranes. The IMU can therefore be configured and mounted on the crane in such a way that the IMU can detect instantaneous torsion of a crane's supporting structure and / or a sensor mount (which, for example, carries an optical sensor).

[0031] As mentioned, the world model preferably further comprises at least one piece of information about a current pose (technical term for "position and orientation" in relation to three different spatial axes, in particular three Cartesian axes x, y, z; a "pose" thus describes six dimensions, for example x / y / z position plus respective rotation about the respective axis) of a gripper of the crane and / or of an object being picked up. In simple terms, the world model can thus know, for example, the current "posture" that the gripper is currently assuming in space.

[0032] To improve accuracy and increase safety, the control system can retrieve information from the world model and take this into account when generating the control signals. The world model serves to coherently represent the current state of the crane, particularly in the sense of a "digital twin"; based on the world model, the current state of the crane can be visualized for a user, for example, using a CAD model of the crane. The world model can therefore include parameters about the current state of the crane (kinematics) as well as about environmental factors (position and / or movement of obstacles; fixed target areas and approach positions, etc.). The control system can also be set up to continuously collect, filter, and, if necessary, predict information in order to

[0033] to continuously update the world model. The control system can also use at least two (but possibly dozens) different coordinate systems (each defining a vector space) to process sensory information (regarding the kinematics of the crane and / or the optical sensor system) and to take this into account when generating the control signals. For example, polar coordinate systems, Cartesian coordinate systems and so-called joint spaces can be used, the latter being suitable for representing angles (and angular velocities) of joints of the crane. Furthermore, at least one of these coordinate systems can map a reference system of the crane.

[0034] The control system can further transform data between the at least two different coordinate systems, for example, using a transformation tree, in particular such that information present in a first coordinate system is transferred to a second coordinate system. For example, status information of the crane, which is typically present in a joint space, can be converted by the control system into Cartesian position data, which can then be processed more easily by the control system.

[0035] In particular, the world model explained can include such a "transformation tree". With this kinematics method borrowed from robotics, the relationship between all joints and parts of the crane and the sensors attached to them can be described (forward and inversely) and updated according to the movements currently recorded by the sensors. The transformation tree thus makes it possible to transform coordinates from the reference system of each sensor and joint of the crane into any other reference system. For example, measured joint angles can be converted into a position in Cartesian coordinates or a target position in Cartesian coordinates can be converted into a position and movement in the joint space.

[0036] In addition to camera data from the optical sensor system, the control system can also evaluate data from an optical distance sensor, in particular a LIDAR sensor, in order to create at least one height map from this. The control system can preferably be set up to automatically create height maps from incoming measurements from the distance sensor (e.g. 3D point cloud of a single measurement) and to update these continuously. These height maps can also be part of the world model. The height maps can in particular cover an environment that the gripper has to move through, in particular in the picking or setting down area. The control system can use the height maps, for example, to avoid collisions, to plan the path (the path along which the crane is to move in space) and to determine a setting down and picking location.The control system can first plan the route to be followed by the crane, taking the elevation map into account, and then, independently of the planning, check whether the grab is currently on a collision course (for example due to a deviation from the plan, an inaccuracy, or a defect); if this is the case, the control system can adapt the current path of the grab and / or brake it.

[0037] It can also be provided that the control system autonomously recognizes the recording area within a live image that is captured by the optical sensor system. This can be done on the basis of optical markers (which is useful, for example, in closed halls, but less so outdoors) or without markers. The artificial intelligence (AI) can then segment the live image into several image areas, whereby the image areas can be delimited by means of a respective contour line. Such contours are helpful for efficient control. The AI ​​assigns a specific predefined object class to each of the image areas segmented in this way; in particular, the AI ​​can differentiate between objects of interest (loading area, gripper, load) and objects of no interest (objects outside the loading area, other parts of the truck).When unloading wood from the back of a truck, these object classes can include, for example, the truck's loading area, the crane's grab, posts as boundaries of the loading area, the truck's cab, and tree trunks. For example, the artificial intelligence can detect and localize the object to be captured (individual tree trunks or bundles of wood) within one of these image areas, to which the predefined object class "log stack" has been assigned, for example. Typically, the AI ​​segments up to 12 objects within the live image. This approach avoids, for example, error-prone edge detection (which can be inaccurate due to changing lighting conditions, shadows, local occlusions, damage, or varying surface properties and textures), as is often used in other state-of-the-art approaches.By limiting the respective image area and robustly identifying the image points (pixels) belonging to the respective object, on the basis of which a previously known object can then be located more precisely and kinematic dimensions of the object can possibly be determined, image processing is made considerably easier, faster and more reliable. The detection of the recording area within the live image can be carried out using image recognition, for example, whereby stationary markers can also be used to mark out the recording area. If, for example, a truck is to be unloaded with the crane, the truck can first drive into the recording area marked out by several markers and parked there. The control system can then use the optical sensor system to first record the recording area and then identify the parked truck within it.The segmentation can then result, for example, in the control system recognizing the loading area of ​​the truck as an image area within the recording area and recognizing and locating tree trunks (as a further image area) within this image area as objects to be unloaded. The AI ​​is trained to recognize that image area and, using the segmentation, to precisely determine the spatial location in which the objects being picked up are currently located. It goes without saying that the current position of this image area must therefore be determined anew by the AI ​​each time depending on the situation. The control system can also use other information from the world model for this purpose, for example the current position of the crane joints and / or the crane kinematics to limit and calculate the respective image areas.

[0038] Preferably, the segmentation is based on a specific unloading situation within the receiving area and / or a specific loading situation within the depositing area. The artificial intelligence can perform the segmentation situation-dependently and each time anew based on the live image, with knowledge of typical (image) situations that occur in this specific situation. The AI ​​has learned this knowledge using training image data.In contrast to previously known systems, the system does not simply use a consistently identical rectangular search window ("region of interest") with Cartesian coordinates to restrict the image area within the live image in which the object to be detected is presumably located; rather, it is proposed that intelligent image segmentation based on training data be used to select a suitable image area within the live image, which can change from situation to situation (for example, the loading area of ​​the truck can be different or the exact parking location of the truck can vary within the recording area), and to identify and localize the object within this image area.After localization, the control system can then position the gripper precisely over the object in the correct orientation using the movable boom and / or the described cable pull and grip the object securely, whereby the current position and / or orientation of the gripper is also determined by the AI ​​using image segmentation.

[0039] The detection of the recording area within the captured live image can alternatively (and preferably) be markerless, for example by the control system dynamically placing a corresponding search window in the live image and identifying the recording area using object recognition. In this case, the position of the search window can be determined by the control system based on knowledge of the current kinematics of the crane (in particular based on joint angles detected by sensors and / or a geometry of the crane). To support this, a projection matrix of a (preferably calibrated) camera of the optical sensor system can be taken into account by the control system when determining the position of the search window.In addition, the control system can also take into account information from the explained world model; for example, the AI ​​can obtain the exact position of a road on which the goods to be unloaded are located from the world model as "world coordinates".

[0040] To make segmentation particularly reliable, it is preferably provided that the artificial intelligence performs the segmentation based on parameters that the artificial intelligence has learned from training image data that has been subjected to human evaluation. This training image data can comprise only a few thousand images, each of which was recorded in a typical unloading or loading situation within the receiving or unloading area. In each case, a human then performed the segmentation, on the basis of which the AI ​​later learns the correct image segmentation, provided that a suitable pre-selection of the training image data has taken place (before the human evaluation). This limits the effort required to train the AI.

[0041] If the crane is to be used in a new area, this type of training can be repeated, allowing the control system to be adapted to the new unloading or loading situation. Furthermore, the artificial intelligence training can be run iteratively several times, even in the same unloading or loading situation, to improve the accuracy and reliability of object recognition / image segmentation and thus the intelligent control system.

[0042] The optical sensor system can comprise a (particularly 2D) camera (primary sensor) which is mounted on the crane (for example on a trolley of the boom) in such a way that a downward view of the gripper and / or of objects to be picked up with the crane is possible. The bird's eye view of this camera makes it particularly easy to observe the vibration behavior of the gripper (by optically measuring the angular deflection of the gripper) and, if necessary, to actively control it, for example in order to implement an optical camera-based and markerless anti-sway control of the gripper with the control system. On the other hand, the gripping of objects to be picked up can be controlled particularly easily and robustly.

[0043] In order to increase safety against collisions of the gripper with obstacles during loading or unloading of objects, it can preferably be provided that a further camera of the optical sensor system is mounted on the crane in such a way that this, in particular second, camera can take images of side views of the gripper (i.e. in particular with a horizontal viewing angle of the gripper). For example, such a secondary sensor can be mounted on a crossbeam of a running gear of the crane. The secondary sensor / the further camera allows in particular a finer control of a running gear of the crane and a height of the gripper.

[0044] In order to enable the fastest possible acquisition of the data from the respective optical sensor of the optical sensor system, one embodiment provides that each optical sensor (camera, LIDAR, etc.) has its own processing unit with a direct interface to the sensor, which is set up to process the sensor signals of this sensor. In this way, individual "sensor nodes" can be formed, which process the sensor signals independently of one another with a high bandwidth, for example within a control cycle of 50 ms (which corresponds to 20 control cycles per second). The respective sensor processing unit then simply transmits the results of the sensor signal processing to the control system / into the respective control loop. Depending on the sensor used, the processing units can be equipped with a conventional CPU or, for example, a "neural

[0045] Inference unit" (e.g.: Jetson from NVIDIA) can be implemented so that sensor data can be processed in a bandwidth range of 1 Gbit / sec per sensor; with better network technology, however, higher bandwidths are also possible, e.g. 10 Gbit / s or more, depending on requirements. This approach enables, in particular, easy horizontal scaling of the number of sensors in the optical sensor system, depending on the application.

[0046] A preferred embodiment provides that the live images from at least one camera of the optical sensor system (i.e., in particular, live images from the described primary and / or secondary sensor) are evaluated by the control system using a multi-layer intelligent image processing chain based on artificial intelligence (in particular the one mentioned above). With this approach, the required information can be extracted from the images particularly reliably and relatively easily. It is particularly advantageous if the respective camera is precalibrated. For example, intrinsic and extrinsic camera parameters can be calibrated. Thus, it is advantageous, for example, if the control system knows the exact position of the camera on the crane (extrinsic parameter) and / or an optical distortion of the camera's optics (intrinsic parameter).

[0047] Preferably, a first layer of the image processing chain is formed by an (artificial) neural network (NN), in particular in the form of an "all-convolutional net" or a "U-Net"-like network topology, which classifies image areas of a (current) live image into different object classes. The result of the first layer can therefore be simple 2D information, for example the contour line of the respective image area that has been assigned to a specific object class. This classification can be carried out in particular with the aid of the described segmentation. Within an object class (gripper), subclassifications (such as "gripper filled" or "gripper empty") can also be carried out by the neural network.This neural network is not a pre-trained network, but is preferably trained based on specific crane data and training image datasets captured by the optical sensor system in a real environment. These datasets can also be supplemented with machine data from the crane's control system and / or with log data from the AI. The individual training steps (data collection; curation; annotation; network training; network evaluation) can be repeated iteratively.

[0048] This intelligent approach enables greater accuracy and robustness to lighting, weather and environmental characteristics and other changes that is not possible with conventional methods alone. For example, an entire image from a camera mounted on a crane trolley can be segmented by the NN in a first step. This means that for each pixel it can be decided which object class (e.g. grab, grab suspension, wood in the grab, wood load on the loading area, truck cab, background, etc.) the respective pixel belongs to. The result is a segmented image (segmentation image) in which the shape and / or orientation of the classified objects can be determined in a subsequent step much more easily and robustly than in the unprocessed original image.

[0049] For this purpose, a second layer of the image processing chain can be set up to localize objects within the segmentation image produced by the first layer, i.e., within the respectively classified and / or segmented image areas, based on their shape and, preferably, to track them (spatially) in a continuous chain of live images (optical tracking). With this approach, the objects are identified based on their respective appearance and not on the basis of markers. The second layer can therefore perform shape tracking of objects that were classified in the first layer. For this purpose, Markov Random Fields (MRF) and particle filters, for example, can be used. In particular, "classic" symbolic methods from the fields of computer vision and probabilistic robotics can be used for this purpose. These find objects in the segmentation image based on their shape and, if necessary,track .

[0050] A third layer of the image processing chain can be configured to extract further kinematic variables from the objects localized by the second layer. For example, based on a respective point mass model of an object (in particular of at least one object) classified in a live image and located based on its shape, angular velocities and / or a pose of the object can be estimated. Furthermore, this approach also makes it possible to estimate the object's center of mass. Such process steps can be implemented and predicted, for example, using a Kalman filter. Using such approaches, the control system can, in particular, perform a dynamic estimation of one of the objects (for example, the currently rotating or oscillating gripper including its load). This approach is capable of dealing with occlusions, superpositions, and only partially visible objects.Based on an estimated uncertainty, the control system can decide whether an object has been detected sufficiently well. Estimates that are not physically plausible, for example a sudden movement, are rejected by the control system in this case. To further increase safety, the control system can include a Safety Overseer instance that checks the control signals generated by the artificial intelligence for unsafe behavior of the crane and overwrites the control signals if at least one safety rule is violated. If such a safety rule is violated, the Safety Overseer instance can transfer the crane to a safe state. In this way, a control signal calculated by the control system can be checked for identifiable unsafe behavior before it is sent to the crane control system. For this purpose, a Simplex architecture, which is common in safety technology, can be implemented.For example, the said overseer can monitor compliance with limit values ​​for positions and / or speeds and / or check a current movement of the crane, in particular of the grab, with regard to a possible collision with obstacles recorded in the world model.

[0051] The control system can further comprise a non-linear model-predictive controller, preferably based on a dynamic model and feedback sensor data. Such a controller can ensure that the crane moves autonomously and in a time-optimized manner to a specific destination, in particular without any significant residual vibration of the grab. The control system or said controller can thus consider trajectories (in particular taking the world model into account) as well as the physical limits of the crane, and can thus implement collision avoidance by monitoring safety distances.

[0052] A preferred embodiment of the method provides that the real-time measurement data acquired with the crane's optical sensor system is obtained using at least two different optical measurement methods. With this approach, the control system can, for example, reliably detect a current position and / or orientation and / or a direction of movement of the gripper. The control system can then adapt / regulate the movement of the gripper with corresponding precision, thus enabling reliable and precise control of the gripper's movement.

[0053] In previously known crane control systems, the control of the crane movement is often based on passive state detection, for example via (typically non-optical) sensors on the crane's swivel joints. This approach is also frequently used in industrial robots that are not equipped with optical sensors. However, a prerequisite for precise control in such approaches is typically that the robot arm is designed to be as rigid as possible, so that the robot can use state detection, for example by reading sensors in the joints of the robot arm, to deduce the current position of the tip of the robot arm in space. The actual position of the tip of the robot arm is only calculated and not actively detected by sensors. In other words, such previously known control systems work passively and have no active measuring devices to actively determine the current position.

[0054] In contrast, the optical sensor system of the crane discussed here allows active optical detection of the position of the grab in real time. This is advantageous, among other things, because the accuracy of this active sensor system does not depend on the rigidity of the supporting structure, in particular of the movable boom of the crane. In addition, relative movements between the grab and boom, which are caused, for example, by gusts of wind or by swinging the grab, can be reliably detected and taken into account by the control system when controlling the crane. A first of the two measuring methods described above can, for example, be based on a computer vision algorithm that processes 2D image data. Such an algorithm can calculate absolute spatial coordinates of the object being picked up and / or of the grab, preferably from relative image coordinates and using a projection matrix.The AI ​​can then process these absolute location coordinates to generate control signals.

[0055] Furthermore, in addition or as an alternative to this, it can be provided that one, in particular a second, of the two measuring methods is based on one of the following approaches: a) triangulation using two cameras; b) optical distance measurement, in particular by means of LIDAR (light detection and ranging); or c) use of structured lighting and a camera to calculate depth information.

[0056] If, for example, the first measuring method only provides relative spatial coordinates, these can be converted into absolute spatial coordinates using measured values ​​from the second measuring method. For example, the control system can calculate relative spatial coordinates of objects captured in the 2D image data from 2D image data of a calibrated camera of the optical sensor system, taking into account a projection matrix of this camera. These relative spatial coordinates can then be converted into absolute spatial coordinates by the control system using a depth map determined with an optical distance sensor (preferably using LIDAR) of the optical sensor system and / or taking into account sensor-captured and / or calculated status information of the crane.However, an absolute coordinate can also be calculated from the (respective) relative coordinates by means of measurements carried out with at least two spatially separated cameras of the optical sensor system, for example with the aforementioned primary and secondary camera.

[0057] Preferably, the second measuring method can be used to generate redundant information regarding the location and / or position of the gripper and / or the object being picked up. Since the artificial intelligence processes all of this measurement data, the autonomous control of the crane can be made more precise and reliable. In particular, robust optical tracking of the gripper in space can be implemented in this way. Preferably, it is provided that during the optical tracking of the gripper, at least one non-Cartesian coordinate system is used by the control system, in particular a coordinate system that defines a joint space of the crane.

[0058] By using the two independent optical measurement methods in parallel or simultaneously, redundant optical object detection can be achieved. With this approach, the control system can, for example, detect the spatial position and / or orientation of unstructured objects, such as tree trunks or other loose goods located on a loading area, with high reliability and safety.

[0059] In addition, by using the two independent measurement methods, the control system can also reliably and safely determine the current position and / or orientation of the gripper, which improves the accuracy of the gripper's control. This approach can be particularly useful for implementing automatic collision prevention with the control system. The control system predicts possible collisions of the gripper with objects within the detection area based on the optically detected current movement of the gripper and then preemptively slows the gripper's movement in space to avoid the expected collision.

[0060] It is therefore proposed, in particular, to use two redundant optical measurement methods, each implemented independently of one another using different procedures, in parallel and simultaneously to detect the relative movement of the boom, in particular a relative movement and / or relative position of the gripper, with the optical sensor system. In this way, the safety of the autonomous crane's operation can be significantly increased.

[0061] The use of two redundant optical measuring methods offers considerable advantages, for example in the application described, in which tree trunks whose position, size and orientation are unknown in advance are to be moved with the crane. This is because such applications do not involve the unloading of known goods that are in a fixed orientation and always of the same size, but rather the unloading of unstructured or randomly arranged goods, which considerably increases the complexity of the control system. The redundancy achieved with the different measuring methods significantly improves the safety during crane operation. In addition, the optical sensor system can precisely record the exact deposit location and deposit orientation.

[0062] In previous approaches, in which the position of the crane boom was only determined passively using sensors, there is always a considerable uncertainty regarding the actual current position of the gripper, since this is not actively recorded by sensors. On the one hand, because the crane boom can twist depending on the load, so that there is an inaccuracy in the position of the tip of the crane boom, unlike with a rigid robot arm; and on the other hand, because the gripper can start to swing, for example when there is a rising wind, so that the current position of the gripper can differ from a nominal position without wind. With the approach according to the invention, all of these factors can be actively recorded with a high degree of reliability using the optical sensor system, so that safe operation of the crane can be guaranteed.

[0063] The method can further be designed such that the real-time measurement data comprise the following: a) a 3D point cloud, the points of which are calculated from distances measured with an optical distance sensor of the sensor system; and b) spatial position and / or spatial attitude data, the latter being obtained with the aid of a computer vision algorithm from image data captured by a camera of the sensor system.

[0064] The mentioned optical distance meter can be implemented, for example, in the form of a laser scanner, in particular based on LIDAR.

[0065] Preferably, said camera and the optical distance meter can be mounted on the movable boom. Accordingly, it can be provided that the optical sensor system always moves with the movable boom. In this way, the sensor system can always sense a target area within which the gripper can detect objects. The measured distances as well as the position data (e.g. xyz coordinates) and attitude data (e.g. rotations along the x / y / z axis) can relate to the gripper of the crane and / or the object being picked up.

[0066] In the method according to the invention, it can thus be provided that the control system determines a spatial position of the gripper and / or the object being picked up from the real-time measurement data and takes this into account in the autonomous control of the crane. Preferably, the control system can also additionally determine a spatial orientation of the gripper and / or the object being picked up from the real-time measurement data. This enables the control system to control a movement of the boom and / or a movement of the gripper more precisely.

[0067] In general, it is advantageous for greater robustness of the autonomous operation of the crane if the detection of the spatial position and / or orientation of the grab and / or the object being picked up is carried out markerless, i.e. without the use of optical reflectors. This markerless approach offers considerable advantages in practice because it makes the crane control much more robust against strong sunlight or, for example, when the objects to be detected are very dirty, which is problematic when markers are used on the objects themselves. Other disadvantages of using markers are possible damage to the markers during loading / unloading or obscuring them under the load.

[0068] With the approaches described above, an optimized control method for the crane can be implemented based on markerless optical tracking of the gripper, with the optical tracking being carried out with the aid of the optical sensor system.

[0069] One embodiment of the method provides that the control system determines an inclination of the object being recorded within the recording area using a gradient method that processes image data of the recording area that is captured with the optical sensor system. The gradient method can take into account eigenvectors of a Hessian matrix in a pixel neighborhood. These eigenvectors can be interpreted as the distribution of the gradient to be calculated in the neighborhood and are often referred to as structure tensors. The gradient method can therefore be used to calculate the structure tensors for individual pixels of a 2D image captured with the optical sensor system in order to ultimately determine the inclination of the object. In order to integrate the information from the neighborhood, a box filter can, for example, first be applied to the image.This results in an average direction for each pixel and a neighborhood. For example, to detect bundles of wood within a 2D image, regions with a common direction can be formed and divided into groups. The resulting groups can then be interpreted by the control system as individual bundles.

[0070] A further variant of the method provides that when the object is placed down within the deposit area, the control system detects an actual final deposit location with the optical sensor system and preferably also documents it. This detection can occur precisely when the gripper has released the object. Preferably, an actual final spatial orientation of the object within the deposit area can also be detected with the optical sensor system. The word "actual" is intended to underline that the final deposit location does not necessarily have to correspond to the location at which the gripper released the gripped object. For example, when depositing tree trunks, it often happens that the tree trunk continues to move during deposit and changes its position and / or orientation until it comes to rest at the final deposit location in a final spatial orientation.The recording of the actual final storage location and the final spatial orientation can be helpful, for example, so that the control system can take into account the previously stored object (more precisely its position and orientation) when storing another object within the storage area.

[0071] The crane's control system can also be designed to implement active collision prevention. For this purpose, a virtual safety zone around the object being picked up (by the crane) can be virtually defined by the control system or artificial intelligence within a live image captured by the optical sensor system. This definition of a virtual safety zone within the captured live image can be based on the previously described and possibly already performed segmentation of the live image and / or based on object recognition.For example, it is possible for the control system to use the optical sensor system to detect an outer boundary of a truck's loading area and, accordingly, to define a safety zone at a certain distance from this boundary, within which the gripper can be positioned collision-free in order to pick up objects from the truck's loading area. With such an embodiment, it is further preferred for the control system to preventively brake a movement of the gripper as soon as the control system detects, based on the real-time measurement data, that the gripper would otherwise stray from the safety zone.

[0072] The described active collision prevention can, in particular, rely on measurement data from a LIDAR sensor. Such collision modification is of considerable importance in practice because reaction times are typically extremely short. The first substep in this task is to determine the gripper's position very precisely in space, which is possible with the optical sensor system. The second task is to virtually define the safety zone within the captured live image to ensure that the gripper moves only within this safety zone without the risk of collision.

[0073] The safety area can therefore be re-defined and reset by the control system depending on the situation before each unloading or loading process begins. The size and / or position of the safety area can change in space (particularly relative to the crane if the crane is permanently installed). This approach can be used to address the problem, for example, that when unloading a truck, the position and orientation of the loading area can vary depending on the truck's current exact parking location. If the objects to be unloaded with the crane are delivered by a different type of truck, this can also be taken into account by the control system by adapting the safety area to the new type of truck. This efficiently and reliably prevents the grab from colliding with the truck when unloading the objects from the loading area.

[0074] When the crane is moving ("navigation"), however, the control system can avoid collisions by determining the current position of the grab in a live image captured by the optical sensor system and converting this into absolute 3D position data for the grab, particularly taking into account calibration data from the optical sensor system and / or current status information on the crane's kinematics. A collision check / estimation is then performed based on the determined 3D position data and a height map captured by the optical sensor system (which can be used to detect obstacles in the grab's current path). The control system can also compare this with the "memory" of this height map.This allows a collision with an object outside the current detection range of the sensors to be prevented, even in the case of very long braking distances, provided that this object was detected in a previous pass (carried out by the crane) and integrated into the elevation map. The elevation map can thus be continuously updated and continued as the crane moves, based on new / ongoing measurements.

[0075] In previously known industrial applications that use a light curtain to implement automatic shutdown for collision prevention, the safety zone defined by the light curtain does not change because the entire system is installed in a fixed location and the picking or setting down areas from which objects are to be picked up or set down with a robotic gripper typically do not change / do not move. The solution proposed here differs from these approaches in that due to the precise object recognition, for example of the loading area of ​​a truck, including recording the size and / or orientation of the loading area, the said virtual safety zone, within which the gripper can move collision-free, can be re-determined with exactitude in each situation, namely with the help of the optical sensor system.

[0076] The control system can also be designed to implement active vibration damping of the gripper. For this purpose, the control system can be provided with a

[0077] The movement of the gripper is preventively braked by appropriately controlling a drive device of the crane as soon as the control system detects from the real-time measurement data that the gripper would otherwise execute an oscillation.

[0078] With this approach, it must be taken into account that the grab, particularly when suspended from the boom by means of a cable pull, is a freely swinging member. The length of the cable pull and thus the length of the pendulum formed by the grab suspended from the cable pull can change during crane operation, so that the vibration behavior can also change. By optically detecting the movement of the grab in space with the aid of the optical sensor system, the control system can evaluate the grab's current vibration tendency and intervene accordingly to prevent excessive vibration of the grab. This can significantly improve safety during crane operation because the grab's position can be controlled more reliably. With this approach, the control system can implement an "anti-sway system".

[0079] A particularly preferred embodiment of the method provides that a drive device of the crane, in particular the one described above, is controlled by the control system by means of a so-called input shaper function, with which a passive vibration compensation can be carried out. This passive method serves to avoid

[0080] To induce vibrations during movements of the crane, for example of the undercarriage and trolley. The input shaper function can in particular be intended and mathematically designed to (passively) prevent or at least minimize vibration of the grab relative to the crane boom. It can also be provided that the control system estimates an end point of a current movement of the grab in space on the basis of a (temporal) integration via the input shaper function. The control system can then decide on the basis of this estimate whether the movement of the grab should be continued or braked. A special feature of this method is that it can predict the braking distance, including the extension due to the implemented vibration damping, in each journal and can thus determine the optimal start of the braking process in order to achieve a specific target with the grab.Because the remaining braking distance can also be continuously checked for collisions by the control system, braking can be applied in good time and collision avoidance intervention, thus preventing the coupling of vibrations (preventive). The active process of the crane / grab used in this approach to vibration damping serves to actively dampen residual vibrations and externally induced vibrations (wind, contact with obstacles). The control system can proceed predictively by exploiting the pendulum dynamics of the grab (in the case of a grab suspended from ropes) and, in particular, estimate the phase, period and amplitude of the vibration, for example using a regression method. On the basis of this estimate, the control system can then intervene to control and correct the crane movement that is currently being carried out.

[0081] With this approach, the control system implements a prediction method in which an estimate is made by means of integration via the input shaper function as to where the final end position of a current movement of the gripper, which is currently being detected by the optical sensor system, is to be located in space. The movement is triggered by applying the input shaper method (i.e. by controlling the drive device with the help of the input shaper function). Such an estimate is particularly valuable for the targeted positioning of the gripper, since the braking of the gripper is operated using the input shaper function in such a way that oscillation of the gripper is (largely) avoided. With this approach, oscillation-free and precise braking of the gripper can be implemented completely autonomously, without any input from a user.

[0082] Another problem that is relevant in practice is the avoidance of torsional vibrations that a crane's grab, which is suspended freely on cables, experiences as soon as it is rotated about a rotation axis by means of a drive (e.g. to change the spatial orientation of a load grasped with the grab) or due to gusts of wind. For example, the crane's grab can be suspended by cables from a rotating block (head bloc) (grab suspension), which in turn is rotatably mounted. If the rotating block / grab suspension is rotated, torsional stress initially builds up in the cables between the rotating block and the grab, particularly if the grab has a load enclosed and therefore has a high rotational moment of inertia. This torsional stress or torsional force accelerates the grab, so that it begins to perform a rotational movement.However, due to its inertia, the gripper would continue this rotational movement when the gripper is aligned again with the rotating block, resulting in an oscillating rotational vibration of the gripper.

[0083] To achieve torsional vibration compensation of the gripper in such situations, it can therefore be provided that the control system is set up to optically measure a torsion of cables of a gripper suspension of the gripper with the aid of the optical sensor system (in particular the aforementioned first camera / primary sensor). At the appropriate moment, the control system builds up a torsion in the cables by rotating the gripper suspension, which counteracts a momentary rotational movement of the gripper. The control system can preferably accomplish the latter by appropriately controlling a rotary drive, with which the gripper can be rotated relative to the gripper suspension (thereby causing a rotational movement of the gripper suspension relative to the gripper). The rotational movement of the gripper can also be detected with the optical sensor system, or for example by means of an IMU on the gripper.By means of the counter-torsion generated, the control system can actively decelerate a momentary rotational movement of the gripper. In particular, the gripper can be stopped / stopped in a desired rotational end position without vibration.

[0084] This active torsional vibration compensation implemented by the control system can therefore take into account optical measurements of the current torsion of the ropes as well as the rotation speed of the grab suspension (detected by sensors or optically) in order to reduce the torsion at the appropriate moment (by turning the rotating block) while taking the dynamics into account and thus to stop the rotation of the load precisely in the desired final rotation end position. If a rotation with the grab through a certain angle of rotation is to be carried out without vibration, the control system initially builds up torsion (center) by twisting the rope suspension in order to start the rotation. To slow down the rotation, torsion is then built up in the opposite direction. The sign of the respective control signals can be reversed in the process.

[0085] It is further proposed that the crane is not operated only partially autonomously as has been the case up to now, but is essentially controlled by a user using an operating unit such as a joystick, so that, for example, a certain direction of travel or speed is specified by the user, but rather via abstract tasks that the control system or the crane processes completely autonomously. For example, the crane can be informed via an appropriate interface that it should process a certain task, such as unloading a certain lorry full of tree trunks. As explained, the crane has an optical sensor system with which it can precisely detect the loading area of ​​the lorry and the tree trunks on it in a live image and calculate the exact position data of the tree trunks (localization of the objects being picked up).This situation is qualitatively very different from that of a typical industrial robot, which grasps objects in a defined environment that are always in the same place and typically in the same orientation. This is because when the truck approaches the pickup area, the position of the loading area will vary and the size or shape of the loading area can also be different depending on which truck is delivering the tree trunks. The AI ​​must take all of this into account. The technical solution presented here enables full autonomy of the crane even in such situations, so that it can perform the desired task completely autonomously and safely.

[0086] Accordingly, the method can in particular provide for the control system to process a catalogue of tasks by autonomously operating the crane, each of the tasks comprising at least two (or all) of the following steps a) to f):

[0087] ( a ) autonomous detection and localization of a current recording area ;

[0088] (b ) autonomous detection and localization of a

[0089] Loading or storage area within the receiving area in which objects to be picked up are stored;

[0090] ( c ) autonomously detecting and locating a suitable or assigned storage area in which the objects can be deposited;

[0091] (d) autonomous detection and localization of a

[0092] Loading or storage area within the storage area in which the objects can be safely stored;

[0093] (f) autonomous loading of a loading area, in particular an additional loading area, within the storage area;

[0094] In other words, it is proposed that the control system's AI independently determine pick-up positions within the pick-up area and / or unloading positions within the drop-off area based on the real-time measurement data. The respective detection and localization can preferably be performed based on the real-time measurement data acquired by the optical sensor system. Of course, other sensor data, such as crane status data, can also be taken into account by the AI ​​when completing such tasks.

[0095] With this approach of controlling the crane via tasks, it must be considered that the driving positions to which the crane's gripper is to be moved autonomously are not predetermined (as is common with industrial robots, for example), but can change depending on the specific application situation. For example, if a new truck delivers tree trunks, these driving positions can change, which must be taken into account by the control system.

[0096] To achieve the stated object, the invention also includes

[0097] Features of the independent device claim are provided. In particular, an autonomously operable crane is proposed according to the invention to achieve the object. This comprises a drive device for driving a movement of a movable boom of the crane, an optical sensor system for (at least indirectly) detecting movements of the boom, wherein the optical sensor system can be configured in particular to detect a movement and / or position of a gripper mounted on the boom, and a control system (according to the invention) for controlling autonomous operation of the crane.The crane is distinguished from previously known cranes in that the control system is based on artificial intelligence, and the artificial intelligence is configured to process real-time measurement data obtained with the optical sensor system and convert it into corresponding control signals with which the crane can be controlled autonomously. The control system and the optical sensor system of such a crane can be configured as described above.

[0098] The drive device can be implemented, for example, by means of electric servo motors and / or by means of hydraulic and / or pneumatic actuators.

[0099] Artificial intelligence (AI) can be designed to implement a machine learning method. For example, AI can be implemented using a neural network, which in turn is implemented using an electronic circuit.

[0100] This crane can, of course, be used particularly advantageously if the control system is configured to carry out a method according to the invention as described above or according to one of the claims directed to such a method. As already explained, the optical sensor system of the crane can comprise a camera configured to record live image data, as well as an optical distance meter for determining a current distance between the crane and the object being picked up and / or between the crane and the gripper.

[0101] If the crane is used indoors, structured lighting can also be used under certain circumstances. In this case, the optical sensor system can comprise a lighting unit for generating this structured lighting. The structured lighting can be used, for example, to illuminate a target area in which the object to be picked up is located and / or in which the gripper can be positioned. When structured (particularly invisible) lighting is used, the sensor system can comprise an additional camera with which the target area illuminated by the lighting can be recorded. This camera can then be used to obtain additional depth information from the target area based on the structured lighting detected by the sensor. This can improve the accuracy of localizing the object to be picked up and / or the gripper.

[0102] The described movable boom of the crane may include a lifting mechanism with which a load can be lifted. However, this is not mandatory; for example, it is also possible to raise the boom directly to lift and move loads.

[0103] The boom may further comprise a gripper as described above, with which a load can be gripped in order to subsequently move the load with the crane. Accordingly, the drive device may comprise a drive for operating the lifting mechanism and / or a drive for operating the gripper. In such configurations, it is naturally advantageous if the control system is configured to control the lifting mechanism and / or the gripper.

[0104] To achieve the object, a retrofit kit is finally proposed which is intended for integration into an existing crane which has its own (unintelligent) crane control system. This retrofit kit comprises an optical sensor system which can be designed in particular as described above and can be installed on the crane; as well as an intelligent control system (according to the invention) which is constructed on the basis of artificial intelligence and is designed to process real-time measurement data from the optical sensor system and convert it into corresponding control signals with which the crane can be controlled autonomously via the existing crane control system. It is understood that the control system can be designed in particular as described above, i.e. the control system can in particular be designed to carry out a calculation / control / regulation / consideration / evaluation etc. as described above.and / or carry out a procedure as previously described.

[0105] Furthermore, the control system can have an interface for transmitting the control signals to the crane control system. This interface can be configured for use with the described user interface (already present on the crane).

[0106] The invention will now be described in more detail with reference to exemplary embodiments, but is not limited to these exemplary embodiments. Further developments of the invention can be derived from the following description of a preferred exemplary embodiment in conjunction with the general description, the claims, and the drawings.

[0107] In the following description of various embodiments of the invention, elements which correspond in their function are given the same reference numbers even if they have a different design or shape.

[0108] It shows :

[0109] Fig. 1 is a schematic representation of a control system according to the invention or of a crane according to the invention,

[0110] Fig. 2 is a schematic representation of an unloading situation in which the crane from Figure 1 is used,

[0111] Fig. 3 and 4 a possible implementation of a control system according to the invention,

[0112] Fig. 5 Images captured and generated by the sensor system of the control system, taken with a side view of the crane's grab,

[0113] Fig. 6 shows an active torsional vibration compensation that can be carried out with a control system according to the invention,

[0114] Fig . 7 a detailed view of Figure 8 and

[0115] Fig. 8: Images captured and generated by the control system's sensor system, taken from above onto the crane's grab.

[0116] Figure 1 schematically shows the components of a control system 4 according to the invention as part of a crane 1. The control system 4 is based on artificial intelligence (AI), which processes real-time measurement data from an optical sensor system 5 mounted on the crane 1 and converts them into corresponding control signals. With the aid of a camera 10, the control system 4 can actively detect the movements of a gripper 6 mounted on a movable boom 3 of the crane 1 and then have a controlling effect on a drive device 2, with which the movements of the boom 3 and the gripper 6 are effected. Optionally, the movements (as in the figure

[0117] I illustrated ) with an additional optical distance meter

[0118] II can be detected by sensors.

[0119] Figure 2 schematically illustrates a typical work situation in which the crane 1, with the aid of its movable gripper 6, is to pick up an object 7 that is located within a picking area 8, whereby only partial components of the crane 1 from Fig. 1 are shown schematically. The control system 4 autonomously detects the picking area 8 with the optical sensor system 4 and, by means of a multi-layer image processing chain that processes live images 17 from the camera 10, also the exact spatial position and orientation of the object 7 to be picked up and of obstacles within the picking area 8. The control system 4 then moves the gripper 6 collision-free into the picking area 8 and there grasps the identified and localized object 7 in order to transport it to a deposit location within a deposit area 9 with the aid of the crane 1.The optical sensor system 5 can also detect the actual final deposit location 12 at which the object 7 finally comes to rest after being deposited by the crane 1.

[0120] Figure 3 illustrates a possible system architecture of a control system 4 according to the invention. Figure 4 provides the corresponding terms in German. The central process, referred to as the core, which is executed by the control system 4, carries out the individual processing steps (receiving sensor data, processing sensor data, updating the world model, querying information from the world model, decision-making, safety check using a safety overseer instance, generating control signals) sequentially in each control cycle. A world model, continuously updated by the control system using sensor data, serves as a central element in which the current state of the crane and its environment is coherently represented.The processing of the individual signals provided by the sensors of the optical sensor system 5 is outsourced to several "sensor nodes," each of which evaluates the sensor signals from the respective sensor (e.g., first and / or second camera; LIDAR, etc.) in parallel. Each sensor node can have its own processing unit with a direct interface to the respective sensor. The results of the sensor signal processing provided by the respective processing unit are synchronously integrated into the central processing in the core before decision-making begins within a control cycle. The central processing and command / control signal generation within the control system 4 is thus organized sequentially and synchronously, as in a classic control loop; sensor information from the optical sensor system 5 and other existing sensors (e.g., on swivel joints) of the crane 1 is processed and subsequently integrated into the world model.Then, a decision is made regarding a target to be pursued currently and, from this, an individual behavior to be carried out by crane 1, and from this, a concrete control command (transmitted by means of the generated control signals) is generated.

[0121] As can be seen in Figures 3 / 4, the control system 4 can transfer the generated control signals via a crane communication layer to a crane control 16 of the crane 1, which was already implemented in the crane 1 by means of a programmable logic controller (PLC). It is understood that the entire control system 4, including the associated optical sensor system 5, can therefore be easily retrofitted to an existing crane 1 with a non-intelligent crane control 16, which can be controlled, for example, via an operating unit or another human-machine interface.

[0122] Figure 5 illustrates in the left half a live image 17 which was recorded with a second camera 10 (secondary sensor) of the optical sensor system 5, this camera 10 enabling a side view of the grab 6 of the crane 1 while it is unloading a truck 23. By means of a neural network, the control system 4 classifies numerous image areas 25 within the live image 17 into different object classes, so that a segmentation image 22 as shown in the right half of Figure 5 can be obtained, in which only image areas 25 of interest within the image are segmented and associated pixels are marked using the respective object class (number).The object classes here are the grab 6, the load / objects 7 to be picked up by the crane 1 (namely the bundle of logs), the chassis 24 of the truck 23 (from which the position of the loading area 15 can be determined) and posts 26 as boundaries of the loading area 15 of the truck 23, which represent possible obstacles.

[0123] Figures 7 and 8 illustrate a similar live image 17 and a segmentation image 22 calculated from it by the AI ​​of the control system 4, in which a plurality of image areas 25 are again segmented, in each of which objects to be localized, such as the grab 6, the logs as cargo, or the chassis 24 of the truck 23, have been recognized and classified accordingly. The contour lines around the respective image area 25, which the AI ​​has marked out to delimit the image areas 25, are clearly visible in the right-hand segmentation image 22. Here, the AI ​​has hidden objects that are not of interest (but are nevertheless recognized), such as the driver's cab at the right end of the truck 23 in the segmentation image 22.

[0124] Figure 6 illustrates a possible embodiment of a crane 1 according to the invention, in which the grab 6 is suspended via a grab suspension 19 by means of steel cables 18 on a trolley 21 which is mounted on a boom 3 of the crane 1 and can be moved along the longitudinal axis of the boom 3. A camera 10 of the optical sensor system 5 is mounted on the trolley 21 (or, for example, on a boom mounted on the trolley 21), with which the grab 6 can be observed from above.

[0125] The gripper 6 is mounted on a head block 20 of the gripper suspension 19 or is carried by it, wherein the head block 20 is suspended from the trolley 21 via the steel cables 18. The head block 20 comprises a rotary drive 28, by means of which the gripper 6 can be rotated about a rotation axis 27 defined by the rotary drive 28. The rotary drive 28 can preferably be designed hydraulically or pneumatically, so that only an electrical power supply needs to be led from the trolley 21 to the head block 20, because the hydraulic / pneumatic rotary drive 28 then has its own electric motor for generating a hydraulic / pneumatic operating pressure.

[0126] As indicated by the block arrows, the vertical height of the head section 20 and thus of the gripper 6 can be adjusted by means of a lifting mechanism 14 designed as a cable drive. Furthermore, the control system 4 can specifically adjust the position of the gripper 6 by pivoting a movable boom 3 of the crane 1 (if the crane 1 is designed as a slewing crane) or by moving a crane bridge of the crane 1 (in the case of a gantry or overhead crane). Furthermore, the position of the gripper 6 can also be adjusted by moving the trolley 21.

[0127] If the gripper 6 is set in rotation about the rotation axis 27 by means of the rotary drive 28, the gripper 6 can transmit a rotational force to the head part 20 due to mass inertia forces (particularly when the approximately 6t heavy gripper 6 has gripped a load of 30t, for example), so that the (relatively light) head part 20 is rotated about the axis 27 relative to the (non-rotating) trolley 21. In this case, however, a restoring torsional force builds up due to the cable suspension 19, which acts via the steel cables 18 on the head part 20 and thus also on the gripper 6.

[0128] The control system 4 now uses the camera 10 to optically measure the current twist that exists between the head section 20 and the trolley 21 and from this estimates the direction and rough strength of the acting torsional force that is transmitted via the cables 18 of the gripper suspension 19, if necessary taking into account a current cable length of the suspension 19 (i.e. e.g. as a function of the vertical height of the gripper 6 or its vertical distance from the trolley 21). In addition, the control system 4 also uses the camera 10 to record the current rotational speed of the gripper 6 about the rotation axis 27. By appropriately controlling the rotary drive 28, the control system 4 can build up a torsion that acts opposite to the current rotational movement of the gripper 6, so that the rotational movement of the gripper 6 can be specifically braked if the control system 4 wants to move the gripper 6 into a specific rotational position without vibration.With this approach, the control system 4 thus realizes an active compensation of torsional vibrations that the gripper 6 performs around the rotation axis 27.

[0129] In summary, the invention enables fully automatic and autonomous crane control based on an optical measuring system 5, which is preferably designed to be redundant. The control system 4 of the crane 1, which is based on artificial intelligence, takes into account not only passively determined status information of the crane 1, but also measurement data that was actively recorded with the help of the optical sensor system 5. From this, the control system 4 can determine precise information about a current position of the crane grab 6, but also about objects 7 to be grasped with the grab 6 (e.g. tree trunks). With this control and regulation approach based on artificial intelligence, vibration-free positioning of the grab 6 is also possible, as well as the avoidance of collisions between the grab 6 and stationary objects whose position can change depending on the situation.Here, the gripper 6 suspended on ropes 18 as a freely swinging member represents an important technical boundary condition which the control system 4 takes into account in an intelligent manner.

[0130] The proposed control system 4 differs, for example, from previously known industrial robotics systems in that the crane 1 is enabled to autonomously pick up unsorted objects 7 from different pick-up locations and place them at a desired deposit location 12. The proposed optical sensor system 5 enables markerless optical tracking of the gripper 6 and the objects 7 to be grasped, so that safe operation of the crane 1 can be guaranteed even under varying weather conditions.With a crane designed according to the invention, almost any load, objects such as trucks, unloading areas, grabs and obstacles can be detected at a sufficient speed (typically with a temporal resolution of 20-60 Hz) and with high accuracy and robustness, even under changing light and weather conditions, without the need for modification, marking or other restrictions of the system environment and relevant objects. For this purpose, it may be sufficient for the artificial intelligence to have learned to recognize objects of a certain object class ("truck in the form of a semi-trailer truck"), i.e., the training does not necessarily have to be limited to the same object (e.g., a very specific type of semi-trailer truck), so that the AI ​​can also recognize different objects (e.g.,several types of different semi-trailer trucks) within the same object class.

[0131] List of reference symbols

[0132] 1 crane

[0133] 2 drive devices (of 1)

[0134] 3 booms (movable, from 1)

[0135] 4 Control system (to control the operation of 1)

[0136] 5 Sensor system (to detect a movement of 3 / 5)

[0137] 6 grippers (mounted on 3)

[0138] 7 Object (is picked up from 6)

[0139] 8 reception area (there is 7)

[0140] 9 Storage area (7 should be moved there using 1)

[0141] 10 Camera

[0142] 11 optical distance meter

[0143] 12 final storage location (and where 7 is within 9)

[0144] 13 Security area (around 7, within 8)

[0145] 14 Lifting mechanism

[0146] 15 Loading or storage area (on which several 7 are currently stored, which are to be unloaded / picked up with 1 or to which several 7 are to be moved with 1)

[0147] 16 crane control (of 1, for example designed as PLC = programmable logic controller)

[0148] 17 Live image (recorded with 5 or 10)

[0149] 18 rope

[0150] 19 Gripper suspension

[0151] 20 Head block (carries 6)

[0152] 21 trolley

[0153] 22 Segmentation image

[0154] 23 trucks (lorries; transports 7, parks in 8)

[0155] 24 chassis (of 23)

[0156] 25 image area (within 17)

[0157] 26 posts (limit 15 of 23)

[0158] 27 Rotation axis (of 6 / 28)

[0159] 28 Rotary drive / rotation drive (for rotating 6)

Claims

Claims 1. Method for the autonomous operation of a crane (1) with the aid of an intelligent control system (4), - preferably wherein the crane (1) is designed according to one of claims 20 to 23, characterized in - that the intelligent control system (4) is based on artificial intelligence and - that the artificial intelligence processes real-time measurement data from an optical sensor system (5) attached to the crane (1) and converts them into corresponding control signals with which the crane (1) is controlled autonomously, without human intervention.

2. Method according to claim 1, wherein the control system (4) controls the crane (1) mediated by a separate crane control (16) and the control system (4) in this case Control behavior and / or a temporal latency of the crane control (16) is taken into account, - in particular wherein the crane control (16) is designed as a programmable logic controller and / or without artificial intelligence and / or - wherein the control system (4) transmits the control signals generated by the artificial intelligence to the crane control (16) via a user interface which is intended for use with a manual control unit.

3. Method according to one of the preceding claims, wherein the control system (4) comprises a world model which contains status information about the crane (1), in particular current sensory information about positions and / or movements of joints of the crane (1), preferably and at least one pose information which contains a current spatial position and orientation of a gripper (6) of the crane (1) and / or of an object (7) to be picked up, and - whereby the control system (4) queries information from the world model and takes this into account when generating the control signals.

4. Method according to one of the preceding claims, wherein a gripper (6) which is attached to a movable boom (3) of the crane (1), an object (7) to be picked up, which is located within a picking area (8), is gripped and then the object (7) is moved by means of the crane (1) into a deposit area (9) and deposited there, - in particular wherein the gripper (6) is mounted on the movable boom (3) in a suspended manner and thus capable of oscillation, in particular by means of at least one cable pull, and / or wherein a distance, in particular a vertical distance, between the gripper (6) and the boom (3) can be changed with the aid of the control system (4).

5. Method according to one of the preceding claims, wherein the control system (4) - during positioning movements of the crane (1) at least two axes of the crane (1) are controlled independently of each other and / or - when gripping or placing objects (7) with the aid of a gripper (6) of the crane (1), at least two axes of the crane (1) are controlled jointly and thus dependently on one another, - preferably and always controlling all axes of the crane (1) simultaneously.

6. Method according to one of claims 4 or 5, wherein the control system (4) controls the recording area (8) within of a live image (17) captured by the optical sensor system (5) autonomously, either marker-based or markerless, and - wherein the artificial intelligence subsequently segments the live image (17) into a plurality of image regions (25) and assigns a specific predefined object class to each of the image regions (25) thus segmented, - preferably wherein the segmentation is based on a specific discharge situation within the recording area (8) and / or is carried out anew by the artificial intelligence on the basis of the live image (17) depending on the situation and / or - where the artificial intelligence performs the segmentation based on parameters that the artificial intelligence has learned from training image data that has been subjected to human evaluation.

7. Method according to one of the preceding claims, wherein the control system (4) evaluates live images from at least one camera (10) of the optical sensor system (5) by means of a multi-layer intelligent image processing chain based on artificial intelligence, - wherein a first layer of the image processing chain is formed by a neural network which classifies image areas (25) within a live image (17) into different object classes, - preferably wherein a second layer of the image processing chain localizes objects within the image areas (25) based on their shape, - particularly preferably wherein a third layer of the image processing chain extracts at least one kinematic variable, for example an angular velocity and / or pose information and / or a center of mass, from one of the localized objects, in particular in order to enable a dynamic estimation of the object.

8. Method according to one of the preceding claims, wherein the control system (4) comprises a Safety Overseer instance which checks the control signals generated by the artificial intelligence for unsafe behavior of the crane (1) and overwrites the control signals if at least one safety rule is violated, - preferably wherein the Safety Overseer instance brings the crane (1) into a safe state in the event of a violation of a safety rule.

9. Method according to one of the preceding claims, wherein the real-time measurement data are obtained with the aid of at least two different optical measurement methods with the optical sensor system (4).

10. Method according to the preceding claim, - where a first of the two measuring methods is based on a computer vision algorithm that processes 2D image data, - in particular, wherein the algorithm calculates absolute location coordinates of the object (7) to be recorded and / or of the gripper (6) from relative image coordinates and using a projection matrix and / or - where a second of the two measurement methods is based on one of the following approaches: - Triangulation using two cameras (10); - optical distance measurement, in particular using LIDAR; - and / or wherein the control system (4) evaluates data from an inertial measuring unit which is mounted on a movable part of the crane (1), in particular in order to sensor-detect a momentary twisting of a supporting structure of the crane (1) and / or the twisting of a sensor holder.

11. Method according to one of the preceding claims, wherein the real-time measurement data - a 3D point cloud, the points of which are calculated from distances measured by an optical distance sensor (11) of the sensor system (4), and - comprise spatial position and / or spatial location data which are obtained with the aid of a computer vision algorithm from image data which are captured by a camera (10) of the sensor system (4).

12. Method according to one of the preceding claims, wherein the control system (4) determines a spatial position, preferably and a spatial orientation, of the gripper (6) and / or of the object to be picked up (7) from the real-time measurement data and takes this into account in the autonomous control of the crane (1), in particular for controlling a movement of the boom (3) and / or for controlling a movement of the gripper (6), - preferably whereby the detection of the spatial position and / or orientation is carried out markerless, i.e. without the use of optical reflectors.

13. Method according to one of the preceding claims, wherein the control system (4) determines an inclination of an object (7) to be recorded within a recording area (8) using a gradient method which processes image data of the recording area (8) which are acquired with the optical sensor system (4).

14. Method according to one of the preceding claims, wherein the control system (4) when depositing the object (7) within a / the deposit area (9) prefers an actual final deposit location (12) and an actual final spatial orientation of the object (7) within of the deposition area (9), in particular after the gripper (6) has released the object (7), is detected with the optical sensor system (4), preferably and documented.

15. Method according to one of the preceding claims, wherein the control system (4) implements an active collision prevention by virtually marking out a virtual safety area (13) around the object (7) to be recorded within a live image (17) captured by the optical sensor system (4), in particular on the basis of the performed segmentation of the live image (17) and / or on the basis of object recognition, - preferably wherein the control system (4) preventively brakes a movement of the gripper (6) as soon as the control system (4) detects from the real-time measurement data that the gripper (6) would otherwise stray from the safety area (13).

16. Method according to one of the preceding claims, wherein the control system (4) implements an active vibration damping of the gripper (6) in that the control system (4) controls a movement of the gripper (6) by corresponding Controlling a drive device (2) of the crane (1) brakes preventively as soon as the control system (4) detects from the real-time measurement data that the gripper (6) would otherwise execute an oscillation.

17. Method according to one of the preceding claims, wherein a / the drive device (2) of the crane (1) is controlled by the control system (4) by means of an input shaper function which is intended to prevent or at least minimize oscillation of the gripper (6) relative to the boom (3) of the crane (1), - in particular, wherein the control system (4) on the basis of an integration via the input shaper function Estimates the end point of a current movement of the gripper (6) in space and then decides on the basis of this estimate whether the movement of the gripper (6) should be continued or decelerated.

18. Method according to one of the preceding claims, wherein the control system (4) for implementing an active torsional vibration compensation of a / the gripper (6) of the crane (1) measures a torsion of ropes (18) of a gripper suspension (19) of the gripper (6) with the aid of the optical sensor system (5) and, at the appropriate moment, builds up a torsion which acts counter to a momentary rotational movement of the gripper (6) by appropriately rotating the gripper suspension (19), in particular by appropriately controlling a rotary drive (28) with which the gripper (6) can be rotated relative to the gripper suspension (19), - in particular so that the rotational movement of the gripper (6) is braked and / or the gripper (6) stops vibration-free in a desired rotational end position.

19. Method according to one of the preceding claims, wherein the control system (4) processes a catalogue of tasks by autonomously operating the crane (1), wherein each of the tasks comprises at least two of the following steps a) to f): a) autonomously detecting and locating a current recording area (8); b) autonomously detecting and locating a Loading or storage area within the receiving area (8) in which objects (7) to be picked up are stored; c) autonomous detection and localization of a suitable or assigned storage area (9) in which the objects (7) can be deposited; d) autonomous detection and localization of a Loading or storage area within the deposit area (9) in which the objects (7) can be safely deposited; e) autonomous picking up of the objects (7) in the receiving area (8) by gripping within the loading or storage area (15); f) autonomous loading of a, in particular further, loading or loading area (15) within the deposit area (9); 20. Autonomously operable crane (1) , comprising - a drive device (2) for driving a movement of a movable boom (3) of the crane (1) - an optical sensor system (4) for detecting movements of the boom (3), in particular for detecting a movement and / or position of a gripper (6) mounted on the boom (3) and - a control system (4) for controlling an autonomous operation of the crane (1), characterized in - that the control system (4) is constructed on the basis of artificial intelligence, which is designed to process real-time measurement data from the optical sensor system (4) and to convert them into corresponding control signals with which the crane (1) can be controlled autonomously.

21. Crane (1) according to the preceding claim, wherein the control system (4) is adapted to carry out a method according to one of claims 1 to 19.

22. Crane (1) according to one of the two preceding claims, wherein the optical sensor system (4) - a camera (10) for recording live image data and - an optical distance meter (11) for determining a current distance between the crane (1) and the object to be picked up (7) and / or between the crane (1) and the gripper (6) and / or to capture a depth map and / or - a lighting unit for generating structured illumination of a target area.

23. Crane (1) according to one of claims 20 to 22, wherein the boom (3) - comprises a lifting mechanism (14) with which a load can be lifted and / or - a gripper (6) with which a load can be gripped in order to then move the load with the crane (1), - in particular wherein the drive device (2) comprises a drive for operating the lifting mechanism (14) and / or a drive for operating the gripper (6), - preferably wherein the control system (4) is arranged to control the lifting mechanism and / or to control the gripper (6).

24. Retrofit kit for integration into an existing crane (1) which has its own crane control system (16), the retrofit kit comprising: - an optical sensor system (5) which can be installed on the crane (1), and - an intelligent control system (4) which is constructed on the basis of artificial intelligence and is designed to process real-time measurement data from the optical sensor system (4) and to convert them into corresponding control signals with which the crane (1) can be controlled autonomously via the existing crane control system (16).

Images