Method for checking and / or monitoring a load carrying system of a crane, and crane comprising a system for checking and / or monitoring a load carrying system

Abstract

The present invention relates to a method for checking and / or monitoring a load carrying system of a crane which comprises a plurality of components and is provided for handling at least one load. In the method, at least one ToF sensor is used to generate at least one point cloud of a working area in which the load carrying system is located at least in part, wherein the load carrying system and / or the at least one load has one or more retroreflective markings which reflect light emitting from the at least one ToF sensor back onto the at least one ToF sensor at least in part. In addition, in the method, the at least one point cloud is processed and an evaluation is carried out, in which a conclusion is drawn as to whether the at least one load has been correctly handled by the load carrying system. The present invention also relates to a crane comprising a system for checking and / or monitoring a load carrying system of the crane which comprises a plurality of components and is provided for handling at least one load.

Description

[0001] METHOD FOR CHECKING AND / OR MONITORING A LOAD LIFTING SYSTEM OF A CRANE AND CRANE COMPRISING A SYSTEM FOR CHECKING AND / OR MONITORING A LOAD LIFTING SYSTEM

[0002] OF THE CRANE

[0003] The present invention relates to a method for checking and / or monitoring a load handling system of a crane comprising a plurality of components for handling at least one load. In the method, at least one point cloud of a work area in which the load handling system is at least partially arranged is generated using at least one ToF sensor, wherein the load handling system and / or the at least one load has one or more retroreflective markings that at least partially reflect light emanating from the at least one ToF sensor back onto the at least one ToF sensor. In addition, the method involves processing the at least one point cloud and evaluating it to determine whether the at least one load is being handled correctly by the load handling system.The present invention further relates to a crane comprising a system for checking and / or monitoring a load handling system of the crane comprising several components and intended for handling at least one load.

[0004] In the case of cranes with a load handling system, it is difficult to check the load pickup / load release or overall incorrect positioning of the load handling system because, due to the flexural flexibility of the ropes and the degrees of freedom of the mechanical joints, the exact position and orientation of the crane hook cannot be reliably derived from the crane control system or the movement-mechanical structure of the crane, the load to be picked up by the crane can be in a position / orientation that cannot be ideally reached by the crane's load handling device, and at least one hook is not visible to the crane operator.

[0005] Reasons for an unknown position and location of the crane hook are:

[0006] Load sway of the rope-guided load beam. Since the rope suspension is not an ideal load sway, load sway cannot be calculated but can only be observed or measured. Deformed rope due to moment or force loading (flexible rope), spontaneously occurring jamming in the joints of the load-handling device (e.g., between the hook fork and the crane hook).

[0007] Reasons for an unclear position / location of the load (e.g. pouring ladle) lie in the nature of the hall floor, harsh environmental conditions (slag, contamination on the lifting device), the introduction of random force impulses by the crane when picking up / removing the object (e.g. diagonal pull when unhooking), the delivery principle by other means of transport.

[0008] In the worst case, these difficulties can lead to the load falling, which would cause immense damage, for example, in the case of a ladle filled with molten steel at a temperature of approximately 1600 °C, as used in process cranes in a steel mill. The load handling system, including the load handling process, and in particular the process of load handling, transport, and / or unloading, must therefore be carefully controlled and monitored.

[0009] AT 192 857 B describes a method for monitoring the load capacity of a hook using a pressure cell. However, this arrangement also results in a reduction in the receiving recess and requires a very robust design, as the main force flow passes through this pressure cell.

[0010] DE 10 304951 A1 describes a method and device for detecting incorrect loads, preferably for casting or charging cranes. This method measures the weight forces on the trolley of the overhead crane. This allows the load attachment to be detected. However, it cannot be determined whether the ladle is resting on the hook tip, since the weight is the same in this case.

[0011] EP 1 979 259 B1 integrates a piezoelectric element into the crane hook's receiving shell, where it is loaded by the weight of a load carried by the hook via the pin held in the hook's mouth. With this system, it is only detected during the lifting process whether the hook has been correctly engaged. In addition, the receiving cross-section is slightly tapered.

[0012] With existing systems installed in the jaw area, there is generally a risk of destruction or functional impairment due to contamination or heat exposure. DE 10 2013 017 803 B4 therefore detects incorrect positioning during load handling based on the magnetic flux passing through both hooks, which is measured with a magnetic flux measuring device. The proposed concept is more robust than existing concepts because no electronic components need to be integrated into the hook. Proximity sensors must also be integrated to determine the correct position of the retaining pins.

[0013] DE 10 2012 015 095 A1 relates to the hook of a crane with at least one angle measuring unit which determines a deflection of the center of gravity of the hook from a desired position, preferably the vertical direction, as well as to a method for detecting incorrect loads on a crane, preferably a metallurgical crane, in which a deflection of the center of gravity of the hook from a desired position, preferably the vertical direction, is determined by means of an angle measuring unit and the determined measured values ​​are processed and prepared in an evaluation unit. If, for example, a ladle hanger pin is correctly inserted into the hook, a certain angle results during lifting due to the position of the center of gravity. An asymmetric load on the hook causes the center of gravity of the hook to deviate from the vertical direction. To measure the deflection of the hook from the desired position, the vertical axis of the hook is defined by the pivot point and the center of gravity of the hook.The deflection of the hook's vertical axis from the vertical direction can be achieved by a bushing used to suspend the hook. For freely suspended hooks, the vertical axis is usually vertical, because, according to the physical laws of gravity and torque, the center of gravity of a freely suspended hook lies vertically below the hook's suspension point in an unloaded state. When comparing the positions of two hooks of a load-handling device, the difference in the measurement signals indicates that the pan was gripped differently by the hooks.

[0014] WO 2022 / 162066 A1 relates to a lifting device with a hoist rope on which a load-handling device for receiving and lifting a load is provided, and with a determination device for determining slack rope on the hoist rope, wherein said determination device has an inclination sensor for detecting an inclination and / or a tipping speed and / or a tipping acceleration of the load-handling device and provides a slack rope signal if the detected inclination and / or tipping speed and / or tipping acceleration of the load-handling device exceeds a predetermined limit value.Based on this, it was the object of the present invention to provide a method for checking and / or monitoring a load handling system of a crane comprising several components and intended for handling at least one load, with which it can be determined in a quick and reliable manner whether the at least one load is handled correctly by the load handling system.

[0015] This object is achieved with respect to a method having the features of patent claim 1 and with respect to a crane having the features of patent claim 10. The respective dependent patent claims represent advantageous developments.

[0016] According to the invention, a method for checking and / or monitoring a device comprising several components for handling (e.g.The invention relates to a load handling system of a crane provided for picking up, transporting and / or delivering at least one load, in which a) at least one (three-dimensional) point cloud of a working area in which the load handling system is at least partially arranged is generated using at least one ToF sensor (Time-of-Flight sensor), wherein the load handling system and / or the at least one load (which can be picked up and / or picked up by the load handling system) has one or more retroreflective markings (located in the working area) which (each) reflect light emanating from the at least one ToF sensor at least partially back onto the (respective) at least one ToF sensor, b) the at least one (three-dimensional) point cloud is processed, and c) an evaluation (at least one product or object obtained by processing the point cloud) is carried out.Result) is carried out, which determines whether at least one load is handled correctly by the load handling system.

[0017] In step a), at least one point cloud of a workspace is first generated in which the load handling system is at least partially, preferably completely, arranged. This is done using at least one ToF sensor (time-of-flight sensor). For example, at least one LiDAR sensor (light detection and ranging sensor) can be used as the at least one ToF sensor. The at least one ToF sensor can also be referred to as at least one ToF camera (i.e., the at least one ToF sensor can also be at least one ToF camera, for example).

[0018] ToF sensors are 3D camera systems that measure distances using the time-of-flight method. The scene is illuminated with a light pulse, and the camera measures the time it takes for the light to travel to the object and back for each pixel. The time required is directly proportional to the distance. The camera thus provides the distance to the object imaged on it for each pixel. This allows a point cloud of the work area examined with the ToF sensor to be obtained. Spatial points scanned by a ToF sensor can be described by their spatial position and an intensity-reflectivity value.

[0019] A (three-dimensional) point cloud can be understood as a set of points in a vector space that has an unorganized spatial structure ("cloud"). A point cloud is described by the points it contains, each of which is recorded by its spatial coordinates. In addition to the points, attributes such as geometric normals, color values, acquisition time, or measurement accuracy can be recorded.

[0020] The at least one ToF sensor may preferably comprise at least one illumination unit, e.g., an LED and / or a laser diode; at least one optic; at least one (image) sensor that measures the time of flight separately for each pixel; at least one control electronics; and / or at least one unit for calculating the distance from the measured values.

[0021] If the at least one ToF sensor comprises multiple ToF sensors, a (three-dimensional) point cloud can be generated using each of the multiple ToF sensors. These can be visualized and displayed superimposed together. The (three-dimensional) point clouds can also be merged into a common (three-dimensional) point cloud. Preferably, the at least one ToF sensor comprises at least one LiDAR sensor and / or at least one ToF camera. Particularly preferably, the at least one ToF sensor is at least one LiDAR sensor and / or at least one ToF camera.

[0022] Laser sensors can actively illuminate a measurement object to determine the distance and, if applicable, the position of the illuminated object. Various methods can be used for this purpose, such as time-of-flight (ToF), structured light, interferometry, chromatic confocal distance measurement, or triangulation. Only laser sensors that use the ToF method can be referred to as ToF sensors.

[0023] The term LiDAR (Light Detection and Ranging) refers to methods that calculate a distance based on an emitted light beam reflected from an object. Various types of light sources can be used (lasers, LEDs, infrared LEDs (IREDs), edge-emitting lasers (EELs), vertical cavity surface-emitting lasers (VCSELs), etc.). Lasers emit light with wavelengths between 250 and 1600 nanometers.

[0024] Time measurement for the ToF method in laser sensors can be carried out in different ways, for example using the pulse time-of-flight method (dToF), the determination of the phase delay (iToF), or the mixed method HDDM. The dToF pulse time-of-flight method determines the distance directly using a precise time measurement of a short light pulse. By measuring at the speed of light, the measured values ​​from LiDAR sensors based on the pulse time-of-flight method are extremely robust and allow measurement accuracy of a few centimeters, even at distances of several hundred meters. The light pulse is usually generated using a pulsed laser diode. However, it can also be generated using an LED. To achieve a larger scan field, the sensor's field of view can be enlarged by deflecting the light pulse using mirrors or lenses. In addition to complex optics and mechanics, micromirrors or MEMS-based mirrors can be used.With a ToF camera, a single point, a partial scene, or the entire scene is illuminated by a light source at a given time, and the time of flight between the sensor and the object is calculated for each pixel. Depending on the method (rolling shutter, global shutter), the camera provides the distance to the object imaged on it for each pixel simultaneously or sequentially, based on the known speed of light.

[0025] A ToF sensor can therefore be, for example, a LiDAR sensor with at least one light beam that scans the scene sequentially without contact, or a ToF camera that can illuminate a partial scene or the entire scene and measure it at a single point in time. Both device classes can use the ToF principle to determine distances.

[0026] According to the invention, the load handling system (i.e. at least one of the plurality of components of the load handling system) and / or the at least one load has / have one or more retroreflective markings, which (each) at least partially reflect light emanating from the at least one ToF sensor back onto the (respective) at least one ToF sensor. A retroreflective marking is understood to be an element that functions as a retroreflector or that has at least one retroreflective (surface) that functions as a retroreflector. A retroreflector is an element or a surface that reflects radiation (e.g. light) back to its source with minimal scattering. This works over a wide range of angles of incidence, in contrast to a planar mirror, which only does this when the mirror is exactly perpendicular to the wavefront, i.e. has an angle of incidence of zero.The reflection of a retroreflector is therefore brighter than that of a diffuse reflector. In other words, a retroreflector has a higher reflectance than a 100% reflective Lambertian surface. A corresponding definition of a retroreflector can be found, for example, in the DIN SAE SPEC 91471:2023-05 standard. Due to their retroreflective properties, the locations with the retroreflective marking(s) in the generated point cloud differ significantly from the rest of the work area or surroundings. The reflectivity is a value derived from the intensity of the laser beam and can be calculated using the formula: Reflectivity = Intensity • Sensor Distance. 2• Factor. The reflectivity value is independent of distance because it is multiplied by the square of the distance. Therefore, distance does not need to be taken into account in the point cloud when searching for areas (points) with high reflectivity. The retroreflective marking(s) can be arranged, in particular, on joints or joint points at connection points between the multiple components of the load-handling system. This generates high reflectivity values ​​for these points in the ToF scan. The reflectivity value is provided together with the point coordinate. The factor can be used, for example, to bring the obtained value into a specific value range or to compensate for internal sensor effects.

[0027] The retroreflective marking(s) may, for example, be one or more flat (or planar, or two-dimensional) markings or plates, each with at least one retroreflective surface. Alternatively, the retroreflective marking(s) may also be one or more three-dimensional markings.

[0028] Preferably, at least two, more preferably at least three, and particularly preferably at least four, of the multiple components of the load handling system are arranged in the work area. For example, at least the load handling device(s), the fastening means(s) for the load handling device(s), and the crossbeam are arranged in the work area. Most preferably, the trolley is arranged at least partially in the work area, and all other components of the load handling system are arranged entirely in the work area.

[0029] In step b), the at least one (three-dimensional) point cloud is processed. During this processing, at least one product or result (of the processing of the at least one point cloud) can be obtained, e.g. at least one visualized point cloud and / or at least one (visualized) digital twin. Preferably, in step b), the at least one (three-dimensional) point cloud is processed, in which the at least one point cloud is visualized and / or, based on the at least one point cloud, at least one digital twin of at least part of the load handling system and optionally of the at least one load is generated (and visualized). Step b) or the processing can be carried out by means of at least one processing unit, e.g. by means of at least one computing unit.In other words, the at least one point cloud can be visualized by at least one computing unit, and / or the at least one digital twin can be generated (and visualized) by at least one computing unit. Preferably, the processing is performed automatically (by a processing unit, e.g., a computing unit). In other words, the at least one point cloud can preferably be visualized automatically (by a processing unit, e.g., a computing unit) and / or the at least one digital twin can be generated (and visualized) automatically (by a processing unit, e.g., a computing unit).

[0030] In step c) an evaluation is carried out (of at least one product or result obtained by processing the point cloud), which determines whether the at least one load is being handled correctly by the load handling system, i.e. whether there is no incorrect position of the load handling system or of the at least one load relative to the load handling system, which could, for example, threaten the at least one load to fall from the load handling system. Preferably, the evaluation in step c) can be used to determine whether the at least one load is being correctly picked up, transported, released, and / or picked up (or held) by the load handling system. In particular, the evaluation in step c) can be used to determine whether the at least one load is being correctly picked up (or held) by the load handling system.Preferably, in step c), an evaluation of the at least one visualized point cloud (obtained in step b) and / or the at least one (visualized) digital twin (obtained in step b) is carried out, from which it is deduced whether the at least one load is being handled correctly by the load handling system. The evaluation can, for example, be carried out by a person, e.g. the crane operator. For example, the person can view the product or result obtained by processing the point cloud - e.g. the at least one visualized point cloud and / or the at least one (visualized) digital twin - which is displayed on a screen, and deduce from this view whether the at least one load is being handled correctly by the load handling system. Alternatively or additionally, the evaluation can also be carried out using at least one evaluation unit, e.g. using at least one computing unit.For example, the evaluation can be performed automatically (by a processing unit, e.g., a computing unit). The evaluation can preferably be performed manually (i.e., by a human) and / or automatically. For example, the entire step c) or steps b) and c) (and thus the entire method) can also be performed autonomously by a computer system.

[0031] A process analysis system can preferably be used for the evaluation in step c). This process analysis system can be used if a person performs the evaluation and / or if an evaluation device, e.g., a computing unit, performs the evaluation. For example, the evaluation in step c) can be performed by a person, e.g., the crane operator, using at least one evaluation unit, e.g., by means of a computing unit, wherein the at least one evaluation unit comprises a process analysis system.

[0032] The process analysis system can rule-based check the movement speeds, maximum / minimum distances, and / or inclination angles between components for each parameter set and can classify the results with respect to the correctness of the handling process for the load handling system. Furthermore, stored parameter sets for correct relative positions of components of a load handling system according to mechanical equilibrium can also be used for process analysis. For example, a jammed hook that no longer realigns can be detected. The positions and orientations of two crane hooks encompassed by the load handling system can be analyzed relative to each other to ensure the parallelism of the hook movement during picking up or dropping off.The movement data of the load handling system components can be interpreted over time to identify and assess movement speed and dynamics. For example, typical movement characteristics of the individual components of the load handling system can be identified from path / time diagrams and examined for abnormalities, such as swaying of the crossbeam. This means that a warning can be issued even if a dangerous equilibrium situation arises. Many of these tests can be performed redundantly to increase the reliability of the results and to make the rules increasingly safe in terms of the functional safety of a work system. For example, tests can be carried out according to one, several or all of the following rules:

[0033] If the position of the hook jaw center and the pin center of the ladle match within a certain tolerance range of, for example, < 100 mm, the pin is securely in the hook jaw.

[0034] When picking up or unloading the ladle, the hook tip must be guided safely below the spigot, because otherwise the ladle may be turned or pulled into an unclear position upon contact with a crane hook or, in extreme cases, tip over.

[0035] If the tip of the crane hook lifts the pin of the ladle and thus the component group comes into a precarious equilibrium, in extreme cases the pin slips outwards from the tip of the crane hook and the ladle tips over.

[0036] In steps b) and c), for example, based on the at least one point cloud, at least one digital twin of at least one part (or the part arranged in the work area) of the load-handling system (e.g. of at least two or at least three of the plurality of components of the load-handling system), preferably of the (entire) load-handling system, and optionally of the at least one load can be generated, and from the at least one digital twin it can be derived whether the at least one load is being handled correctly by the load-handling system. Based on the at least one digital twin, an equilibrium position of at least two of the plurality of components, preferably of the (or all) components, of the load-handling system can first be determined, and from the determined equilibrium position it can be derived whether the at least one load is being handled correctly by the load-handling system.The digital twin describes the position and orientation of at least one part (of the multiple components) of the load-handling system. Thus, the generated twin also represents the equilibrium position. Thus, it is also possible to directly verify from the shape of the (visualized) digital twin whether at least one load is being correctly handled by the load-handling system, in particular whether it is being picked up (or held). The one or more retroreflective markings can simplify and accelerate the creation of the digital twin, while also improving the accuracy of the digital twin creation.

[0037] Alternatively or additionally, in steps b) and c), for example, the at least one point cloud can also be visualized, it then being possible to deduce from the at least one visualized point cloud whether the at least one load is being handled correctly by the load handling system, in particular whether it is being picked up (or held). This is possible in particular by using the retroreflective markings. For example, the equilibrium position of at least two of the plurality of components, preferably of the (or all) components, of the load handling system can be determined using the retroreflective markings visible in the visualized point cloud, and it can be deduced from the determined equilibrium position whether the at least one load is being handled correctly by the load handling system. Alternatively, for example, a check can be carried out based on the at least one visualized point cloud.It can be determined whether the plurality of retroreflective markings are arranged together in a predefined pattern from one or more viewing directions, and from this it can be deduced whether the at least one load is handled correctly by the load handling system.

[0038] Furthermore, steps b) and c) can also be carried out, for example, in such a way that neither a visualization of the at least one point cloud occurs nor a digital twin is generated based on the at least one point cloud. In this way, a correspondingly configured computing unit can also calculate or derive, using an alternative processing and evaluation variant, whether the at least one load is being handled correctly by the load handling system. The entire steps b) and c) can be carried out autonomously by a computer system, for example by at least one ASIL-tested LiDAR sensor. Subsequently, the entire process can also be carried out autonomously by a computer system, e.g. by at least one ASIL-tested LiDAR sensor.

[0039] By using at least one ToF sensor in combination with one or more retroreflective markings on at least one of the multiple components of the load handling system and / or on the at least one load, it is possible to quickly and reliably determine whether the at least one load is being handled correctly by the load handling system. The at least one ToF sensor can generate a point cloud of the work area contactlessly and with minimal time expenditure. This point cloud contains information on the precise position of the load handling system and the load relative to one another. After processing, this point cloud enables an evaluation to determine whether the at least one load is being handled correctly by the load handling system. Due to the contactless measurement, any influence or obstruction of the load handling system by the measuring sensors can be avoided.In particular, measuring systems attached to the load handling system can be dispensed with.

[0040] The use of one or more retroreflective markers significantly increases the speed and accuracy of processing and evaluation, and thus of the method as a whole. If, for example, a digital twin of the load-handling system and / or the load is generated during processing, this generation can be simplified and accelerated using the one or more retroreflective markers, while also improving the accuracy of the digital twin generation. The retroreflective marker(s) can be used as "landmark(s)" for reliable and precise positioning of bodies and / or joints. On the other hand, the use of retroreflective markers even makes it possible to completely dispense with the generation of a digital twin, which further significantly increases the speed of the method without compromising accuracy.Ultimately, this exploits the fact that the retroreflective markings – due to their increased reflection values ​​compared to the surroundings – stand out clearly in the point cloud generated by at least one ToF sensor (and annotated with reflection values) and clearly distinguish themselves from the surroundings. In this way, specific marked areas can be assigned very easily and quickly, or the position of individual markings and multiple markings relative to each other can be determined very quickly and easily, without the need to know the position of the areas marked with the markings beforehand. The retroreflective marking(s) thus improve the method, particularly in terms of robustness, speed, and reliability.

[0041] The visualized point cloud can be annotated in various ways (using different colors), for example, by distance (between the ToF sensor and the respective point in the point cloud) or by reflectance values ​​(of the points in the point cloud). A point cloud annotated with reflectance values ​​enables particularly easy recognition of retroreflective markings when viewing the visualized point cloud.

[0042] In the method according to the invention, the verification and / or monitoring or detection of whether the at least one load is being handled correctly by the load handling system can be carried out exclusively based on knowledge of the positions of components of the load handling system (or specific parts thereof, such as joints) and optionally also of the load, which are determined contactlessly using at least one ToF scan. Knowledge of the body orientation (x rotation, y rotation, z rotation) of the crane components is not necessary. Therefore, an angle measuring system is not required.

[0043] The position and orientation of the load do not need to be detected, since even without including the load, it can be clearly determined whether the at least one load is being handled correctly by the load handling system. For example, the equilibrium position of the load handling system and / or the position of the retroreflective markings can be used to clearly detect an incorrect position, or to determine whether the at least one load is being handled correctly by the load handling system. To further increase the accuracy of the method, the at least one load can optionally be included in the measurement.

[0044] Preferably, a combination of contour detection and detection of retroreflective markings can enable even more reliable position determination and even shorter latency. The retroreflective markings can be used, for example, to parameterize a 3D geometry model of the body used in the process during the ongoing process so that the geometry fitting process for fitting the 3D model into the point cloud functions optimally. In this case, the body type (e.g., pouring ladle) of the partial point cloud is known. This variant can be used in particular if the body is partially obscured and only an incomplete point cloud is available, or if the dimensional variant of the body type (e.g., a pouring ladle) is unknown. To determine the exact shape of the 3D model, for example, different model variants can be generated by varying the parameters and checked for the best possible match with the point cloud.The best shape can then be used for the further process. This variant can be used particularly when the body's point cloud is complete in the essential areas.

[0045] The crane can be any crane with a load handling system. Preferably, the crane is a process crane, such as those used in the steel industry.

[0046] The at least one load can be any load that can be handled and / or picked up by the load handling system. Preferably, the at least one load is a casting ladle, such as that used in the steel industry. The process crane can preferably be selected from the group consisting of casting cranes, charging cranes, ladle transport cranes, slab transport cranes / slab turning cranes, coil and bundle transport cranes, tong cranes, forging cranes, magnetic traverse cranes, automatic cranes, and combinations thereof.

[0047] The load handling system is preferably a cable-guided load handling system. The method according to the invention is preferably a method for checking and / or monitoring a load handling system of a process crane (in the steel industry) comprising several components for handling (e.g., for picking up, transporting, and / or delivering) at least one load, wherein the at least one load is preferably at least one pouring ladle, e.g., for transporting liquid metals and / or alloys.

[0048] The method according to the invention for checking and / or monitoring a load handling system of a crane comprising a plurality of components and intended for handling at least one load can preferably also be a method for detecting correct picking up of at least one load by a load handling system of a crane comprising a plurality of components.

[0049] The method according to the invention can be used, for example, to check the load handling system by carrying out steps a) and b) (each) once at a specific point in time or at several specific points in time (e.g. during the load handling process and / or the load release process), so that a three-dimensional point cloud is generated for the specific point in time(s) and thus it can be checked for the specific point in time(s) whether the at least one load is being handled correctly by the load handling system, in particular whether it is being picked up or held (or whether there is an incorrect position with respect to the load handling system or the relative position of the load to the load handling system instead).

[0050] The method according to the invention can, for example, also be used to monitor the load handling system by performing steps a) and b), preferably steps a) to c), multiple times in succession during a specific period of time—e.g., during the (entire) load handling process and / or the load unloading process. For example, steps a) and b), preferably steps a) to c), can be performed at least once per second, preferably at least twice per second, particularly preferably at least five times per second, and most particularly preferably at least ten times per second.As a result, one or more images per second and thus a type of video can be obtained with which the position of the load handling system can be continuously monitored so that the load handling process and / or load unloading process can be continuously monitored and it can be precisely determined whether and when at least one load is being correctly handled by the load handling system.

[0051] A preferred embodiment of the method according to the invention is characterized in that the one or more retroreflective markings

[0052] - each contain or consist of at least one (reflective) primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or

[0053] - are each flat (or flat or planar), preferably rectangular, and / or,

[0054] - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or

[0055] - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor, and / or the one or more retroreflective markings each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably 40° to 50°, particularly preferably 44° to 46°, to a main direction of movement of the crane, and / or have a reflection value (determined according to the manual "LiDAR Sensor Livox Horizon", 2019) in the range of 151 to 255, preferably in the range of 160 to 255, particularly preferably in the range of 180 to 255, most preferably in the range of 200 to 250, and / or a retroreflection coefficient of at least 35 mcd / m 2 *lx, preferably at least 150 mcd / m 2*lx, and / or are designed such that a contrast (or ratio) with respect to the coefficient for retroreflection between the retroreflective markings and other surface materials in the crane process is at least 3:1, preferably at least 4:1.

[0056] The developments in assisted and autonomous driving in road traffic are suitable as a paradigm for the use of retroreflective markers in situation analysis in a crane process.

[0057] Cameras are often used to detect road markings because camera images contain rich texture information of the environment. However, cameras are limited by their susceptibility to lighting variations and distortions in the road surface from a bird's-eye view, making them less robust for certain applications.

[0058] LiDAR sensors are less sensitive to varying lighting conditions than cameras and provide a precise 3D representation of the environment. For example, road markings can be extracted from road surfaces using LiDAR point clouds for driver assistance, leveraging the high reflectivity of the retroreflective materials. However, these LiDAR-based methods face the challenge of balancing the need for denser point clouds with the essential requirement for real-time performance.

[0059] To reliably implement machine vision in road traffic, international minimum standards for road markings are recommended to sustainably increase road safety. The 11th edition of the MUTCD (Manual on Uniform Traffic Control Devices for Streets and Highways) recommends a stripe width of 150 mm and a retroreflection coefficient of 150 mcd / m for retroreflective road markings. 2 *lx. In wet conditions, the recommended retroreflection is 35 mcd / m 2 *lx. Innovative road markings already reach values ​​of 1000 mcd / m 2 *lx. The contrast between the markings and the road surface should preferably be 3:1, or even better, 4:1.

[0060] The coefficient of retroreflection (RA) describes the retroreflective performance of a material by measuring the ratio of retroreflected luminance to incident illuminance. It indicates the efficiency of a material in returning light to its source.

[0061] Retroreflection measurements indicate how efficiently a retroreflector reflects light in a specific direction. They do not measure brightness. The measurements are taken at a specific point within the retroreflection cone. They do not measure all the reflected light. The closer or farther the distance to a retroreflector, the more or less light reaches the detector, but the retroreflection measurements do not change.

[0062] There are numerous standard specifications for retroreflective markings in roadway applications worldwide. However, many standards refer to ASTM D4956 (Standard Specification for Retroreflective Sheeting for Traffic Control) or use it as a starting point.

[0063] The retroreflection coefficient can be determined, for example, as follows: To measure retroreflection in the laboratory, a fixed light source is directed at a sample. The light source illuminates the sample with a specific light intensity. Depending on its size, the sample collects some of this light and redirects some of it back to a sensor mounted near the light source. The sensor then measures the amount of returning light to determine the retroreflection coefficient in cd / m². 2 *lx to be determined.

[0064] It is important to note that the sensor does not measure the entire cone of retroreflection, but rather a point within that cone, so it calculates how well a retroreflector returns light in a particular direction, not how much light is reflected by the entire surface.

[0065] LiDAR sensor manufacturers specify company-specific values ​​for the retroreflection coefficient.

[0066] According to the 2019 "LiDAR Sensor Livox Horizon" manual, the reflectance value of an object can in principle range from 0 to 255, with the range from 0 to 150 corresponding to the reflectance of objects in the range from 0 to 100% in the Lambertian reflection model, and the range from 151 to 255 corresponding to the reflectance of objects with retroreflective properties. The at least one (reflective) primer preferably contains or consists of at least one component selected from the group consisting of xylene isomers, hydrocarbons, preferably C6-C7 hydrocarbons, isoalkanes, cyclics, n-hexane, and mixtures thereof. The at least one clear coat protective sealant preferably contains or consists of at least one component selected from the group consisting of acetone, 2-methoxy-1-methylethyl acetate, butan-1-ol, n-butyl acetate, and mixtures thereof.The clear coat protective seal protects the retroreflective glass beads from external influences and simultaneously serves to adhere the retroreflective glass beads to the primer. All materials of the retroreflective marking(s) are preferably suitable for high temperature exposure, e.g., temperatures above 600°C, preferably above 800°C.

[0067] The mean diameter of the retroreflective glass beads can, for example, be determined according to ISO 13320:2020-01.

[0068] The retroreflective glass beads preferably contain or consist of soda-lime glass, and / or preferably have a refractive index in the range of 1.6 to 2.2, and / or preferably have a retroreflection value of at least 500 mcd / m 2 -lx, preferably at least 1000 mcd / m 2 -lx, particularly preferably at least 1500 mcd / m 2 -lx, on.

[0069] The refractive index can, for example, be determined according to DIN EN 1423:2013-03.

[0070] The retroreflection value can be determined, for example, according to DIN EN 1436: 2018-03.

[0071] The one or more retroreflective markers can each have at least one retroreflective surface. The size and shape of the retroreflective marker(s) can be defined depending on the task, and / or can correlate with the scanning pattern of the ToF scanner or LiDAR scanner, and / or can influence the detection latency.

[0072] The load handling system and / or the at least one load has / have at least one retroreflective marking that at least partially reflects light emanating from the at least one ToF sensor back to the at least one ToF sensor. Preferably, the load handling system and / or the at least one load has / have at least two, particularly preferably at least three, very particularly preferably at least five, retroreflective markings that (each) at least partially reflect light emanating from the at least one ToF sensor back to the (respective) at least one ToF sensor.

[0073] Preferably, the one or more retroreflective markings have a minimum width of 150 mm and / or a minimum size of 150 mm * 150 mm and / or a retroreflection coefficient of (at least) 150 mcd / m 2 *lx or at least 35 mcd / m 2*lx. The contrast between the retroreflective markings and the other surface materials in the crane process should preferably be (at least) 3:1, particularly preferably (at least) 4:1.

[0074] A further preferred embodiment of the method according to the invention is characterized in that the plurality of components of the load-handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings, and / or the plurality of components of the load-handling system (or the load-handling system) have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings, and / or the at least one load has at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings.

[0075] For a three-dimensional determination of position and orientation, three (or more) retroreflective markers are preferably used. However, with a priori knowledge of the relative position of the components (e.g., hook hanging over joint under traverse), only one retroreflective marker or two retroreflective markers can be used.

[0076] A further preferred embodiment of the method according to the invention is characterized in that the at least one ToF sensor is attached to the crane and / or is not attached to the load handling system and / or comprises at least two, preferably at least four, ToF sensors and / or at least one LiDAR sensor (light detection and ranging sensor), preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested (Automotive Safety Integrity Level-tested) LiDAR sensor, particularly preferably at least one ASIL-tested LiDAR with "embedded" intelligence orwith embedded intelligence, comprises at least one ToF sensor with a limited field of view, preferably at least one LiDAR sensor with a limited field of view, or is at least one ToF sensor with a limited field of view, preferably at least one LiDAR sensor with a limited field of view, wherein the limited field of view is defined by two angles comprising a vertical angle and a horizontal angle, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°, wherein particularly preferably the vertical angle is a maximum of 120° and / or the horizontal angle is a maximum of 30°. By attaching the at least one ToF sensor to the crane, there is no relative movement between the sensor system and the crane, so that assembly, calibration, measurement and evaluation are simplified.Particularly preferably, the at least one ToF sensor is mounted on the crane, but not mounted on the load handling system (i.e. mounted on a part of the crane that is not part of the load handling system).

[0077] Using at least one ToF sensor with a limited field of view increases the proportion of points that can be assigned to the work area when generating the point cloud.

[0078] The at least one ASIL-tested LiDAR sensor preferably comprises at least two, particularly preferably at least four, ASIL-tested LiDAR sensors.

[0079] The at least one ASIL-tested LiDAR sensor with embedded intelligence preferably comprises at least two, particularly preferably at least four, ASIL-tested LiDAR sensors with embedded intelligence.

[0080] The at least one LiDAR sensor can be a pulsed LiDAR sensor. This can increase measurement accuracy.

[0081] The at least one LiDAR sensor with a limited field of view preferably comprises at least two, particularly preferably at least four, LiDAR sensors with a limited field of view.

[0082] The at least one LiDAR sensor may preferably comprise at least one illumination unit, e.g., an LED and / or a laser diode; at least one optic; at least one (image) sensor that measures the time of flight separately for each pixel; at least one control electronics; and / or at least one unit for calculating the distance from the measured values.

[0083] LiDAR sensors as can be used in the present invention are defined and described, for example, in DIN SAE SPEC 91471:2023-05.

[0084] A LiDAR-based measurement system and method comprising a LiDAR sensor, preferably multiple LiDAR sensors, generates a consistent three-dimensional, time-dependent point cloud of high measurement quality, enabling robust, fast, and reliable determination of the position and orientation of bodies, joints, and retroreflective markings in the work area. The at least one LiDAR sensor can preferably be at least one ASIL-certified (Automotive Safety Integrity Level-certified) LiDAR sensor, particularly preferably with embedded intelligence.In this case, it is particularly preferred that the at least one LiDAR sensor comprises at least one processing unit configured to perform processing of the at least one point cloud, in which the at least one point cloud is visualized and / or at least one digital twin of at least a part of the load-handling system and optionally of the at least one load is generated, and the at least one processing unit is additionally configured to perform an evaluation of the at least one visualized point cloud (annotated with reflection values) and / or the at least one digital twin, in which it is deduced whether the at least one load is being handled correctly by the load-handling system. The entire method can thus be carried out autonomously in this case by the at least one ASIL-tested LiDAR sensor.

[0085] Preferably, the at least one ToF sensor comprises multiple ToF sensors with overlapping, preferably identical, viewing areas. By using multiple ToF sensors with overlapping, preferably identical, viewing areas, redundancy in the results can be achieved, thus increasing the reliability of the results for an automated solution.

[0086] A further preferred embodiment of the method according to the invention is characterized in that the plurality of components of the load-handling system comprise at least one rope, and / or comprise a crossbeam, and / or comprise a trolley, and / or comprise at least one, preferably at least two, load-handling means, wherein the at least one load-handling means is preferably at least one crane hook, particularly preferably at least two crane hooks, and / or comprises at least one, preferably at least two fastening means for at least one (or for the at least one) load-handling means, wherein the at least one fastening means is preferably at least one hook fork, preferably at least two hook forks.

[0087] For example, the traverse can be attached to the trolley via the at least one rope and / or the at least one load-carrying device can be attached to the traverse via the at least one fastening device.

[0088] Particularly preferably, the plurality of components of the load handling system comprise a cross member, wherein the cross member has one or more retroreflective markings (or the one retroreflective marking or one or more of the retroreflective markings).

[0089] Particularly preferably, the plurality of components comprise a trolley, at least one cable, a cross member, at least two load-handling devices, preferably at least two crane hooks, and at least two fastening means for the load-handling devices, preferably at least two hook forks. It is preferred that the cross member has one or more retroreflective markings (or the one retroreflective marking or one or more of the retroreflective markings). For example, the cross member can be fastened to the trolley via the at least one cable, preferably via several cables, and / or the at least two load-handling devices can be fastened to the cross member via the at least two fastening means.

[0090] Crane and load components can preferably be designed so that they assume different center of gravity positions and orientations in the unloaded / loaded state.

[0091] Preferably, at least one load-handling device, at least one fastening device, and the crossbeam are arranged in the working area. Most preferably, the trolley is arranged at least partially in the working area, and all other components of the load-handling system are arranged entirely in the working area.

[0092] If at least one digital twin of at least one part of the load-handling device is created in step b) (ua), the at least one part of the load-handling system preferably comprises at least the at least one load-handling device, the at least one fastening means, and the crossbeam. Most preferably, the at least one part of the load-handling system comprises at least part of the trolley and all other components of the load-handling system.

[0093] A further preferred embodiment of the method according to the invention is characterized in that during the processing of the at least one point cloud in step b), the at least one point cloud is visualized and / or based on the at least one point cloud, at least one digital twin of at least a part of the load-handling system (or of the part of the load-handling system arranged in the work area or of a part of the part of the load-handling system arranged in the work area) and / or of the at least one load is generated, and during the evaluation in step c), an evaluation of the at least one visualized point cloud and / or of the at least one digital twin is carried out, in which it is derived whether the at least one load is handled correctly by the load-handling system.

[0094] In step b), the at least one (three-dimensional) point cloud is preferably processed, in which the at least one point cloud is visualized and / or based on the at least one (three-dimensional) point cloud, at least one digital twin of at least one part (e.g. of at least one or at least two of the plurality of components) of the load handling system (preferably of the entire load handling system) and optionally of the at least one load is generated (and visualized). The at least one part of the load handling system preferably comprises at least the at least one load handling means, the at least one fastening means and the crossbeam. Very particularly preferably, the at least one part of the load handling system comprises at least a part of the trolley and all other components of the load handling system.

[0095] In step c), an evaluation of the at least one visualized point cloud and / or the at least one (visualized) digital twin is then preferably carried out, in which it is deduced whether the at least one load is being handled correctly by the load handling system, i.e. whether there is no incorrect position of the load handling system or of the at least one load relative to the load handling system, which could, for example, threaten the at least one load to fall from the load handling system. It can preferably be deduced whether the at least one load has been / is being correctly picked up, transported or released by the load handling system. In particular, it can be deduced whether the at least one load has been or is being correctly held by the load handling system.

[0096] In steps b) and c), at least one digital twin of at least a part of the load-handling system (e.g., of at least two of the plurality of components of the load-handling system), preferably of the (entire) load-handling system, and / or of the at least one load can thus be generated (and visualized), preferably based on the at least one point cloud, and it can be derived from the at least one (visualized) digital twin whether the at least one load is being handled correctly by the load-handling system. Based on the at least one (visualized) digital twin, an equilibrium position of at least two of the plurality of components, preferably of the (or all) components, of the load-handling system can first be determined, and from the thus determined equilibrium position it can be derived whether the at least one load is being handled correctly by the load-handling system.The digital twin describes the position and orientation of at least one part (of the multiple components) of the load handling system. Thus, the generated twin also represents the equilibrium position. This allows one to directly verify from the shape of the digital twin whether at least one load is being handled correctly by the load handling system.

[0097] Alternatively or additionally, in steps b) and c), the at least one point cloud can be visualized, it then being possible to deduce from the at least one visualized point cloud whether the at least one load is being handled correctly by the load handling system. This is possible in particular through the use of the retroreflective markings. For example, the equilibrium position of at least two of the plurality of components, preferably of the (or all) components, of the load handling system can be determined using the retroreflective markings visible in the visualized point cloud, and it can be deduced from the determined equilibrium position whether the at least one load is being handled correctly by the load handling system. Alternatively, for example, a check orIt can be determined whether the plurality of retroreflective markings are arranged together in a predefined pattern from one or more viewing directions, and from this it can be deduced whether the at least one load is handled correctly by the load handling system.

[0098] In particular, in steps b) and c), both the at least one point cloud can be visualized and, based on the at least one point cloud, at least one digital twin of at least a part of the load-handling system and optionally of the at least one load can be generated (and visualized). An evaluation of the at least one visualized point cloud and / or the at least one (visualized) digital twin is performed to determine whether the at least one load is being handled correctly by the load-handling system. Thus, ultimately, the at least one visualized point cloud and the at least one (visualized) digital twin can both be evaluated and used to check and / or monitor whether the at least one load is being handled correctly by the load-handling system.

[0099] For example, the at least one visualized point cloud and / or the at least one visualized digital twin can be displayed on a screen (of an assistance system). The crane operator can then use the screen display to determine or assess whether the at least one load is being handled correctly by the load handling system. The visualized digital twin can also be overlaid with the visualized point cloud in the assistance system display, allowing for easy verification of the accuracy of the assistance system.

[0100] A digital twin is generally understood to be a virtual representation or digital representation of a tangible or intangible object from the real world in the digital world. The at least one digital twin created is ultimately a (visually representable) calculated model of (at least part of) the load handling system or (at least part of) the multiple components of the load handling system. The at least one digital twin can be visualized within step b).

[0101] A kinematic system is in equilibrium when a moment equilibrium and a force equilibrium are established. A crane's load handling system with rope guidance assumes a stable geometric shape when only tensile forces act, since the chain of rigid bodies (crossbeam, hook fork, crane hook, load) and slack ropes is not dimensionally stable. The equilibrium of a rope-guided load handling system is determined by the load with its own weight. A suspended load changes the equilibrium position of the equilibrium group and can cause a change in the position and orientation of the rigid bodies or the joints between the bodies. These changes in the position and orientation of bodies can be further provoked by specially shaped bodies.An incorrectly attached load also provokes a change in the equilibrium position and thus the position and orientation of the rigid bodies and joints in this kinematic chain. By contactless detection of the position of several components (and / or joint points) of the load handling system, the equilibrium position can be clearly detected and analyzed for incorrect positioning, whereby measuring systems attached to the load handling system can be dispensed with. The position of bodies or joints of the load handling system and / or the load can be determined based on (continuous) ToF scans, which generate a point cloud of the work area, and the subsequent generation of a digital twin, e.g., using a geometry fitting. The retroreflective markings, preferably at exposed locations (e.g., at the joint points), improve the detection process in terms of robustness, speed, and reliability.

[0102] The current equilibrium position of the load-handling system can be determined from the position / position change (x, y, z) of the bodies / joints of the load-handling system. The position / position change (x, y, z) of the bodies / joints of the load-handling system can be used to reliably detect both correct and incorrect load handling, or to reliably detect an incorrect position in the entire load-handling system. The physical principle of equilibrium can be used for this purpose under the special conditions of flexible systems. Optionally, the load can be included in the incorrect position detection.

[0103] A further preferred embodiment of the method according to the invention is characterized in that in step b) based on the at least one point cloud at least one digital twin of at least a part of the load-handling system (e.g. of at least two of the plurality of components of the load-handling system), preferably of the (entire) load-handling system and optionally of the at least one load, is generated, and in step c) based on the at least one digital twin an equilibrium position of at least two of the plurality of components, preferably of the (or all) components, of the load-handling system is determined and from the determined equilibrium position it is derived whether the at least one load is handled correctly by the load-handling system.

[0104] According to a further preferred variant of the method according to the invention, the at least one digital twin is generated with the aid of at least one geometry fitting process, in which at least one three-dimensional model of at least part of the load-bearing system, preferably of the (entire) load-bearing system, is transformed into the at least one point cloud, wherein in the at least one geometry fitting process at least one shape-based (orcontour-based) geometry fitting is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that an at least partial geometric (preferably greatest possible) correspondence is achieved between the at least one point cloud and the at least one three-dimensional model, wherein preferably an average deviation of the points of the at least one three-dimensional model from the points of the point cloud is at most 20 cm, preferably at most 15 cm, particularly preferably at most 10 cm, very particularly preferably at most 5 cm, and / or at least one geometry fitting based on the one or more retroreflective markings is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that one or more regions of the at least one point cloud which are defined by a region defined by the one or more retroreflective markings are transformed into the at least one point cloud in such a way that one or more regions of the at least one point cloud which are defined by a region defined by the one or more retroreflective markings are transformed into the at least one point cloud in such a way thatthe increased reflection caused by the plurality of retroreflective markings are assigned to one or more corresponding regions of the at least one three-dimensional model.

[0105] The at least one geometry fitting process can also be referred to as at least one registration of a three-dimensional geometry with a point cloud (transformation of the rigid 3D geometry so that the distances between the point cloud and the 3D geometry are as small as possible).

[0106] The geometry fitting process can preferably be contour-based (body appearance) and / or based on the retroreflective marker(s). Contour-based or shape-based geometry fitting and geometry fitting based on the retroreflective marker(s) can be used as alternative or complementary localization methods. This further reduces latency and further increases localization accuracy.

[0107] The shape-based geometry fitting process can be carried out as follows: Using a priori knowledge (the truss is suspended under ropes, etc.), the point cloud can be reduced to the relevant area containing the truss, providing an initial estimate of the truss's position. This initial estimate will generally not accurately reflect the actual position of the truss and can be subsequently corrected. For this purpose, a standard procedure (e.g., ICP - "Iterative Closest Point" with the extension of compatible normals) can be used to transform / fit the 3D model of the truss into the point cloud from the initial position. This process iteratively determines corresponding pairs of points on both objects based on their distance. Only those pairs are considered whose normal vectors point in similar directions. A rotation and translation are then determined from these pairs to minimize the pairwise distances.Limiting the selection to pairs with similar normals prevents the registration of sides of the model facing away from the sensor, which cannot be represented in the point cloud, and thus stabilizes the geometry fitting process. A similar approach is used for hook detection. Here, too, a priori knowledge (hooks are located in two areas under the traverse) can be used to reduce the point cloud accordingly, and an initial estimate of the hook positions can be derived. This estimate can then be improved using a geometry fitting (e.g., ICP with compatible normals). However, only the lower section of a crane hook is used as the 3D model here, since the upper section is mounted in a hook fork, which obstructs the sensor's view of the hook geometry. The detection of the load (e.g.,Casting ladle) also uses a priori knowledge to reduce the point cloud, since a load can only be located below the traverse and between the crane hooks. If corresponding 3D models of the load are available, these can be used for the subsequent geometry fitting process. In the specific case of a casting ladle, a generic cylindrical model can also be used.

[0108] For example, the geometry fitting process based on retroreflective marker(s) can be carried out as follows: Due to their significantly higher reflectivity, the points generated on the marker(s) can be easily extracted from the entire set of points. The current position of the marker(s) can then be calculated from the extracted points. The actual positions of the markers on the objects to be detected (e.g., traverse, hook, ladle) are known. By assigning the extracted areas to the known markers, the position of the objects in the point cloud can be derived. This detection process can be used as an alternative detection method to the previously described procedure, or as a complementary analysis to increase the stability of the detection system.

[0109] A further preferred embodiment of the method according to the invention is characterized in that before carrying out the at least one geometry fitting process, the at least one three-dimensional model is parameterized, wherein preferably during an initial recording of the point cloud, parameters which characterize the plurality of components of the load-handling system and / or the at least one load in the three-dimensional model are varied and then a suitable parameter sequence is selected and this is used for the geometry fitting process, and / or the size of the at least one three-dimensional model is adjusted, wherein preferably the one or more retroreflective markings are arranged at a predefined distance from at least one edge or at least one surface of the at least one load or one of the plurality of components of the load-handling system and based on a determined position and / or orientation of the one or more retroreflective markings.the size of the at least one three-dimensional model is adjusted to the size of the plurality of retroreflective markings.

[0110] The geometry fitting process can improve the accuracy of position determination.

[0111] The parameters that characterize the multiple components of the load handling system and / or the at least one load in the three-dimensional model can preferably be diameters and lengths of the at least one load and / or of one or more of the multiple components of the load handling system. A casting ladle, for example, is essentially characterized by the overall height, various diameters and the position of the pins for the hook holder, which can serve as parameters, for example. The hook holder is the spatial location for the hook mouth as a counterpart to the pin. A crane hook, for example, is determined by the hook suspension and the position of the hook mouth, which can serve as parameters, for example.

[0112] During geometry fitting, the three-dimensional model should match the real object with sufficient accuracy. In reality, the appearance of an object is generally known (e.g. the object type is a casting ladle). However, the exact dimensions vary, e.g. because different sizes of casting ladle are in circulation in a foundry, for which no geometric information is normally known and the individual casting ladle cannot be identified. The different real designs and thus characteristics for the point cloud mean that the generic three-dimensional model of the object, e.g. the casting ladle, cannot be fitted exactly. This is because during the execution of the geometry fitting algorithm, different subsets of the point cloud are used at different times to fit the generic three-dimensional model.As a result, the three-dimensional model jumps between different positions and orientations depending on the selection during the fitting routine. This problem can be solved by parameterizing the at least one three-dimensional model and / or adjusting the size of the at least one three-dimensional model before generating the at least one digital twin of the load-handling system. During parameterization, the dimensional parameters of the ladle model can be varied randomly or according to a rule, arranged using geometry fitting / registration, and evaluated for agreement with the point cloud. The highest agreement means that this dimensional variant is used for the digital twin. When adjusting the size of the at least one three-dimensional model, retroreflective markers, for example, can be used on the real object (i.e., the load and / or the component of the load-handling system).One meter below the top edge and one meter above the bottom edge. The markers are located in the point cloud, and the height of the pan can be derived from their position. A parametric pan model can then be taken and converted into a concrete three-dimensional model with the appropriate dimensions (scaled). The resulting three-dimensional model can then be used for the geometry fitting as before.

[0113] According to a further preferred variant of the method according to the invention, the load handling system and / or the at least one load have several, preferably at least three, particularly preferably at least four, very particularly preferably at least five, of the retroreflective markings, wherein the retroreflective markings only when the at least one load is correctly handled by the load handling system (e.g. is picked up orheld), together from one or more viewing directions (each) in a predefined pattern, preferably in a predefined ASI L-compliant (Automotive Safety Integrity Level-compliant) pattern (according to ISO 26262-1:2018-12), wherein in the evaluation in step c) based on the at least one visualized point cloud it is determined whether the plurality of retroreflective markings are arranged together from one or more viewing directions in the predefined pattern, and from this it is derived whether the at least one load is handled correctly by the load handling system, wherein preferably at least two of the plurality of components of the load handling system each have at least one of the plurality of retroreflective markings, and / or additionally the at least one load also has one of the plurality of retroreflective markings.

[0114] ASIL stands for Automotive Safety Integrity Levels and is a risk classification system defined within the framework of the ISO 26262-1:2018-12 standard.

[0115] In this embodiment, retroreflective markings can be applied in such a way that the retroreflective markings on the at least one load and / or on at least one of the multiple components of the load handling system form a common typical pattern (e.g. a crosshair or a line between two beams) when the at least one load is handled correctly by the load handling system, whereby the common typical pattern is not formed when the at least one load is not handled correctly by the load handling system. Ultimately, it can be deduced directly from the visualized point cloud whether the at least one load is handled correctly by the load handling system by reading directly from the visualized point cloud whether the multiple retroreflective markings are arranged together (from one or more viewing directions) in the predefined pattern.If the correct pattern is recognizable in the visualized point cloud, it can be concluded that at least one load is being handled correctly by the load handling system. If the correct pattern is not recognizable in the visualized point cloud, it can be concluded that at least one load is not being handled correctly by the load handling system, but is instead incorrectly positioned.

[0116] The reflectors can be positioned so that they are recognized as patterns in one or more desired views, e.g., as a target crosshair. The resulting situation can be easily and reliably recognized by the crane operator on an assistance screen displaying the visualized point cloud, but is also suitable for efficient automated analysis. The analysis can be supported by automatically assigned auxiliary lines if necessary. The result is a reliable process control method based solely on the output of a point cloud and thus does not require the creation of a digital twin. Consequently, this is a very fast process variant with a very easy-to-implement evaluation.

[0117] The predefined pattern can be, for example, a line, parallel lines, a cross, a U-shaped pattern or an H-shaped pattern.

[0118] It may be useful for the observer to analyze a point cloud with retroreflective markings that are not located in the main axis of the sensor alignment. In this case, only a thin line of the retroreflective marking would be visible in the main orthogonal view. Therefore, the retroreflective markings can be designed so that they are equally clearly visible in several selected orthogonal main views. This can be achieved by inclining the retroreflective markings, e.g., by 45° (e.g., with respect to a main direction of movement of the crane). For example, a retroreflective marking can be illuminated in the trolley direction, but in the main orthogonal viewing direction in the crane direction, it also results in a sufficiently large area for visualization in the crane assistance system.

[0119] Preferably, the at least one predefined pattern is at least one predefined ASI L-compliant (Automotive Safety Integrity Level-compliant) pattern (according to ISO 26262-1:2018-12). This can, for example, be a pattern with two parallel lines across several, preferably all, components of the load handling system (analogous to road markings).

[0120] According to a further preferred variant of the method according to the invention, in particular the points of the at least one point cloud lying behind the component or the at least one load from the viewing direction (of an observer) can also be visualized (the observer).

[0121] The present invention additionally relates to a crane comprising a system for checking and / or monitoring a load handling system of the crane comprising a plurality of components and provided for handling (e.g. for picking up, transporting, and / or delivering) at least one load, wherein the system further comprises at least one processing unit for processing the at least one point cloud, and wherein the load handling system has one or more retroreflective markings which (each) at least partially reflect light emanating from the at least one ToF sensor back onto the (respective) at least one ToF sensor. The at least one processing unit can be at least one computing unit or at least one computer which is configured to process the at least one (three-dimensional) point cloud.

[0122] A preferred embodiment of the crane according to the invention is characterized in that the one or more retroreflective markings

[0123] - each contain or consist of at least one (reflective) primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or

[0124] - are each flat (or flat or planar), preferably rectangular, and / or,

[0125] - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or

[0126] - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor, and / or the one or more retroreflective markings each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably 40° to 50°, particularly preferably 44° to 46°, to a main direction of movement of the crane, and / or have a reflection value (determined according to the manual "LiDAR Sensor Livox Horizon", 2019) in the range of 151 to 255, preferably in the range of 160 to 255, particularly preferably in the range of 180 to 255, most preferably in the range of 200 to 250.

[0127] The at least one (reflective) primer preferably contains or consists of at least one component selected from the group consisting of xylene isomers, hydrocarbons, preferably C6-C7 hydrocarbons, isoalkanes, cyclics, n-hexane, and mixtures thereof. The at least one clear coat protective sealant preferably contains or consists of at least one component selected from the group consisting of acetone, 2-methoxy-1-methylethyl acetate, butan-1-ol, n-butyl acetate, and mixtures thereof. The clear coat protective sealant protects the retroreflective glass beads from external influences and simultaneously serves to adhere the retroreflective glass beads to the primer. All materials of the retroreflective marking(s) are preferably suitable for high temperature loads, e.g., temperatures above 1000°C, preferably above 1500°C.

[0128] The mean diameter of the retroreflective glass beads can, for example, be determined according to ISO 13320:2020-01.

[0129] The retroreflective glass beads preferably contain or consist of soda-lime glass, and / or preferably have a refractive index in the range of 1.6 to 2.2, and / or preferably have a retroreflection value of at least 500 mcd / m 2 -lx, preferably at least 1000 mcd / m 2 -lx, particularly preferably at least 1500 mcd / m 2 -lx, on.

[0130] The refractive index can, for example, be determined according to DIN EN 1423:2013-03.

[0131] The retroreflection value can be determined, for example, according to DIN EN 1436: 2018-03.

[0132] The one or more retroreflective markings may each have at least one retroreflective (surface) surface.

[0133] The size and shape of the retroreflective marker(s) can be defined task-dependently, and / or can correlate with the scanning pattern of the ToF scanner or LiDAR scanner, and / or can influence the detection latency.

[0134] A further preferred embodiment of the crane according to the invention is characterized in that the plurality of components of the load handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings, and / or the plurality of components of the load handling system have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings, and / or the at least one load has at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings.

[0135] A further preferred embodiment of the crane according to the invention is characterized in that the at least one ToF sensor is attached to the crane and / or is not attached to the load handling system and / or comprises at least two, preferably at least four, ToF sensors and / or at least one LiDAR sensor (light detection and ranging sensor), preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested (Automotive Safety Integrity Level-tested) LiDAR sensor, particularly preferably at least one ASIL-tested LiDAR with "embedded" intelligence orwith embedded intelligence, and / or comprises at least one ToF sensor with a limited field of view, preferably at least one LiDAR sensor with a limited field of view, or at least one ToF sensor with a limited field of view, preferably at least one LiDAR sensor with a limited field of view, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°, wherein particularly preferably the vertical angle is a maximum of 120° and / or the horizontal angle is a maximum of 30°.

[0136] Particularly preferably, the at least one ToF sensor is mounted on the crane, but not mounted on the load handling system (i.e. mounted on a part of the crane that is not part of the load handling system).

[0137] The at least one LiDAR sensor can preferably be at least one AS IL-tested (Automotive Safety Integrity Level-tested) LiDAR sensor, particularly preferably with embedded intelligence. In this case, it is particularly preferred that the at least one LiDAR sensor comprises the at least one processing unit and that the at least one processing unit is additionally configured to perform an evaluation of the at least one visualized point cloud and / or the at least one digital twin, from which it is deduced whether the at least one load is being handled correctly by the load handling system.

[0138] The at least one ASIL-tested LiDAR sensor preferably comprises at least two, particularly preferably at least four, ASIL-tested LiDAR sensors.

[0139] The at least one ASIL-tested LiDAR sensor with embedded intelligence preferably comprises at least two, particularly preferably at least four, ASIL-tested LiDAR sensors with embedded intelligence.

[0140] The at least one LiDAR sensor with a limited field of view preferably comprises at least two, particularly preferably at least four, LiDAR sensors with a limited field of view.

[0141] The at least one LiDAR sensor can be a pulsed LiDAR sensor. This can increase measurement accuracy.

[0142] A further preferred embodiment of the crane according to the invention is characterized in that the plurality of components of the load-handling system comprise at least one rope, and / or comprise a crossbeam, and / or comprise a trolley, and / or comprise at least one, preferably at least two, load-handling means, wherein the at least one load-handling means is preferably at least one crane hook, particularly preferably at least two crane hooks, and / or comprises at least one, preferably at least two fastening means for a load-handling means, wherein the at least one fastening means is preferably at least one hook fork, preferably at least two hook forks.

[0143] For example, the traverse can be attached to the trolley via the at least one rope and / or the at least one load-carrying device can be attached to the traverse via the at least one fastening device.

[0144] Particularly preferably, the plurality of components of the load handling system comprise a cross member, wherein the cross member has one or more retroreflective markings (or the one retroreflective marking or one or more of the retroreflective markings).

[0145] Particularly preferably, the plurality of components comprise a trolley, at least one cable, a cross member, at least two load-handling devices, preferably at least two crane hooks, and at least two fastening means for the load-handling devices, preferably at least two hook forks. It is preferred that the cross member has one or more retroreflective markings (or the one retroreflective marking or one or more of the retroreflective markings). For example, the cross member can be fastened to the trolley via the at least one cable, preferably via several cables, and / or the at least two load-handling devices can be fastened to the cross member via the at least two fastening means.A further preferred embodiment of the crane according to the invention is characterized in that the at least one processing unit is configured to carry out a processing of the at least one point cloud, in which the at least one point cloud is visualized and / or at least one digital twin of at least a part of the load handling system and / or the at least one load is generated (and visualized), and optionally.

[0146] • the system additionally comprises at least one evaluation unit which is configured to carry out an evaluation of the at least one visualised point cloud and / or the at least one (visualised) digital twin, in which it is derived whether the at least one load is correctly handled by the load handling system (preferably picked up, transported, delivered, and / or picked up or held), or

[0147] • the at least one processing unit is additionally configured to carry out an evaluation of the at least one visualized point cloud and / or the at least one (visualized) digital twin, in which it is derived whether the at least one load is correctly handled by the load handling system (preferably picked up, transported, delivered, and / or picked up or held), and / or the load handling system has several, preferably at least three, particularly preferably at least four, most preferably at least five, of the retroreflective markings, wherein the retroreflective markings are arranged together from one or more viewing directions in a predefined pattern, preferably in a predefined ASIL-compliant pattern,wherein preferably at least two of the plurality of components of the load-handling system each have at least one of the plurality of retroreflective markings, and / or the at least one ToF sensor comprises the at least one processing unit, wherein preferably the at least one processing unit is additionally configured to carry out an evaluation of the at least one visualized point cloud and / or the at least one (visualized) digital twin, in which it is deduced whether the at least one load is handled correctly by the load-handling system.

[0148] The at least one processing unit may be at least one computing unit or at least one computer that is configured to process the at least one point cloud, in which the at least one point cloud is visualized and / or at least one digital twin of at least part of the load handling system and / or the at least one load is generated (and visualized).

[0149] The at least one evaluation unit can be at least one computing unit or at least one computer that is configured to carry out an evaluation of the at least one visualized point cloud and / or the at least one (visualized) digital twin, in which it is derived whether the at least one load is correctly handled by the load handling system (preferably picked up, transported, released, and / or picked up or held).

[0150] The at least one processing unit and / or the at least one evaluation unit may comprise a process analysis system.

[0151] The predefined pattern can be, for example, a line, parallel lines, a cross, a U-shaped pattern or an H-shaped pattern.

[0152] A further preferred embodiment of the crane according to the invention is characterized in that the system (for checking and / or monitoring a load handling system of the crane) is suitable for carrying out the method according to the invention.

[0153] The present invention also relates to the following aspects:

[0154] Aspect 1

[0155] Method for checking and / or monitoring a load handling system of a crane (10) comprising a plurality of components and intended for handling at least one load (6), in which a) at least one point cloud of a working area in which the load handling system is at least partially arranged is generated using at least one ToF sensor (8), wherein the load handling system and / or the at least one load (6) has one or more retroreflective markings (7) which at least partially reflect light emanating from the at least one ToF sensor (8) back onto the at least one ToF sensor (8), b) the at least one point cloud is processed, and c) an evaluation is carried out in which it is deduced whether the at least one load (6) is being handled correctly by the load handling system.

[0156] Aspect 2

[0157] Method according to the preceding aspect, characterized in that the one or more retroreflective markings (7)

[0158] - each contain or consist of at least one primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or

[0159] - are each flat, preferably rectangular, and / or,

[0160] - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor (8), and / or the one or more retroreflective markings (7) each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably of 40° to 50°, particularly preferably of 44° to 46°, to a main direction of movement of the crane.

[0161] Aspect 3

[0162] Method according to one of the preceding aspects, characterized in that the plurality of components of the load-handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7), and / or the plurality of components of the load-handling system have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings (7), and / or the at least one load (6) has at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7).

[0163] Aspect 4

[0164] Method according to one of the preceding aspects, characterized in that the at least one ToF sensor (8) is attached to the crane (10), and / or comprises at least two, preferably at least four, ToF sensors (8), and / or comprises at least one LiDAR sensor, preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested LiDAR sensor, and / or comprises at least one ToF sensor with a limited field of view or is at least one ToF sensor with a limited field of view, wherein the limited field of view is defined by two angles comprising a vertical angle and a horizontal angle, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°.

[0165] Aspect 5

[0166] Method according to one of the preceding aspects, characterized in that the plurality of components of the load-handling system comprise at least one rope, and / or comprise a crossbeam (2), and / or comprise a trolley (1), and / or comprise at least one, preferably at least two, load-handling means, wherein the at least one load-handling means is preferably at least one crane hook (4), particularly preferably at least two crane hooks (4), and / or comprise at least one, preferably at least two fastening means for a load-handling means, wherein the at least one fastening means is preferably at least one hook fork (3), preferably at least two hook forks (3).

[0167] Aspect 6

[0168] Method according to one of the preceding aspects, characterized in that during the processing of the at least one point cloud in step b), the at least one point cloud is visualized and / or at least one digital twin of at least a part of the load handling system and / or of the at least one load is generated based on the at least one point cloud, and during the evaluation in step c), an evaluation of the at least one visualized point cloud and / or of the at least one digital twin is carried out, in which it is derived whether the at least one load is handled correctly by the load handling system.

[0169] Aspect 7

[0170] Method according to aspect 6, characterized in that the at least one digital twin is generated using at least one geometry fitting process, in which at least one three-dimensional model of at least a part of the load-bearing system is transformed into the at least one point cloud, wherein in the at least one geometry fitting process at least one shape-based geometry fitting is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that an at least partial geometric match is achieved between the at least one point cloud and the at least one three-dimensional model, wherein an average deviation of the points of the at least one three-dimensional model from the points of the point cloud is at most 20 cm, preferably at most 15 cm, particularly preferably at most 10 cm, most particularly preferably at most 5 cm,and / or at least one geometry fitting based on the one or more retroreflective markings (7) is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that one or more regions of the at least one point cloud, which are characterized by an increased reflection caused by the one or more retroreflective markings (7), are assigned to one or more corresponding regions of the at least one three-dimensional model.

[0171] Aspect 8

[0172] Method according to aspect 7, characterized in that before carrying out the at least one geometry fitting process, the at least one three-dimensional model is parameterized, wherein preferably during an initial recording of the point cloud, parameters which characterize the plurality of components of the load-handling system and / or the at least one load (6) in the three-dimensional model are varied and then a suitable parameter sequence is selected and this is used for the geometry fitting process, and / or the size of the at least one three-dimensional model is adjusted, wherein preferably the one or more retroreflective markings (7) are arranged at a predefined distance from at least one edge or at least one surface of the at least one load (6) or one of the plurality of components of the load-handling system and are based on a determined position and / or orientation of the one or more retroreflective markings (7).the size of the at least one three-dimensional model is adjusted to the size of the plurality of retroreflective markings (7).

[0173] Aspect 9

[0174] Method according to aspect 6, characterized in that the load-handling system and / or the at least one load (6) have a plurality of the retroreflective markings (7), wherein the retroreflective markings (7) are arranged together from one or more viewing directions in a predefined pattern, preferably in a predefined A-SIL-compliant pattern, only when the at least one load (6) is correctly handled by the load-handling system, wherein during the evaluation in step c) it is determined based on the at least one visualized point cloud whether the plurality of retroreflective markings (7) are arranged together from one or more viewing directions in the predefined pattern, and from this it is derived whether the at least one load (6) is correctly handled by the load-handling system,wherein preferably at least two of the plurality of components of the load-handling system each have at least one of the plurality of retroreflective markings (7), and / or additionally the at least one load also has one of the plurality of retroreflective markings (7). Aspect 10,

[0175] Crane (10) comprising a system for checking and / or monitoring a load handling system of the crane (10) comprising a plurality of components and provided for handling at least one load (6), wherein the system comprises at least one ToF sensor (8) for generating at least one point cloud of a work area in which the load handling system is at least partially arranged, wherein the system further comprises at least one processing unit for processing the at least one point cloud, and wherein the load handling system has one or more retroreflective markings (7) which at least partially reflect light emanating from the at least one ToF sensor (8) back onto the at least one ToF sensor (8).

[0176] Aspect 11

[0177] Crane according to aspect 10, characterized in that the one or more retroreflective markings (7)

[0178] - each contain or consist of at least one primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or

[0179] - are each flat, preferably rectangular, and / or,

[0180] - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or

[0181] - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor (8), and / or the one or more retroreflective markings (7) each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably 40° to 50°, particularly preferably 44° to 46°, to a main direction of movement of the crane. Aspect 12

[0182] Crane according to aspect 10 or 11, characterized in that the plurality of components of the load-handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7), and / or the plurality of components of the load-handling system have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings (7).

[0183] Aspect 13

[0184] Crane according to one of aspects 10 to 12, characterized in that the at least one ToF sensor is attached to the crane (10), and / or comprises at least two, preferably at least four, ToF sensors (8), and / or comprises at least one LiDAR sensor, preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested LiDAR sensor, and / or comprises at least one ToF sensor with a limited field of view or is at least one ToF sensor with a limited field of view, wherein the limited field of view is defined by two angles comprising a vertical angle and a horizontal angle, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°.

[0185] Aspect 14

[0186] Crane according to one of aspects 10 to 13, characterized in that the plurality of components of the load-handling system comprise at least one rope, and / or comprise a crossbeam (2), and / or comprise a trolley (1), and / or comprise at least one, preferably at least two, load-handling means, wherein the at least one load-handling means is preferably at least one crane hook (4), particularly preferably at least two crane hooks (4), and / or comprises at least one, preferably at least two fastening means for a load-handling means, wherein the at least one fastening means is preferably at least one hook fork (3), preferably at least two hook forks (3).

[0187] Aspect 15

[0188] Crane according to one of aspects 10 to 14, characterized in that the at least one processing unit is configured to carry out a processing of the at least one point cloud, in which the at least one point cloud is visualized and / or at least one digital twin of at least a part of the load handling system and / or the at least one load (6) is generated, and optionally

[0189] • the system additionally comprises at least one evaluation unit which is configured to carry out an evaluation of the at least one visualised point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load (6) is handled correctly by the load handling system, or

[0190] • the at least one processing unit is additionally configured to carry out an evaluation of the at least one visualized point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load (6) is handled correctly by the load handling system, and / or the load handling system has a plurality of the retroreflective markings (7), wherein the retroreflective markings (7) are arranged together from one or more viewing directions in a predefined pattern, preferably in a predefined ASI L-compliant pattern, wherein preferably at least two of the plurality of components of the load handling system each have at least one of the plurality of retroreflective markings, and / or the at least one ToF sensor (8) comprises the at least one processing unit, wherein preferably the at least one processing unit is additionally configured toto carry out an evaluation of the at least one visualized point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load is handled correctly by the load handling system and / or the system is suitable for carrying out a method according to one of aspects 1 to 9.

[0191] The present invention will be explained in more detail with reference to the following figures and examples, without limiting it to the specific embodiments and parameters shown here.

[0192] Fig. 1 shows two views of an exemplary embodiment of the crane 10 according to the invention, with a side view on the left and a top view on the right. The crane 10 comprises a system for checking and / or monitoring a load handling system of the crane 10 comprising a plurality of components and provided for handling at least one load, wherein the system comprises four ToF sensors 8 for generating at least one point cloud of a work area in which the load handling system is at least partially arranged, which sensors are attached to the crane 10. The four TOF sensors 8 are LiDAR sensors with a limited field of view. The lines emanating from the LiDAR sensors in Fig. 1 indicate the limited field of view, which is defined by a vertical angle and a horizontal angle.Furthermore, it comprises a processing unit for processing the at least one point cloud, wherein the processing unit is not shown in Fig. 1.

[0193] The load handling system of the crane 10 of the exemplary embodiment shown in Fig. 1 is shown in Figs. 2 and 3 in enlarged illustrations, each showing two views from different sides. The load handling system comprises a trolley 1, a crossbeam 2, two hook forks 3, two crane hooks 4, and ropes connecting the crossbeam 2 to the trolley 1. Also shown is a load 6, which is a pouring ladle with ladle pins 5. The ladle pins 5 can be hooked into the crane hooks 4 to receive the load from the pouring ladle by the load handling system.

[0194] The load handling system of the crane 10 has several retroreflective markings 7, which at least partially reflect light emitted by the ToF sensors 8 back to the ToF sensors 8. For better clarity, the retroreflective markings 7 are shown only in Fig. 3 but not in Figs. 1 and 2.

[0195] The retroreflective markings are applied to the crossbeam 2, the hook forks 3, the crane hook 4 and the load 6.

[0196] As can be seen in Fig. 3, the retroreflective markings are arranged together in a predefined pattern from at least two viewing directions. In the left view in Fig. 3, the retroreflective markings form a cross-shaped pattern. In the right view in Fig. 3, the retroreflective markings form a "U"-shaped pattern. If the retroreflective markings are arranged in the respective predefined pattern from both viewing directions, it can be deduced that the at least one load is being correctly handled by the load handling system, in particular that it is being picked up or held.

[0197] Fig. 4 shows an alternative exemplary embodiment of the crane 10 according to the invention in a top view of a cross-section. Retroreflectors 7 can be attached to a component of the load-handling system or a casting ladle at an angle of preferably 44° - 46° to a main direction of movement of the crane 10. If each point from the LiDAR scan in the resulting point cloud is annotated with a different color based on the strength of the reflectivity, the angled retroreflective surfaces can be clearly seen in several main views of the point cloud without the need for extensive computational steps. In this example, the retroreflective surfaces are seen in the same size in both the side view and the front view with the virtual eye 9.

[0198] Example

[0199] An exemplary variant of the method according to the invention is intended to inspect and / or monitor a crane load handling system comprising several components intended for handling a casting ladle. The crane has four ToF sensors. Furthermore, several components of the load handling system and / or the load (or the casting ladle) have retroreflective markings that at least partially reflect light emitted by four ToF sensors back to the ToF sensors.

[0200] It should be checked and / or monitored whether the ladle is handled correctly by the load handling system.

[0201] The four sensors mounted on the crane continuously transmit their data (3D point with reflectivity value) to the object detection system. The system collects the sensor data and retains it for a certain period of time (integration time) for data processing.

[0202] For an upcoming analysis of the current state, the data from, for example, the last 200 ms is queried. The point clouds of the individual sensors are not yet in a uniform coordinate system, as each sensor transmits the data in its own coordinate system. Therefore, all points are first transformed. The necessary transformations were determined from the sensor calibration during commissioning of the entire system. The positions and orientations of the traverse, crane hook, and pouring ladle are determined in the point cloud using two different methods in parallel. In the geometry-based analysis, 3D geometric models of the components are fitted into the point cloud based purely on the appearance of the point cloud. In the marker-based analysis, the positions of the objects are determined based on the location of the retroreflective markers in the point cloud.The results of the two approaches are then combined and tested for consistency. This increases the reliability of the system, as both analysis methods must produce similar results to ensure that all objects have been correctly identified and located.

[0203] In the geometry-based approach, the load ropes of the truss are first determined. Due to the rigid mounting of the sensors on the crane, various regions of interest (ROI) are defined in the point cloud in the form of bounding boxes aligned with the coordinate axes. The points within these boxes, for the left and right load ropes, are analyzed, and the ropes are located based on their high point density and their vertically running, thin structures. Since it is known that the truss must be located below the ropes, the ropes are traced downwards in the point cloud. At the end of the rope point clouds, the 3D model of the truss is placed in space. This initial estimate of the actual truss position now needs to be further improved.To reduce the effort, points are first extracted within a ROI around the estimated crossbeam position, and the crossbeam model is then iteratively fitted into the point cloud using the ICP algorithm. The crane hooks are located below the crossbeam. Here, too, an initial estimate of the position of the crane hooks is derived from the design of the load system based on the previously determined crossbeam position. This initial position estimate is then again improved using the ICP algorithm. The casting ladle is the last object to be located in the point cloud. Here, it is known that casting ladles relevant to the monitoring process must be located below and in front of the crossbeam in the trolley travel direction, as well as between the two crane hooks. Again, based on this prior knowledge and using an appropriate ROI, the point cloud can be reduced to the relevant area.Since the diameter of the ladle is known, suitable regions with the appropriate curvature for the ladle position are identified in the remaining point cloud. These point cloud segments are then evaluated for their suitability based on the number of points that can be reliably assigned to the outer surface of a ladle, and the best point cloud segment is selected. Finally, the position of the 3D model for the ladle is further improved using an ICP algorithm. After this, the positions and orientations of all objects to be detected in the crane's working area are known.

[0204] In the marker-based approach, the point cloud is first analyzed based on its reflectivity values. The points generated on the retroreflective markers exhibit significantly higher reflectivity than the remaining points in the point cloud. The points above the threshold value of 151 are extracted and grouped into local point cloud groups based on their spatial proximity. The positions of the retroreflective markers in the point cloud are then determined. The relative arrangement of the retroreflective markers and their exact location on the objects to be detected are known. From this, an assignment of the retroreflective markers in the point cloud to the positions stored in the model is derived. Based on this assignment, the transformations to their real position in the point cloud are finally determined for each object.

[0205] After both procedures have delivered a result and this result has been checked for consistency, the transformations of the objects are converted into an exchange data format and transmitted to a process analysis system and a visualization system via a data protocol.

[0206] The next detection procedure then starts.

[0207] The visualization system visualizes the point cloud and / or the digital twin, whereby the visualized point cloud and / or the visualized digital twin can then be displayed on a screen. By viewing the visualized point cloud and / or the visualized digital twin, a person and / or an evaluation unit can determine whether the ladle is being handled correctly by the load handling system. The process analysis system can also be used for this purpose.

[0208] The movement data of the individual crane components and the load are transferred to the process analysis system 4 to 10 times per second with regard to their position (x, y, z) and their orientation (x rotation, y rotation, z rotation).

[0209] The process analysis system uses rules-based checks for each parameter set, including the movement speeds, maximum / minimum distances, and inclination angles between components, and classifies the results with respect to the correctness of the handling process for the load handling system. Furthermore, stored parameter sets for the correct relative positions of components of a load handling system according to mechanical equilibrium are used for process analysis. This allows detection of a jammed hook that no longer aligns. The positions and alignments of the two crane hooks are analyzed relative to each other to ensure the parallelism of the hook movement during picking up or dropping off.

[0210] The movement data of the load handling system's components is interpreted over time to detect and assess the movement speed and dynamics. Typical movement characteristics for the individual components of the load handling system are identified from the travel / time diagrams and examined for abnormalities such as oscillation of the crosshead. This allows a warning to be issued even if a dangerous equilibrium situation arises.

[0211] Many tests are performed redundantly to increase the reliability of the results and to make the rules increasingly safe in terms of the functional safety of a work system. The following rules can be checked:

[0212] If the position of the hook mouth center and the pin center of the ladle match within a certain tolerance range of, for example, < 100 mm, the pin is securely in the hook mouth.

[0213] When picking up or unloading the ladle, the hook tip must be guided safely below the spigot, because otherwise the ladle may be turned or pulled into an unclear position upon contact with a crane hook or, in extreme cases, tip over.

[0214] If the tip of the crane hook lifts the pin of the ladle and thus the component group comes into a precarious equilibrium, in extreme cases the pin slips outwards from the tip of the crane hook and the ladle tips over.

Claims

Patent claims 1. A method for checking and / or monitoring a load handling system of a crane (10) comprising a plurality of components for handling at least one load (6), in which method a) at least one point cloud of a working area in which the load handling system is at least partially arranged is generated using at least one ToF sensor (8), wherein the load handling system and / or the at least one load (6) has one or more retroreflective markings (7) which at least partially reflect light emanating from the at least one ToF sensor (8) back onto the at least one ToF sensor (8), b) the at least one point cloud is processed, and c) an evaluation is carried out in which it is deduced whether the at least one load (6) is being handled correctly by the load handling system.

2. Method according to the preceding claim, characterized in that the one or more retroreflective markings (7) - each contain or consist of at least one primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or - are each flat, preferably rectangular, and / or, - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor (8), and / or the one or more retroreflective markings (7) each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably of 40° to 50°, particularly preferably of 44° to 46°, to a main direction of movement of the crane.

3. Method according to one of the preceding claims, characterized in that the plurality of components of the load-handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7), and / or the plurality of components of the load-handling system have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings (7), and / or the at least one load (6) has at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7).

4. Method according to one of the preceding claims, characterized in that the at least one ToF sensor (8) is mounted on the crane (10) and / or is not mounted on the load-carrying system, and / or at least two, preferably at least four, ToF sensors (8), and / or at least one LiDAR sensor, preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested LiDAR sensor, and / or comprises at least one ToF sensor with a limited field of view or is at least one ToF sensor with a limited field of view, wherein the limited field of view is defined by two angles comprising a vertical angle and a horizontal angle, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°.

5. Method according to one of the preceding claims, characterized in that the plurality of components of the load-handling system comprise at least one rope, and / or comprise a crossbeam (2), and / or comprise a trolley (1), and / or comprise at least one, preferably at least two, load-handling means, wherein the at least one load-handling means is preferably at least one crane hook (4), particularly preferably at least two crane hooks (4), and / or comprises at least one, preferably at least two fastening means for a load-handling means, wherein the at least one fastening means is preferably at least one hook fork (3), preferably at least two hook forks (3).

6. Method according to one of the preceding claims, characterized in that during the processing of the at least one point cloud in step b), the at least one point cloud is visualized and / or at least one digital twin of at least one part of the load handling system and / or of the at least one load is generated based on the at least one point cloud, and during the evaluation in step c), an evaluation of the at least one visualized point cloud and / or of the at least one digital twin is carried out, in which it is deduced whether the at least one load is handled correctly by the load handling system.

7. The method according to claim 6, characterized in that the at least one digital twin is generated using at least one geometry fitting process, in which at least one three-dimensional model of at least a part of the load-bearing system is transformed into the at least one point cloud, wherein in the at least one geometry fitting process at least one shape-based geometry fitting is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that an at least partial geometric correspondence is achieved between the at least one point cloud and the at least one three-dimensional model, wherein an average deviation of the points of the at least one three-dimensional model from the points of the point cloud is at most 20 cm, preferably at most 15 cm, particularly preferably at most 10 cm, most particularly preferably at most 5 cm,and / or at least one geometry fitting based on the one or more retroreflective markings (7) is carried out, in which preferably the at least one three-dimensional model is transformed into the at least one point cloud in such a way that one or more areas of the at least one point cloud, which are caused by an increased reflection caused by the one or more retroreflective markings (7), are assigned to one or more corresponding areas of the at least one three-dimensional model.

8. The method according to claim 7, characterized in that before carrying out the at least one geometry fitting process, the at least one three-dimensional model is parameterized, wherein preferably during an initial recording of the point cloud, parameters which characterize the plurality of components of the load-handling system and / or the at least one load (6) in the three-dimensional model are varied and then a suitable parameter sequence is selected and this is used for the geometry fitting process, and / or the size of the at least one three-dimensional model is adapted, wherein preferably the one or more retroreflective markings (7) are arranged at a predefined distance from at least one edge or at least one surface of the at least one load (6) or one of the plurality of components of the load-handling system and based on a determined position and / or orientation of the one or more retroreflective markings (7).the size of the at least one three-dimensional model is adjusted to the size of the plurality of retroreflective markings (7).

9. Method according to one of claims 6 to 8, preferably according to claim 6, characterized in that the load-handling system and / or the at least one load (6) have a plurality of the retroreflective markings (7), wherein the retroreflective markings (7) are arranged together from one or more viewing directions in a predefined pattern, preferably in a predefined ASI L-compliant pattern, only when the at least one load (6) is correctly handled by the load-handling system, wherein during the evaluation in step c) it is determined based on the at least one visualized point cloud whether the plurality of retroreflective markings (7) are arranged together from one or more viewing directions in the predefined pattern, and from this it is deduced whether the at least one load (6) is handled correctly by the load handling system, wherein preferably at least two of the plurality of components of the load handling system each have at least one of the plurality of retroreflective markings (7), and / or additionally the at least one load also has one of the plurality of retroreflective markings (7).

10. Crane (10) comprising a system for checking and / or monitoring a load handling system of the crane (10) comprising a plurality of components and provided for handling at least one load (6), wherein the system comprises at least one ToF sensor (8) for generating at least one point cloud of a work area in which the load handling system is at least partially arranged, wherein the system further comprises at least one processing unit for processing the at least one point cloud, and wherein the load handling system has one or more retroreflective markings (7) which at least partially reflect light emanating from the at least one ToF sensor (8) back onto the at least one ToF sensor (8).

11. Crane according to claim 10, characterized in that the one or more retroreflective markings (7) - each contain or consist of at least one primer and retroreflective glass beads and optionally additionally at least one clear varnish protective seal, wherein the retroreflective glass beads preferably have an average diameter in a range of 0.1 mm to 0.3 mm, and / or - are each flat, preferably rectangular, and / or, - each have a length and / or width of at least 150 mm, preferably at least 200 mm, and / or - each have a size and / or shape that correlates with a scanning pattern of the at least one ToF sensor (8), and / or the one or more retroreflective markings (7) each have a retroreflective surface that is arranged at an angle of 30° to 60°, preferably of 40° to 50°, particularly preferably of 44° to 46°, to a main direction of movement of the crane.

12. Crane according to claim 10 or 11, characterized in that the plurality of components of the load handling system each have at least one, preferably at least two, particularly preferably at least three, very particularly preferably at least five, of the retroreflective markings (7), and / or the plurality of components of the load handling system have a total of at least two, preferably at least three, particularly preferably at least five, very particularly preferably at least ten, of the retroreflective markings (7).

13. Crane according to one of claims 10 to 12, characterized in that the at least one ToF sensor is attached to the crane (10) and / or is not attached to the load-carrying system, and / or comprises at least two, preferably at least four, ToF sensors (8), and / or comprises at least one LiDAR sensor, preferably at least two LiDAR sensors, particularly preferably at least four LiDAR sensors, wherein the at least one LiDAR sensor is preferably at least one ASIL-tested LiDAR sensor, and / or comprises at least one ToF sensor with a limited field of view or is at least one ToF sensor with a limited field of view, wherein the limited field of view is defined by two angles comprising a vertical angle and a horizontal angle, wherein one of the two angles is a maximum of 120° and the other of the two angles is a maximum of 30°.

14. Crane according to one of claims 10 to 13, characterized in that the plurality of components of the load handling system comprise at least one rope, and / or comprise a crossbeam (2), and / or comprise a trolley (1), and / or comprise at least one, preferably at least two, load handling means, wherein the at least one load handling means is preferably at least one crane hook (4), particularly preferably at least two crane hooks (4), and / or comprises at least one, preferably at least two fastening means for a load handling means, wherein the at least one fastening means is preferably at least one hook fork (3), preferably at least two hook forks (3).

15. Crane according to one of claims 10 to 14, characterized in that the at least one processing unit is configured to carry out a processing of the at least one point cloud, in which the at least one point cloud is visualized and / or at least one digital twin of at least a part of the load handling system and / or the at least one load (6) is generated, and optionally • the system additionally comprises at least one evaluation unit which is configured to carry out an evaluation of the at least one visualised point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load (6) is handled correctly by the load handling system, or • the at least one processing unit is additionally configured to carry out an evaluation of the at least one visualized point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load (6) is handled correctly by the load handling system, and / or the load handling system has a plurality of the retroreflective markings (7), wherein the retroreflective markings (7) are arranged together from one or more viewing directions in a predefined pattern, preferably in a predefined ASI L-compliant pattern, wherein preferably at least two of the plurality of components of the load handling system each have at least one of the plurality of retroreflective markings, and / or the at least one ToF sensor (8) comprises the at least one processing unit, wherein preferably the at least one processing unit is additionally configured toto carry out an evaluation of the at least one visualized point cloud and / or the at least one digital twin, in which it is deduced whether the at least one load is handled correctly by the load handling system and / or the system is suitable for carrying out a method according to one of claims 1 to 9.

Images