Method and device for controlling a crane, an excavator, a crawler or a similar construction machine

Description

[0001] The present invention generally relates to the control of material handling and / or construction machines with camera support. The invention relates, on the one hand, to a method for controlling a material handling and / or construction machine, in particular in the form of a crane, an excavator, or a crawler, wherein an image of the work tool is provided to a machine operator and / or a machine control system by an imaging sensor. The invention further relates to the material handling and / or construction machine itself, in particular a crane, with a display device for displaying an image of the work tool and / or the work tool environment.

[0002] A construction machine of this type is known from DE 10 2014 218 749 A1, in which a drone provides the machine operator with a camera image of the load hook, which automatically follows the load hook. Other drones for construction machines are known from EP 2 993 620 A1, DE 10 2014 009 165 A1, DE 10 2013 019 098 B3, US 2016 / 050840 A1, and WO 2015 / 179797 A1.

[0003] During operation, cranes such as tower cranes, telescopic cranes, and harbor cranes, or construction machinery such as excavators or industrial trucks such as surface miners, often result in the machine operator no longer being able to see the work tool they are supposed to control. For example, if a load is picked up or set down using the load hook of a crane behind a building or a visible edge, the crane operator may not be able to see the location where the load is picked up or set down, meaning the crane operator has to work blind, so to speak, and rely on instructions from a signalman. Similarly, with other construction or material handling equipment such as excavators, the machine operator may no longer be able to see the work tool, for example when working with a digging bucket or grab in a deeper pit or behind an embankment.

[0004] In order to improve the control of the work tool during such work in areas with obstructed visibility, it has already been proposed to provide the machine operator or the machine control system with a camera image of the work tool, wherein said image can advantageously be a real-time or live image in the sense of a video image in order to visualize, for example, pendulum movements of the load hook of a crane or an obstacle in the vicinity of the work tool.

[0005] For tower cranes, for example, it has already been proposed to mount a camera on the trolley that looks down toward the load hook in the direction of the pay-out hoist rope, with the load hook position being automatically determined using image analysis (see DE 20 2012 012 116 U1). However, with this previously known camera arrangement, the height of the load hook must be calculated, and the camera image is not particularly helpful when checking the load hook height, for example, when setting a load down on the ground or threading it into an eyelet, because the camera looks vertically downwards.

[0006] Furthermore, DE 198 07 989 A1 discloses attaching a video camera to the crane's hook block, which transmits a camera image of the load attached to the hook to a monitor located in the crane operator's cab. However, this downward-facing camera is also subject to the aforementioned restrictions.

[0007] The present invention is therefore based on the object of creating an improved method and an improved device for controlling a material handling and / or construction machine, as well as an improved material handling and / or construction machine itself, which avoid the disadvantages of the prior art and advantageously develop the latter. In particular, improved visualization of the work tool relative to its surroundings is to be created, even in difficult, obstructed-visibility areas.

[0008] According to the invention, the stated object is achieved by a method according to claim 1, a material handling and / or construction machine according to claim 5. Preferred embodiments of the invention are the subject of the dependent claims.

[0009] It is therefore proposed to use a remote-controlled aerial drone equipped with at least one imaging sensor, by means of which the desired image of the work tool and / or the tool's surroundings can be provided from different viewing directions. In particular, such an aerial drone can also provide a perspective, three-dimensional image of the work tool and its surroundings from oblique viewing axes, which can be obtained from a point spaced from the machine and its operator's cab and above the ground, so that the machine operator can visualize the work tool and its surroundings from a viewing direction as an external observer would also see them. In the case of a crane, for example, images of the load hook and its surroundings can be provided, looking from the side at an angle or perpendicular to the vertical crane center plane through the boom.Images of the load hook can also be provided from drone positions located in this vertical crane center plane passing through the boom. Such perspective images from different viewing axes allow the relative positions between the work tool and its surroundings or a target point to be visualized much more clearly.

[0010] Various types of imaging sensors are used on the drone. A camera, particularly a video or television camera, is advantageously mounted on the drone, which can provide a highly realistic image of the load hook area, giving the operator the feeling of seeing it with their own eyes.

[0011] An infrared sensor is also provided as the imaging sensor, although a radar sensor and / or a photonic mixer and / or a time-of-flight sensor, known as a time-of-flight detector, may also be included. With such a time-of-flight or PMD sensor, the measurement objects are illuminated by light pulses, and the signal propagation time is measured. Based on the propagation time, the distance between the camera and the object can be calculated. The resulting distance image can then be displayed in various ways (e.g., colors as distances). In addition to the distance, a grayscale image can be calculated from the intensity of the reflected light.

[0012] According to the invention, the drone is provided with several imaging sensors of different image types, including a camera in conjunction with an infrared sensor, for capturing different images. The different images can be displayed on different display units or different sections of the same display unit. Advantageously, the images from at least two different sensors are also superimposed by an image processing module and displayed as a single image in such a way that the warmer parts of an infrared image, which can depict people, for example, are superimposed into a "normal" camera image.

[0013] In an advantageous development of the invention, the camera or the imaging sensor system can also be configured to provide a three-dimensional image of the work tool or its surroundings. For this purpose, a TOF or time-of-flight sensor system, as previously explained, can be used. 3D imaging technology can create a three-dimensional representation that also gives the machine operator a sense of the depth of the displayed space.

[0014] To enable simple operation of the drone, the drone is controlled depending on the machine position and / or the work tool position in such a way that the drone automatically follows machine movements, in particular work tool movements, and at least approximately maintains a desired position relative to the machine and / or its work tool, even during machine movements, in particular work tool movements, or attempts to maintain this position and follows suit. If the drone is used in conjunction with a crane, for example, the drone can automatically follow the crane's load hook if the automatic follow mode is activated. If, for example, the drone's relative position is approximately at load hook height orIf desired and set a little way above with lateral spacing to the vertical crane center plane by the boom, the drone can automatically lower or increase its flight altitude when the load hook is lowered or raised, and / or fly forwards or backwards parallel to the vertical crane center plane when the crane's trolley is moved and / or fly sideways to the left or right when the crane is rotated.

[0015] Advantageously, however, the drone can also be remotely controlled autonomously, allowing the drone to freely fly to various desired positions relative to the machine and / or its working tool. This can be achieved, for example, by entering a desired position for the drone relative to the load hook or the working tool. For example, a position relative to the load hook is entered into a position control module, which can be provided in the crane operator's cab, a driver's station, or a remote control station, for example, in the form of "2 m above, to the right of the load hook." Additionally, the drone can also be flown completely freely relative to the machine and its working tool, for example, using a joystick to control the drone until the camera position and its viewing angle of the working tool and its surroundings are available to the crane operator or operator.The operator agrees. Additionally, gesture control can be provided for the drone, in which a detection device, such as a camera with a downstream image evaluation device, records the operator's gestures, such as hand movements, and converts them into control commands for the drone.

[0016] In order to position the drone relative to the machine or its working tool and, for example, to automatically track working tool movements, the drone can be position-controlled in a relative coordinate system fixed to the machine or crane. For this purpose, a position-determining device can be provided that continuously or cyclically determines the flight position of the drone relative to the machine. Such a position-determining device can, for example, have a signal locating device that can locate signals coming from and / or sent to the drone and / or evaluate them with regard to certain signal properties in order to determine the relative position of the drone to the working machine.

[0017] Such a signal locating device can, for example, be implemented in such a way that several transceiver units are attached to the crane or machine, which communicate with a transceiver unit on the aerial drone, so that the position of the aerial drone relative to the crane or machine can be determined from the signal propagation times and / or signal strengths and / or signal directions in the sense of the connecting lines between the various crane- or machine-side transceiver units and the transceiver unit of the aerial drone. The said transceiver units can, for example, be transponders or short-range transceiver units. In the case of a crane, the said transceiver units can, for example, be attached to the boom, the trolley, the tower and / or the load hook itself. In particular, the signal propagation times from the respective transceiver unit on the crane orthe machine to the drone and / or back from the drone to the machine-side transmitting / receiving unit and / or signal strengths are recorded and / or the directions in which maximum signal strengths occur are determined in order to determine the position of the drone relative to the machine from the signal propagation times and / or signal strengths and / or signal directions of maximum signal strength.

[0018] Alternatively or in addition to such a relative position determination in a machine-fixed coordinate system, the positions of the drone on the one hand and of the machine and / or its work tool on the other hand can also be determined in an absolute coordinate system, so that the relative position can in turn be determined from the two absolute positions and, for example, the drone can be controlled in the manner described above in such a way that the drone automatically follows or attempts to follow a load hook or a work tool and its movements.

[0019] The aforementioned absolute position determination can be carried out, for example, using a positioning system, such as a GPS system. For example, the drone, on the one hand, and the load hook, on the other hand, can each be equipped with a GPS unit to determine the absolute spatial position of the load hook, on the one hand, and the absolute spatial position of the drone, on the other hand. However, the spatial position of the load hook can also be determined approximately from the known movement and / or position data of the work machine components, such as the angle of rotation of a tower crane, the trolley position, and the load hook height. From these, the load hook position can be determined at least approximately, particularly while neglecting pendulum movements and / or wind influences, given a known installation location.

[0020] Alternatively or in addition to readjusting the flight position of the drone, in a further development of the invention, the viewing axis and / or the focal length of the camera or the imaging sensors of the drone can also be readjusted relative to the body of the drone, in particular such that the camera or sensors on the drone are pivoted and / or their focal length is adjusted in order to automatically follow a load hook or work tool movement. If, for example, the load hook of a crane is lowered slightly, the imaging sensors on the drone can pivot slightly downwards in order to follow the load hook movement or to keep the crane hook in the image. In this case, the drone can maintain its flight altitude or, if necessary, also descend slightly.The camera pan angle and / or focal length can be calculated, in particular, depending on the relative position between the work tool and the drone. Alternatively or additionally, these settings can also be controlled using an image processing process, which can determine the position of the work tool, in particular the load hook, in the image and monitor movements from the image center. This allows the viewing axis and / or focal length of the imaging sensors to be readjusted to maintain the work tool or load hook image in the image center with an approximately constant display size. Changes in the viewing angle and / or focal length can be overlaid with changes in the flight position of the drone.

[0021] Such a changeable alignment of the viewing axis of the imaging sensors relative to the drone and / or changing the focal length can be particularly advantageous if the drone cannot change its position at will, for example due to existing building elements or environmental obstacles such as trees.

[0022] According to the invention, the position control of the drone is controlled as a function of work area limitations, wherein the position control of the drone is additionally controlled as a function of construction site model data and / or obstacle detection data that can be obtained from the drone itself. If, for example, the drone is operated in the previously described automatic follow mode, in which the drone automatically follows the load hook of a crane, it could happen, for example, that when the crane rotates about its upright tower axis, the drone flies sideways in order to maintain its position relative to the load hook, and in doing so collides with part of the building, even though the load hook itself has not yet reached the building. If the position control device of the drone takes into account work area limitations and, if applicable,Additionally, construction site model data is also included, whereby the drone itself can also have obstacle detection, for example, using a radar or ultrasonic sensor, to detect obstacles. The automatic follow-up control is then overridden and the automatic follow-up mode deactivated when a building section from the construction site model data set or an obstacle is reached. Advantageously, the drone can then also automatically calculate an alternative route that takes the work area boundaries into account and / or avoids an obstacle. The alternative route is advantageously determined in such a way that the load hook or the work tool remains in the field of view of the drone's imaging sensors.

[0023] Advantageously, the drone can be connected to various work machines, for example, various cranes or various work machines on a construction site, such as an excavator or a crane, depending on which work machine the drone currently requires. Advantageously, the drone can have a control interface that allows control from various machines. In particular, preconfiguration can be performed in the flight control and / or camera control module of the drone depending on the control signal of the respective work machine, for example, by reading a preconfiguration data set from a configuration library.For example, when controlled by a crane, the drone can be preconfigured so that the imaging sensors adjust their focal length to a load hook, whereas when preconfigured for an excavator, for example, the drone is preconfigured for different flight heights than for a crane and for different focal lengths.

[0024] If a drone can be used for various construction machines or work machines in this way, it can, for example, be parked on the construction site and requested by the construction machine required at the time.

[0025] The invention is explained in more detail below using a preferred embodiment and the accompanying drawings. In the drawings: Fig. 1: a schematic side view of a material handling machine in the form of a crane, on whose load hook a load is maneuvered behind a building and thus outside the field of vision of the crane operator's cabin, whereby a flying drone with a camera provides the crane operator with a camera image of the load hook and its surroundings.

[0026] How Fig. 1 shows, the crane 1 can be designed as a tower crane, the tower 2 of which carries a boom 3 on which a trolley 4 is mounted for movement. The boom 3 can be rotated about an upright axis together with the tower 2 or without the tower 2 - depending on whether the crane is designed as a top-slewing or bottom-slewing crane - for which purpose a slewing drive is provided. The boom 3 could also, if necessary, be designed so that it can be luffed up and down about a horizontal transverse axis, in which case a suitable luffing drive could be provided, for example, in conjunction with the boom guying. The aforementioned trolley 4 can be moved by means of a trolley winch or another trolley drive.

[0027] The aforementioned drive devices can be controlled by a control device 5, which can comprise a stationary control unit with suitable input means, for example in the form of joysticks in the crane operator's cabin 6 or at the crane control station or a remote control station, and / or can also have a mobile control unit with corresponding input means. Such a mobile control unit can, for example, be designed in the form of a radio remote control, which the crane operator can carry with him when walking across the construction site in the crane's working area in order to be able to control the crane even outside the crane operator's cabin 6. The aforementioned remote control station can, however, also be a remote control station set up remotely from the construction site, for example in the form of a simulator.

[0028] In order to be able to see the load hook 8, which can be connected to a hoist rope 7 running from the trolley 4, or a load carried thereon or the surroundings of the load hook 8 even when the load hook 8 is outside the field of vision of the crane operator's cabin 6 or the crane operator, for example when - as Fig. 1shows - the load is to be set down behind a building, according to the invention an aerial drone 9 is provided, on which at least one camera 10 is mounted, by means of which a camera image of the load hook 8 and / or the load hook surroundings can be provided. Said camera image is advantageously a live or real-time image in the sense of a television or video image and is transmitted wirelessly from the camera 10 of the aerial drone 9 to a display unit 11 and / or the control device 5 of the crane 1, wherein said display unit 11 can be, for example, a machine operator display in the manner of a tablet or a screen or a monitor that can be mounted in the crane operator's cabin 6. If a remote control station or a mobile operating unit is used to control the crane 1 in the manner described above, said display unit 11 can be provided in the remote control station or on the mobile operating unit.

[0029] The drone 9 is provided with a remote control device 12, which allows the drone 9 to be remotely controlled, in particular to control the flight control units such as rotor blades, in order to remotely control the flight position of the drone 9 and / or to remotely control the camera 10, in particular with regard to the pan angle or the viewing axis of the camera 10 relative to the body of the drone 9 and / or the focal length of the camera 10.

[0030] A corresponding remote control module can be provided in the crane operator's cabin 6 and / or the remote control station or the mobile control unit, for example, equipped with appropriate joysticks. To enable simple operation, however, voice control and / or menu control can also be provided for the drone 9, for example, to select a desired relative position of the drone 9 relative to the crane from several predetermined relative positions. This can be achieved, for example, by entering "Drone Position 1" via voice control and / or menu control, which can be preprogrammed or stored in the position control device 13.

[0031] Advantageously, the position of the flying drone 9 relative to the crane 1 and / or its load hook can be controlled at least largely autonomously and independently of the crane, for example in a manner known per se via the aforementioned joysticks of the remote control device 12. A desired position of the flying drone 9 relative to the load hook 8 can be flown to via the autonomous control module of the position control device 13.

[0032] Alternatively or in addition to such an autonomous position control module, the position control device 13 can have an automatic follow-up control module in order to maintain a predetermined position of the flying drone 9 - for example, the desired position randomly flown to by the autonomous position control module and / or a predetermined, pre-programmed position - even if the crane 1 executes crane movements and / or the load hook 8 is moved, so that the flying drone 9 largely automatically follows the load hook 8 and maintains the predetermined relative position thereto.

[0033] Advantageously, a position determination device 18 is provided which automatically, continuously or cyclically, determines the position of the flying drone 9 relative to the crane 1 and / or its load hook 8, so that the position control device 13 can control the flying drone 9 depending on the determined relative position.

[0034] For this purpose, the drone 9 can, for example, comprise a GPS unit 14, by means of which the absolute spatial position of the drone 9 is determined and transmitted to the position control device 13. On the other hand, the position of the load hook 8 can be determined, so that the position control device 13 can remotely control the drone 9 to maintain the relative position.

[0035] The load hook position can also be determined by GPS, for example, by integrating a GPS unit into the load hook. Alternatively or additionally, however, the load hook position can also be determined from the position of the crane components, in particular calculated by the crane's control device 5, for example, by recording the angle of rotation of the boom, the position of the trolley 4 on the boom 3, and the unwinding length of the hoist rope 7. From this, the load hook position can be determined at least approximately, given a known installation location of the crane 1, if dynamic pendulum movements or wind influences are neglected.

[0036] Alternatively or in addition to such absolute position determination, the position of the drone 9 can also be determined relatively in a coordinate system fixed to the crane, i.e., one that rotates with the crane. For this purpose, transmitting / receiving units, for example in the form of transponder units 15, can be provided on the crane 1, for example on its boom 3 and its tower 2, and possibly also on its trolley 4 and / or its load hook 8. These units are advantageously mounted at several spaced-apart locations on the crane 1. The aforementioned transmitting / receiving units 15 can communicate with a corresponding transmitting / receiving unit 16 on the drone 9.For example, a locating device 17, which can be integrated into the control device 5 of the crane 1, can then determine the distances of the flying drone 9 from the respective transmitting / receiving units 15 on the crane 1 and from this the position of the flying drone 9 relative to the crane 1 from the signal propagation times of a signal between the transmitting / receiving unit 16 on the flying drone 9 and the respective transmitting / receiving units 15 on the crane 1.

Claims

1. Method of controlling a material transfer machine and / or a construction machine, in particular a crane, in which an image of a piece of working equipment, in particular of a lifting hook (8), and / or of an environment of the piece of working equipment is provided to a machine operator and / or to a machine control (5), characterized in that the image is provided by a remote controlled aerial drone (9) that is equipped with at least one imaging sensor (10), wherein the aerial drone (9) - is controlled in an automatic follow mode depending on a position of a piece of working equipment such that the aerial drone (9) automatically follows movements of the piece of working equipment and maintains a desired position relative to the piece of working equipment, also in the case of downward movements of the piece of working equipment, and - is autonomously remote-controlled in an autonomous remote control mode such that different desired positions relative to the machine and / or to its piece of working equipment are flown to by the aerial drone (9), wherein a desired position for the aerial drone (9) relative to the piece of working equipment is input into a position control module and the input desired position is then flown to by the aerial drone, characterized in that by means of a plurality of imaging sensors of different image categories provided at the aerial drone (9), comprising a camera and an infrared sensor, different images, comprising a camera image and an infrared image, are detected, wherein the different images comprising the camera image and the infrared image are superimposed by an image processing module and are shown as a common image on a display unit, wherein the drone is controlled by a position control apparatus (13) depending on working area delimitations, wherein the automatic follow control is overridden and the automatic follow mode is invalidated when the areal drone reaches a working area delimitation.

2. Method in accordance with the preceding claim, wherein the position of the aerial drone (9) relative to the material transfer machine and / or to the construction machine, in particular its piece of working equipment, is automatically determined continuously or cyclically by a position determination device (18), wherein the aerial drone (9) is controlled by the position control apparatus (13) in dependence on a signal of the position determination device (18) such that the relative position between the aerial drone (9) and the machine, in particular the piece of working equipment, remains at least approximately constant.

3. Method in accordance with one of the preceding claims, wherein the at least one imaging sensor (10) of the aerial drone (9) is controlled relative to a drone carrier body in its orientation and / or in its focal length in dependence on a machine position and / or on a position of the piece of working equipment such that the viewing axis and / or the focus of the imaging sensor also automatically follows movements of the machine, in particular of the piece of working equipment on position changes of the aerial drone (9) relative to the machine and / or to its piece of working equipment.

4. Method according to one of the preceding claims, wherein the aerial drone (9) is connected to various working machines for control purposes and is controlled on the side of the various working machines via a control interface, wherein a flight control module of the aerial drone (9) is preconfigured in response to a control signal from a respective working machine by reading a preconfiguration data set from a configuration library for the respective working machine.

5. Material handling and / or construction machine, in particular a crane (1), with a movable working tool, in particular in the form of a lifting hook (8), and at least one imaging sensor (10) for providing an image of the working tool and / or the working tool environment to a machine operator display unit (11) and / or a machine control (5), wherein the imaging sensor (10) is mounted on a remote-controlled aerial drone (9), wherein the machine control (5) and / or the machine operator display unit (11) has an image receiver for receiving the image of the imaging sensor (10) from the aerial drone (9), wherein a position control device (13) comprises - an automatic follow mode control module for controlling the aerial drone (9) in an automatic tracking mode in dependence on position of the piece of working equipment such that the aerial drone (9) automatically follows movements of the piece of working equipment and maintains a desired position relative to the piece of working equipment even during movements of the piece of working equipment, and - an autonomous control module for autonomously remote controlling the aerial drone such that the aerial drone (9) flies to different desired positions relative to the piece of working equipment, wherein the autonomous control module has input means for inputting a desired position for the aerial drone (9) relative to the piece of working equipment and the inputted desired position can be flown to by the aerial drone (9), characterized in that - a plurality of imaging sensors of different image categories including a camera in conjunction with an infrared sensor, for capturing different images are provided at the aerial drone (9), wherein the images from at least two different sensors can be superimposed by an image processing module to form a common image, and the common, superimposed image can be displayed on the machine operator display unit (11), - wherein the position control device (13) is configured to control the aerial drone (9) in dependence on working area delimitations, namely to override the automatic follow mode control and to disable the automatic tracking mode when the aerial drone reaches a working area delimitation.

6. Material transfer machine and / or construction machine in accordance with the preceding claim, wherein a / the position control apparatus (13) has a voice recognition device for inputting control commands by means of voice and / or has a gesture recognition device, in particular comprising a camera having a downstream image evaluation device, for inputting control commands by means of gestures.

7. Material transfer machine and / or construction machine in accordance with one of the preceding claims 5 to 6, wherein a position determination device (18) is provided for determining the position of the aerial drone (9) relative to a machine element of the material transfer machine and / or of the construction machine, in particular its piece of working equipment, wherein the position control apparatus (13) is configured to control the aerial drone (9) in dependence on the automatically determined relative position.

8. Material transfer machine and / or construction machine in accordance with the preceding claim, wherein the aerial drone (9) has a GPS unit for the absolute position determination of the aerial drone (9), with the position control apparatus (13) for controlling the position of the aerial drone (9) relative to the machine being configured to control the aerial drone (9) in dependence on the absolute position data of the areal drone (9) and on the absolute position data of the material transfer machine and / or the construction machine and / or its piece of working equipment.

9. Material transfer machine and / or construction machine in accordance with the preceding claim, wherein the position determination device (18) has a signal location apparatus for locating a signal output by the aerial drone (9), with said signal location apparatus having transceivers (15) attached to the material transfer machine and / or construction machine and spaced apart from one another for communicating with a transceiver (16) at the aerial drone (9) and having an evaluation device for evaluating the transmitted signals between the transceivers (15) at the machine side and the transceiver (16) at the drone side with respect to predetermined signal properties, in particular with respect to the signal time of flight and / or to the signal strength, and for determining the position of the aerial drone (9) from the signal properties.

10. Material transfer machine and / or construction machine in accordance with one of the preceding claims 5 to 9, wherein a radar sensor and / or a photonic mixing device and / or a time of flight sensor for light is / are provided as the imaging sensor (10).

11. Material transfer machine and / or construction machine in accordance with one of the preceding claims 5 to 10, wherein the at least one imaging sensor is configured to provide a three-dimensional image; and / or wherein the image displayed by the machine operator display unit (11) is a 3D image.

12. Material transfer machine and / or construction machine in accordance with one of the preceding claims 5 to 11, wherein the material transfer machine and / or construction machine is designed as a crane and comprises a lifting hook (8), wherein the automatic follow mode control module of the position control apparatus (13) is configured to control the aerial drone (9) in dependence on the lifting hook position, in such a way, that the aerial drone (9) automatically follows movements of the lifting hook, and that the input means of the autonomous control module are configured for inputting a desired position of the aerial drone (9) relative to the lifting hook.