Industrial truck and method for operating an industrial truck

Abstract

The invention relates to a forklift truck (100) for transporting load carriers (160), for example, pallets or wire mesh containers. The forklift truck (100) comprises a load-handling device, in particular a pair of load forks (114a,b), which is configured to be inserted into one or more recesses, in particular pockets (161a,b), of a load carrier (160) in order to receive the load carrier (160), as well as a drive device (110) which is configured to move the forklift truck (100) and the load-handling device (114a,b) relative to the load carrier (160). Furthermore, the forklift truck (100) comprises an image acquisition device (120) which is configured to acquire a plurality of images of the load carrier (160) when the forklift truck (100) moves relative to the load carrier (160).The industrial truck (100) further comprises a control and evaluation unit (130) configured to detect one or more recesses (161a,b) of the load carrier (160) in the multitude of images and, based on the one or more recesses (161a,b) of the load carrier (160) detected in the multitude of images, to control the drive unit (110) in order to move the industrial truck (100) and the load handling device (114a,b) relative to the load carrier (160) such that the load handling device (114a,b) is inserted into the one or more recesses (161a,b) of the load carrier (160). A method for operating such an industrial truck (100) is also described.

Description

[0001] The invention relates to an industrial truck for transporting load carriers and a method for operating such an industrial truck. The industrial truck can be an autonomous, semi-autonomous, and / or manually operated industrial truck, for example a forklift or an industrial robot.

[0002] Load carriers, such as pallets or wire mesh crates, are often used for the transport and storage of products, goods, and materials. Industrial trucks, such as autonomous, semi-autonomous, and / or manually operated forklifts, industrial robots, and the like, are used to handle such load carriers, for example, in intralogistics—i.e., the internal flow of materials, e.g., in a warehouse or factory hall. For this purpose, industrial trucks typically have a load-handling device, such as two load forks, which can be inserted into corresponding insertion openings (also called entry pockets or pockets for short) of the load carrier in order to pick it up and transport it.

[0003] In a dynamic environment, such as a warehouse or a factory hall, with a large number of static and dynamic objects, such as load carriers, people, other industrial trucks and the like, it must be ensured that when an autonomously, semi-autonomously and / or manually operated industrial truck is used to transport load carriers, such a load carrier is correctly picked up by the load handling device, in particular the load forks, of the industrial truck.

[0004] For this purpose, automated driverless transport systems (AGVs) are known, for example, in which loads are picked up by a driverless industrial truck based on a global position of a respective load, for example a load carrier, which is usually available to such a system with high accuracy (see e.g. Ullricht, G, “Driverless Transport Systems”, 2nd edition, Springer, 2014).

[0005] Using vehicle localization systems, the driverless industrial truck transporting load carriers can determine its global position in order to carry out the entry and pick-up process based on its own global position and the known global position of the load carrier. However, automated transport systems often require a strong adaptation of the industrial environment to the requirements of an automated transport system, which means that such automated transport systems have only limited application capabilities. Furthermore, determining the global position of the industrial truck and the load requires highly accurate and therefore complex and costly localization systems. Furthermore, mixed operation with human operators is difficult in automated transport systems, since modification of the position of a load, for example, by a human operator, can lead to the loss of the known global position.

[0006] In recent times, systems with driverless or autonomous industrial trucks (commonly known as “autonomous mobile robots (AMR)”) have been developed in which, instead of global referencing of the truck and load or load carrier, a relative position reference between the truck and the load is used to carry out the load entry and pickup process. However, load pickup is usually carried out “blindly”, i.e. by means of a controller using open loop control. The position of the load is detected at a specific time, and on the basis of this information, a path for the truck’s movement is determined. The truck then follows this path without receiving any feedback as to whether there is a deviation from the path.In AMR systems, the advantages of a relative positional reference between the vehicle and the load compared to the use of global positions are offset by the fact that the position of the load to be picked up must be determined with high accuracy and the demands on the precision of the vehicle's drive system increase. This arises from the fact that, as already described above, it is no longer possible to continuously adjust the relative positional reference between the vehicle and the load during the retraction and pick-up process to compensate for detection errors in the load position and deviations of the drive system from the planned trajectory.

[0007] The present invention is based on the object of providing an improved industrial truck for transporting load carriers and a method for operating such an industrial truck, wherein the picking up of a load carrier and / or the depositing of a load carrier can be carried out more precisely.

[0008] This object is achieved according to a first aspect of the invention by an industrial truck for transporting load carriers, for example, in an industrial environment, such as a warehouse or a factory hall. The industrial truck can, in particular, be an autonomous, semi-autonomous, and / or manually operated, i.e., operator-controlled, industrial truck, such as a forklift truck or a mobile industrial robot. The load carrier can, for example, be a pallet or a mesh box.

[0009] The industrial truck according to the first aspect comprises a suitable load-handling means, in particular a pair of load forks, which is designed to be introduced into one or more recesses, in particular pockets, of a load carrier in order to receive the load carrier, as well as a drive device which is designed to move the industrial truck and the load-handling means, in particular the pair of load forks, relative to the load carrier in order to, for example, change the orientation and / or the distance between the industrial truck and the load carrier and / or to raise or lower the load-handling means, preferably the pair of load forks.

[0010] Furthermore, the industrial truck according to the first aspect comprises an image capture device which is designed to capture a plurality of images of the load carrier during the movement of the industrial truck relative to the load carrier. The industrial truck according to the first aspect further comprises a control and evaluation device which can, for example, comprise one or more processors and is designed to detect or evaluate one or more recesses, in particular one or more pockets, of the load carrier in the plurality of images.to recognize which are designed to receive the load-handling device, in particular the pair of load forks, and to control the drive device on the basis of the one or more recesses, in particular pockets, of the load carrier detected in the plurality of images in order to move the industrial truck and the load-handling device, in particular the pair of load forks, relative to the load carrier in such a way that the load-handling device, in particular the pair of load forks, is introduced into the respective recess, in particular pockets, of the load carrier in order to be able to receive and transport the load carrier.

[0011] According to one embodiment, the image capture device is mounted on the industrial truck such that the field of view of the image capture device lies substantially along a forward direction of movement (or load-side direction of movement) of the industrial truck and also encompasses at least part of the load-handling device, in particular the pair of load forks. In addition to the image capture device with the field of view along the forward direction of movement of the industrial truck, the industrial truck can also comprise further image capture devices, for example, an image capture device with a field of view along a reverse direction of movement of the industrial truck and / or an image capture device with a field of view perpendicular to the forward direction of movement of the industrial truck.

[0012] In one embodiment, the image capture device comprises a 2D camera configured to capture the plurality of images of the load carrier as a plurality of 2D images of the load carrier.

[0013] According to one embodiment, the control and evaluation device of the industrial truck is designed to detect one or more corner points and / or one or more marker points and / or one or more edges of the one or more recesses, in particular pockets, of the load carrier in the plurality of images and, on the basis of the corner points and / or marker points and / or edges of the one or more recesses, in particular pockets, of the load carrier detected in the plurality of images, to control the drive device in order to move the industrial truck and the load handling device, in particular the pair of load forks, relative to the load carrier in such a way that the load handling device, in particular the pair of load forks, is introduced into the one or more recesses, in particular pockets, of the load carrier.

[0014] In one embodiment, the control and evaluation device is designed to implement a neural network, wherein the neural network is designed to detect the one or more recesses, in particular pockets, of the charge carrier in the plurality of images. Preferably, the neural network comprises a neural network for object detection.

[0015] According to one embodiment, the control and evaluation device for detecting the one or more recesses, in particular pockets, of the load carrier in the plurality of images of the load carrier is designed to define a respective image area (ie a ROI, "region of interest") in the plurality of images of the load carrier, which image area comprises an end face of the load carrier with the one or more recesses, in particular pockets, of the load carrier.

[0016] In one embodiment, the control and evaluation device for detecting the one or more recesses, in particular pockets, of the charge carrier in the plurality of images of the charge carrier can be configured to transform the plurality of images of the charge carrier using a homography matrix. For this purpose, the individual images of the plurality of images of the charge carrier can be provided with a respective time stamp, for example.

[0017] According to one embodiment, the control and evaluation device is designed to continuously control the drive device on the basis of the recesses, in particular pockets, of the load carrier detected in the plurality of images as part of a closed loop control in order to move the industrial truck and the load handling device, in particular the pair of load forks, relative to the load carrier in such a way that the load handling device, in particular the pair of load forks, is introduced into the one or more recesses, in particular pockets, of the load carrier.

[0018] The above-mentioned object is achieved according to a second aspect of the invention by a method for operating an industrial truck for transporting load carriers, wherein the industrial truck comprises a load-handling device, in particular a pair of load forks, which is designed to be inserted into one or more recesses, in particular pockets, of a load carrier in order to receive the load carrier, as well as a drive device designed to move the industrial truck and the load-handling device relative to the load carrier. As already described above, the industrial truck can be autonomous, semi-autonomous and / or manually operated, i.e., driven by an operator, in particular with a driver as the operator. The method according to the second aspect comprises the following steps: Capturing a plurality of images of the load carrier as the industrial truck moves relative to the load carrier by means of the drive device; Detecting one or more recesses, in particular pockets, of the load carrier in the plurality of images of the load carrier, wherein the one or more recesses, in particular pockets, are designed to receive the load-handling means, in particular the pair of load forks; and Controlling the drive device on the basis of the one or more recesses, in particular pockets, of the load carrier detected in the plurality of images in order to move the industrial truck and the load handling device, in particular the pair of load forks, relative to the load carrier in such a way that the load handling device, in particular the pair of load forks, is introduced into the one or more recesses, in particular pockets, of the load carrier in order to be able to pick up and transport the load carrier.

[0019] The method according to the second aspect of the invention can be carried out using the industrial truck according to the first aspect of the invention. Therefore, further possible embodiments of the method according to the second aspect of the invention arise from the further embodiments of the industrial truck according to the first aspect of the invention described above and below.

[0020] Further advantages and details of the invention are explained in more detail using the exemplary embodiments illustrated in the schematic figures. Herein: Fig. 1 a schematic representation of an industrial truck in the form of a forklift truck according to an embodiment for transporting a load carrier; and Fig. 2 a flowchart illustrating steps of a method for operating an industrial truck according to an embodiment for transporting a load carrier.

[0021] Fig. 1 shows a schematic representation of an industrial truck 100 in the form of a forklift truck 100 according to one embodiment for transporting a load carrier 160 in an industrial environment, such as a warehouse or a factory hall. The industrial truck 100 can, in particular, be an autonomous, semi-autonomous, and / or manually operated, i.e., operator-driven, industrial truck 100. The load carrier 160 can, for example, be a pallet 160, in particular a Euro pallet 160 or a wire mesh box 160.

[0022] As in Fig. 1, the industrial truck 100 comprises a load-handling device in the form of a pair of load forks 114a,b, which are designed to be inserted into respective recesses, in particular pockets 161a,b, on an end face 161 of the load carrier 160 in order to receive the load carrier 160 and a product 170 arranged thereon. Fig. In the preferred embodiment illustrated in Figure 1, the load-handling means 114a,b, as mentioned, are a pair of load forks 114a,b, which are designed to be inserted into corresponding pockets 161a,b of the load carrier 160. According to further embodiments, the load-handling means can also be designed as a mandrel, for example, for receiving film rolls or wire coils, as an under-hooking load-handling means (e.g., comparable to garbage trucks for picking up garbage bins), or as bale and roll clamps, for example, for picking up paper rolls.

[0023] The Fig. The industrial truck 100 shown in Figure 1 further comprises a drive device 110, for example at least one motor 110, in particular an electric motor 110, wherein the drive device 110 is designed to move the industrial truck 100 and the pair of load forks 114a,b relative to the load carrier 160, for example to change the orientation and / or the distance between the industrial truck 100 and the load carrier 160 and / or to raise or lower the pair of load forks 114a,b. For this purpose, as in Fig. 1, the drive device 110 can be suitably connected to wheels 112a-d and / or the pair of load forks 114a,b of the industrial truck 100.

[0024] According to the invention, the industrial truck 100 further comprises an image capture device 120, which is designed to capture a plurality of images of the load carrier 160 during the movement of the industrial truck 100 relative to the load carrier 160. Preferably, the image capture device 120 comprises a 2D camera 120, in particular a monocular 2D camera, which is designed to capture the plurality of images of the load carrier 160 as a plurality of 2D images of the load carrier 160.

[0025] As in Fig. As indicated in Figure 1, the image capture device 120 is preferably mounted on the industrial truck 100 such that the field of view of the image capture device 120 lies substantially along a forward movement direction A of the industrial truck 100 and thus also encompasses at least part of the load-handling device, in particular the pair of load forks 114a,b. The image capture device 120 is preferably mounted in the plane of symmetry between the two load forks 114a,b. In addition to the image capture device 120 with the field of view along the forward movement direction A of the industrial truck 100, the industrial truck 100 can also comprise further image capture devices, for example an image capture device with a field of view along a reverse movement direction of the industrial truck 100 and / or an image capture device with a field of view perpendicular to the forward movement direction A of the industrial truck 100.

[0026] According to the invention, the industrial truck 100 further comprises a control and evaluation device 130, which can, for example, comprise one or more processors or microcontrollers with suitable software and is designed to detect or recognize the recesses, in particular pockets 161a, b, of the load carrier 160 in the plurality of images of the load carrier 160 captured by the image capture device 120. The control and evaluation device 130 is further designed to control the drive device 110 on the basis of the pockets 161a, b of the load carrier 160 detected in the plurality of images (for example via suitable Fig. 1) to move the industrial truck 100 and the pair of load forks 114a,b relative to the load carrier 160 such that the pair of load forks 114a,b are inserted into the respective pockets 161a,b of the load carrier 160.

[0027] According to one embodiment, the control and evaluation device 130 of the industrial truck 100 is designed to detect one or more corner points, and / or one or more marker points and / or one or more edges of the recesses, in particular pockets 161a,b, of the load carrier 160 in the plurality of images and, on the basis of the corner points and / or marker points and / or edges of the recesses, in particular pockets 161a,b, of the load carrier 160 detected in the plurality of images, to control the drive device 110 in order to move the industrial truck 100 and the pair of load forks 114a,b relative to the load carrier 160 such that the pair of load forks 114a,b are inserted into the respective pockets 161a,b of the load carrier 160 in order to be able to pick up and transport the load carrier 160.The marker points can be, for example, a section of the load carrier 160 provided with a QR code or other characteristic points of the load carrier 160 that are in a fixed positional relationship to the positions of the recesses 161a,b for the load handling device 114a,b.

[0028] According to a preferred embodiment, the control and evaluation device 130 is designed to continuously control the drive device 110 on the basis of the pockets 161a, d of the load carrier 160 detected in the plurality of images as part of a closed loop control in order to move the industrial truck 100 and the pair of load forks 114a, b relative to the load carrier 160 such that the pair of load forks 114a, b are inserted into the respective pockets 161a, b of the load carrier 160 in order to be able to pick up and transport the load carrier 160. In other words: according to one embodiment, the image capture device 120, the control and evaluation device 130 and the drive device 110 form a closed control loop in which, by continuously analyzing the plurality of images of the load carrier 160, manipulated variables are generated in order to control the drive device 110 or the load forks 114a,b.

[0029] In one embodiment, the control and evaluation device 130 is configured to implement a neural network, wherein the neural network is configured to continuously detect the pockets 161a,b of the load carrier 160 in the plurality of images. Preferably, the neural network comprises a neural network for object detection. The tips of the load forks 114a,b, which are also visible in the plurality of images of the load carrier 160, can be used as a control reference on the industrial truck 100 side. The pixel coordinates of the load forks 114a,b, which are static relative to the position of the image capture device 120, can be determined in an initialization step.

[0030] According to one embodiment, the control and evaluation device 130 is configured to detect the pockets 161a,b of the load carrier 160 in the plurality of images of the load carrier 160, defining a respective relevant image area (i.e., a ROI, "region of interest") in the plurality of images of the load carrier 160, which includes an end face 161 of the load carrier 160 with the pockets 161a,b of the load carrier 160. To determine a respective relevant image area or ROI, the control and evaluation device 130 can use one or more of the following, partly known approaches: Object recognition of the load carrier 160 based on classification approaches, in particular a neural network for object detection; geometric approaches for object recognition based on the external dimensions of the load carrier 160 (e.g.using 2D / 3D sensors); Determination of interruptions in the free spaces with interruption dimensions that correspond to the dimensions of the load carrier 160; End face detection, e.g., using object segmentation; Localization of the end face position and reference to known dimensions of the load carrier 160 (e.g., 800 mm for a load carrier 160 in the form of a Euro pallet 160); Localization of the entry pockets 161a,b (Regional Lines Extraction).

[0031] In one embodiment, the control and evaluation device 130 for detecting the pockets 161a,b of the charge carrier 160 in the plurality of images of the charge carrier 160 can be configured to transform or process the plurality of images of the charge carrier 160 using a homography matrix. For this purpose, the individual images of the plurality of images of the charge carrier 160 can be provided with a respective time stamp, for example.

[0032] As in Fig. 1, the industrial truck 100 may further comprise a display 150 which is designed to display the plurality of images of the load carrier 160 to a user of the industrial truck 100.

[0033] Fig.2 shows a flowchart illustrating steps of a method 200 for operating the industrial truck 100 for transporting load carriers 160. The method 200 comprises a step 201 of capturing a plurality of images of the load carrier 160 during the movement of the industrial truck 100 relative to the load carrier 160 by means of the drive device 110. Furthermore, the method 200 comprises a step 203 of detecting the one or more recesses, in particular pockets 161a,b, of the load carrier 160 in the plurality of images of the load carrier 160.The method 200 further comprises a step 205 of controlling the drive device 110 on the basis of the one or more recesses, in particular pockets 161a,b, of the load carrier 160 detected in the plurality of images, in order to move the industrial truck 100 and load handling means, in particular the pair of load forks 114a,b, relative to the load carrier 160 such that the load handling means, in particular the pair of load forks 114a,b, are introduced into the respective recesses, in particular the pockets 161a,b, of the load carrier 160 in order to be able to pick up and transport the load carrier 160.

[0034] According to one embodiment, step 203 of detecting the recesses, in particular the pockets 161a,b, of the load carrier 160 in the plurality of images of the load carrier 160 and / or step 205 of controlling the drive device 110 of the industrial truck 100 on the basis of the recesses, in particular pockets 161a,b, of the load carrier 160 detected in the plurality of images can also be carried out by an external device, for example a server or a cloud service, which is designed to manage and control a plurality of industrial trucks 100.In other words: according to one embodiment, the industrial truck 100 only captures the plurality of images of the load carrier 160 during the movement of the industrial truck 100 relative to the load carrier 160 and transmits the obtained image data (possibly pre-processed) to the external server, which uses this image data to detect the recesses, in particular the pockets 161a,b, of the load carrier 160 and controls the drive device 110 of the industrial truck 100 such that the load handling means, in particular the pair of load forks 114a,b, are introduced into the respective recesses, in particular the pockets 161a,b, of the load carrier 160 in order to be able to pick up and transport the load carrier 160.

[0035] According to the embodiments described herein, the industrial truck 100 is configured to use a control system for the approach and pickup of the load carrier 160, based on which an automatically controlled approach and pickup takes place based on features in the form of the recesses, in particular the pockets 161a,b, of the load carrier 160, in particular the corners and / or edges of the pockets 161a,b of the load carrier 160. As already described above, a homography matrix can be used to analyze temporally offset images of the load carrier, which is known to compensate for the displacement and rotation of chronologically successive images. On this basis, the control and evaluation device 130 can generate a control command. An advantage of this approach over known methods is that the position of the load carrier 160 does not have to be known exactly.In addition, the industrial truck 100 according to the invention implements a “real” control, ie in the event of deviations in the position of the load carrier 160, the industrial truck 100 according to the invention can react directly and readjust.

[0036] Embodiments described herein have, in particular, the following advantages: Elimination of the need for global localization of the load carrier 160 by determining the relative position and / or orientation (also referred to as relative pose) between the industrial truck 100 and the load carrier 160; Continuous recording of the relative pose, use of a compact, cost-effective image capture device 120, for example in the form of a monocular 2D camera 120; Elimination of complex, precise calibration of the image capture device 120 by exploiting symmetry properties during image processing using the homography matrix (because as long as the aberrations are axially symmetrical to the height axis, this still results in a usable homography, which applies in particular to radial distortions such as pincushion / barrel distortions).

Claims

[1] Industrial truck (100) for transporting load carriers (160), the industrial truck (100) comprising: a load-carrying means (114a,b) which is designed to be inserted into one or more recesses (161a,b) of a load carrier (160) in order to receive the load carrier (160); a drive device (110) which is designed to move the industrial truck (100) and the load-handling means (114a,b) relative to the load carrier (160); an image capture device (120) which is designed to capture a plurality of images of the load carrier (160) during the movement of the industrial truck (100) relative to the load carrier (160); and a control and evaluation device (130) which is designed to detect the one or more recesses (161a,b) of the load carrier (160) in the plurality of images and, on the basis of the one or more recesses (161a,b) of the load carrier (160) detected in the plurality of images, to control the drive device (110) in order to move the industrial truck (100) and the load handling means (114a,b) relative to the load carrier (160) in such a way that the load handling means (114a,b) is introduced into the one or more recesses (161a,b) of the load carrier (160). [2] Industrial truck (100) according to claim 1, wherein the image capture device (120) is mounted on the industrial truck (100) such that a field of view of the image capture device (120) lies substantially along a forward movement direction (A) of the industrial truck (100). [3] Industrial truck (100) according to claim 1 or 2, wherein the image capture device (120) comprises a 2D camera (120) and wherein the 2D camera (120) is designed to capture the plurality of images of the load carrier (160) as a plurality of 2D images of the load carrier (160). [4] Industrial truck (100) according to one of the preceding claims, wherein the control and evaluation device (130) is designed to detect one or more corner points and / or one or more marker points and / or one or more edges of the one or more recesses (161a,b) of the load carrier (160) in the plurality of images and, on the basis of the corner points and / or marker points and / or edges of the one or more recesses (161a,b) of the load carrier (160) detected in the plurality of images, to control the drive device (110) in order to move the industrial truck (100) and the load handling means (114a,b) relative to the load carrier (160) in such a way that the load handling means (114a,b) is introduced into the one or more recesses (161a,b) of the load carrier (160). [5] Industrial truck (100) according to one of the preceding claims, wherein the control and evaluation device (130) is designed to implement a neural network, wherein the neural network is designed to detect the one or more recesses (161a,b) of the load carrier (160) in the plurality of images. [6] Industrial truck (100) according to claim 5, wherein the neural network comprises a neural network for object detection. [7] Industrial truck (100) according to one of the preceding claims, wherein the control and evaluation device (130) for detecting the one or more recesses (161a,b) of the load carrier (160) in the plurality of images of the load carrier (160) is designed to define a respective image area in the plurality of images of the load carrier (160) which comprises an end face (161) with the one or more recesses (161a,b) of the load carrier (160). [8] Industrial truck (100) according to one of the preceding claims, wherein the control and evaluation device (130) for detecting the one or more recesses (161a,b) of the load carrier (160) in the plurality of images of the load carrier (160) is designed to transform the plurality of images of the load carrier (160) by means of a homography matrix. [9] Industrial truck (100) according to one of the preceding claims, wherein the control and evaluation device (130) is designed to continuously control the drive device (110) on the basis of the one or more recesses (161a,b) of the load carrier (160) detected in the plurality of images as part of a closed control loop in order to move the industrial truck (100) and the load handling means (114a,b) relative to the load carrier (160) in such a way that the load handling means (114a,b) is introduced into the one or more recesses (161a,b) of the load carrier (160). [10] Method (200) for operating an industrial truck (100) for transporting load carriers (160), wherein the industrial truck (100) comprises a load-handling means (114a,b) which is designed to be introduced into one or more recesses (161a,b) of a load carrier (160) in order to receive the load carrier (160), and a drive device (110) which is designed to move the industrial truck (100) and the load-handling means (114a,b) relative to the load carrier (160), wherein the method (200) comprises: Capturing (201) a plurality of images of the load carrier (160) during the movement of the industrial truck (100) relative to the load carrier (160) by means of the drive device (110); Detecting (203) the one or more recesses (161a,b) of the charge carrier (160) in the plurality of images of the charge carrier (160); and Controlling (205) the drive device (110) on the basis of the one or more recesses (161a,b) of the load carrier (160) detected in the plurality of images in order to move the industrial truck (100) and the load handling means (114a,b) relative to the load carrier (160) such that the load handling means (114a,b) is introduced into the one or more recesses (161a,b) of the load carrier (160).

Images