Method for detecting at least one object in the environment of a forklift truck, logistics system, industrial truck and data processing system

Abstract

The invention relates to a method for detecting at least one object (5) in the environment of a forklift truck (2), a logistics system (1), the forklift truck (2), and an external data processing device (20). The environment of the forklift truck (2) is captured as image data (4) by means of at least one camera (3) of the forklift truck (2). The image data (4) is transmitted to a communication interface (9) of the forklift truck (2). The communication interface (9) transmits the image data (4) to an external data processing device (20), which is separate from the forklift truck (2). The external data processing device (20) evaluates the image data (4) to recognize the at least one object (5) in the environment of the forklift truck (2). The external data processing device (20) then transmits response data (24) based on the evaluated image data back to the communication interface (9).

Description

[0001] The invention relates to a method for detecting at least one object in the vicinity of a forklift truck. Furthermore, the invention relates to a logistics system comprising a forklift truck and an external data processing device. Finally, the invention relates to the forklift truck and the external data processing device.

[0002] Modern industrial trucks are often equipped with one or more digital camera systems that capture images of the truck's surroundings. Such a camera system can, for example, be positioned above the forks to assist the operator during pallet loading and unloading by providing a precise view of the forks' position relative to the pallet. Camera systems are also used for automatic object recognition, where the captured images are analyzed by the truck's data processing units. For instance, the fork tips and pallets can be automatically detected in the images, thus supporting or even automating the loading process. This technology also enables the detection of potential obstacles such as other industrial trucks, people, or shelving.

[0003] However, image processing algorithms are computationally intensive. The higher the quality requirements regarding the accuracy of object recognition, the greater the computational effort. A compromise must therefore be made between object recognition accuracy on the one hand and short processing times and the availability of hardware capacity on the other. The cost of the forklift's internal data processing device, the available installation space, and the resulting heat generation also complicate the use of a high-performance data processing device.

[0004] The object of the invention is to improve the accuracy of the detection of at least one object in the vicinity of a forklift truck using a camera, while simultaneously reducing the computing time.

[0005] This problem is solved by a method for detecting at least one object in the environment of a forklift truck, comprising an internal data processing device, wherein at least one camera of the forklift truck captures the environment of the forklift truck as image data, wherein the image data is transmitted to a communication interface of the forklift truck, wherein the communication interface transmits the image data to an external data processing device, which is designed separately from the forklift truck and the internal data processing device, wherein the external data processing device evaluates the image data in order to detect at least one object in the environment of the forklift truck, wherein the external data processing device transmits response data based on the evaluated image data to the communication interface.

[0006] Advantageously, in the method according to the invention, the image data is not evaluated on the internal data processing device of the industrial truck, but on an external data processing device that is separate from the industrial truck. Internal data processing devices are necessarily limited in their computing power due to the restricted installation space within the industrial truck's housing. Furthermore, integrating a high-performance data processing device into an industrial truck is costly and often not economically viable. Finally, the internal data processing devices must also perform other tasks related to the control and monitoring of the industrial truck. For these reasons, it is advantageous to outsource the computationally intensive evaluation of the image data and transmit the image data to the external data processing device.

[0007] The evaluation of the image data includes the detection of at least one object in the vicinity of the industrial truck. This is done, for example, using object detection methods known to those skilled in the art, such as classical methods or neural networks. Object detection is, in particular, a component of obstacle detection and / or localization of the industrial truck.

[0008] The camera is, in particular, a 2D camera. Specifically, the communication interface is part of the internal data processing device and / or the camera. According to one embodiment, the camera, the internal data processing device, and the communication interface are combined into a single functional unit.

[0009] Preferably, the camera and / or the internal data processing device compresses the image data before it is transmitted. Compression reduces the size of the image data, allowing for faster and easier transmission. In particular, the image data is transmitted in a predefined format. Specifically, the external data processing device compresses the response data before it is transmitted to the communication interface. While compression may result in a loss of image information, it is designed so that the losses are imperceptible or barely noticeable to a human observer. Therefore, the impact of compression on image data processing is also minimal. Examples of such compression formats are JPG, H.264, and H.265.The image data is embedded in a container format that allows the image data to be stored on a storage medium or streamed. Examples of such container formats are MP4 and RTP.

[0010] According to one embodiment, the image data is encoded before transmission. The encoding depends on the vehicle hardware used and the transmission distance between the communication interface and the external data processing device. In particular, a dedicated video unit, such as a VPU or GPU, is provided on the external data processing device. Such a dedicated video unit is advantageous for decoding the encoded data.

[0011] According to one embodiment, data is transmitted wirelessly between the communication interface and the external data processing device. For example, the data is transmitted via a wireless data connection, in particular a WLAN connection. Specifically, the communication interface establishes the wireless data connection with an access point that is connected to a network or the internet. A connection to the external data processing device is established via the network or the internet. In this embodiment, the external data processing device is, for example, a computer in a data center. According to another embodiment, a point-to-point connection is established between the communication interface and the external data processing device.The wireless data connection can also be an LTE or 5G connection, through which the external data processing device directly accesses the internet and / or the external data processing device.

[0012] According to another embodiment, the image data is transmitted between the communication interface and the external data processing device via a wired connection, in particular a USB interface. A wired data connection enables faster data transmission than wireless data transmission, but requires that the external data processing device be carried with the forklift. The communication interface can be connected directly to the external data processing device via a USB cable, for example. The wired data connection utilizes the USB OTG standard for data transmission. The USB OTG (On-The-Go) standard enables a quick and easy connection of the communication interface to various external data processing devices.According to another embodiment, a proprietary USB connection is provided by the communication interface. In this case, the communication interface provides USB interface support.

[0013] Preferably, the external data processing device is a mobile data processing device, in particular a mobile phone or a tablet, wherein the industrial truck is specifically configured to install image processing software on the mobile data processing device when connected to it, and wherein the industrial truck has a holder shaped to accommodate, hold, and supply electrical power to the mobile data processing device. The mobile data processing device is connected to the communication interface, in particular via WLAN or cable, for example, using a USB interface. Modern mobile phones and tablets have very high computing power, which is used in this way to evaluate the image data.Since mobile data processing devices such as mobile phones or tablets can easily be carried on the forklift, they can also be connected to the communication interface via cable. For simplified use, the communication interface is specifically designed to automatically install image processing software on the mobile data processing device. The mounting bracket makes it easier to transport the mobile data processing device in forklifts. In particular, the bracket is designed to accommodate mobile data processing devices of different sizes. For example, the bracket includes an adjustable clamping device. Because the bracket is designed to supply the mobile data processing device with electrical power, it can be used for extended periods, even under high load.In particular, the mount supplies the mobile data processing device with electrical energy from the forklift. For example, the electrical energy from a forklift's battery is used.

[0014] According to one embodiment, the external data processing device is located on the premises of the industrial truck or in an external data center off-site. For example, the external data processing device is an in-house computer or a computer in the external data center.

[0015] Preferably, the response data comprises the evaluated image data, wherein the communication interface transmits the evaluated image data to the internal data processing device, the internal data processing device generating, in particular, a notification signal and / or a control signal and / or a warning signal based on the evaluated image data. By transmitting the evaluated image data to the communication interface and then to the internal data processing device, the internal data processing device is enabled to react based on the evaluated image data. For example, if it is detected that the forks are not correctly positioned relative to a pallet being loaded during a loading process, a control signal is transmitted to the vehicle control system of the industrial truck to correct the position of the forks.Alternatively, the analyzed image data can be transmitted to a display unit, such as a screen, where it is visible to the forklift operator. The display provides visual guidance, for example, on the position of the forks relative to the pallet. If the fork positioning is found to be correct, no warning signal is generated. However, if the image analysis detects that a person is in the forklift's path, a warning signal is generated and transmitted to the display unit.

[0016] In particular, the internal data processing device determines, based on the selected image data, whether a warning situation and / or a hazard situation exists. If a warning or hazard situation is present, a notification signal and / or a control signal and / or a warning signal is generated. The warning signal is displayed on the display unit or output as a warning signal. The display unit is, for example, a screen or an acoustic output unit. The control signal is transmitted to the vehicle control system, particularly via a bus system. The vehicle control system is configured to perform a limiting, semi-automated, or fully automated intervention. In the case of a limiting intervention, for example, the speed of the industrial truck is reduced. This is called target braking.In a semi-automated intervention, for example, automatic adjustment occurs as long as the user operates a control element. In a fully automated intervention, the adjustment occurs completely automatically. If only a warning signal is generated, instructions for correcting the fork position are displayed, for example. In this case, the driver corrects the fork position manually.

[0017] According to one embodiment, the response data transmitted from the external data processing device to the communication interface includes an indicator signal, a control signal, and / or a warning signal generated by the external data processing device based on the evaluated image data. The communication interface transmits the control signal to a vehicle control unit, the warning signal to an output unit, and / or the indicator signal to the output unit. In this embodiment, the indicator, control, and warning signals are not generated by the internal data processing device but directly by the external data processing device after evaluating the image data. These signals can be transmitted either directly from the communication interface to the vehicle control unit or via the internal data processing device.In this embodiment, the external data processing device takes over the interpretation of the evaluated image data.

[0018] According to one embodiment, the camera transmits the image data to the internal data processing device for preprocessing before the image data is further transmitted to the communication interface. In particular, the internal data processing device selects a region of the image data and transmits it to the communication interface as preprocessed image data. For preprocessing, a region of the image data is specifically cropped. Within this image region, the internal data processing device has identified an object. In particular, the preprocessing includes the identification of significant points in the image data. Specifically, the preprocessing of the image data includes upstream object recognition. Preprocessing the image data reduces the size of the transmitted data, since, for example, only the image section or sections in which objects have been identified are transmitted.However, the significantly more computationally intensive evaluation of the pre-processed image data still takes place on the external data processing device.

[0019] According to one embodiment, during the preprocessing of the image data, the internal data processing device decides whether the image data should be sent to the external data processing device for evaluation. For example, if no object is detected during obstacle detection in the preprocessing stage, the image data does not need to be transmitted to the external data processing device. This avoids unnecessary processing power.

[0020] Preferably, the communication interface and / or the external data processing device performs flow control of the data transmitted between them. Flow control monitors whether the connection is functioning correctly, enabling continuous data transmission without loss. The data throughput between the communication interface and the external data processing device is dynamically controlled, preventing overload of the data connection.

[0021] Preferably, the image data, particularly from the camera, is timestamped, and the external data processing device uses this timestamp to reconstruct the chronological sequence of the images contained within the data. Using a timestamp ensures that the image data can always be associated with the time it was captured throughout the entire processing process. This guarantees that the chronological order of the image data is known and that the transmission and processing time is taken into account when generating the response data. This allows for the inclusion of latency during data transmission. Specifically, a warning is displayed on the screen if the latency becomes too high or if the image data transmission is interrupted. Latency is considered too high if it exceeds an application-specific threshold.This informs the driver that the improved object recognition provided by the external data processing device is no longer available. Alternatively or additionally, the speed of the forklift is automatically reduced if the data connection between the communication interface and the external data processing device is interrupted.

[0022] In particular, the internal data processing device uses the image data to create a checksum and / or a cryptographic signature and transmits this to the external data processing device. This ensures that the data has not been corrupted in transit and originates from the intended communication partner.

[0023] According to one embodiment, a selection function switches between transmitting and evaluating the image data on the external data processing device and evaluating the image data on the internal data processing device. In particular, the selection function is activated by means of a control element, which is provided, in particular, in the industrial truck or on the external data processing device. The selection function determines whether the image data is evaluated externally or internally in the industrial truck.

[0024] According to one embodiment, the selection function allows the evaluation of image data on the external data processing device to be set only if this function has been enabled by a provider, e.g., the manufacturer of the industrial truck. For example, this function could be designed as a paid, unlockable add-on.

[0025] The task is further solved by a logistics system comprising a forklift truck and an external data processing device, which is designed separately from the forklift truck, wherein the forklift truck comprises at least one camera, an internal data processing device and a communication interface, wherein the camera is configured to capture an environment of the forklift truck as image data and to transmit the image data to the communication interface, wherein the communication interface is configured to transmit the image data to the external data processing device, wherein the external data processing device is configured to evaluate the image data in order to recognize at least one object in the environment of the forklift truck and to transmit response data based on the evaluated image data to the communication interface.

[0026] In particular, the industrial truck includes a BUS system and / or a vehicle control system.

[0027] Furthermore, the task is solved by a forklift truck comprising the following components of the logistics system described above: at least one digital camera, the internal data processing device and the communication interface.

[0028] Finally, the task is solved by a data processing device that is set up and configured as an external data processing device according to one of the previously discussed embodiments.

[0029] The logistics system, the industrial truck, and the data processing device embody the same advantages, features, and properties as the previously described method for detecting at least one object in the environment of an industrial truck.

[0030] Further features of the invention will become apparent from the description of embodiments according to the invention, together with the claims and the accompanying drawings. Embodiments according to the invention may fulfill individual features or a combination of several features.

[0031] Within the scope of the invention, features marked with "in particular" or "preferably" are to be understood as optional features.

[0032] The invention is described below, without limiting the general concept of the invention, with reference to exemplary embodiments and the drawings, whereby for all details of the invention not explained in detail in the text, explicit reference is made to the drawings. The drawings show: Fig. 1 A schematically simplified representation of a logistics system with a forklift truck that transmits image data from its camera to an external computer, Fig. 2 a schematically simplified representation of the logistics system Fig. 1, where the image data is instead transmitted to a mobile phone connected via a USB cable, Fig. 3a and Fig. 3b a schematically simplified representation of an output unit in the form of a screen on which an indicator signal and a warning signal are displayed, and Fig. 4 A schematically simplified representation of a logistics system in which the industrial truck transmits the image data from the camera to an external computer via WLAN and a network.

[0033] In the drawings, identical or similar elements and / or parts are provided with the same reference numbers, so that a re-presentation is omitted.

[0034] Fig. Figure 1 schematically illustrates a simplified logistics system 1 with a forklift truck 2 and an external data processing device 20. The forklift truck 2 is located, for example, on the premises of a goods logistics facility and comprises a camera 3, a load handling device 6, a vehicle control unit 7, a BUS system 8, a communication interface 9, an internal data processing device 10, and an output unit 11. In the illustrated embodiment, the camera is positioned above the load handling device 6 and captures image data 4 of the forklift truck 2's surroundings during a loading process. The image data 4 shows an object 5 in the form of a pallet, which is being approached by the forks 6a of the load handling device 6. The image data 4 can thus be used to support the loading process. For this purpose, the image data 4 is transmitted to the internal data processing device 10 or directly to the communication interface 9.In the illustrated embodiment, the communication interface 9 transmits the image data 4 via a wireless data connection 13 to the external data processing device 20, which in this embodiment is a computer 21. If the image data 4 is first transmitted to the internal data processing device 10, the internal data processing device 10 performs preprocessing of the image data 4 before the preprocessed image data 4 is transmitted to the external data processing device 20 via the communication interface 9. Preprocessing includes, for example, object identification, selection of image sections, and / or recognition of significant points. The external data processing device 20 then evaluates the image data 4 to identify the at least one object 5.For example, the pallet entry openings and the position of the fork tines 6a are identified in the image data 4.

[0035] From the image data 4 selected in this way, the external data processing device 20 creates response data, which it transmits back to the communication interface 9 via the wireless data connection 13. For example, the response data includes a control signal 30, which is transmitted from the communication interface 9 via a BUS system 8 of the industrial truck directly to the vehicle control unit 7 in order to control the load handling device 6 so that the forks 6a engage directly in the pallet's entry openings. Instead of a control signal 30, an indication signal 32 or a warning signal 34 can also be generated, which is displayed on the output unit 11, for example, a screen (see Fig. 3a and Fig. 3b). The response data can also include the evaluated image data 24 itself, which is forwarded from the communication interface 9 to the internal data processing device 10. The internal data processing device 10 then generates control signals 30, indicator signals 32, or warning signals 34 based on the evaluation of the evaluated image data.

[0036] The external data processing device 20 is located in the Fig. In the embodiment shown in Figure 1, the external data processing device 2 is arranged on the operating premises of the industrial truck 2. Alternatively, the external data processing device 20 can also be located outside the operating premises, for example in an external data center.

[0037] Fig. Figure 2 shows another embodiment of the logistics system 1. Fig. 1. In this embodiment, the external data processing device 20 is not a computer 21, but a mobile data processing device 22, for example, a smartphone. The communication interface 9 is connected to the mobile data processing device 22 via a USB interface 14b using a cable 14 and a USB connector 14a. The connection is made, for example, via the USB OTG standard. After connection, software is automatically installed on the mobile data processing device 22, which handles the evaluation of the image data 4. The evaluated image data 24 or image information can then be transmitted back to the communication interface 9 via the cable 14. To prevent the operator of the industrial truck 2 from having to hold the mobile data processing device 22, a holder 12 is provided on the industrial truck 2. A further difference is that Fig. 2. No charging process is recorded, but rather image data 4 of the environment in front of the industrial truck 2 while the industrial truck 2 is moving. In the example shown, object 5 is a person standing in front of the industrial truck 2. After evaluating the image data 4, the mobile data processing device 22 or the internal data processing device 10 generates a control signal 30, which is transmitted to the vehicle control unit 7 to brake the vehicle and thus avoid a collision with the person. Additionally or alternatively, a warning signal 34 can also be generated.

[0038] Fig. 3a and Fig. Figure 3b shows schematically simplified representations of the display unit 11, which is designed as a screen. Fig. 3a A warning signal 32 is displayed on the display unit 11. In the example shown, the warning signal 32 is an arrow indicating that the driver must move the load handling device 6 to the right in order to insert the fork tines 6a into the entry openings of a pallet. Fig. In contrast, in 3b a warning signal 34 is displayed. Warning signal 34 is intended, for example, to indicate that there is an obstacle in front of the industrial truck 2. The warning signal 34 displayed on the display unit 11 can also be supported or replaced by an acoustic warning signal.

[0039] Fig.Figure 4 schematically illustrates a simplified logistics system 1 with a forklift truck 2, which is connected via a wireless data connection 13 to an access point 15 of a WLAN network. In the illustrated embodiment, the access point 15 is connected via a network 16, for example, the internet, to the external data processing device 20 in the form of a computer 21. However, a connection to a mobile data processing device 22 can also be established via such a WLAN connection. The computer 21 can be located on the company premises to implement edge computing. The computer 21 can also be located in an external data center separate from the company premises.

[0040] The functionality whereby the image data 4 is evaluated by the external data processing device 20 can be activated and deactivated, in particular by means of a selection function. For example, the functionality of evaluation by the external data processing device 20 can be offered as an additional service, which the operator of the industrial truck 2 can activate for an additional fee in order to increase the accuracy of the evaluation and / or save computing power in the internal data processing device 10.

[0041] All features mentioned, including those discernible from the drawings alone as well as individual features disclosed in combination with other features, are considered essential to the invention, both individually and in combination. Inventive embodiments may be fulfilled by individual features or by a combination of several features. Reference symbol list 1 Logistics system 2 industrial trucks 3 cameras 4 Image data 5 objects 6 Lifting devices 6a Fork tines 7 Vehicle control 8 bus system 9 Communication interface 10 internal data processing devices 11 Output unit 12 bracket 13 wireless data connection 14 cables 14a USB plug 14b USB interface 15 Access point 16 Network 20 external data processing devices 21 computers 22 mobile data processing devices 24 evaluated image data 30 Control signal 32 Warning signal 34 Warning signal

Images