A method for controlling a conveyor system for automated conveying of autonomous mobile robots

Abstract

A method for controlling a conveyor system 10 to allow automated conveying of at least one autonomous mobile robot (AMR) 50, the method comprising monitoring with a first sensor unit 12 a waiting zone

Description

TECHNICAL FIELD The invention relates to a method for controlling a conveyor system. The invention further relates to said conveyor system and a kit-of-parts for retrofitting. BACKGROUND Autonomous mobile robots (AMRs) are widely used in modern plants to improve efficiency and save manpower for routine tasks. A fleet of AMRs has been a solution for warehouses to carry and transport cargo among different positions on the site. As part of the warehouse automation, the AMRs typically move along planned paths autonomously, except loading and unloading of cargo boxes that are generally assisted by human operators. In addition, some tasks can require the AMR to travel between different floors by taking a conveyor system, such as a lift. When the AMRs travel up / down floors using the lift, the lift operation involves, for example, pressing the lift buttons, which in a conventional system requires human assistance. It can be cumbersome or just not feasible to upgrade older lift installations for automation. Furthermore, a complete replacement is generally not ecological and can involve huge costs. Thus, there is a demand for better efficiency to automate the process including the scenarios of lift usage. SUMMARY OF THE INVENTION It is the object of the invention to provide measures that allow easier automation of conveyor systems. The object is achieved by the subject-matter of the independent claims. Preferred embodiments are subject-matter of the dependent claims. The invention provides method for controlling a conveyor system to allow automated conveying of at least one autonomous mobile robot (AMR), the method comprising: - monitoring with a first sensor unit a predetermined waiting zone for the presence of an AMR and monitoring with a second sensor unit in a conveyor zone of the conveyor system; - a controller actuating a first control button of the conveyor system with a first actuator unit in response to the controller detecting the presence of the AMR in the waiting zone, wherein the first control button causes the conveyor system to move the conveyor zone towards the AMR and to allow access for the AMR to the conveyor zone; and - the controller actuating a second control button of the conveyor system with a second actuator unit in response to the controller detecting the presence of the AMR in the conveyor zone, wherein the second control button causes the conveyor system to prevent the AMR from leaving the conveyor zone. The conveyor system can be automated to convey AMRs automatically from a waiting zone to a destination. The controller is able to control the conveyor system with actuators based on the sensor data gathered from the sensor units. The whole system is modular and suitable for retrofit such that it is not necessary to change the programming of the conveyor system. At the same time, it is possible to increase the safety of using the conveyor system, as the more important elements can be monitored constantly. The time for simultaneous presence of humans and AMRs in the system can be reduced, thereby being able to reduce the risk for collisions or injuries. Preferably, the method further comprises the controller actuating a third control button of the conveyor system in response to the control zone having arrived at the destination, wherein the third control button causes the conveyor system to allow the AMR to leave the conveyor zone. An active control on the access to the control zone can improve safety, as the risk of the AMR misinterpreting the current state can be reduced. Preferably, the controller determines whether the AMR has left the conveyor zone with the second sensor unit, and in response to having determined that the AMR has left the conveyor zone the controller actuates the second actuator unit to causes the conveyor system to block access to the conveying zone. An active blocking of the conveyor zone can further improve safety, as unauthorized AM Rs or people can be prevented from entering the conveying zone. Preferably, any of the first sensor unit and the second sensor unit are configured as an imaging sensor and the monitoring is performed by detecting image data of the respective waiting zone or conveyor zone. An imaging sensor allows for more information depth. In addition to merely determine the presence or absence of the AMRs or other objects or humans, the image data allow for a more detailed description of the state in which the system currently is in. Preferably, the controller comprises a machine learning model that includes a vision module that performs object recognition on the image data and outputs possible bounding boxes with classification and a similarity evaluation module that determines a similarity value for each bounding box based on a comparison between the results of the vision module and ground truth data and the similarity evaluation module outputs the most similar bounding box as the final detection result. Computer vision can help to improve the overall safety. The imaging data can be interpreted by advanced deep machine learning models and better determine the current state of the system, such as the presence or absence of the AMR, its current condition, etc. Preferably, the controller generates control signals based on the final detection result and the control signals cause each actuator unit to actuate. The whole system can be controlled based on the recognized objects and their current state. Preferably, any of the first actuator unit, the second actuator unit, and the third actuator unit are configured as a mechanical actuator that actuates the respective first to third control button by pressing said control button. Mechanical actuators can be easily retrofitted without changing the electronics of a legacy system. Preferably, the conveyor system comprises a conveyor cabin, wherein the second sensor unit and the conveyor zone are disposed within the conveyor cabin. Preferably, the AMR moves from the waiting zone into the cabin to the conveying zone, and the conveyor system conveys the AMR to the destination by moving the cabin. Preferably, the AMR leaves the conveyor zone by exiting the cabin at the destination. The conveyor system may have a cabin that moves. The cabin can further increase safety. Preferably, the controller conveyor system comprises at least one access opening with a closing element, wherein the first control button and / or the third control button cause the closing element to open to allow access for the AMR. Preferably, the second control button cause the closing element to close to prevent access for the AMR. The system may have an access opening that is able to control the access to the cabin, which can increase safety or prevent unauthorized access. Preferably, the first sensor unit monitors the waiting zone from a position above and / or that the second sensor unit monitors the conveying zone from a position above. Mounting the sensor units with a more birds eye view allows for a better overview of the current state. The invention provides a conveyor system configured for automated conveying of AMRs, the conveyor system comprising means that are adapted to execute the preferred method. The invention provides a computer program comprising instructions to cause the conveyor system to execute the preferred method. The invention provides a computer-readable medium or a computer-readable signal comprising the computer program. The invention provides a kit-of-parts for retrofitting a conventional conveyor system to birds eye a preferred conveyor system, wherein the kit-of-parts comprises a plurality of sensor units, a plurality of actuator units, a controller, and the computer program or the computer-readable medium or the computer-readable signal. With this kit-of-parts it is possible to retrofit or upgrade existing conveyor systems to be automated such that AMRs are able to use them with no human assistance. The idea is to provide an automated system comprising actuator units, e.g., fingerbots, and a vision subsystem that can be retrofitted to conventional conveyor systems. A fingerbot is typically an loT device that can physically trigger pushing force to a button surface. The fingerbot can be fixed next to the control buttons and allows remote operation via a wireless connection, e.g., Bluetooth®. The vision subsystem includes imaging sensors, e.g., IP cameras, that can be installed both inside and outside the conveyor zone, such as a lift cabin. To have a decent field of view, the sensors are preferably positioned at roof level. Vision software from a host control unit is designed with intelligence of object recognition, to detect the presence and position of the AMR. The object recognition approach is preferably based on a templated-based model which can detect the new object without re-training by utilizing several template images. The model may leverage a multi-scale convolutional neural network to learn the correlation between the template and the captured image to propose several possible bounding boxes, then a similarity evaluation model is utilized to find out the most possible one as the final detection result. It is very suitable for the industrial scenario. The image signal from the IP camera can be transmitted over wireless LAN to the host control unit for processing. Based on the AMR state from the recognition result, the control unit then can determine the operational logic, i.e., pressing / releasing of the buttons with the connected fingerbots. It is not necessary to store the image data in any process. The whole system can function without access to any external network. Therefore, the risk from IT security and privacy can be reduced. The system does not require any direct communication with the lift control module and can therefore be retrofitted to almost any conveyor system without modifying it. The system is also independent of the type of AMR and can work with almost any AMR model. Furthermore, the system does not require communication with the AMR. The safety of the automation can be enhanced with cameras used for task monitoring. The solution presented here uses generally available systems that are adapted to work together in a specific way. The solution can therefore have a greater flexibility regarding configuration and programming. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings. Fig. 1 depicts an embodiment of a conveyor system; and Fig. 2 depicts an embodiment of a method for controlling the conveyor system. DETAILED DESCRIPTION OF EMBODIMENT Referring to Fig. 1 a conveyor system 10 is illustrated schematically. The conveyor system 10 comprises a first sensor unit 12 and a second sensor unit 14. The first sensor unit 12 and the second sensor unit 14 are configured as an imaging sensor, e.g., an IP camera. The first and second sensor units 12, 14 can be connected using a wireless LAN connection. The first sensor unit 12 is arranged to monitor a waiting zone 16. The second sensor unit 14 is arranged to monitor a conveying zone 18. In a preferred embodiment, the conveyor system 10 is configured as a lift. The conveyor system 10 comprises a conveyor cabin 20 that can move between different levels usually within a lift shaft. On each level there is an access opening 22 that is provided with one or more closing elements 24 that can be opened and closed. The conveyor system 10 comprises a plurality of control buttons and associated actuator units. The actuator unit can receive control signals via a short range wireless device network, e.g., Bluetooth® or similar networks. The actuator unit is sometimes called fingerbot and may include a pushing rod that can be actuated by a solenoid or small electric motor. The actuator unit may include an adhesive surface to be bonded to a surface close to the button the respective actuator unit is supposed to push. A first control button 26 can be arranged near the access opening 22 to call the conveyor cabin 20. A first actuator unit 28 is arranged to mechanically press the first control button 26 upon being actuated by a corresponding control signal. The first control button 26 and the first actuator unit 28 are arranged outside the conveyor cabin 20. A second control button 30 can be arranged on a control panel within the conveyor cabin 20. The second control button 30 preferably causes the closing element 24 to open. A second actuator unit 32 is arranged to mechanically press the second control button 30 upon being actuated by a corresponding control signal. A third control button 34 can be arranged on the control panel within the conveyor cabin 20. The third control button 34 preferably causes the closing element 24 to close. A third actuator unit 36 is arranged to mechanically press the third control button 34 upon being actuated by a corresponding control signal. The conveyor system comprises a controller 38. The controller 38 may be composed of one or more control units 40 that can communicate with each other. The controller 38 is in communication with the sensor units and the actuator units. The controller 38 includes a machine learning model 42 that is configured for object recognition and bounding box determination. The controller 38 includes a logic control module 44 that can process the output of the machine learning model 42 and generate control signals. The controller 38 includes an actuator interface 46 that communicates the control signals to the actuator units. The machine learning model 42 comprises a vision module 48. The vision module 48 preferably includes a multi-scale convolutional neural network, such as Faster R-CNN. Faster R-CNN is disclosed in S. Ren, et al., „Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks” available on https;. / / grxiy,orq / abs / 1506,01497. Implementations of Faster R-CNN are widely available in machine learning libraries such as PyTorch or TensorFlow and may be obtained from https: / / github.^ for example. The Faster R-CNN comprises a feature extraction portion that includes a convolutional neural network that is configured for feature extraction, such as ResNet50, to extract high-level features from the input image data. ResNet is disclosed in K. He, et al., “Deep Residual Learning for Image Recognition” available on https; / / arxsv.orQ / abs / 151203385. The Faster R-CNN further comprises a region proposal network (RPN) that is a neural network that operates on the feature maps generated by the feature extraction to generate region proposals. The region proposals are potential bounding boxes that may contain objects. The RPN slides a small network (typically a few convolutional layers) over the CNN feature maps and at each location, it predicts multiple anchor boxes and their associated objectness scores (the probability of containing an object) and bounding box coordinates (offsets from the anchor boxes). The use of anchor boxes at different scales and aspect ratios allows the RPN to handle objects of various sizes. The Faster R-CNN further includes a region of interest (Rol) pooling portion. Once the RPN generates region proposals, the Rol pooling portion is applied to the CNN feature maps. Rol pooling resizes the features within each proposal to a fixed spatial dimension so that they can be used by subsequent layers. The Faster R-CNN further includes classification and regression Heads. After Rol pooling, the features are passed through these separate heads. The classification head assigns a class label to each region proposal. The classification head is preferably formed by fully connected layers followed by a softmax layer. The regression head refines the bounding box coordinates for each proposal and adjusts the positions of the proposals to better fit the objects within them. The Faster R-CNN includes a non-maximum suppression (NMS) layer that is applied to the final set of bounding box predictions to remove duplicates and keep only the most confident and non-overlapping detections. Referring to Fig. 2, an embodiment for controlling the conveyor system 10 is described in more detail. An autonomous mobile robot (AMR) 50 moves on a lower level 52 of a plant transporting some goods. The AMR 50 has a destination that is on an upper level 54. The AMR 50 has a planned route and follows the route towards the conveyor system 10. The AMR 50 moves into the waiting zone 16. The first sensor unit 12 that monitors the waiting zone 16 generates image data that include the AMR 50 being present in the waiting zone 16. The image data are transmitted from the first sensor unit 12 via wireless LAN to the controller 38, specifically to the vision module 48. The vision module 48 using the machine learning model 42 analyses the image data and recognizes the objects and their respective boundary boxes. Specifically, the vision module 48 will detect the AMR 50 and mark it with an appropriate bounding box. The output of the machine learning model 42 is processed by the logic control module 44. If the vision module 48 has detected the presence of the AMR 50 in the waiting zone 16, the logic control module 44 generates a control signal that actuates the first actuator unit 28. The first actuator unit 28 is activated and presses the first control button 26. In response to the first control button 26 the conveyor system 10 moves the conveyor cabin 20 to the lower level and opens the closing elements 24, or if the conveyor cabin 20 is already on the lower level, opens the closing elements 24. The AMR 50 is capable of recognizing the access opening 22 using its systems and, in response to determining that the access opening 22 is open, proceeds to move into the conveyor cabin 20, where it stops in the conveyor zone 18. The second sensor unit 14 monitors the conveyor zone 18 and generates image data that show the AMR 50 being present in the conveyor zone 18. The image data are again analyzed by the vision module 48, which recognizes the AMR 50. The output of the vision module 48 is fed to the logic control module 44 which generates a control signal that causes the second actuator unit 32 to actuate the second control button 30. In response to the second control button 30 being pressed the conveyor system 10 closes the closing elements 24 and - after closing - moves the conveyor cabin 20 from the lower level 52 to the upper level 54. Depending on the conveyor system 10, the closing elements 24 may open automatically or be opened in response to the third control button 34 being pressed. The third control button 34 is pressed by the third actuator unit 36. Again, the control signal is generated based on image data detected by the second sensor unit 14 and processed by the vision module 48. Upon the AMR 50 detecting that the closing elements 24 are open again, the AMR 50 leaves the conveyor zone 18 and proceeds to its destination on the upper level 54. As a result, the conveyor system 10 can be automated without modifying the inner workings of the conveyor system 10, instead retrofitting or upgrading the conveyor system 10 with additional components. REFERENCE SIGNS conveyor system first sensor unit second sensor unit waiting zone conveying zone conveyor cabin access opening closing element first control button first actuator unit second control button second actuator unit third control button third actuator unit controller control unit machine learning model logic control module actuator interface vision module autonomous mobile robot (AMR) lower level upper level

Claims

1. A method for controlling a conveyor system (10) to allow automated conveying of at least one autonomous mobile robot (AMR) (50), the method comprising:- monitoring with a first sensor unit (12) a predetermined waiting zone (16) for the presence of an AMR (50) and monitoring with a second sensor unit (14) in a conveyor zone (18) of the conveyor system (10);- a controller (38) actuating a first control button (26) of the conveyor system (10) with a first actuator unit (28) in response to the controller (38) detecting the presence of the AMR (50) in the waiting zone (16), wherein the first control button (26) causes the conveyor system (10) to move the conveyor zone (18) towards the AMR (50) and to allow access for the AMR (50) to the conveyor zone (18); and- the controller (38) actuating a second control button (30) of the conveyor system (10) with a second actuator unit (32) in response to the controller (38) detecting the presence of the AMR (50) in the conveyor zone (18), wherein the second control button (30) causes the conveyor system (10) to prevent the AMR (50) from leaving the conveyor zone (18).

2. The method according to claim 1, further comprising the controller (38) actuating a third control button (34) of the conveyor system in response to the control zone having arrived at the destination, wherein the third control button (34) causes the conveyor system (10) to allow the AMR (50) to leave the conveyor zone.

3. The method according to any of the preceding claims, characterized in that the controller (38) determines whether the AMR (50) has left the conveyor zone with the second sensor unit (14), and in response to having determined that the AMR (50) has left the conveyor zone the controller (38) actuates the second actuator unit (32) to causes the conveyor system (10) to block access to the conveying zone (18).

4. The method according to any of the preceding claims, characterized in that any of the first sensor unit (12) and the second sensor unit (14) are configured as an imaging sensor and the monitoring is performed by detecting image data of the respective waiting zone (16) or conveyor zone.

5. The method according to claim 4, characterized in that the controller (38) comprises a machine learning model (42) that includes a vision module that performs object recognition on the image data and outputs possible bounding boxes with classification and a similarity evaluation module that determines a similarity value for each bounding box based on a comparison between the results of the vision module and ground truth data and the similarity evaluation module outputs the most similar bounding box as the final detection result.

6. The method according to claim 5, characterized in that the controller (38) generates control signals based on the final detection result and the control signals cause each actuator unit (28, 32, 36) to actuate.

7. The method according to any of the preceding claims, characterized in that any of the first actuator unit (28), the second actuator unit (32), and the third actuator unit (36) are configured as a mechanical actuator that actuates the respective first to third control button (26, 30, 34) by pressing said control button (26, 30, 34).

8. The method according to any of the preceding claims, characterized in that the conveyor system (10) comprises a conveyor cabin (20), wherein the second sensor unit (14) and the conveyor zone are disposed within the conveyor cabin (20), wherein the AMR (50) moves from the waiting zone (16) into the cabin to the conveying zone (18), and the conveyor system (10) conveys the AMR (50) to the destination by moving the cabin.

9. The method according to claim 8, characterized in that the AMR (50) leaves the conveyor zone by exiting the cabin at the destination.

10. The method according to any of the preceding claims, characterized in that the controller conveyor system comprises at least one access opening with a closing element (24), wherein the first control button (26) and / or the third control button (34) cause the closing element (24) to open to allow access for the AMR (50), wherein the second control button (30) cause the closing element (24) to close to prevent access for the AMR (50).

11. The method according to claim 10, characterized in that the first sensor unit (12) monitors the waiting zone (16) from a position above and / or that the second sensor unit (14) monitors the conveying zone (18) from a position above.

12. A conveyor system (10) configured for automated conveying of AMRs (50), the conveyor system (10) comprising means that are adapted to execute the method of any of the previous claims.

13. A computer program comprising instructions to cause the conveyor system (10) of claim 12 to execute the method of any of the claims 1 to 11.

14. A computer-readable medium or a computer-readable signal comprising the computer program of claim 13.

15. A kit-of-parts for retrofitting a conventional conveyor system to controller a conveyor system (10) according to claim 12, wherein the kit-of-parts comprises a plurality of sensor units (12, 14), a plurality of actuator units (28, 32, 36), a controller (38), and a computer program according to claim 13 or a computer-readable medium or computer-readable signal according to claim 14.

Images