Picking station system and method

Abstract

A computer-implemented method 500 of detecting an operational state of a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which o

Description

Technical Field The present disclosure relates generally to the field of robotic picking stations for use in warehouses or fulfilment centres. Background Some commercial and industrial activities require systems that enable the storage and retrieval of a large number of different products. For example, WO2015 / 185628A2 (Ocado) describes an automated storage and fulfilment system (ASRS) in which stacks of storage containers are arranged within a grid storage structure. The containers are accessed from above by load-handling devices operative on rails or tracks located on the top of the grid storage structure. The load-handling devices may be those described in WO2015 / 019055A1 (Ocado). In such an ASRS, items within the containers may be accessed by robotic picking stations that use a robotic manipulator or arm, such as those described in WO2017081281A1 (Ocado). Within the storage and fulfilment system, it is important that malfunctioning robotic picking stations and / or load-handling devices are identified. It is against this background that the present invention has been devised. Summary In a first aspect, there is a computer-implemented method of detecting an operational state of a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, the method comprising: obtaining, from the image sensor of a first picking station, image data of a second picking station; processing the image data with an object detection model trained to detect an operational state of a picking station on the grid; determining, based on the processing, an operational state of the second picking station; and outputting annotation data indicative of the operational state of the second picking station. This allows a picking station to determine the operational status of an adjacent picking station. This means any required maintenance or action can be automatically identified, which ensures major operational faults on the picking station or the wider system in which it is used, can be proactively averted. The method may further comprise orienting the robotic manipulator of the first picking station to direct the image sensor of the robotic manipulator of the first picking station towards the second robotic manipulator of the second picking station. This means an advantageous view of the second picking station can be obtained. The advantageous view assists with determining the operational state of the second picking station. Determining the operational state may comprise determining whether a warning signal is engaged on the second picking station, and / or whether an item within a container in one of the respective cells of the second picking station extends above a top surface of the grid or a top of the container, and / or whether an item and / or debris and / or a spillage is located on a top surface of the grid, and / or wherein each robotic manipulator further comprises trunking to protect supply lines of the robotic manipulator, wherein determining the operational state may comprise determining whether the trunking is displaced from an optimal operational position for a current orientation of the robotic manipulator of the second picking station, and / or determining whether a component, such as a filter, or suction assembly of an end-effector, of the second picking station requires replacement. This means the state of key or specific operational aspects (e.g. sub-systems) of the picking station can be automatically determined. Each robotic manipulator may comprise an end effector, wherein the end effector has full mechanical degrees of freedom via orientation of the end-effector and / or via orientation of the robotic manipulator. The end effector may comprise the image sensor. This means the image sensor can be oriented across a range of elevation and azimuth to obtain an advantageous view of the second picking station. The method may comprise outputting an updated version of the image data including the annotation data. The annotation data may comprise a bounding box. The object detection model may comprises a convolutional neural network. This means providing the operational status of the second picking station is fully automated. The robotic manipulator of the first picking station may execute a first pre-determined movement sequence to facilitate inspection of the second picking station by the image sensor, and / or the robotic manipulator of the second picking station may execute a second pre-determined movement sequence to facilitate inspection by the image sensor of the first picking station. This means the image sensor can obtain all possible views of the second picking station. The method may further comprise using the image sensor of the first picking station to control the second picking station. The method may further comprise using the image sensor of the first picking station to set-up an exclusion zone around the second picking station. The first image sensor thus provides a useful input to control the second picking station based on the operational status of the second picking station. The computer-implemented method may further comprise obtaining, from the image sensor of a third picking station, further image data of the second picking station, processing the further image data with the object detection model trained to detect an operational state of a picking station on the grid, determining, based on the processing, an operational state of the second picking station, and outputting annotation data to verify the operational state of the second picking station. The method may further comprise orienting the robotic manipulator of the third picking station to direct the image sensor of the robotic manipulator of the third picking station towards the second robotic manipulator of the second picking station. This increases the confidence of the automatically determined operational status of the second picking station. In a second aspect, there is a computer-implemented method, wherein the method of any preceding claim is applied iteratively such that an operational state of each picking station of the plurality of picking stations is determined. This allows the operational status of a fleet of picking stations to be determined. In a third aspect, there is a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding aspect. In a fourth aspect there is a computer readable medium comprising the computer program of the third aspect. In a fifth aspect, there is a data processing system comprising means for carrying out the method of the first and second aspects. In a sixth aspect, there is a system comprising: a grid comprising a plurality of grid cells forming part of a grid-based storage system; one or more picking stations mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station; and a processor configured to carry out the method of the first and second aspects. Brief Description of Drawings The invention is described with reference to the accompanying drawings, wherein: Figure 1 shows a known automated storage and retrieval system that uses load-handling devices; Figure 2 shows a known single load-handling device with container-lifting means in a lowered configuration; Figure 3 shows a known robotic picking station; Figure 4 shows a known end-effector used in the robotic picking station of Figure 3; Figure 5 shows a method of using a first picking station to determine the operational status of a second picking station; Figure 6 shows a plurality of robotic picking stations used in the method of Figure 5; Figure 7 shows a method of verifying the method of Figure 5; Figure 8 shows a fleet of picking stations, where each picking station performs the method of Figures 5 and 7 to determine the operational status of the fleet; and Figure 9 shows an architecture that can be used for an object detection model. Detailed Description WO2015 / 185628A (Ocado), hereby incorporated by reference, describes a known ASRS in which stacks of containers are arranged within a grid framework structure. The containers are accessed by one or more load-handling devices, otherwise known as “bots”, operative on tracks located on the top of the grid framework structure. A system of this type is illustrated schematically in Figure 1. As shown in Figures 1, stackable containers 10, also known as “bins” or “totes”, are stacked on top of one another to form stacks 12. The stacks 12 are arranged in a grid framework structure 14. The grid framework structure 14 is made up of a plurality of storage columns or grid columns. Each grid in the grid framework structure has at least one grid column to store a stack of containers. Each bin 10 typically holds a plurality of product items (not shown). The grid framework structure 14 comprises a plurality of upright members 16 that support horizontal members 18, 20. A first set of parallel horizontal grid members 18 is arranged perpendicularly to a second set of parallel horizontal members 20 in a grid pattern to form a horizontal grid structure 15 supported by the upright members 16. The members 16, 18, 20 are typically manufactured from metal. The bins 10 are stacked between the members 16, 18, 20 of the grid framework structure 14, so that the grid framework structure 14 guards against horizontal movement of the stacks 12 of bins 10 and guides the vertical movement of the bins 10. The top level of the grid framework structure 14 comprises a grid or grid structure 15, including rails 22 arranged in a grid pattern across the top of the stacks 12. The rails or tracks 22 guide a plurality of load-handling devices 30. A first set 22a of parallel rails 22 guide movement of the robotic load-handling devices 30 in a first direction (e.g. an X- direction along track 22a) across the top of the grid framework structure 14. A second set 22b of parallel rails 22, arranged perpendicular to the first set 22a, guide movement of the load-handling devices 30 in a second direction (e.g. a Y-direction along track 22b), perpendicular to the first direction. In this way, the rails 22 allow the robotic load-handling devices 30 to move laterally in two dimensions in the horizontal X-Y plane. A loadhandling device 30 can be moved into position above any of the stacks 12. A known form of load-handling device 30 shown in Figure 2 is described in WO2015 / 019055 (Ocado), hereby incorporated by reference. The load-handling device 30 comprises a vehicle 32, which is arranged to travel on the rails 22 of the frame structure 14. A first set of wheels 34, consisting of a pair of wheels 34 on the front of the vehicle 32 and a pair of wheels 34 on the back of the vehicle 32, is arranged to engage with two adjacent rails of the first set 22a of rails 22. Similarly, a second set of wheels 36, consisting of a pair of wheels 36 on each side of the vehicle 32, is arranged to engage with two adjacent rails of the second set 22b of rails 22. Each set of wheels 34, 36 can be lifted and lowered, by way of a direction-change assembly, so that either the first set of wheels 34 or the second set of wheels 36 is engaged with the respective set of rails 22a, 22b at any one time. For example, when the first set of wheels 34 is engaged with the first set of rails 22a and the second set of wheels 36 is lifted clear from the rails 22, the first set of wheels 34 can be driven, by way of a drive assembly housed in the vehicle 32, to move the load-handling device 30 in the X-direction. To achieve movement in the Y-direction, the first set of wheels 34 is lifted clear of the rails 22, and the second set of wheels 36 is lowered into engagement with the second set 22b of rails 22. The drive assembly can then be used to drive the second set of wheels 36 to move the load-handling device 30 in the Y direction. The load-handling device 30 is equipped with a container-lifting device or assembly, e.g. a crane mechanism, to lift a storage container from above. The lifting device comprises a winch tether or cable 38 wound on a spool or reel and a gripper device 39. The lifting device shown in Figure 2 comprises a set of four lifting tethers 38 extending in a vertical direction. The tethers 38 are connected at or near the respective four corners of the gripper device 39, e.g. a lifting frame, for releasable connection to a storage container 10. The gripper device 39 is configured to releasably grip the top of a storage container 10 to lift it from a stack of containers in a storage system of the type shown in Figure 1. To remove a bin 10 from the top of a stack 12, the load-handling device 30 is first moved in the X- and Y-directions to position the gripper device 39 above the stack 12. The gripper device 39 is then lowered vertically in the Z-direction to engage with the bin 10 on the top of the stack 12. The gripper device 39 grips the bin 10, and is then pulled upwards by the cables 38, with the bin 10 attached. At the top of its vertical travel, the bin 10 is held above the rails 22 accommodated within the vehicle body (or skeleton) 32. In this way, the load-handling device 30 can be moved to a different position in the X-Y plane, carrying the bin 10 along with it, to transport the bin 10 to another location. On reaching the target location (e.g. another stack 12, an access point in the storage system, or a conveyor belt) the bin or container 10 can be lowered from the container receiving portion and released from the grabber device 39. With reference to Figures 3, the ASRS may further comprise a robotic picking station 50 mounted on top of the storage and retrieval structure 1, e.g. alongside the load-handling devices 30 (not shown). The robotic picking station 50 comprises a robotic manipulator 52 comprising a robotic arm 54 and an end effector 56 for releasably engaging a product to be manipulated, together with several designated grid cells 60, 62. The end effector 56 may be a suction device 64 connected to a vacuum source by a vacuum line within trunking 66. The robotic manipulator 52 is mounted on a plinth 58 above a single grid cell 60 and, depending on its location on the structure 1, can be surrounded by up to eight other grid cells 62 as shown in Figure 6. In general, the robotic manipulator 52 is configured to pick an item or product from any one of the containers located in one of the designated grid cells 62 and place it in a container located in another of the designated grid cells 62. The load-handling devices collect containers from, and deliver them to, the designated grid cells 62 as necessary. In this way, the robotic picking station 50 and the load-handling devices 30 work in conjunction via a master controller to fulfil a customer order or redistribute products throughout the storage and retrieval system 1. A picking station on top of the storage and retrieval structure provides the most efficient point to carry out a picking operation. Absent this, the containers would have to continually exit and enter the ASRS to perform a picking operation via the access point, which typically is located at a bottom of the grid framework structure. This may involve lowering and raising containers over a significant height a significant number of times with a corresponding power demand. With reference to Figure 4, the end effector 56 comprises a suction device 64 and an integrated vacuum generator, both of which form part of a suction assembly. The vacuum generator is supplied by a pressurised fluid, which is used to produce a vacuum or suction pressure to releasably engage an object or item with the suction device 64. The suction assembly, and in particular the vacuum assembly may involve filters. A vacuum pressure line within trunking 66 is used to route the pressurised fluid along the robotic arm 54 from a pressure source, possibly located at ground level at the bottom of the framework structure 1, to the end effector 56 through the framework structure 1. Although not shown, the end effector 56 might include a parallel jaw gripper or the like for manipulating objects. The parallel jaw gripper may be in addition to or instead of the suction device 64. Moreover, the robotic manipulator 52 may also include one or more vision systems or image sensors 68 (e.g. cameras) to facilitate its control via an Artificial Intelligence (Al) or Machine Learning (ML) based vision system. It will be appreciated that the vision systems or image sensors need not be located on the end effector and can be located on the robotic arm instead. Accordingly, other power supply lines such as hydraulic, pneumatic and / or electrical power and / or communication supply lines may be provided within trunking 66. Trunking 66 may also be routed through the framework structure 1 to provide services to the robotic manipulator 52 as necessary. Whilst two trunkings are shown in Figure 3, a single trunking or more than two trunkings can be used instead. In general, the use of trunking is to allow multiple supply lines (e.g. power, electrical and / or communication, pressure etc.) to be gathered for their protection and to ease their collective routing along the robotic arm and through the structure 1. A number of measures help ensure and / or verify that the picking station operates as intended. One such measure is the provision of a warning signal 70, which can in principle be located anywhere on the robotic arm or picking station. The warning signal of the picking station comprises a predetermined light, or colour of light, emitted by a light source on the picking station such as a light emitting diode (LED). For example, the picking stations include an LED which is configured to emit a first wavelength (colour) of light when responsive to communications from the master controller and emit a second, different, colour wavelength (colour) of light when unresponsive to communications from the master controller. The picking station may be in an unresponsive state when communication with the master controller is lost, for example, causing the warning signal to be engaged. Other types of warning signal from the light source are possible, for example a predetermined pattern of emission such as flashing. The warning signal may be used to indicate a failure in the supply of power, supply of pressure, or electrical communication with any part of the robotic arm, such as the imaging system for example. The warning signal can indicate the operational status of specific systems or sub-systems of the picking station. Another measure is precise routing and securing of the trunking. In general, the robotic arm 54 has a number of joints (i.e. points of articulation) to enable full mechanical degrees of freedom. That is, the end effector can ‘point’ in any direction along the XYZ axes in Figure 3, and rotate in pitch, roll, and yaw along an axis of the XYZ axes. It is important that the trunking is routed and secured in a way that does not hinder this movement. The trunking is thus configured to provide an optimal level of slack to accommodate for all possible movements of the robotic arm. In other words, for correct operation, the trunking has an optimal operational position for a given orientation of the robotic manipulator. A further measure is to use a depth camera as part of the vision system 68. This ensures that during a picking operation, in which one item is transferred from a first container to a second container, the item is placed such that it does not protrude beyond a top of the second container. Otherwise, this presents an operational risk where a load-handling device passing over the second container could derail, especially if the protrusion is beyond a top surface of the grid. Also, a load-handling device attempting to lift the second container after the item has been placed may fail to do so since it cannot fully receive the second container within. The depth camera ensures that any item placed does not protrude beyond the top of the second container. This is also known as ‘overheight’ condition. If an over-height condition is detected, the robotic arm can attempt to reposition the item so that it no longer protrudes beyond the top of the second container. Yet a further measure is to use the imaging system to ensure that an item has not been placed on a top surface of the grid, such as on the tracks. Otherwise, this presents an operational risk where a load-handling device passing over a top surface of the grid on which the item has been placed could derail. The robotic arm can attempt to remove the item from the top surface of the grid. It will be appreciated that the item may no longer be intact and spillage and / or debris is now on the top surface of the grid. In such a case, the robotic arm may only be able to retrieve parts of the item. For example, if a cardboard carton of milk has failed, the carton itself may be retrieved, but the milk will remain on the top surface of the grid. Another example is a plastic bag of pasta, where the bag may be retrievable but not the pasta. Yet a further measure is to replace certain components as part of a regular maintenance cycle. One such component is at least one filter used as part of the suction assembly. However, a filter may become ineffective before it is due to be replaced, perhaps due to a build up of dirt. Another such component is the suction assembly, or other gripping element used in the end-effector. However, the suction assembly may become ineffective before it is due to be replaced, perhaps due to wear and tear including holes for example. Each of the above measures, in practice, may not be effective at all times. Failures in these measures can be difficult to detect. One option involves using an image sensor located above the ASRS. A human operator can receive a feed from the image sensor located above the ASRS to detect the operational state of the picking station, i.e. whether a warning light is engaged, or a trunking is not in the precise configuration, or an over-height condition has been detected, or an item or debris or a spillage is located on a top surface of the grid, or that a component requires replacement The image sensor located above the ASRS may not have sufficient resolution, even with zoom functionality, for the human operator to detect the failure of the above measures. Further the image sensor above the ASRS may be centrally located relative to a plurality of picking stations and cannot cover all angles necessary to enable the human operator to see that a given measure has failed on a given picking station. It can be appreciated that a plurality of picking stations operating correctly improves the overall efficiency and throughput of the ASRS. If picking stations are not operating correctly, load-handling devices will instead have to move containers to the access point, which typically is located at the bottom of the grid framework structure. This may involve lowering and raising containers over a significant height a significant number of times with a corresponding power and time demand. Thus, ASRS throughput and thus power consumption / demand is adversely affected when picking stations are not operating correctly. It is thus essential to determine the operational state of each picking station at any given time to maintain system throughput. With reference to Figure 5, a method according to the invention is described. The method is used to detect an operational state of a system comprising a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station. The system may be that described above in reference to Figures 1-4. In step 510, image data of a second picking station is obtained from the image sensor of the first picking station. With reference to Figure 6, the first picking station may be picking station 50a and the second picking station may be picking station 50b. Thus, picking station 50a may orient its image sensor towards picking station 50b. In step 520, the image data is processed with an object detection model trained to detect an operational state of a picking station on the grid. An example object detection model that may be used is described in further detail below. In general, the operational state corresponds to any indication as to whether the picking station is operating correctly. Accordingly, the operational state is determined by at least whether the warning signal is engaged, and / or whether trunking is displaced from an optimal operational position fora current orientation of the picking station, and / or an over-height condition is detected, and / or an item and / or debris and / or a spillage is located on a top of the grid near a picking station, and / or a component requires replacement. In step 530, as a result of the processing in step 520, the operational state of the second picking station is determined. In one example, it is determined whether the warning signal is engaged, and / or whether trunking is displaced from an optimal operational position for a current orientation of the picking station, and / or an over-height condition is detected, and / or an item and / or debris and / or a spillage is located on a top of the grid near a picking station, and / or a component requires replacement. In step 540, following on from determining the operational state in step 530, annotation data indicating the operational state is output. The annotation data may be general and indicate in a binary manner whether the second picking station is operating correctly or it is not. The annotation data may alternatively or additionally indicate whether the warning signal is engaged, and / or whether trunking is displaced from an optimal operational position for a current orientation of the picking station, and / or an over-height condition is detected, and / or an item and / or debris and / or a spillage is located on a top of the grid near a picking station, and / or or a component requires replacement. The method of Figure 5 may further comprise outputting an updated version of the image data including the annotation data. A bounding box may be used to generally indicate whether a picking station is operating correctly. Additionally or alternatively, further bounding boxes can be used to indicate whether the warning signal is engaged, and / or whether trunking is displaced from an optimal operational position for a current orientation of the picking station, and / or an over-height condition is detected, and / or an item and / or debris and / or a spillage is located on a top of the grid near a picking station, and / or a component requires replacement. For example, a bounding box may be colour coded and one example is green to indicate correct operation and red to indicate incorrect operation. The method of Figure 5 has operational advantages compared to an overhead imaging sensor. With reference to Figure 6, it can be appreciated that the picking stations 50a-d can be located close to each other. A closer view of each picking station can be obtained using the imaging sensor of an adjacent picking station. Accordingly the resolution of the imaging sensor is not as critical to the process compared to an overhead imaging sensor. To assist with this process, step 520 may involve orientating the robotic manipulator 52a to direct the image sensor of the robotic manipulator of the first picking station 50a towards the second robotic manipulator of the second picking station. This may include orientating the robotic arm and / or end effector to direct the image sensor to the second picking station. In other words, the part of the robotic manipulator 52a on which the imaging sensor is mounted is oriented accordingly. Additionally and / or alternatively, step 520 may involve instructing the robotic manipulator of the first picking station to execute a first pre-determined movement sequence to facilitate inspection of the second picking station by the image sensor of the first picking station. In other words, the robotic manipulator of the first picking station may ensure the image sensor of the first picking stations images the second picking stations from advantageous elevation and azimuth ranges. This means that the operational state of the second picking station is more likely to be observed by the first station. In particular, it is more likely that a warning light and / or trunking, and / or that an over-height condition and an item (and / or debris and / or a spillage) on a top surface of the grid, and / or a component can be observed. Additionally and / or alternatively, step 520 may involve instructing the robotic manipulator of the second picking station to execute a second pre-determined movement sequence to facilitate inspection by the image sensor of the first picking station. For example, the second pre-determined movement sequence may involve moving each joint / articulation of the robotic manipulator (i.e. the robotic arm and end effector) of the second picking station through a respective full range of motion such that all possible poses are adopted. This means that the operational state of the second picking station is more likely to be observed by the first station. In particular, it is more likely that a warning light and / or trunking can be observed, and / or that an over-height condition and / or an item on a top surface of the grid is unobscured. With reference to Figure 5, once the annotation data has been output, it may in certain instances be used to control the second picking station as per optional step 550. For example, in the event an over-height condition has been detected, the second picking station can be instructed to remove the item giving rise to the over-height condition. The image sensor of the first picking station can be used to verify that the over-height condition has been removed. Similarly, in the event an item and / or debris on a top surface of the grid has been detected, the second picking station can be instructed to remove the item. The image sensor of the first picking station can be used to verify that the item has been removed. This control process may be automated or it may involve a manual override where a human operator controls the second picking station via a visual feed and the master controller. It will be appreciated that the image sensor of a first picking station alone may be used as an input independent of the method of Figure 5. That is, the image sensor of the first picking station can be used as an input to control the second picking station without first having to apply the object detection model of Figure 5. Additionally and / or alternatively, upon determining an operational state of the second picking station, it may be necessary to set up an exclusion zone around the second picking station. The exclusion zone is set up by the master controller to ensure that loadhandling device are prevented from entering the exclusion zone. This can prevent the load-handling devices potentially derailing or falling over due to an over-height condition and / or an item located on a top surface of the grid. With reference to Figure 7, there is a method 700 to improve the confidence with which an operational state of the second picking station is determined. Upon using the method of Figure 5 to determine an operational state of the second picking station, in step 710, image data of the second picking station is obtained from the image sensor of a third picking station. With reference to Figure 6, the third picking station may be picking station 50c. Thus, picking station 50c may orient its image sensor towards picking station 50b. In step 720, the image data of the third picking station is processed with the object detection model trained to detect an operational state of a picking station on the grid described above. In step 730, as a result of the processing, in step 720, the operational state of the second picking station is determined. In step 740, following on from determining the operational state in step 730, annotation data verifying the operational state, is output. That is, if the both the image data from the first and third picking devices, when processed using the object detection model, result in the determining the same operational state of the second picking device, it can be assumed with high confidence that the second picking station is operating correctly or incorrectly as the case may be. It will be appreciated that a further picking station, 50d, may also perform the steps of Figure 7 for further verification of the operational state of the second picking station. It will also be appreciated that optional step 750 allows the image sensor(s) of the third (and / or fourth) picking station to control the second picking station as described above for step 550. In such as case, all of the image sensors used may be used to generate an augmented visual feed. Again, the image sensor(s) of the third (and / or fourth) picking station(s) alone may be used as an input independent of the method of Figure 5. That is, the image sensor(s) of the third (and / or fourth) picking station(s) can be used as an input to control the second picking station without first having to apply the object detection models of Figures 5 and 7. The above methods can also be used to determine the operational state of a fleet 800 of picking stations such as that shown in Figure 8. That is, each picking station can be the first picking station that performs the method of Figure 5 on an adjacent second picking station. In other words, the method of Figure 5 is iterated until each picking station of a fleet of picking stations has had its operational state determined. The method of Figure 7 may be applied to verify the operational state of each picking station. That is, at least two picking stations are used to verify the operational status of a given picking station in the fleet. This allows the current operating state of the fleet to be determined. With reference to Figure 8, one or more of picking stations 50a, 50e, and 50f are used to inspect picking station 50b using the method(s) of Figure 5 and / or Figure 6. Then, one or more of picking stations 50b, 50e, and 50f may be used to inspect picking station 50a using the method(s) of Figure 5 and / or Figure 6. The order or combination chosen does not matter provided each of picking stations 50a-o have been inspected by at least one other of picking stations using the method(s) of Figure 5 and / or Figure 6. Each picking station may carry out the method method(s) of Figure 5 and / or Figure 6 when it currently has no picking tasks assigned. This ensures throughput is maintained. In one example, the robotic arm of the picking station with no outstanding picking tasks can extend fully vertically whilst maintaining the end effector in a substantially horizontal orientation to inspect the desired picking station. Ideally, 100% of the fleet will be operating correctly. If this is not the case, and for example 90% of the fleet is operating correctly, further analysis can be performed. If the 10% of picking stations not operating correctly are distributed randomly throughout the fleet, they can be inspected as part of a regular maintenance cycle. However, if the 10% of picking stations not operating correctly are in the same area of the fleet, an urgent maintenance operation can be performed. An exclusion area isolating the section of the grid with the 10% of picking stations not operating correctly may be set up. This ensures ASRS throughput reduction is not exacerbated further such as multiple load-handling devices derailing if there are several items on the top of the grid, for example. Whilst the above methods concern determining the operating state of the picking stations, the method can be used to additionally or alternatively determine the operating state of the load-handling devices. In such a case, the imaging sensor of a first picking station obtains image data of a load-handling device. To assist in the process, the load handling device may move to a location that is central to and equidistant to at least two picking stations. In such a case, each picking device should be able to provide images of 3 sides and a top of the load-handling device. This allows all angles of a load-handling devices to be obtained. Four picking devices may be used to maximise the angles available so the load-handling device may be located central to and equidistant to fours picking stations. The object detection model used in any of these scenarios may be that described in patent applications PCT / EP2022 / 083344 or PCT / EP2022 / 083361, both of which are incorporated by reference. Thus, the operating state of the entire fleet of robotic devices (i.e. the load-handling devices and the picking stations) used in the ASRS can be determined. For example, the location (relative to the picking station for example) of any given load-handling device, as well as its operational state can be determined, such as whether a warning signal is engaged, or whether components need replaced, such as a wheel or body part or any part / component shown in PCT / EP2022 / 051652, herby incorporated by reference, or whether trunking needs reconfigured, or whether the load-handling device has derailed. Thus, the multiple image sensors of the picking devices allows the operational state of the ASRS devices (i.e. the picking and load-handling devices) to be continuously monitored. Although the above methods discuss the using the image sensor of the first picking station to inspect another device, such as another picking station or a load-handling device, the first picking station can instead inspect itself. In such a case, the first picking station can use the object detection model to determine whether its warning signal is engaged, and / or whether its trunking is displaced from an optimal operational position for its current orientation, and / or an over-height condition is detected nearby, and / or an item and / or debris and / or a spillage is located on a top of the grid near a picking station, and / or a component requires replacement. This may be an independent process as part of a routine maintenance cycle or a first step in which the first picking station becomes the second picking station of the above methods. Object detection model Figure 9 shows an example of a neural network architecture. The example neural network 90 is a convolutional neural network (CNN). An example of a CNN is the U-Net architecture developed by the Computer Science Department of the University of Freiburg, although other CNNs are usable e.g. the VGG-16 CNN. An input 91 to the CNN 90 comprises image data in this example. The input image data 91 is a given number of pixels wide and a given number of pixels high and includes one or more colour channels (e.g. red, green and blue colour channels). Convolutional layers 92, 94 of the CNN 90 typically extract particular features from the input data 91, to create feature maps, and may operate on small portions of an image. Fully connected layers 96 use the feature maps to determine an output 97, e.g. classification data specifying a class of objects predicted to be present in the input image 91. In the example of Figure 9, the output of the first convolutional layer 92 undergoes pooling at a pooling layer 93 before being input to the second convolutional layer 94. Pooling, for example, allows values for a region of an image or a feature map to be aggregated or combined, e.g. by taking the highest value within a region. For example, with 2x2 max pooling, the highest value of the output of the first convolutional layer 92 within a 2x2 pixel patch of the feature map output from the first convolutional layer 92 is used as the input to the second convolutional layer 94, rather than transferring the entire output. Thus, pooling can reduce the amount of computation for subsequent layers of the neural network 90. The effect of pooling is shown schematically in Figure 9 as a reduction in size of the frames in the relevant layers. Further pooling is performed between the second convolutional layer 94 and the fully connected layer 96 at a second pooling layer 95. It is to be appreciated that the schematic representation of the neural network 90 in Figure 9 has been greatly simplified for ease of illustration; typical neural networks may be significantly more complex. In general, neural networks such as the neural network 90 of Figure 9 may undergo what is referred to as a “training phase”, in which the neural network is trained for a particular purpose. A neural network typically includes layers of interconnected artificial neurons forming a directed, weighted graph in which vertices (corresponding to neurons) or edges (corresponding to connections) of the graph are associated with weights, respectively. The weights may be adjusted throughout training, altering the output of individual neurons and hence of the neural network as a whole. In a CNN, a fully connected layer 96 typically connects every neuron in one layer to every neuron in another layer, and may therefore be used to identify overall characteristics of an image, such as whether the image includes an object of a particular class, or a particular instance belonging to the particular class. In the present context, the neural network 90 is trained to perform object identification by processing image data, e.g. to determine whether an object of a predetermined class of objects is present in the image (although in other examples the neural network 90 may have been trained to identify other image characteristics of the image instead). Training the neural network 90 in this way for example generates weight data representative of weights to be applied to image data (for example with different weights being associated with different respective layers of a multi-layer neural network architecture). Each of these weights is multiplied by a corresponding pixel value of an image patch, for example, to convolve a kernel of weights with the image patch. The object detection model may be a neural network, e.g. a convolutional neural network, trained to perform object detection of picking stations 50 and / or load-handling devices on the grid 15 of the workspace. The description of neural networks with respect to Figure 9 therefore applies to the methods described above. Accordingly the object detection model, e.g. CNN 90, is trained to perform object identification by processing the obtained image data to determine whether an object of a predetermined class of objects. Training the neural network 90, for example, involves providing training images of: • picking stations with different operational states including where a warning signal is engaged and not engaged, and / or trunking is displaced or not displaced from an optimal operational position for a current orientation of the picking station, and / or an over-height condition is or is not detected, and / or an item and / or debris and / or a spillage is or is not located on a top of the grid near a picking station, and / or a component (e.g. the filter or suction assembly) that does or does not need replaced; it will be appreciated that this will use images taken from different viewpoints and involving different poses of the robotic manipulator / arm of the picking station; and / or • load-handling devices at different positions on the grid (e.g. relative to a picking station) in different operational states including moving and / or derailment, and / or with a warning engaged and not engaged, and / or with labels / identifiers / markings used to identify the load-handling device, and / or with components that need replaced and do not need replaced, and / or with trunking that is displaced or not displaced from an optimal operational position. Weight data then is generated for the respective (convolutional) layers 92, 94 of a multilayer neural network architecture and stored for use in implementing the trained neural network. In examples, the object detection model comprises a “You Only Look Once” (YOLO) object detection model, e.g. YOLOv4 or Scaled-YOLOv4, which has a CNNbased architecture. Other example object detection models include neural-based approaches such as RetinatNet or R-CNN (Regions with CNN features) and non-neural approaches such as a support vector machine (SVM) to do the object classification based on determined features, e.g. Haar-like features or histogram of oriented gradients (HOG) features. In examples employing storage to store data, the storage may be a random-access memory (RAM) such as DDR-SDRAM (double data rate synchronous dynamic randomaccess memory). In other examples, the storage may include non-volatile memory such as Read-Only Memory (ROM) or a solid-state drive (SSD) such as Flash memory. The storage in some cases includes other storage media, e.g. magnetic, optical or tape media, a compact disc (CD), a digital versatile disc (DVD) or other data storage media. The storage may be removable or non-removable from the relevant system. In examples employing data processing, a processor can be employed as part of the relevant system. The processor can be a general-purpose processor such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof designed to perform the data processing functions described herein. In examples involving a neural network, a specialised processor may be employed as part of the relevant system. The specialised processor may be an NPU, a neural network accelerator (NNA) or other version of a hardware accelerator specialised for neural network functions. Additionally or alternatively, the neural network processing workload may be at least partly shared by one or more standard processors, e.g. CPU or GPU. The term “annotation data” has been used throughout the description and is envisaged to correspond with prediction data or inference data in alternative nomenclature. For example, the object detection model (e.g. comprising a neural network) may be trained using annotated images, e.g. images with annotations such as bounding boxes, which serve as a ground truth for the model, e.g. a prediction or inference with a confidence of 100% or 1 when normalised. These annotations may be made by a human for the purposes of training the model, for example. Thus, the object detection of the present disclosure can be taken to involve outputting prediction data or inference data (e.g. instead of “annotation data”) to indicate a prediction or inference of the picking device and / or load-handling device in the image. The prediction data or inference data may be represented as an annotation applied to the image, e.g. a bounding box and / or a label. The prediction data or inference data includes a confidence associated with the prediction or inference of the transport device in the image, for example. The annotation can be applied to the image based on the generated prediction data or inference data, for example. For instance, the image may be updated to include a bounding box surrounding the picking device and / or load-handling device with a label indicating the confidence level of the prediction, e.g. as a percentage value or a normalised value between 0 and 1. It is also to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the accompanying claims. The following is a list of embodiments which may be or are claimed. 1. A computer-implemented method of detecting an operational state of a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, the method comprising: obtaining, from the image sensor of a first picking station, image data of the first picking station; processing the image data with an object detection model trained to detect an operational state of a picking station on the grid; determining, based on the processing, an operational state of the first picking station; and outputting annotation data indicative of the operational state of the first picking station. 2. The computer-implemented method of embodiment 1, wherein the method further comprises orienting the robotic manipulator of the first picking station to direct the image sensor of the robotic manipulator of the first picking station towards the first robotic manipulator and / or the respective grid cells. 3. The computer-implemented method of embodiments 1-2, wherein determining the operational state comprises determining whether a warning signal is engaged on the first picking station. 4. The computer-implemented method of embodiments 1-3, wherein determining the operational state comprises determining whether an item within a container in one of the respective cells of the first picking station extends above a top surface of the grid or a top of the container. 5. The computer-implemented method of embodiments 1-4, wherein determining the operational state comprises determining whether an item and / or debris and / or a spillage is located on a top surface of the grid. 6. The computer-implemented method of embodiments 1-5, wherein each robotic manipulator further comprises trunking to protect supply lines of the robotic manipulator, wherein determining the operational state comprises determining whether the trunking is displaced from an optimal operational position for a current orientation of the robotic manipulator of the first picking station. 7. The computer-implemented method of embodiments 1-6, wherein determining the operational state comprises determining whether a component, such as a filter or suction assembly of an end-effector, of the second picking station requires replacement. 8. The computer-implemented method of embodiments 1-7, wherein each robotic manipulator comprises an end effector, wherein the end effector has full mechanical degrees of freedom via orientation of the end-effector and / or via orientation of the robotic manipulator. 9. The computer-implemented method of embodiments 1-8, wherein the end effector comprises the image sensor. 10. The computer-implemented method of embodiments 1-9, wherein the method comprises outputting an updated version of the image data including the annotation data. 11. The computer-implemented method of embodiments 1-10, wherein the annotation data comprises a bounding box. 12. The computer-implemented method of embodiments 1-10, wherein the object detection model comprises a convolutional neural network. 13. The computer-implemented method of embodiments 1-12, wherein: the robotic manipulator of the first picking station executes a first pre-determined movement sequence to facilitate inspection of the first picking station by the image sensor. 14. The computer-implemented method of embodiment 1, wherein the method further comprises using the image sensor of the first picking station to set-up an exclusion zone around the first picking station. 15. The computer-implemented method of embodiments 1-14, further comprising: obtaining, from the image sensor of a second picking station, further image data of the first picking station; processing the further image data with the object detection model trained to detect an operational state of a picking station on the grid; determining, based on the processing, an operational state of the first picking station; and outputting annotation data to verify the operational state of the first picking station. 16. The computer-implemented method of embodiment 15, wherein the method further comprises orienting the robotic manipulator of the second picking station to direct the image sensor of the robotic manipulator of the second picking station towards the first robotic manipulator of the first picking station. 17. A computer-implemented method, wherein the method of any preceding embodiment is applied iteratively such that an operational state of each picking station of the plurality of picking stations is determined. 18. A computer-implemented method of detecting an operational state of a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, the method comprising: obtaining, from the image sensor of a first picking station, image data of a second picking station; and using the image sensor of the first picking station to control the second picking station. 19. A computer-implemented method of detecting an operational state of a loadhandling device on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image sensor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, each load-handling device configured to move along the grid in transverse directions and lift and / or lower a container from and / or to beneath a top of the grid the method comprising: obtaining, from the image sensor of a first picking station, image data of a loadhandling device; processing the image data with an object detection model trained to detect an operational state of a load-handling device on the grid; determining, based on the processing, an operational state of the load-handling device; and outputting annotation data indicative of the operational state of the load-handling device. 20. The computer-implemented method of embodiment 19, wherein the method further comprises orienting the robotic manipulator of the first picking station to direct the image sensor of the robotic manipulator of the first picking station towards the loadhandling device. 21. The computer-implemented method of embodiments 19-20, wherein determining the operational state comprises determining whether the load-handling device is moving, and / or has derailed, and / or whether a warning signal is engaged on the load-handling device, and / or whether a component, such as a wheel or body part, needs replaced. 22. The computer-implemented method of embodiments 19-21, wherein each robotic manipulator comprises an end effector, wherein the end effector has full mechanical degrees of freedom via orientation of the end-effector and / or via orientation of the robotic manipulator. 23. The computer-implemented method of embodiments 19-22, wherein the end effector comprises the image sensor. 24. The computer-implemented method of embodiments 19-23, wherein the method comprises outputting an updated version of the image data including the annotation data. 25. The computer-implemented method of embodiments 19-24, wherein the annotation data comprises a bounding box. 26. The computer-implemented method of embodiments 19-25, wherein the object detection model comprises a convolutional neural network. 27. The computer-implemented method of embodiments 19-26 wherein: the robotic manipulator of the first picking station executes a first pre-determined movement sequence to facilitate inspection of the load-handling device by the image sensor. 28. The computer-implemented method of embodiment 27, wherein the method further comprises using the image sensor of the first picking station to set-up an exclusion zone around the load-handling device. 29. The computer-implemented method of embodiments 19-28, further comprising: obtaining, from the image sensor of a second picking station, further image data of the load-handling device; processing the further image data with the object detection model trained to detect an operational state of a load-handling device on the grid; determining, based on the processing, an operational state of the load-handling device; and outputting annotation data to verify the operational state of the load-handling device. 30. The computer-implemented method of embodiment 29, wherein the method further comprises orienting the robotic manipulator of the second picking station to direct the image sensor of the robotic manipulator of the second picking station towards the load-handling device. 31. The computer-implemented method of embodiments 19-30, wherein the loadhandling device is located central to and equidistant to at least two picking stations. 32. The computer-implemented method of embodiments 19-31, wherein the grid based storage structure comprises a first set of parallel rails or tracks and a second set of parallel rails or tracks extending substantially perpendicular to the first set of rails or tracks in a substantially horizontal plane to form the grid comprising the plurality of grid spaces, wherein the grid is supported by a set of uprights to form a plurality of vertical storage locations beneath the grid for containers to be stacked between and be guided by the uprights in a vertical direction through the plurality of grid spaces. 33. The computer-implemented method of embodiment 32, wherein the loadhandling device comprises: a body or skeleton mounted on a first set of wheels being arranged to engage with the first set of parallel tracks and a second set of wheels being arranged to engage with the second set of parallel tracks; a drive assembly configured to drive the first or second sets of wheels to move the load-handling device along the first or second set of parallel rails respectively; a direction-change assembly configured to raise or lower the first set of wheels and / or lower or raise the second set of wheels with respect to the body or skeleton to engage and disengage the wheels with the parallel tracks; and a container-lifting assembly configured to raise or lower a gripping device in the vertical direction. 34. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding embodiment. 35. A computer readable medium comprising the computer program of embodiment 34. 36. A data processing system comprising means for carrying out the method of embodiments 1-33. 37. A system comprising: a grid comprising a plurality of grid cells forming part of a grid-based storage system; one or more picking stations mounted on the grid and / or one or more loadhandling devices, each picking station comprising a robotic manipulator comprising an image sensor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, each loadhandling device configured to move along the grid in transverse directions and lift and / or lower a container from and / or to beneath a top of the grid; and a processor configured to carry out the method of claims 1-32. 38. The system of embodiment 37, wherein the grid based storage structure comprises a first set of parallel rails or tracks and a second set of parallel rails or tracks extending substantially perpendicular to the first set of rails or tracks in a substantially horizontal plane to form the grid comprising the plurality of grid spaces, wherein the grid is supported by a set of uprights to form a plurality of vertical storage locations beneath the grid for containers to be stacked between and be guided by the uprights in a vertical direction through the plurality of grid spaces. 39. The system of embodiment 38, wherein the load-handling device comprises: a body or skeleton mounted on a first set of wheels being arranged to engage with the first set of parallel tracks and a second set of wheels being arranged to engage with the second set of parallel tracks; a drive assembly configured to drive the first or second sets of wheels to move the load-handling device along the first or second set of parallel rails respectively; a direction-change assembly configured to raise or lower the first set of wheels and / or lower or raise the second set of wheels with respect to the body or skeleton to engage and disengage the wheels with the parallel tracks; and a container-lifting assembly configured to raise or lower a gripping device in the vertical direction.

Claims

1. A computer-implemented method of detecting an operational state of a picking station on a grid comprising a plurality of grid cells, the grid forming part of a grid-based storage system in which one or more picking stations are mounted on the grid, each picking station comprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station, the method comprising:obtaining, from the image sensor of a first picking station, image data of a second picking station;processing the image data with an object detection model trained to detect an operational state of a picking station on the grid;determining, based on the processing, an operational state of the second picking station; andoutputting annotation data indicative of the operational state of the second picking station.

2. The computer-implemented method of claim 1, wherein the method further comprises orienting the robotic manipulator of the first picking station to direct the image sensor of the robotic manipulator of the first picking station towards the second robotic manipulator of the second picking station.

3. The computer-implemented method of claims 1-2, wherein determining the operational state comprises determining whether a warning signal is engaged on the second picking station.

4. The computer-implemented method of claims 1-3, wherein determining the operational state comprises determining whether an item within a container in one of the respective cells of the second picking station extends above a top surface of the grid or a top of the container.

5. The computer-implemented method of claims 1-4, wherein determining the operational state comprises determining whether an item and / or debris and / or a spillage is located on a top surface of the grid.

6. The computer-implemented method of claims 1-5, wherein each robotic manipulator further comprises trunking to protect supply lines of the robotic manipulator, wherein determining the operational state comprises determining whether the trunking is displaced from an optimal operational position for a current orientation of the robotic manipulator of the second picking station.

7. The computer-implemented method of claims 1-6, wherein determining the operational state comprises determining whether a component, such as a filter, or suction assembly of an end-effector, of the second picking station requires replacement8. The computer-implemented method of claims 1-7, wherein each robotic manipulator comprises an end effector, wherein the end effector has full mechanical degrees of freedom via orientation of the end-effector and / or via orientation of the robotic manipulator.

9. The computer-implemented method of claims 1-8, wherein the end effector comprises the image sensor.

10. The computer-implemented method of claims 1-9, wherein the method comprises outputting an updated version of the image data including the annotation data.

11. The computer-implemented method of claims 1-10, wherein the annotation data comprises a bounding box.

12. The computer-implemented method of claims 1-11, wherein the object detection model comprises a convolutional neural network.

13. The computer-implemented method of claims 1-12 wherein:the robotic manipulator of the first picking station executes a first pre-determined movement sequence to facilitate inspection of the second picking station by the image sensor; and / orthe robotic manipulator of the second picking station executes a second predetermined movement sequence to facilitate inspection by the image sensor of the first picking station.

14. The computer-implemented method of claim 13, wherein the method further comprises using the image sensor of the first picking station to control the second picking station.

15. The computer-implemented method of claim 14, wherein the method further comprises using the image sensor of the first picking station to set-up an exclusion zone around the second picking station.

16. The computer-implemented method of claims 1-15, further comprising: obtaining, from the image sensor of a third picking station, further image data of the second picking station;processing the further image data with the object detection model trained to detect an operational state of a picking station on the grid;determining, based on the processing, an operational state of the second picking station; andoutputting annotation data to verify the operational state of the second picking station.

17. The computer-implemented method of claim 16, wherein the method further comprises orienting the robotic manipulator of the third picking station to direct the image sensor of the robotic manipulator of the third picking station towards the second robotic manipulator of the second picking station.

18. A computer-implemented method, wherein the method of any preceding claim is applied iteratively such that an operational state of each picking station of the plurality of picking stations is determined.

19. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding claim.

20. A computer readable medium comprising the computer program of claim 19.

21. A data processing system comprising means for carrying out the method ofclaims 1-18.

22. A system comprising:a grid comprising a plurality of grid cells forming part of a grid-based storage system;5 one or more picking stations mounted on the grid, each picking stationcomprising a robotic manipulator comprising an image senor, wherein the robotic manipulator is configured to transfer items between containers received in respective grid cells adjacent the picking station; anda processor configured to carry out the method of claims 1-18.10

Images