Systems and methods for determining container status

Abstract

The disclosure relates to a method for use in an automated storage and retrieval system comprising a storage grid including a plurality of stacked containers, the method comprising: capturing, by at least one sensor on a robotic container-handling vehicle, data indicative of a status of a container of the plurality of stacked containers; determining the status of the container based on the data; and controlling an operation of the robotic container-handling vehicle based on the status of the container.

Description

SYSTEMS AND METHODS FOR DETERMINING CONTAINER STATUSTECHNICAL FIELD

[0001] The disclosure relates to systems and methods for determining a status of a container. More particularly, it relates to systems and method for determining a status of a container in an automated storage and retrieval system or in a vertical farming system. More particularly, it relates to methods of determining a status of a container using a sensor on a robotic container-handling vehicle, and to a robotic containerhandling vehicle for determining a status of a container.BACKGROUND

[0002] Traditional storage solutions usually involve the arrangement of goods on rows of shelves within a warehouse. The shelf location for each item is recorded in an inventory, and goods are retrieved from the shelves by a stock picker. The shelves are restocked and the inventory updated, as needed, as goods enter and leave the warehouse.

[0003] Warehouse workers may be assisted by robotic pickers and by automated inventory management systems. Automated transit systems may also be implemented in traditional warehouse set-ups to move goods from their inventory location to a picking and / or packing station.

[0004] An alternative to a traditional warehouse set-up is an automated storage and retrieval system in which robots retrieve items from their logged location within the warehouse and deliver the items to a packing station or port. Such systems can reduce or eliminate the space needed to pass between rows of shelves to access stock, thereby removing the need for broad aisles within the warehouse. One example of such a system involves placing goods in bins or containers that are configured to be stacked, side by side, within a three-dimensional grid. A rail system is arranged on top of the grid, along which robotic container-handling vehicles configured to lift containers from the grid can travel. The container-handling vehicles are configured to transport containers from the grid and to deliver them to ports or stations at the periphery of the grid so that the goods within the container can be picked and / or packed.

[0005] Another example use of an automated storage and retrieval system is vertical farming in which storage containers within the system can be used to grow and / or store organisms (e.g. crops, plants or other biological matter). One knownvertical farming system uses frames which serve as storage containers to be stored one on top of another in a vertical stack.

[0006] Given the vast selection of items that can be stored in the automated storage and retrieval system and the potentially high frequency at which items are added to and removed from containers, it can be difficult to monitor the status of containers. It is important to accurately track the status of each container, so that orders can be correctly fulfilled and so that robotic container-handling vehicles do not spend time transporting or attempting to transport containers unnecessarily. For example, it is desirable that robotic container-handling vehicles do not transport empty containers that are mistakenly thought to contain items, since this reduces the overall efficiency of the automated storage and retrieval system. Likewise it is important that robotic container-handling vehicles do not transport containers containing items that are mistakenly thought to be empty.

[0007] In another example, it is important that robotic container-handling vehicles do not attempt to lift containers which, due to the status of the container, cannot be lifted by the robotic container-handling vehicle.

[0008] In yet another example, it is important that containers that have content which exceeds a height or volume capacity of the container do not interfere with the normal operation of the automated storage and retrieval system.

[0009] In a yet further example, it is important that containers or modules containing plants or organisms are handled (e.g. situated in the correct position within the automated storage and retrieval system) for optimum growth, health, development or harvest.

[0010] In view of these and associated problems, one solution is to keep track of the status (e.g. contents) of containers by periodically removing containers from the storage grid and assessing them at another area of the automated storage and retrieval system, e.g. by an operator at an access station. However, this solution is not efficient, and is prone to inaccuracies.SUMMARY

[0011] One or more aspects of an invention are set out in the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The disclosure will now be described in more detail in connection with a number of exemplary embodiments shown in the accompanying drawings, in which:Fig. 1 shows a perspective view of a storage system or vertical farming system comprising a grid and a plurality of robotic container-handling vehicles configured to retrieve and / or rearrange goods stored within the grid;Fig. 2 shows a top view of the system of Fig. 1;Fig. 3A shows a side view of a first robotic container-handling vehicle suitable for use in the system of Fig. 1;Fig. 3B shows a side view of a second robotic container-handling vehicle suitable for use in the system of Fig. 1;Fig. 3C is an orthographic view of the robot of Fig. 3B;Fig. 4 shows a computing device for implementing the operations described herein;Fig. 5 shows a bottom perspective view of a robot of the type shown in Fig. 3B;Fig. 6 shows a flow chart of a method for collecting data on container content;Fig. 7 shows a flow chart of a method for processing collected data on container content;Fig. 8 shows a flowchart of a method of determining a position of a robotic container-handling vehicle;Fig. 9 shows a flowchart of a method of identifying a position of a storage container from a plurality of candidate positions of the storage container;Fig. 10 shows a schematic diagram of weights of storage containers and stack heights of storage containers.Fig. 11 shows a flowchart of a method of determining a content of a storage container.Fig. 12 shows a flowchart of a method of determining an orientation of a storage container.Fig. 13 shows a flowchart of a method of determining a status of one or more plants stored in a storage container.Fig. 14 shows a first storage container storing an item that exceeds a height threshold of the container.Fig. 15 shows a second storage container storing an item that exceeds a height threshold of the container.Fig. 16 shows a stack of storage containers in which one of the containers is improperly stacked.DETAILED DESCRIPTION

[0013] In overview, the disclosure relates to a system or computer-implemented method for determining a status of a container in an automated storage and retrieval system. In general, the method involves capturing data associated with a container and / or its contents, inferring information about (i.e. the status of) the container and / or its contents from the data, and then controlling a robotic container-handling vehicle based on this information. First, data indicative of a status of the container is captured by a sensor on a robotic container-handling vehicle (e.g. by a sensor disposed on a lifting device or a cantilever portion of the robotic container-handling vehicle). Then, the operation of the robotic container-handling vehicle is controlled based on the status of the container (e.g. the robotic container-handling vehicle is controlled to transport the container to another location within the storage grid, or out of the storage grid; or to avoid, abort or cease lifting of the container).

[0014] The sensor enables monitoring of the status of a container positioned under or below the robotic container-handling vehicle so that the container does not have to be removed from its position for the container and / or its contents to be inspected. Example implementations include but are not limited to:• sensing the contents of the container and operating the robotic containerhandling vehicle based on whether the container contains the expected content, for example: o the container is expected to be empty and it contains items; o the container is expected to contain items and it is empty; o the container is expected to contain a certain set of items, but instead it contains another set of items different from the expected set;• sensing data indicative of the orientation of the container and operating the robotic container-handling vehicle based on whether the container is in an expectedorientation, for example based on whether the container is in a skewed or tilted position relative to other storage containers in the automated storage and retrieval system.• sensing data indicative of the condition (such as growth stage, health or development stage) of plants or organisms in the container and operating the robotic container-handling vehicle based on the condition of the plants or organisms.• sensing the height and / or volume of the contents of the container and operating the robotic container-handling vehicle based on whether the contents of the container protrude above an opening of the container (such as when the volume and / or height of the contents of the container exceed the internal volume and / or internal height of the container).• sensing the weight of the container and its contents and operating the robotic container-handling vehicle based on the weight of the container. In this case the weight may indicate to the robotic container-handling vehicle the location of the container and hence the robotic container-handling vehicle.

[0015] Based on the sensed data, the robotic container-handling vehicle may be operated to either handle the container in a particular manner or to avoid or cease handling the container altogether, for example by disengaging with the container or by moving away from the container location. If handling the container in the particular manner, the robotic container-handling vehicle may be operated to move the container to an access port or station, to an inspection area, or to another storage location in the automated storage and delivery system. If avoiding or ceasing handling the container, the method may include operating another robotic container-handling vehicle to engage or move the container in place of the (original) robotic container-handling vehicle.

[0016] The systems and methods described herein can be used to improve management of operation of the robotic container-handling vehicles and / or improve management of the storage location(s) of the containers to provide an improved (e.g., more efficient and / or more productive) automated storage and retrieval system.The preceding overview is provided to introduce in simplified form a selection of concepts that are further described herein. The overview is not intended to identify key or essential features of the invention.Automated storage and retrieval systemReferring to the embodiment shown in Fig. 1, a grid too comprises a frame formed by a plurality of generally rectilinear, adjacent vertical columns 102 formed between vertical frame members 104 and extending in the X and Y directions 108, no. The grid elements may be fabricated of any appropriate material; for example, the frame members may be formed of extruded aluminium. Storage containers or bins 112 are stacked on top of each other, preferably in a self-supporting manner, in the Z direction 114 in the columns 102, forming a storage volume of storage cells for respective bins 112 extending in the X, Y and Z directions 108, no, 114.

[0017] A rail system or network 116 is formed on top of the grid too and comprises pairs of vehicle rails or tracks 118a, 118b and 120a, 120b, respectively extending in the X and Y directions 108, no. Robotic container-handling vehicles, or robots, 122, which can be of a range of size, shape and function, are provided and configured to run on the rails 118, 120 and to transport bins 112 in both the X and Y directions 108, no. The robots 122 are additionally configured to lift and lower bins 112 from / into the columns 102 in the Z direction 114, the bins 112 optionally being guided by the vertical frame members 104. The robots 122 access the bins 112 via access openings 124 above the columns 102 and formed between the rails 118, 120.

[0018] Some columns 102 may be used for alternative purposes than bin storage. For example, port columns 126, 128 comprise port or access columns allowing transfer of a bin 112 in and / or out of the grid too. Port columns 126, 128 provide a vertical channel for lifting of a bin 112 from, or lowering of a bin 112 to, a port or ports 130, 132. The ports 130, 132 are shown in Fig. 1 at the lowest level of the grid, however ports can be located at any vertical position along the column. The respective port columns 126, 128 can be assigned for removing (‘drop-off) and / or returning or delivering (‘pick-up’) bins 112 from / to the grid too. The ports 130, 132 are therefore configured to allow bins 112 to be removed and reintroduced (horizontally) into the associated port column. As such, a port 130, 132 can comprise a conveyor (not shown in Fig. 1) onto which a bin 112 may be lowered and transported horizontally out of the port column. The port columns 126, 128 include an opening or access point through which bins 112 can enter and leave the column.

[0019] The storage grid can be used for storage and retrieval of containers and / or for vertical farming applications as described herein.

[0020] Bins (or containers or stackable modules) 112 can be transported along the top of the grid too to and / or from a port column 126, 128 by robots 122, and from a port 130, 132 to a location outside the grid too, which may be an access station (not shown) for processing of the bin 112 or its contents, such as a picking station for adding content to, or removing content from, the bin 112. In alternative examples (not shown), the bin 112 may be transported to a port of another grid on the same or another level, or to an external facility. Transport of bins 112 to and from ports 130, 132 may be by any appropriate means (not shown) including conveyors, transport vehicles, lifts or robots.

[0021] Referring to the embodiment shown in Fig. 2, the X-Y configuration 200 of the rail system 116 can be seen in more detail, together with robots 202, 204 of different types. The rail system includes rails 206 defining between them vertical column access openings 124 for access to bins 112. The rails 206 can be any appropriate type for permitting travel of the robots 202, 204 in the X and Y directions 108, 110 thereon, including (not shown) groove-type rails for receiving vehicle wheels, or protrusion-type rails for engaging wheel recesses. Each rail 206 may comprise a single track or multiple parallel tracks in each of the X and Y directions 108, no.

[0022] A first, ‘cantilever’ type of robot 202 is shown in more detail in Fig. 3A and includes a body 300, a set of wheels 302 and a lifting device 304. The body 300 contains operational equipment (not shown) for the robot 202 including drive, power and control systems. The wheels 302 permit movement of the robot 202 in one of the X and Y directions, an additional set of wheels (not visible in this view) permitting movement in the other of the X and Y directions, in both cases along the respective rails or tracks 206. One or both sets of wheels can be raised or lowered to permit selective engagement of the rails for movement in the desired direction. The lifting device 304 includes a cantilever element 306 extending in the X-Y plane from the top of the body 300, and a gripping device 308, which is raisable and lowerable from the cantilever element 306. The gripping device 308 is configured to grip or engage a bin 112; for example, by gripping a part of the bin 112, or by passively or actively engaging a suitably configured part of the bin 112.

[0023] A second, ‘internal cavity’ type of robot 204 is shown in more detail in Fig. 3B and includes, as an alternative to the cantilevered lifting system, an internal cavity 310 within the body 300 and in which the lifting device 312 including a gripping device (not shown) is located. In this case, the body 300 includes the robot’s operationalequipment and a storage space for one or more bins 112, for use, for example, while transporting the bin 112.

[0024] Fig. 3C shows a perspective side view of the robot of Fig. 3B in which the first set of wheels 302 from Fig. 3B are visible. The additional set of wheels referenced above but not shown in Fig. 3B are shown as wheels 303 in Fig. 3C. The additional set of wheels 303 is arranged perpendicular to the first set of wheels 302, to allow rolling of the robot 204 in the X and Y directions on the first and second set of wheels 302, 303 respectively. The first and second set of wheels 302, 303 shown in Fig. 3C may be configured to be independently lowered into engagement with the rails (and conversely raised out of engagement with the rails) to allow the robot 202 to move in the X and Y direction across the arrangement of rails shown in Fig. 2. Although the perspective view shown in Fig. 3C is of the robot 204 of Fig. 3B, it will be appreciated that a similar perpendicular wheel arrangement may be applied to the robot 202 of Fig. 3A.Control and monitoring system

[0025] Control and monitoring of the automated storage and retrieval system, including monitoring and storing bin position and controlling bin delivery, retrieval and transport and robot routing and collision avoidance, is performed by a control system shown in Fig. 4 in communication with the robots and / or other controllable system components. Control can be performed locally or remotely and may be implemented by a processing system, for example in the form of a computing device. Accordingly, the methods described herein may form all or part of a computer-implemented method, or a system configured to perform the methods described herein.

[0026] With reference to Fig. 4, a processing system 400 suitable for carrying out the methods described herein will now be described. The processing system can be described as one or more controllers. Fig. 4 shows a block diagram of one implementation of a processing system 400 in the form of a computing device within which a set of instructions for causing the computing device to perform any one or more of the methods described herein may be executed. In some implementations, the computing device may be connected (e.g., networked) to other machines in a Local Area Network (LAN), an intranet, an extranet, or the Internet. The computing device may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The computing device may be a personal computer (PC), a tablet computer, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, aweb appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single computing device is illustrated, the term ‘computing device’ shall also be taken to include any collection of machines (e.g., computers) that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods described herein.

[0027] The example processing system 400 includes a processor 402, a main memory 404 (e.g., read-only memory (ROM), flash memory, dynamic random-access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 406 (e.g., flash memory, static random-access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 418), which communicate with each other via a bus 430.

[0028] Processor 402 represents one or more general-purpose processors such as a microprocessor, central processing unit, or the like. More particularly, the processor 402 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 402 may also be one or more special-purpose processors such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processor 402 is configured to execute the processing logic (instructions 422) for performing the operations and steps described herein.

[0029] The processing system 400 may further include a network interface device 408. The processing system 400 also may include any of a video display unit 410 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 412 (e.g., a keyboard or touchscreen), a cursor control device 414 (e.g., a mouse or touchscreen), and an audio device 416 (e.g., a speaker).

[0030] It will be apparent that some features of the processing system 400 shown in Fig. 4 may be absent. For example, the processing system 400 may have no need for display device 410 (or any associated adapters). This may be the case, for example, for particular server-side computer apparatuses which are used only for their processing capabilities and do not need to display information to users. Similarly, user input device 412 may not be required. In its simplest form, processing system 400 comprises processor 402 and main memory 404.

[0031] The data storage device 418 may include one or more machine-readable storage media (or more specifically one or more non-transitory computer-readable storage media) 428 on which is stored one or more sets of instructions 422 embodying any one or more of the methods or functions described herein. The instructions 422 may also reside, completely or at least partially, within the main memory 404 and / or within the processor 402 during execution thereof by the processing system 400, the main memory 404 and the processor 402 also constituting computer-readable storage media 428.

[0032] The various methods described herein may be implemented by a computer program. The computer program may include computer code arranged to instruct a computer (or processor, or one or more controllers) to perform the functions of one or more of the various methods described herein. The computer program and / or the code for performing such methods may be provided to an apparatus, such as a computer, on one or more computer-readable media or, more generally, a computer program product. The computer-readable media may be transitory or non-transitory. The one or more computer-readable media could be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or a propagation medium for data transmission, for example for downloading the code over the Internet. Alternatively, the one or more computer-readable media could take the form of one or more physical computer-readable media such as semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random-access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, or an optical disk, such as a CD-ROM, CD-R / W or DVD.

[0033] The computer program is executable by the processor 402 to perform functions of the systems and methods described herein.

[0034] In an implementation, the modules, components, and other features described herein can be implemented as discrete components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs, or similar devices.

[0035] A ‘hardware component’ is a tangible (e.g., non-transitory) physical component (e.g., a set of one or more processors) capable of performing certain operations and may be configured or arranged in a certain physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be or include a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardwarecomponent may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations.

[0036] Accordingly, the phrase ‘hardware component’ should be understood to encompass a tangible entity that may be physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein.

[0037] In addition, the modules and components can be implemented as firmware or functional circuitry within hardware devices. Further, the modules and components can be implemented in any combination of hardware devices and software components, or only in software (e.g., code stored or otherwise embodied in a machine- readable medium or in a transmission medium).Operation of the automated storage and retrieval system

[0038] In operation, each bin 112 is given a unique identifier, which may be marked on the bin 112 using a computer-readable identifier (e.g., a barcode, quickresponse code or radio-frequency identification tag) to ease identification of the bin 112. A database of the processing system 400 stores, in association with the unique identifier, the position and, optionally, content of each bin 112. When a bin 112 is moved (e.g., when it is retrieved from the grid 100), the database is updated to record its change in position.

[0039] When it is desired to retrieve a bin 112 from the grid 100, under control of the processing system 400, a robot 202, 204 is routed via the rail system 116 to the vertical column 102 including the storage cell where, according to the database, the bin 112 is positioned, and the lifting device 304, 312 is positioned (according to robot type) over the corresponding access opening 124, either adjacent or below the robot 202, 204. The robot 202, 204 lowers the gripping device 308 which engages, grips and lifts the bin 112 to the robot 202, 204. The robot 202, 204 then transports the bin 112, for example, to the drop-off port column 126, 128 for delivery to the port 130, 132 and subsequent processing external to the grid too. In the event that the target or designated bin 112 is below other bins in the stack then the robot 202, 204 or multiple robots, which may be dedicated to the task, are controlled in a ‘digging’ operation to sequentially lift and reposition, temporarily or permanently, bins above the target bin 112 in order for it to be retrieved. It will be appreciated that other operations in relation to the bin 112 can be carried out in a similar manner. For example, a bin 112 can be delivered for storage in the grid too at the port 130, 132 of the pick-up port column 126, 128, gripped and liftedby a robot 202, 204 and delivered to the desired storage cell, bins above the desired position being repositioned if necessary as discussed above.Controlling lifting of containers

[0040] The first, ‘cantilever’ type of robot 202 and the second, ‘internal cavity’ type of robot 204 each include a lifting device 304, 312. The lifting device 312 of the second type of robot 204 is shown in Fig. 5. The robot moving on top of the grid too can move over a vertical column 102 of bins 112. The lifting device 304, 312 may then be lowered down to grip a bin 112 and then retracted to lift the bin 112 out of the grid too, e.g., into the cavity 310 of the robot 204 or into a space between the cantilever element 306 of the robot 202 and the top of the grid too. As described below with reference to Fig. 6, a sensor attached to the robot 202, 204 can be triggered to capture data by this retraction of the lifting device 304, 312. This may be a proximity-based trigger, that is configured to sense when the bin 112 has been lifted out of the grid too.Collecting bin content data

[0041] Fig. 6 shows a flow diagram of a computer-implemented method, in an automated storage and retrieval system comprising a storage grid too including a plurality of stacked bins 112, of collecting data on bin content.

[0042] As a computer-implemented method, the method of Fig. 7 may be performed by any data processing system, including the processing system 400, and / or a data processing system on the robot 202, 204. Each of the steps of the method may thus be performed by any such data processing system.

[0043] At step S100, an instruction to transport the first bin to a destination may be received. This step may be performed by a data processing system of the robot 202, 204. The instruction may be received from the robot 202, 204 (e.g., from the processing system 400), or it may have been pre-programmed in a list of tasks for the robot 202, 204 to execute. The destination may be a storage cell in the grid too, a position on top of the grid too, or a destination outside the grid such as a port 300 on the edge of the grid too (e.g., a picking station).

[0044] Then, at step S105, the robot 202, 204 may be controlled to move to the first bin. The exact route taken by the robot 202, 204 may have been calculated separately in dependence on the geometry of the grid too, for example. This enables the robot to be in position for lifting the bin and capturing data on its content. The controlling may comprise a data processing system (e.g., processing system 400)sending an instruction to the robot 202, 204 and / or a data processing system (e.g., a data processing system on the robot 202, 204) controlling the drive system of the robot 202, 204.

[0045] At step S110, the robot 202, 204 may be controlled on the storage grid too to lift a first one of the plurality of stacked bins 112 (a ‘first bin’) out of the storage grid too. The lifting may be carried out by the cantilever-type robot 202 as shown in Fig. 3A, or by the cavity-type robot 204 as shown in Fig. 3B. The lifting may also be carried out by any other robot configured to transport the bin 112 from one position in the grid too to another. The controlling may comprise a data processing system (e.g., processing system 400) sending an instruction to the robot 202, 204, and / or a data processing system (e.g., a data processing system on the robot 202, 204) controlling the lifting device of the robot 202, 204.

[0046] As the first bin is lifted out of the storage grid too (e.g., from the vertical column 102) by the robot 202, 204, the first bin approaches the robot 202, 204. An appropriate, or optimal, time to capture data on its content will depend on what data is to be captured and what type of sensor is used to capture that data. The capturing of data may thus be triggered in many ways. An appropriate, or optimal, time to capture the data may, in general, be at some point after the robot 202, 204 has been controlled to lift the first bin out of the storage grid too and before the robot 202, 204 has been controlled to lift another bin out of the storage grid too. As a result, the captured data can be straightforwardly associated with the bin to which it pertains because the specific bin 112 which is being lifted is already being tracked (e.g., by the processing system 400).

[0047] For example, when capturing image data, an appropriate time to capture the data may be when the first bin is at a distance that results in the image being in focus. As another example, when capturing image data, an appropriate time to capture the data may be when the first bin has not yet been completely lifted by the robot 202, 204, i.e., it has not yet reached its minimum distance with respect to the robot 202, 204 (e.g., not yet fully inside the cavity 310, for robot 204) so that ambient lighting is available. As yet another example, an appropriate time to capture the data may be when the first bin has been completely lifted by the robot 202, 204, i.e., it has reached its minimum distance with respect to the robot 202, 204; this allows data to continue to be captured even while the robot 202, 204 travels to its destination, and may allow more accurate or consistent data to be captured.

[0048] At step S115, it may be determined that the first bin is proximate to the robot 202, 204. Such a determination may, more specifically, be a determination that a position of the first bin meets a (predetermined) proximity criterion. The proximity criterion could, for example, be that the position of the first bin is within a predetermined threshold distance of the robot 202, 204.

[0049] The determination may be made using a sensor of the robot 202, 204 (e.g., in, on or mounted to the robot 202, 204), or based on a predetermined amount of time having elapsed since the robot was (first) controlled to lift the first container out of the storage grid too.

[0050] The determination may be a determination that lifting means 304, 312 of the robot 202, 204 are in a retracted configuration. The retracted configuration may correspond to the first bin being fully lifted by the robot, or at least partially lifted. For example, the retracted configuration of the lifting device 312 shown in Fig. 5 may be when the gripping device has gripped the bin 112 and is fully back up inside the cavity 310 of the robot 202, 204.

[0051] At step S120, data is captured which is indicative of a content of the first bin. The data may be captured using a sensor of the robot 202, 204. As explained above, the capturing may be performed at any time although, in general, the capture is responsive to the controlling (or the lifting itself). The capturing may be responsive to any of the determination(s) made in step S115, e.g., responsive to determining that the first container is proximate to the robot 202, 204, that a position of the first container with respect to the robot 202, 204 meets a proximity criterion, that a predetermined amount of time has elapsed since the robot was controlled to lift the first container out of the storage grid too, that lifting means 304, 312 of the robot 202, 204 are in a retracted configuration, etc. The types of data that may be captured will be set out in more detail below. The sensor of the robot 202, 204 may be any sensor in, on or mounted to the robot 202, 204. This sensor may be the same sensor as used in step S115, or any another sensor described in the present disclosure.

[0052] At step S125, the captured data is stored in association with a unique identifier of the first bin. The manner in which the captured data is stored may depend on the type of data collected. The captured data may be stored in a database, or indeed anywhere in a memory (e.g., a transitory or temporary memory) of, or communicatively coupled to, whatever apparatus performs the storing. The captured data may be stored in a memory of the robot 202, 204.

[0053] The unique identifier may be any identifier capable of uniquely identifying the first bin. In some examples, the unique identifier is a numerical identifier, an alphabetical identifier, or a combination thereof. In some examples, the unique identifier identifies a position of the first bin in or on the grid too (e.g., in the form of three- dimensional coordinates); in this case, the unique identifier of the first bin changes as the first bin moves around the grid too. The unique identifier may not directly identify the first bin; for example, the unique identifier may indicate the sequence in which the data is captured and may then identify the first bin when correlated with a sequence of bins 112 which have been lifted by the robot 202, 204. The unique identifier may be the same identifier used to track the bin in the storage and retrieval system, using processing system 400. The unique identifier may also be used in a warehouse management or inventory management system. The data and the unique identifier may be transmitted for processing at step S135 which is described in more detail below.

[0054] At step S130, the robot 202, 204 may be controlled to move the first bin 112 to the destination contained in the instruction of step S100. The controlling of step S130 may be subsequent to, or consequent to, the capturing of step S120 and / or the storing of step S125 but, more generally, will simply be subsequent to the controlling of step S110 (e.g., when the capturing continues even while the robot 202, 204 travels to its destination). As a result, the operation of the storage and retrieval system is unaffected by the data capture. In some examples, the robot 202, 204 may be controlled to move the first bin 112 to a different destination responsive to processing of the captured data, e.g., if it is determined that the bin contents are low quality for example. This is described in more detail below with reference to Fig. 7.

[0055] Capturing data about the contents of the bins 112 when the bins 112 are moved around the grid too allows an indication of the status of the items inside the bins 112 to be obtained during normal operation, in passing, without needing to remove the bins 112 from the system (e.g., to take them to a picking station). This enables more efficient and continuous monitoring of the contents of bins 112. This is particularly useful when the nature of the items stored is highly time-dependent, e.g., because items are perishable or because items are frequently added to or removed from the bins 112.

[0056] Depending on where the captured data is to be processed, the captured data may need to be transmitted to a(nother) data processing system (e.g., processing system 400). Thus, at step S135, the captured data may be transmitted to a dataprocessing system in association with the unique identifier. Alternatively, the captured data may be processed by a data processing system on the robot 202, 204 itself.

[0057] The captured data may also be transmitted to the warehouse management system for storage. This allows the captured data to be tracked over time, and stored in association with the management data for the container. For example, the warehouse management system may store one or more of the most recent image(s) of the container content, one or more of the most recent temperature reading(s), or one or more of the most recent weight reading(s).

[0058] In one example, the robot 202, 204 is controlled through instructions transmitted via a first network, the captured data is transmitted via a second network, and the first and second networks are different. As a result, transmission of the captured data does not disrupt traffic on the network that controls movement of the robot 202, 204, and vice versa. This may be a particular consideration when many vehicles operate on the grid 100, or when a large amount of data is captured and transmitted, for example. In another example, the robot is controlled through instructions transmitted with a first network traffic priority (e.g., a first quality of service, QoS), the captured data is transmitted with a second network priority (e.g., a second quality of service, QoS), and the second network traffic priority is lower than the first network priority. That is, the instructions controlling the movement of robots on the grid 100 are transmitted in preference over the captured data. As a result, non-essential, captured data does not interrupt the flow of instructions and impede the movement of robots 202, 204 on the grid too.Bin content data types

[0059] The type of data captured by the sensor at step S120 may depend on the type of items (or ‘products’) that are stored in the bins 112 in the grid too. For example, different data may be useful for monitoring fresh produce versus clothing and apparel. The type of data collected can therefore be chosen in dependence on the expected contents of the bin.Sensor

[0060] The type of sensor used in the methods of the present disclosure are suitable for capturing the data indicative of the status of the container or the contents of the container.

[0061] The sensor may be one of or a combination of several types, e.g. a distance sensor, a colour sensor, a contrast sensor, or a sensor arranged to determine a type (e.g. ‘food’, ‘homeware’), shape (e.g. ‘circular’, ‘rectangular’), or marker (e.g. a QR code or a barcode) of an item in the container.

[0062] The sensor may capture the data indicative of the status of the container by visually inspecting the status of the container. For example, the sensor may comprise a camera arranged to capture an image of the inside or exterior of the container, optionally alongside a light source that is used to illuminate the inside and / or exterior of the container. As one example, the sensor may be a camera capturing one or more optical images. Image recognition and processing algorithms can therefore be applied in order to analyse the data. These are described in more detail below with reference to Fig. 7-

[0063] The image sensor may be arranged to capture only low fidelity or low resolution images thus reducing a cost and / or complexity of the sensor. For example, the sensor may be configured to detect only basic shape, contrast and / or colour in a low resolution manner so as to detect, for example, whether or not a container contains any items or contains certain types of items. Alternatively, the image sensor may be arranged to capture high-resolution or high fidelity images so as to be able to discern, for example, detail including markers, labels and / or text on items in the container or on the container itself. Optical character recognition techniques may be used to determine or interpret markers or text on the container or on items in the container. Image recognition techniques may be used to determine or interpret visual characteristics of items or organisms in the container.

[0064] As another example, the sensor may be an infrared camera arranged to capture one or more infrared images so as to be able to detect, for example, temperature of items or organisms in a container or to be able to detect the container or its contents in low lighting conditions.

[0065] In a further example, the sensor may be arranged to capture a mass or weight measurement of the container including its content (from which the mass or weight of the content may be derived by subtracting a predetermined container weight), or of the container content. Such a sensor may take the form of a Newton meter attached to the container handling parts (e.g. lifting device) of the robotic containerhandling vehicle and arranged to determine the mass of the container including its contents when the container is lifted by the robotic container-handling vehicle.Alternatively, the weight may be estimated using motor torque as described elsewhere herein.[oo66] As a further example, the sensor may be a LIDAR or other distance sensor arranged to capture a distance between the sensor and the container and / or its contents. One or more such captured distances can be used to build up a topographical map or 3D representation of the container or the contents of the container. The distance or map / representation data can be used to determine the height of a part of the container relative to other parts of the container. Alternatively, or in addition, the distance or map data can be used to ascertain an angle of a surface of the container. This data can, in turn, be used to determine an orientation of the container, such as a tilt or skew of the container relative to other containers in the vicinity of the container or relative to a frame of the automated storage and retrieval system or storage grid or vertical farming environment. Alternatively, or in addition, the distance or map / representation data can be used to determine the height or volume of the contents of the container relative to the internal height or volume of the container itself. This data can, in turn, be used to determine whether the contents of the container protrude above an opening at the top of the container.

[0067] The sensor may be a gas sensor arranged to determine the presence of a gas emanating from items or organisms in the container. The gas may be ethylene or any other gas indicative of the state of ripeness or sourness of a product contained in the container or the state of growth, development or health of an organism contained in the container. The gas sensor may be disposed on the container handling parts (e.g. a lifting device) of the robotic container-handling vehicle.

[0068] Any of the sensors described herein may be used independently, or in any combination to capture the data indicative of the status of the container and / or the contents of the container. The sensor may include a plurality of sensors operating independently or in conjunction with one another to provide the data indicative of the status of the container or status of the contents thereof. For example, the sensor may include a plurality of image sensors each arranged to capture an image of the container and / or its contents and the data indicative of the status of the container and / or its contents may be extracted from a composite image created from two or more of the captured images.

[0069] The sensor may be disposed on container-handling parts (e.g. a lifting device) of a robotic container-handling vehicle, e.g. the lifting device 312 of the roboticcontainer-handling vehicle depicted in Fig. 5. The sensor may be disposed for example at the centre of the lifting device 312. The lifting device 312 may be arranged to be lowered down into a storage column, thereby allowing the sensor to capture data relating to the inside and / or exterior of containers stored within the storage column. For example the sensor may be disposed on a gripping device 308 which is raisable and lowerable from e.g. the main body or a cantilever portion of the robotic container-handling vehicle. If the cantilever element 306 includes more than one gripping device 308, then there may be multiple sensors, one disposed on each of several gripping devices 308. The sensor may be activated or operated when the container handling parts are close to, adjacent to or attached to the container. In this way, the sensor may provide a more accurate measurement of the data indicative of the status of the container or its contents.

[0070] Alternatively, or in addition, the sensor may be disposed on a part of the robotic container-handling vehicle that is positioned over the container handling parts (e.g. over the lifting device). For example the sensor may be disposed on cantilever element 306 of a robotic container-handling vehicle. Alternatively, or in addition, the sensor may be disposed on a downward facing portion of the robotic container-handling vehicle over the container handling parts.Capturing and processing of data

[0071] Fig. 7 shows a flow diagram of a computer-implemented method of processing data on bin content that may be performed on the data captured using the method shown in Fig. 6, or any other data on bin content. The steps of Fig. 7 may thus be performed independently, or in combination with those of Fig. 6 - thus resulting in a method of collecting and processing data on bin content.

[0072] As a computer-implemented method, the method of Fig. 7 may be performed by any data processing system, including the processing system 400, and / or a data processing system on the robot 202, 204. Each of the steps of the method may thus be performed by any such data processing system.Determination of bin content

[0073] At step S200, captured data indicative of a content of the first bin is received. The data may be received from any source. For example, the data may be received at a data processing system (e.g., processing system 400) from whatever apparatus performed step S120 or, if performed, step S135. As another example, the datamaybe received internally at a data processing system on the robot 202, 204 (e.g., when step S135 is omitted).

[0074] At step S205, which may be performed in parallel with step S200, warehouse management information may also be received, e.g., from an inventory database of the automated storage and retrieval system. This could include information regarding stock levels and the expected content of the first bin, for example. It could also include information regarding the expected (but unverified) type of the content of the first bin. In yet another example, the warehouse management information may include an image of the bin content. In yet another example, the warehouse management information may include data from an earlier (e.g., the previous) data capture for comparison (e.g., an earlier image of the bin content, or an earlier weight of the container and its content or of the container content).

[0075] At step S210, at least one type of item and / or a number (or ‘quantity’) of items in the first bin may be determined. In some examples, this determination is made by reading from the warehouse management information, thus indicating what is expected to be in the first bin. In some examples, this determination is additionally or alternatively based on the captured data, thus indicating what is likely to actually be in the first bin. This is described in more detail below.

[0076] The type of item(s) may be determined at varying levels of specificity depending on the content of the bins 112. For example, in a grocery context, the types could include ‘fruit & vegetables’, ‘meat & fish’, etc., or could be more specific and include ‘carrots’, ‘peaches’, ‘cod’, ‘beef, etc.

[0077] The type of item (e.g. produce) may also be determined from an image of the bin contents by using (e.g. optical) character recognition. This can be used to read a barcode or other product information from the packaging of an item, for example.

[0078] When bins 112 store multiple types of items, a plurality of types of items in the first bin may be determined. When bins 112 store only one type of item, a (single) type of item in the first bin may be determined.

[0079] When the captured data is an image, the type of the items may be determined using image classification. For example, a machine learning model maybe trained to recognise the content of the bin from a predetermined set of options.

[0080] When there are multiple types of items in the first bin, the determining of the number of items in the first bin may be a determination of how many of each type of item there are in the first bin.

[0081] Any of the above approaches for determining the type or number of items in the first bin may be used independently, or in any combination.

[0082] The method of Fig. 7 may also comprise identifying a discrepancy between an expected number of items and the determined number of items in the first bin, or between at least one expected type of items and the determined at least one type of items in the first bin. There may for example be a different number of items in the bin than expected according to the warehouse management information.

[0083] Similarly, the method of Fig. 7 may additionally or alternatively comprise comparing the captured data to expected content data for the first bin and, responsive to the comparing, identifying a discrepancy between the expected content data and the captured data.

[0084] The expected content data may be a previous image or other previously captured data. Identifying a discrepancy may therefore comprise determining that there has been a change in the content data, or a change which meets a predetermined significance criterion, since the last time data was captured.If a discrepancy is identified in any manner, the discrepancy can be flagged to an operator for (manual) confirmation, or a message can be sent back to the warehouse management system for an inventory database to be updated.Determination of bin content quality

[0085] In an automated storage and retrieval system, the quality of items stored may change - and in particular deteriorate - over time. For example, perishable items may spoil, thus in turn spoiling other adjacent items. The method of Fig. 7 may therefore determine a quality of the item(s) in the first bin.

[0086] The manner in which the quality is determined may depend on which type of item is determined to be in the first bin. In this case, at step S215, a quality determination algorithm may be selected in dependence on the determined type(s) of item in the first bin. The selected quality determination algorithm is then used to determine the quality indicator in step S220. For example, if at a first stage the type of item in the first bin is determined to be fresh produce, the selected algorithm may be analgorithm to determine whether the produce is spoiled. Otherwise, if a shelf-stable item is determined to be in the first bin, the selected algorithm may be an algorithm to read a ‘best-before’ or expiry date marked on the item (e.g., using optical character recognition). As yet another example, a machine learning model could be trained in order to determine when fresh produce has become spoiled or rotten, or when glass is broken, and the selected algorithm could apply the machine learning model. As yet another example, fruit and vegetables release various different gases (e.g, ethylene) as they ripen and begin to spoil, and one or more of these gas levels could therefore be measured using a gas sensor as below (or as described elsewhere herein), and the selected algorithm may compare these levels to respective threshold values to determine a quality indication as to the freshness of the produce.

[0087] At step S220, a quality indicator for the content of the first bin may be determined based on the captured data. The quality indicator may be ‘good’ or ‘bad’, or ‘low’, ‘medium’ and / or ‘high’. Alternatively, the quality indicator may be a numeric value, which may be within a predetermined range. For example, the quality indicator could be an integer with a value between o and 10 or too, where a higher value indicates a higher quality, or vice versa. When ‘low’ quality is referred to herein, this may mean a ‘bad’ or ‘low’ quality indicator has been determined, or a quality indicator within a predetermined low quality range has been determined. Similarly, when ‘high’ quality is referred to herein, this may mean a ‘good’ or ‘high’ quality indicator has been determined, or a quality indicator within a predetermined high quality range has been determined.

[0088] Determining the quality indicator may also comprise comparing the captured data to at least one of: expected content data for items of an acceptable quality; expected content data for items of a low quality; or expected content data for items of a high quality. For example, in the case where a temperature reading is an indication of quality, an expected temperature range for low, acceptable and high quality items may be defined and the quality indication determined based on which range the captured data falls within.

[0089] The quality indicator may be determined using image classification. This may be useful if, for example, a low quality item is visibly different to a high quality item.

[0090] The quality indicator may be stored and monitored over time. This could then enable the prediction of how quickly items will become low quality. This information may then be used to prioritise items for picking before they become lowquality or encourage sales of that item in the future for example. The quality indicator may therefore be sent to the warehouse management system for storage.Handling of low quality content

[0091] Once a quality indicator has been determined, this may be used as a guide for a human to review the bin content. At step S225, therefore, responsive to determining that the quality indicator indicates low quality for the content of the first bin, at least a portion of the captured data may be sent to an operator for review. The quality indicator may therefore be used to trigger a more thorough, reliable or accurate inspection of the captured data (e.g., image data), but such an inspection need not be performed when the quality of the content of the first bin has been determined to be high, thus reducing the need for operator review. Whether step S225 is performed may depend on the data captured and the likelihood that an operator will be able to accurately assess quality from the data. For example, it may not be useful or necessary for an operator to check a ‘best-before’ date, but it may be possible for the operator to provide a more accurate assessment of whether produce has spoiled. Alternatively, responsive to determining that the quality indicator indicates low quality for the content of the first bin, at step S245, the robot 202, 204 may be controlled to transport the first bin to a disposal station.

[0092] At step S230, user input may be received from the operator regarding a quality of the content of the first bin. This could take a variety of forms. The user input may include a simple confirmation or rejection of the quality indicator. The user input may additionally or alternatively include a request to manually inspect the first bin (e.g., at an inspection station), either at that time, or whenever the first bin is next moved to an inspection station. The user input may additionally or alternatively include a degree to which the operator is certain of the quality of the content.

[0093] In the case where the user input indicates a low degree of certainty with regard to the quality of the content of the first bin, or the user input comprises a request to inspect the first bin, at step S235, the robot 202, 204 may be controlled to transport the first bin to an inspection station. The format and timing of the request would be dependent on the configuration of the storage and retrieval system. In some examples, an inspection station may be a dedicated port 300 to which the bin is presented and inspected by an operator. In other examples, a picking port may be used and the bin flagged to the operator to indicate that items do not need to be picked from the bin but simply inspected.

[0094] Where the user input of step S230 or the determination of step S220 indicates that the content of the first bin is of acceptable quality, it may be refrained from controlling the robot 202, 204 to transport the first bin to an inspection station. The system therefore continues to operate as normal following confirmation that the quality of the item(s) in the bin is acceptable to be kept in the grid too. This is shown at step S240 of Fig. 7. As a result, data may be captured when bins are moved from one position to another within the grid too, and the bins 112 do not always need to be presented to an operator or an inspection station in order to determine the quality of items.

[0095] Where the user input indicates that the content of the first bin is not of acceptable quality or the user input comprises a request to dispose of the content of the first bin, at step S245, the robot 202, 204 may be controlled to transport the first bin to a disposal station. As a result, spoiled items may automatically be disposed of, without otherwise interrupting the flow of bins around the grid too. The disposal station may be a dedicated trash port at which bin content is disposed of. It may alternatively be a picking port, at which the bin is presented with an instruction for the operator to remove the low quality item from the bin. The whole content of the bin may be disposed of, or alternatively only items determined to be of low quality may be disposed of, with other items remaining in the bin.

[0096] The warehouse management information may be updated to reflect the quality determination, either with the result of the quality indicator determination and / or confirmation from an operator. The warehouse management information or inventory may also be updated if the bin is moved to a disposal station and the content removed from the bin.

[0097] Once completed, either of the methods of Figs. 6 and 7 may be repeated. The methods of Figs. 6 and 7 may be repeated with the same first bin (at a later time), or with another one of the bins 112. Additionally or alternatively, the methods of Figs. 6 and 7 may be repeated with the bin being handled (lifted, transported, etc.) by another one of the robots 202, 204. The repetition of these methods may be at a frequency that depends on the type of item determined at step S210; as a result, items that are more perishable or otherwise more prone to spoilage may be inspected more frequently.Hardware and software implementations

[0098] The methods described herein may be performed by a data processing system (which may take the form of one or more controllers) configured to perform themethod(s). The data processing system may comprise any of the components of processing system 400 shown in Fig. 4, such as one or more processors 402 configured to perform the method(s). The data processing system may be on board a robot 202, 204. The method(s) may be split so that different steps are performed by different data processing systems. Any of the method steps, which may be relatively computationally intensive, may for example be performed in a cloud computing environment.

[0099] The methods described herein may be performed by a data processing system configured to execute instructions stored on a computer-readable medium, which may be transitory or non-transitory.

[0100] The data processing system may form part of an automated storage and retrieval system or vertical farming system, which may in turn comprise any of the above robot(s); the first bin / containers; the plurality of stacked bins / containers; and / or the storage grid.

[0101] In general terms, the methods described above in relation to Figs. 6 and 7 include capturing, by a sensor on a robotic container-handling vehicle, data indicative of a status of a container, determining the status of the container (e.g. the quality or type of items in the container) based on the data; and operating the robotic container-handling vehicle based on the status. In an example, the data indicative of the status of the container may take the form of image data (including e.g. contrast data, colour data); height data; volume data; weight data; temperature data; gas data or surface angle data. The status of the container and / or its contents ascertained from any of these types of data include: a type, quality or quantity of items in the container; an orientation of the container; a height or volume of items in the container relative to the internal height or volume of the container (and hence, for example, whether any whole or part of an item protrudes beyond the opening of the container); a health status, growth status or development status a type of item in the container; a number of items in the container; or a quality indicator for the content of the container.Determining a position of a robot with an unknown location

[0102] Fig. 8 is a flowchart of a method 500 of determining a position of a robot 202, 204 on the rail system 116. The method 500 may be implemented by the processing system 400, or by a distinct processing system (e.g., by a processing system of the robot 202, 204). As explained above, a database of the processing system 400 stores, in association with the unique identifier of a bin 112, the position of the bin 112 and,optionally, the content of the bin 112. In the examples described herein, the database also stores, in association with the position of the bin 112, a weight value for the bin 112.

[0103] As explained above, a robot 202, 204 operating on the rail system 116 may be unable to determine its position, meaning that the robot 202, 204 is unable to carry out an instruction to follow a route to a particular location on the rail system 116. In addition, the unknown position of the robot 202, 204 means that the robot 202, 204 is a safety hazard for other robots 202, 204, in view of the risk of collision with the robot 202, 204 with unknown position.

[0104] At S502, a robot position is identified as unknown (meaning that the robot 202, 204 is ‘lost’). For example, a robot 202, 204 may detect that its sensing device for detecting rail crossings on the rail system 116 is not functioning. In response, the robot 202, 204 may inform the processing system 400 that its position is unknown.Alternatively, an external device may detect (for example, as a result of sensor failure), that a position of a robot 202, 204 is unknown.

[0105] At S504, an exclusion zone is established around the ‘lost’ robot 202, 204. For example, the processing system 400 may determine an exclusion zone around an estimated position of the ‘lost’ robot 202, 204 and / or the last known position of the ‘lost’ robot 202, 204. The processing system 400 may then prevent other robots 202, 204 from entering the exclusion zone. For example, the processing system 400 may alter routes of other robots 202, 204 moving on the rail system 116 so that those routes do not pass within the exclusion zone. The exclusion zone extends beyond the footprint of the ‘lost’ robot in view of the uncertainty as to the position of the ‘lost’ robot 202, 204. The exclusion zone may be large enough to allow the ‘lost’ robot 202, 204 to conduct a plurality of tests to determine its position while remaining within the exclusion zone.

[0106] At S506, the ‘lost’ robot 202, 204 is instructed to move to a nearby column 102 of the grid too. In one example, each robot 202, 204 includes a plurality of sensors that detect whether the robot 202, 204 (and specifically the gripping device 308) is positioned directly over a column 102 of the grid too (as opposed to over one of the rails 118, 120 of the rail system 116). These sensors are different from the sensors that detect the number of rail crossings (which may have failed in order for the robot’s position to be unknown). Accordingly, the ‘lost’ robot 202, 204 can be instructed to move in a particular direction until the sensors detect that the robot 202, 204 is positioned directly over a column 102.

[0107] At S508, a first value indicative of a weight of a first bin 112 stored in the column 102 is received. In one example, the first value is received from the ‘lost’ robot 202, 204. For example, the ‘lost’ robot 202, 204 may lower its gripping device 308 (e.g., under instruction) and to lift a bin 112 within the column 102. The ‘lost’ robot 202, 204 may determine a weight of the bin 112 from a torque value of its motor (as described in more detail below), and send the determined weight of the bin 112 to the processing system 400. Alternatively, the ‘lost’ robot 202, 204 may transmit its torque value(s) to the processing system 400, and the processing system 400 may subsequently determine a weight value for the bin 112 from the received torque value(s).

[0108] Optionally, at S510, an additional value indicative of a height of a stack of bins 112 in the column 102 is received. In one example, the additional value is received from the ‘lost’ robot 202, 204. For example, when lowering its gripping device 308 at S508, the ‘lost’ robot 202, 204 may determine how far the gripping device 308 is lowered into the column 102. The ‘lost’ robot 202, 204 or the processing system 400 may then correlate that value to a height of the stack of bins 112 in the column 102 and / or to the position of the uppermost bin 112 in that column 102 in the Z-direction 114.

[0109] At S512, a position of the bin 112 is identified using the first value. The position of the bin 112 is identified from the database of stored weights and positions of bins 112 in the grid too. For example, the processing system 400 queries the database using the first value indicative of the weight of the bin 112, in order to identify one or more candidate positions of the bin 112 in the grid too. If there is only one bin 112 in the grid too having a stored weight that matches the first value, then the position of the bin 112 in the grid too can be identified. It will be appreciated that only the stored weights of the uppermost bins 112 in each column 102 need to be queried, because the first value is for the uppermost bin 112 in the column 102 that the ‘lost’ robot 202, 204 is positioned over.

[0110] If the additional value indicative of the height of the stack of bins 112 in the column 102 was received at S510, then the additional value can be used in combination with the first value to identify the position of the bin 112 at S512. The additional value provides information about the position of the bin 112 in the Z-direction 114. This information can be used to narrow down the candidate positions for the bin 112. For example, if the first bin 112 is stacked on top of two other bins 112 in the column 102, then only the stored weights of bins 112 having a position Z=3 in the grid too need to be queried using the first value.[O111] If multiple candidate positions are identified for the bin 112 at S512, then the processing system 400 carries out the method 600 (described further below). If only one position for the bin 112 is identified at S512, the method 500 proceeds to S514, at which a position of the robot 202, 204 is determined from the position of the bin 112 identified at S512. For example, the position of the robot 202, 204 in the X-direction 108 and the Y- direction 110 can be determined from the stored position of the bin 112 in the X-direction 108 and the Y-direction 110.

[0112] Once the position of the robot 202, 204 has been determined at S514, the method optionally proceeds to S516, at which the robot 202, 204 is instructed to move to another position on the rail system 116. For example, the robot 202, 204 may be instructed to return to a service area so that its sensing device(s) can be serviced and / or replaced. Alternatively, if the robot position was unknown as a result of temporary failure of the sensing device(s), then the robot 202, 204 may be instructed to carry out a particular task in the automated storage and retrieval system.Procedure where no unique position for the first bin can be determined

[0113] In some cases, the processing system 400 may be unable to determine a position of the robot 202, 204 from the first value received at S508 and from the additional value optionally received at S510. Fig. 10 shows an example where a first value Wi indicative of the weight of the bin 112 is received at S508 and, optionally, an additional value Hi indicative of the position of the bin 112 in the Z-direction 114 is received at S510. As shown in Fig. 10, it can be seen that there are three candidate positions of columns 102 storing bins 112 with weight Wi. If the additional value Hi is received at S510, it can be seen that the number of candidate positions for the bin 112 is reduced to two. However, no unique position of the bin 112 can be determined from the received data.

[0114] In this case, the method 600, depicted in Fig. 9, is carried out. The method 600 may be implemented by the processing system 400, or by a distinct processing system (e.g., by a processing system of the robot 202, 204).

[0115] At S602, a plurality of candidate positions of the first bin 112 are identified. For example, the plurality of candidate positions of the first bin 112 may be identified from the results of querying the database at S512 with the first value.

[0116] At S604, the robot 202, 204 is instructed to move to a location from which a second bin 112 can be accessed (e.g., another column 102). For example, the robot 202,204 may be instructed to move to a column 102 that is adjacent to the column 102 that the robot 202, 204 is positioned directly over. In the event that the sensing devices that detect rail crossings are still not functioning, the robot 202, 204 can be instructed to move a short distance (i.e., less than one column width) in one direction, and then to continue moving in that direction until other sensing devices of the robot 202, 204 detect that the robot 202, 204 is positioned directly over the adjacent column 102.[oii7]At S606, a second value indicative of a weight of a second bin 112 stored in the adjacent column 102 is received. The second value may be received in the same way as the first value received at S508. The relative position between the first bin 112 and the second bin 112 is known. For example, if the robot 202, 204 is instructed to move to an adjacent column at S604, then it is known that the second bin 112 is in a column 102 that is adjacent to the column 102 storing the first bin 112. Optionally, a second additional value indicative of a position of the second bin 112 in the Z-direction is also received at S606, in the same way as the additional value is received at S510.

[0118] At S608, one or more candidate positions of the second bin 112 are identified. The position of the second bin 112 is identified from the database of stored weights and positions of bins 112 in the grid too. For example, the processing system 400 queries the database using the second value indicative of the weight of the bin 112, in order to identify one or more candidate positions of the second bin 112 in the grid too. If there is only one bin 112 in the grid too having a stored weight that matches the second value, then it will be appreciated that the position of the second bin 112 in the grid too can be identified, and that the method may then return to S514. Optionally, the one or more candidate positions of the second bin 112 may be identified using the second additional value indicative of the position of the second bin 112 in the Z-direction, if this is received at S606.

[0119] If there are multiple candidate positions for the second bin 112, then the multiple candidate positions for the second bin 112 can be used to identify, at S610, the position of the first bin 112 from the plurality of candidate positions of the first bin 112 identified at S602. The candidate positions for the second bin 112 can be used to narrow down the candidate positions for the first bin 112 using the known relative position between the first bin 112 and the second bin 112.

[0120] Returning to the example shown in Fig. 10, it can be seen that for a second value W2 indicative of the weight of the second bin 112, there are three candidate positions. In this example, a second additional value H3 indicative of the position of thesecond bin 112 in the Z-direction 114 does not provide a unique position of the second bin 112, because there are two bins 112 with weight W2 and position H3 in the Z- direction. However, only one of these candidate positions for the second bin 112 is adjacent to a candidate position for the first bin 112. Accordingly, the position of the second bin 112 can be determined, and the method 600 can return to S514.

[0121] If a unique position of the robot 202, 204 cannot be determined from the candidate positions of the second bin 112, then S604 to S610 can be repeated until a unique position of the robot 202, 204 has been determined. In some examples, operation of the robots 202, 204 on the rail system 116 may be ceased if the position of the ‘lost’ robot 202, 204 has not been determined after a certain number of iterations of method 600.Bin weight measurements

[0122] In order to store weights of bins 112 in the database, bins 112 can be weighed at ports 130, 132 (i.e., ports 130, 132 comprising a scale weight) and / or by robots 202, 204. In one example, a unique identifier of a bin 112 is stored in association with a first tag indicating the bin weight, and a second tag for a bin weight measurement. When a bin 112 is weighed, the bin weight measurement tag is updated. A weight indicated by the bin weight tag may then be updated using the bin weight measurement tag, if the new weight is precise enough.

[0123] For example, when a bin 112 is opened or removed, the bin weight tag can be updated to indicate that the weight of the bin 112 is unknown. When a bin 112 has an unknown weight and a bin weight measurement is received (via the bin weight measurement tag), the bin weight is set as the value indicated by the bin weight measurement.

[0124] When a bin 112 already has a weight value indicated by the bin weight tag, the bin weight is set as the new weight value indicated by the bin weight measurement if the new weight is precise enough. Weight values are stored with tolerances (or precisions) indicating the precision of the weight measurement.Tolerances may be provided by the module (port 130, 132, robot 202, 204, etc.) that weighs the bin 112, or may be preset values that are accessed by the processing system 400 and are associated with the module that weighs the bin 112. A bin weight measurement provided by a port 130, 132 may have a precision of + / - 500g (for a quickmeasurement) or + / - 100g (for a more thorough measurement). In contrast, a bin weight measurement provided by a robot 202, 204 may have a precision of + / - 2000g.

[0125] If the new weight value could fall within the existing weight value(accounting for the tolerances of both weight values) such that there is an overlap between the new weight value and the existing weight value when accounting for the tolerances of both weight values, then the bin weight tag is updated with the weight value that has higher precision. On the other hand, if the new weight value is outside the existing weight value (accounting for the tolerances of both weight values) such that there is no overlap between the new weight value and the existing weight value when accounting for the tolerances of both weight values, then the bin weight tag is updated with the new weight value.

[0126] As an example, if a bin weight tag has a value “unknown” and a bin weight measurement of 3500g + / - 100g is received, then the bin weight tag is updated with the bin weight of 3500g and the tolerance of 100g. If a subsequent bin weight measurement of 3900g + / - 1000g is received, then the bin weight tag is not updated, because (i) the subsequent weight value could fall within the existing weight value when accounting for the tolerances of the weight values, and (ii) the existing weight value has higher precision. However, if a subsequent bin weight measurement of 5000g + / - 1000g is received, then the bin weight tag is updated with the subsequent bin weight value, because the subsequent weight value does not fall within the existing weight value when accounting for the tolerances of the weight values. Therefore, the bin weight tag is updated with the bin weight of 5000g and the tolerance of 1000g.

[0127] In some examples, a tolerance is determined or received for a bin weight value received at S508 from a ‘lost’ robot 202, 204. The identification of the position of the bin 112 weighed by the ‘lost’ robot 202, 204 at S512 may comprise comparing the weight value received from the ‘lost’ robot 202, 204 and the tolerance of that weight value with weight values of bins 112 and respective tolerances of those weight values stored in the database. Candidate positions for the bin 112 identified at S602 may comprise any bins 112 for which the bin weight value received at S508 could fall within the stored bin weight value, when accounting for the tolerances of both weight values.Estimating bin weight using robots

[0128] In one example, a robot 202, 204 (or the processing system 400) estimates a weight value of a bin 112 from the torque of the robot’s motor during liftingof the bin 112. The motor torque is measured at a point where it has stabilised (i.e. , when the lift velocity reaches a certain value, such as 0.6 m / s). Motor torque is assumed to increase linearly with bin weight, such that a linear relationship exists between stabilised motor torque and bin weight. That is: bin_weight = a * motor_torque + c. The parameters a (gradient) and c (intercept) can be found through calibration of a number of motor torque values measured when lifting bins 112 of known weight. Calibrated parameters may then be stored by the robots 202, 204 or at the processing system 400 so that a measured torque value can be converted to an estimate of bin weight.

[0129] In general terms, the methods described above in relation to Figs. 8 and 9 include capturing, by a sensor on a robotic container-handling vehicle, data indicative of a status of a container, and optionally, controlling an operation of the robotic containerhandling vehicle based on the status of the container. In an example, the data indicative of the status of the container may represent a weight of a storage container, from which a position (status) of the container may be determined. In an example, controlling the operation of the robotic container-handling vehicle based on the status of the container may include instructing the robotic container-handling vehicle to move from the determined position to a further position in the automated storage and retrieval system.Bin content verification

[0130] As described above, an automated storage and retrieval system may make use of a warehouse management or inventory management system in order to maintain and keep track of containers stored within the system and the contents and positions of these containers. In order to properly fulfil orders, it is therefore important that the warehouse management or inventory management system maintains an accurate record of containers and their contents. For instance, a container including one or more items should not be recorded in the inventory management system as being empty, in order to avoid robots mistakenly transporting containers for stocking that already contain items. Likewise, an empty container should not be recorded in the inventory management system as being stocked with items, in order to avoid robots mistakenly transporting empty containers, for example. Similarly, it is important that a record of items stored within a container is consistent with the items that are actually stored within the container. For instance, a container having contents consisting of a football and a rugby ball should be recorded in the inventory management system as storing two items, a football and a rugby ball, and not, for example, that it is empty or that it stores one item or three or more items. Maintaining accurate records avoidsrobots mistakenly transporting containers, which may result in wasted time and inefficiencies. In response to determining that the content of a container does not match a reference content (e.g. as recorded in an inventory management system), it may be desired that the container is transported to, for example, an inspection area of the automated storage and retrieval system for further inspection. The inspection area may be a port or access station.

[0131] Fig. 11 shows a flow diagram of a computer-implemented method 1100, in an automated storage and retrieval system comprising a storage grid too including a plurality of stacked containers 112, of determining a status of a container 112.

[0132] At S1102, the method 1100 captures, by a sensor on a robotic containerhandling vehicle, data indicative of a status of a container and / or its contents. The sensor may take the form of any of the sensors described herein. The captured data may represent, for example, data indicative of a content of the container. The method 1100 may then proceed to S1106, where an operation of the robotic container-handling vehicle is controlled based on the status of the container.

[0133] If the data captured at S1102 represents data indicative of a content of the container (e.g. an indication of the items stored in the container), then, optionally, before proceeding to S1106 the method 1100 may proceed to S1104. At S1104, the method 1100 determines a more detailed status of the content of the container, possibly in relation to a reference content associated with the container (e.g. an associated record of the contents of the container in an inventory management system). In particular, at S1104, the method 1100 determines, based at least in part on the data indicative of the content of the container, whether: i. the container is empty, ii. the container is empty and a reference content associated with the container indicates that the container is not empty (e.g. the inventory management system indicates that the container contains items), iii. the container is not empty and a reference content associated with the container indicates that the container is empty (e.g. the inventory management system indicates that the container is empty), or iv. the container is not empty and the content of the container does not match a reference content associated with the container (e.g. the inventory management system correctly records that the container contains items but there is a discrepancy betweenthe items stored in the container and the items expected to be stored in the container according to the inventory management system).

[0134] If at S1104 it is determined that the container is empty, then at S1106 the method 1100 may control the robotic container-handling vehicle to leave the container in its place, for example to avoid mistakenly transporting an empty container, and may control another robotic container-handling vehicle to retrieve the empty container. The retrieved empty container may be transported to another area within the automated storage and retrieval system, such as an access station or picking station, for example for adding one or more items to the container or for removal of the container from the storage grid. Thus, in more general terms, the method may comprise, responsive to determining that the container is empty, controlling the robotic container-handling vehicle to leave the container in its place within the storage grid, and controlling a second robotic container-handling vehicle to retrieve the container.

[0135] Similarly, if at S1104 it is determined that the container is empty but the reference content associated with the container is not empty (e.g. if the inventory management system indicates that the container contains items), then at S1106 the method 1100 includes controlling the robotic container-handling vehicle or another robotic container-handling vehicle to transport the container to another location within the storage grid or to an access port of the automated storage and retrieval system. There, the empty container may be filled with items, for example in accordance with a list of items considered as being stored in the container by the inventory management system.

[0136] In a similar way, if at S1104 it is determined that the container is not empty but the reference content associated with the container is empty (e.g. if the inventory management system indicates that the container is empty), then at S1106 the method 1100 controls the robotic container-handling vehicle to transport the container to another location within the storage grid or to an access port of the automated storage and retrieval system. At the access port, the items stored in the container may for example be removed from the container if they were erroneously stored in the container. Additionally or alternatively, the automated storage and retrieval system may be notified of the discrepancy so that the incorrect record in the inventory management system may be corrected. The container may then be returned to the storage grid.

[0137] If at S1104 it is determined that the content of the container does not match the reference content associated with the container (e.g. if the container containsa first number of items but the inventory management system records the container as containing a second number of items different from the first number), then the method 1100 may notify the automated storage and retrieval system that there is a discrepancy in the content of the container, and / or control the robotic container-handling vehicle or another robotic container-handling vehicle to transport the container to an inspection area of the automated storage and retrieval system. At the inspection area, an operator may assess the contents of the container to determine whether there is in fact a discrepancy in its contents, and update the inventory management system accordingly or fill the container so that the discrepancy is removed. The container may then be returned to the storage grid.Container content height and volume

[0138] In order for one or more containers to properly stack within a column of the storage grid, the contents of each container within the stack must not exceed a top height of the storage container. That is, a maximum height of the contents of a container must not protrude above the top height of the storage container or an opening at the top of the container. If this is not the case, then the containers may improperly stack, possibly causing one or more containers to become stuck within a storage column. Such a situation is best avoided, since this may result for example in certain parts of the grid being closed down in order to retrieve a stuck container. Additionally, before it can be ascertained that the container is stuck, the stuck container may result in repeated failed attempts by one or more robotic container-handling vehicles to retrieve the stuck container without determining a reason why the container cannot be retrieved in the normal manner. Once it has been ascertained that the container is stuck, removing the stuck container may include controlling a special type of robotic vehicle to retrieve the stuck container. The special type of robotic vehicle may include a lifting device specially arranged to grip and lift a container which is improperly stacked. The lifting device may be arranged to move the container by brute force, which may damage or destroy the stuck container. These corrective procedures may reduce the overall efficiency of the automated storage and retrieval system.

[0139] The contents of a container can exceed the height of the container in several ways. For example, as depicted in Fig. 14, an item 1406 stored in a container may exceed the height of the container 1404 if it is improperly placed within the container 1406. The item 1406 depicted in Fig. 14 for example may not exceed the height threshold if stored horizontally, but may exceed the height threshold if it is improperly placedwithin the container, e.g. vertically or at an angle with respect to the horizontal plane. In another example, as depicted in Fig. 15, an item 1506, such as a vacuum-packed item of clothing, may exceed the height of the container 1404 if the vacuum-sealed packing leaks causing the volume of the item 1506 to expand to such an extent that the item 1506 exceeds the height of the container 1404.

[0140] To address this problem, the method 1100 depicted in Fig. 11 may therefore further include a step of determining, based at least in part on the data indicative of the content of the container, that the content of the container exceeds a height or volume threshold of the container. For example, it may be determined from an image captured by a camera of the robotic container-handling vehicle that the container includes an item that protrudes from the top of the container. Additionally or alternatively, if a distance to the top of the container is known or measured, it may be determined that the content of the container exceeds a height threshold of the container by determining a distance (e.g. by a distance sensor) to one or more items within the container, and comparing the determined distance to the distance to the top of the container. If, for example, it is determined that the distance to the top of the container is a first distance (e.g. 5 meters) and the distance to an item in the container is a second distance shorter than the first distance (e.g. 4.9 meters, then it may be concluded that the item exceeds the height threshold of the container by at least a difference between the first and second distances (e.g. 0.1 meters). Responsive to determining that the content of the container exceeds the threshold, the method 1100 may involve controlling the robotic container-handling vehicle or another robotic container-handling vehicle to retrieve the container, for example for an operator to restock the container or to rearrange contents of the container at an access station.

[0141] A related problem is that of storage containers being stored at an incorrect angle within a column of a storage grid. For example, if a storage container is stored at an incorrect angle within a storage column of the storage grid, then robotic containerhandling vehicles may be unable to lift the container out of the column. This may introduce delays into order fulfilment processes, and may pose additional obstacles in relation to the other containers stored in the same column as the stuck container. For example, as depicted in Fig. 16, a storage column of a storage grid may contain at least two containers 1602, 1604. In Fig. 16, a first container 1604 is skewed (or tilted or rotated) with respect to the storage column and a second container 1602, and therefore a robotic container-handling vehicle may be unable to retrieve the container 1604 from thecolumn using the lifting device. Furthermore, due to the obstacle posed by the first container 1604, a robotic container-handling vehicle may also be unable to retrieve the second container 1602 from the storage column. It is therefore desirable to be able to determine when a storage container is improperly stored within a column of a storage grid.

[0142] Fig. 12 depicts a flowchart of a method of determining an orientation of a container. As in method 1100, the first step S1202 of the method 1200 captures, by a sensor on a robotic container-handling vehicle, data indicative of a status of a container. In particular, in method 1200, capturing data indicative of the status of the container comprises capturing data indicative of an orientation of the container. The method 1200 may therefore determine at S1204 an orientation of the container. The orientation of the container may be determined in several ways. In one example, the orientation of the container may be determined from an image captured by a camera of the robotic container-handling vehicle. In another example, a distance sensor on the robotic container-handling vehicle may be used to make several distance measurements to various points of the container, and an orientation of the container may be determined by comparing these distance measurements. If, for example, the distance measurements are all the same, then it may be concluded that the container is stored in proper alignment with the storage column. If, however, some of the distance measurements are different, then it may be concluded that the container is stored at an improper angle. Alternatively, the sensor may be arranged to determine the angle of a surface of the container by emitting a beam of light toward a face or surface of the container and measuring the angle of reflection from the face or surface. From the angle of reflection it may be determined that the face or surface is at an angle oblique to the horizontal plane of the storage grid and therefore that the container is skewed or otherwise improperly oriented. The beam of light may be directed toward a surface which would be horizontal if the container was stacked normally. The sensor may therefore include a light source (e.g. LED or laser), a light detector (e.g. CCD) arranged to receive a reflection from the surface of the container and a controller arranged to determine the angle of reflection from the position of the reflected beam on the light detector and the position of the light source using the law of reflection and simple geometrical calculations including the distance to the container (which may be determined by another sensor arranged to determine distance to the surface of the detector).

[0143] At S1206, responsive to determining the orientation of the container, the method 1200 controls the robotic container-handling vehicle or another robotic container-handling vehicle to retrieve the container. The robotic container-handling vehicle used to retrieve the container may be a standard robotic container-handling vehicle or a robotic vehicle specifically designed for the retrieval of stuck containers. This type of robotic vehicle may include a lifting device specifically arranged to grip (for example using a gripper such as a clamp) and lift a container which is improperly stacked using an actuator (e.g. including a winch assembly) operably coupled to the gripper and the robotic vehicle. The lifting device may be arranged to move the container by brute force to remove it from the storage grid or may be arranged to lift one side of the container or otherwise move the container to restore it to the correct orientation.Vertical farming applications

[0144] In some embodiments, the automated storage and retrieval system may be arranged to store organisms. For example, containers in a storage grid may store organisms, or the automated storage and retrieval system may be arranged as a vertical farming system for growing organisms in receptacles (containers or stackable modules) that can be stacked in the storage columns of a storage grid, and which can be lifted, lowered and transported by robotic container-handling vehicles. It is important to ensure that organisms stored within such systems receive, for example, adequate nutrition and ventilation, that they are stored in an appropriate temperature and climate, so that organisms are meeting an expected growth and / or development rate. Examples of organisms suitable for growth or development in the vertical farming system include crops or plants of any variety suitable for growth or development in a vertical farming environment, including but not limited to vegetables, herbs, fruit, medicinal plants or fungi. However, the present disclosure is not limited thereto and organisms may alternatively include other organisms suitable for growth in a vertical farming environment, such as biological tissues or insects.

[0145] Containers (or bins) for use in a vertical farming system may be of the type described with reference to the automated storage and retrieval system of Fig. 1. Alternatively, the containers may be a subcomponent of stackable modules deployed in stacks within columns of a framework structure (e.g. storage grid), such as that described with reference to Fig. 1. The stackable modules are used as building blocks to build vertical farming columns within the framework structure. The columns are analogous tothe storage columns of the previously described automated storage and retrieval systems, e.g., as illustrated in Fig. 1. Features referred to in relation to the storage containers or bins of the automated storage and retrieval system, e.g., as described in relation to figures 1 to 4, apply equally to the stackable modules except where described elsewhere herein. Robotic container-handling vehicles such as those described with reference to Figs. 1-5 may be arranged to carry / transport the stackable modules in the same way as a container or bin of the automated storage and retrieval system is carried.

[0146] The stackable modules may comprise support members for supporting functional elements (containers) required for plant or organism growth. An example of such a container is a(n e.g. vertically arranged) growth board for growing the organisms or a(n e.g. vertically arranged) plate member for supporting other functional components, like sources of light, heat, fluids or air movement. The growth board is as an example of a container of sorts which is carried / transported by the robotic containerhandling vehicle as part of the stackable modules. Alternatively, the stackable modules themselves, together with the growth boards, may be described as containers.

[0147] Fig. 13 depicts a flowchart of a method 1300 of determining one or more statuses associated with one or more organisms in a container of a vertical farming system. At S1302, the method 1300 includes capturing, by a sensor on a robotic container-handling vehicle, data indicative of a status of at least one organism supported by a container disposed at a first location within the storage grid. Put another way, at S1302, the method includes capturing, by a sensor on a robotic container-handling vehicle, data indicative of a status of a container, wherein the status of the container in this case is the status of one or more organisms (e.g. plants in the stackable growth module, or growth board contained or supported therein) supported by or contained in the container.

[0148] Example indicators of the status of organisms in (or supported by) the container are temperature, weight, size (length, area or volume), fullness, colour, turgidity, surface texture, surface patterns or contrast but the present disclosure is not limited thereto and the indicators may be any characteristic which is indicative of a state of growth, a state of health and / or a state of development of the organism.

[0149] At S1304, the method further comprises determining a status of the at least one organism based on the data captured by the sensor. The status may be determined by comparing the data indicative of the status of the organism to reference data of organisms of known status and determining an exact or closest match between avalue of the sensed data and one or more values of the reference data. For example, the colour data collected by the sensor (e.g. colour of the leaves of a plant) is compared with known (reference) colour data (e.g. colour of the leaves of plants at one or more known stages of growth or development) and a match between the sensed colour data and the one or more colours of the known colour data is used to determine the status of the organism (e.g. the stage of growth or development of the at least one organism). The reference data may be stored in a memory, generated by an Al model trained on data from known or previously recorded growth, health or development cycles, or captured by the sensor itself from other organisms in the vicinity of the subject organism within the vertical farming system.

[0150] At S1306, the method further comprises controlling operation of the robotic container-handling vehicle based on the status of the at least one organism. Controlling operation of the robotic container-handling vehicle based on the status of the at least one organism can include: controlling movement of the (whole of the) robotic container-handling vehicle relative to the first location (or relative to the storage grid), and / or controlling movement of container-handling parts of the robotic containerhandling vehicle (e.g., relative to the rest of the robotic container-handling vehicle).

[0151] Controlling movement of the robotic container-handling vehicle relative to the first location can include moving away from the first location without the containerhandling parts of the vehicle interacting with the subject container (i.e. the container including the at least one organism sensed by the sensor). For example, once it has been determined that the container does not have to be removed, the robotic containerhandling vehicle does not lower the lifting device, or retracts the lifting device already lowered without picking up the container, and moves away from the location of the container. This can be useful to save energy expended by the vehicle in handling the container if it is determined that the container does not need to be moved, or should be moved by another (e.g. second) robotic container-handling vehicle. The second robotic container-handling vehicle may have the same or different capabilities than the (e.g. first) robotic container-handling vehicle that sensed the data indicative of the status of the at least one organism.

[0152] Alternatively, the controlling movement of the robotic container-handling vehicle relative to the first location can include moving away from the first location (or moving relative to the storage grid) after the container-handling parts of the robotic container-handling vehicle have engaged the container and lifted it to a position relativeto the robotic container-handling vehicle that allows the robotic container-handling vehicle to transport the container across the storage grid. The robotic containerhandling vehicle can then be controlled to move the container to another location, for example to a port or access station of the storage grid or to another location in the storage grid (e.g. the second location described herein).

[0153] The kind of operation of the robotic container-handling vehicle is determined based on the status of the at least one organism. For example, based on the determined status of the organism, the method 1300 of Fig. 13 can include determining whether a change in storage or farming conditions is required to maintain, progress or improve the growth, development or health of the organism. Example conditions include a temperature condition, an irrigation condition, a nutrient supply, a humidity and / or an illumination condition. The temperature condition includes, for example, a maximum, minimum or average temperature level and / or a temperature schedule. The irrigation condition includes, for example, a maximum, minimum or average irrigation level and / or an irrigation schedule. The humidity includes, for example, a maximum, minimum or average humidity level and / or a humidity schedule. The nutrient supply includes, for example, a maximum, minimum or average nutrient supply level and / or a nutrient supply schedule and / or a type of nutrient supplied. The illumination condition includes, for example, an illumination intensity, an illumination schedule and / or or a wavelength spectra. The temperature / irrigation / humidity / illumination schedule can be, for example, a temperature / irrigation / humidity / illumination which varies according to time, e.g. a time of day, a time during any other fixed (repeating) period or an ad hoc time schedule.

[0154] In embodiments, the method 1300 of Fig. 13 further comprises, following determining that the change in conditions is required, notifying the automated storage and retrieval system of any required change. Following the notification, action can be taken to change the conditions to which the at least one organism is subjected in the storage grid.

[0155] The conditions can be changed by changing conditions in the region of the storage grid in which the container supporting the at least one organism is located. For instance, if it is determined based on the colour of leaves of a plant that a temperature status is too high, then a temperature in the storage grid in the vicinity of the at least one organism may be lowered, for example by controlling a heater or air conditioning system operating in the vicinity. In another example, if it is determined from the size or healthof the organism that a particular nutrient is deficient in the organism, then a type or quantity of nutrient supplied (e.g. included in the irrigation fluid or surrounding air) at the first location may be modified. The storage grid may contain any one or more of a humidity control system, an irrigation system, a nutrient supply system, a temperature control system or an air conditioning (or air composition control) system arranged to control the conditions at different locations in the storage grid (e.g. the first and second locations described herein).

[0156] Alternatively, the conditions can be changed by moving, using the robotic container-handling vehicle, the container supporting the at least one organism from a first location in the storage grid to a second location in the storage grid. For example, if it is determined that the leaves of a plant have reached a certain size, action can then be taken to move the plant from a first location in the storage grid to a second location in the storage grid. The conditions at the first location are different from the conditions at the second location. For example, conditions at the first location may have been suitable for a first stage of growth or development, for example in which the leaves of the plant are grown to a certain size, and the conditions at the second location may be more suitable for or conducive to a second stage of growth or development, for example in which another part of the plant is made to ripen, grow or otherwise develop. That is, in embodiments, the method 1300 of Fig.13 includes controlling the robotic containerhandling vehicle to move the container and the one or more organisms contained therein to an alternative location within the storage grid. The conditions (e.g. temperature, climate, irrigation rate or volume, and / or nutritional provisions) in the alternative (e.g. second) location are sufficiently different than in the original location at which the plant status is determined so that the growth, development or health of the one or more plants can be improved, progressed or otherwise positively influenced.

[0157] Alternatively, if it is determined from the status of the at least one organism that a plant has satisfied a growth requirement, then the method 1300 may include controlling the robotic container-handling vehicle to retrieve the container so that the one or more organisms contained therein can be transported out of the storage grid or vertical farming environment, for example via transportation of the container to a port or access station of the storage grid or vertical farming environment.

[0158] Any of the above-described methods may be performed at least in part by a system including a robotic container-handling vehicle of an automated storage and retrieval system. For example, a robotic container-handling vehicle may comprise asensor configured to capture data indicative of a status of a container. The system includes one or more controllers arranged to control the robotic container-handling vehicle based on the status of the container. The one or more controllers can take the form of those described with reference to Fig. 4. The robotic container-handling vehicle and / or the one or more controllers of the automated storage and retrieval system may be configured to perform any of the methods described herein.Penultimate comments

[0159] It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure has been described with reference to specific example implementations, it will be recognized that the disclosure is not limited to the implementations described, but can be practiced with modification and alteration within the scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0160] Also disclosed herein are the following numbered clauses: Clause 1. A computer-implemented method, in an automated storage and retrieval system comprising a storage grid including a plurality of stacked containers, of collecting data on container content, comprising: controlling a robotic vehicle on the storage grid to lift a first one of the plurality of stacked containers out of the storage grid; responsive to the controlling, capturing data indicative of a content of the first container using a sensor of the robotic vehicle; and storing the captured data in association with a unique identifier of the first container.Clause 2. The method of clause 1, wherein the data is captured responsive to determining that a position of the first container with respect to the robotic vehicle meets a proximity criterion.Clause 3. The method of any preceding clause, further comprising: prior to the controlling:receiving an instruction to transport the first container to a destination, and controlling the robotic vehicle to move to the first container; and subsequent to the capturing: controlling the robotic vehicle to move the first container to the destination.Clause 4. The method of any preceding clause, further comprising transmitting the captured data to a data processing device in association with the unique identifier.Clause 5. The method of any preceding clause, wherein the captured data comprises at least one of: one or more optical images, one or more infrared images, air composition data, humidity data, weight data, or temperature data.Clause 6. The method of any preceding clause, further comprising determining at least one type of item in the first container.Clause 7. The method of clause 6, wherein the determining of the at least one type of item in the first container is based on at least one of: an inventory database of the automated storage and retrieval system; or the captured data.Clause 8. The method of any preceding clause, further comprising determining, based on the captured data, a number of items in the first container.Clause 9. The method of clause 8, further comprising identifying a discrepancy between an expected number of items and the determined number of items in the first container.Clause 10. The method of any preceding clause, further comprising determining, based on the captured data, a quality indicator for the content of the first container.Clause 11. The method of clause 10 when dependent on clause 6, wherein the determining of the quality indicator comprises: selecting, from a plurality of algorithms, a quality determination algorithm in dependence on the determined at least one type of item in the first container; and applying the selected quality determination algorithm to determine the quality indicator.Clause 12. The method of any of clauses 10 to 11, further comprising, responsive to determining that the quality indicator indicates low quality for the content of the first container, sending at least a portion of the captured data to an operator for review.Clause 13. The method of any preceding clause, further comprising: comparing the captured data to expected content data for the first container; and responsive to the comparing, identifying a discrepancy between the expected content data and the captured data.Clause 14. A data processing system configured to perform the method of any preceding clause, or a computer-readable medium comprising instructions which, when executed by a data processing system, cause the data processing system to perform the method of any preceding clause.Clause 15. An automated storage and retrieval system comprising at least one of a data processing system and a robotic vehicle, the automated storage and retrieval system being configured to perform the method of any of clauses 1 to 13, and the system optionally further comprising at least one of: the first container; the plurality of stacked containers; or the storage grid.

[0161] Also disclosed herein are the following numbered items:Item 1. A computer-implemented method for determining a position of a robotic container-handling vehicle in an automated storage and retrieval system, the method comprising:receiving a first value indicative of a weight of a first storage container handled by the robotic container-handling vehicle; identifying, using the first value, a position of the first storage container in the automated storage and retrieval system from a database of stored weights and positions of storage containers in the automated storage and retrieval system; and determining, from the identified position of the first storage container, the position of the robotic container-handling vehicle.Item 2. The method of item 1, wherein the first value is received from the robotic container-handling vehicle.Item 3. The method of item 1 or item 2, wherein the first value is determined from a torque value of a motor of the robotic container-handling vehicle during handling of the storage container.Item 4. The method of any of items 1 to 3, wherein identifying the position of the storage container in the automated storage and retrieval system comprises: identifying, from the database, a plurality of candidate positions of the first storage container using the first value; receiving a second value indicative of a weight of a second storage container handled by the robotic container-handling vehicle, wherein a position of the second storage container relative to the first storage container is known; identifying, using the second value, a plurality of candidate positions of the second storage container in the automated storage and retrieval system from the database; and identifying, using the plurality of candidate positions of the second storage container, the position of the first storage container from the plurality of candidate positions of the first storage container.Item 5. The method of item 4, further comprising: in response to identifying the plurality of candidate positions of the first storage container, instructing the robotic container-handling vehicle to move to a location from which the second storage container can be accessed by the robotic container-handling vehicle.Item 6. The method of any of items 1 to 5, further comprising:receiving an additional value indicative of a height of a stack of storage containers stored in the automated storage and retrieval system; and identifying, from the database, the position of the first storage container using the first value and the additional value, wherein the database stores heights of stacks of storage containers stored in the automated storage container and retrieval system.Item 7. The method of item 6, wherein the stack of storage containers includes the first storage container.Item 8. The method of any of items 1 to 7, further comprising: determining an exclusion zone relative to the robotic container-handling vehicle; and preventing other robotic container-handling vehicles in the automated storage and retrieval system from entering the exclusion zone.Item 9. The method of any of items 1 to 8, further comprising instructing the robotic container-handling vehicle to move from the determined position to a further position in the automated storage and retrieval system.Item 10. The method of any of items 1 to 9, further comprising, for each of a plurality of storage containers: receiving a value indicative of a weight of the storage container in the automated storage and retrieval system; and storing the value indicative of the weight of the storage container in the database in association with a position of the storage container.Item 11. The method of item 10, wherein the value is received from a robotic container-handling vehicle in the automated storage and retrieval system.Item 12. The method of item 10 or item 11, wherein the value is received from a port of the automated storage and retrieval system, wherein the port comprises a weighing scale.Item 13. The method of any of items 10 to 12, further comprising: determining a tolerance of the received value;wherein storing the value indicative of the weights of the storage container in the database in association with the position of the storage container comprises storing the tolerance of the received value in association with the value.Item 14. The method of item 13, wherein identifying the position of the first storage container in the automated storage and retrieval system comprises: comparing, using the database, the first value and a tolerance of the first value with stored weights of storage containers in the automated storage and retrieval system and respective tolerances of the stored weights.Item 15. A computer-readable medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to carry out the method of any of items 1 to 14.Also described herein are the following numbered embodiments:Embodiment 1. A method for use in a vertical farming system comprising a storage grid including a plurality of stacked containers, each container arranged to support at least one organism, the method comprising: capturing, by a sensor on a robotic container-handling vehicle, data indicative of a status of at least one organism supported by a container disposed at a first location within the storage grid; determining the status of the at least one organism based on the data, and controlling an operation of the robotic container-handling vehicle based on the status of the at least one organism.Embodiment 2. The method of embodiment 1, wherein the status is a state of growth, a state of health or a state of development of the at least one organism.Embodiment 3. The method of embodiment 1 or 2, wherein the data comprises an image of the at least one organism, optionally wherein the image indicates a colour, size, turgidity or contrast of at least one region of the at least one organism.Embodiment 4. The method of any preceding embodiment, further comprising, based on the status of the at least one organism,determining that a change in a condition at the first location is not required to progress or improve the growth, development and / or health of the at least one organism, controlling the robotic container-handling vehicle to leave the container at the first location within the storage grid and maintaining the condition at the first location; or determining that a change in a condition at the first location is required to progress or improve the growth, development and / or health of the at least one organism, and either: controlling the robotic container-handling vehicle to retrieve the container from the first location and move the container to a second location within the storage grid, or changing the condition at the first location.Embodiment 5. The method of embodiment 4, wherein the condition at the first location is different from the condition at the second location.Embodiment 6. The method of embodiment 5, wherein the condition includes a temperature condition, an irrigation condition, a nutrient supply, a humidity and / or an illumination condition, optionally wherein the temperature condition includes a maximum, minimum or average temperature level and / or a temperature schedule, optionally wherein the irrigation condition includes a maximum, minimum or average irrigation level and / or an irrigation schedule, optionally wherein the humidity includes a maximum, minimum or average humidity level and / or a humidity schedule, optionally wherein the nutrient supply includes a maximum, minimum or average nutrient supply level and / or a nutrient supply schedule and / or a type of nutrient supplied, optionally wherein the illumination condition includes an illumination intensity, an illumination schedule and / or or a wavelength spectra.Embodiment 7. The method of embodiment 3 or 4, further comprising, responsive to determining that the status of the at least one organism matches a reference status or meets a threshold status:notifying a controller of the vertical farming system that there is a match between the determined status and the reference status or that the threshold status has been met; and / or controlling the robotic container-handling vehicle or a second robotic containerhandling vehicle to transport the container to an access station or port of the vertical farming system.Embodiment 8. The method of any preceding embodiment, further comprising: determining, based at least in part on the data indicative of the status of the at least one organism, that the height or volume of the at least one organism respectively exceeds a height or volume threshold of the container; and responsive to determining that the at least one organism exceeds the threshold, controlling the robotic container-handling vehicle or a second robotic container-handling vehicle to retrieve the container and deliver the container to a port or access station of the storage grid.Embodiment 9. The method of any preceding embodiment, wherein the sensor is at least one of: an image sensor; LIDAR; an infrared sensor; a distance sensor; a temperature sensor; a weight sensor; a colour sensor; or a contrast sensor.Embodiment 10. The method of embodiment 9, wherein capturing the data comprises using an image sensor to capture the data, optionally while using a light source to illuminate the at least one organism.Embodiment 11. The method of any preceding embodiment, wherein the sensor is disposed on: a lifting device of the robotic container-handling vehicle, or a downward facing portion of the robotic container-handling vehicle, optionally a cantilever portion of the robotic container-handling vehicle.Embodiment 12. A system including a robotic container-handling vehicle for use in a vertical farming system comprising a storage grid including a plurality of stacked containers, each container arranged to support at least one organism, the robotic container-handling vehicle comprising:at least one sensor configured to capture data indicative of a status of at least one organism supported by a container disposed at a first location within the storage grid; wherein the system comprises at least one controller arranged to: determine the status of the at least one organism based on the data; and control the robotic container-handling vehicle based on the status of the at least one organism.Embodiment 13. The system of embodiment 12, wherein the status is a state of growth, a state of health or a state of development of the at least one organism.Embodiment 14. The system of embodiment 12 or 13, wherein the data comprises an image of the at least one organism, optionally wherein the image indicates a colour, size, turgidity or contrast of at least one region of the at least one organism.Embodiment 15. The system of any of embodiments 12-14, wherein, responsive to determining that the status of the at least one organism does not match a reference or threshold status, the at least one controller is further arranged to: control the robotic container-handling vehicle to leave the container at the first location within the storage grid, or control the robotic container-handling vehicle to retrieve the container from the first location and move the container to a second location within the storage grid.Embodiment 16. The system of embodiment 15, wherein a condition at the second location is different from the condition at the first location.Embodiment 17. The system of embodiment 16, wherein the condition includes a temperature condition, an irrigation condition, a nutrient supply, a humidity and / or an illumination condition, optionally wherein the temperature condition includes a maximum, minimum or average temperature level and / or a temperature schedule, optionally wherein the irrigation condition includes a maximum, minimum or average irrigation level and / or an irrigation schedule, optionally wherein the humidity includes a maximum, minimum or average humidity level and / or a humidity schedule,optionally wherein the nutrient supply includes a maximum, minimum or average nutrient supply level and / or a nutrient supply schedule and / or a type of nutrient supplied, optionally wherein the illumination condition includes an illumination intensity, an illumination schedule and / or or a wavelength spectra.Embodiment 18. The system of any of embodiments 12-17, wherein, responsive to determining that the status of the at least one organism matches a reference status or meets a threshold status, the at least one controller is arranged to: determine that there is a match between the determined status and the reference status or that the threshold status has been met; and / or control the robotic container-handling vehicle to transport the container to an access station or port of the vertical farming system.Embodiment 19. The system of any of embodiments 12-18, wherein the at least one controller is arranged to determine, based at least in part on the data indicative of the status of the at least one organism, that the height or volume of the at least one organism respectively exceeds a height or volume threshold of the container; and responsive to determining that the at least one organism exceeds the threshold, the at least one controller is arranged to control the robotic container-handling vehicle to retrieve the container and deliver the container to a port or access station of the storage grid.Embodiment 20. The system of any of embodiments 12-19, wherein the sensor is at least one of: an image sensor; an infrared sensor; a distance sensor; a colour sensor; or a contrast sensor.Embodiment 21. The system of any of embodiments 12-20, wherein capturing the data comprises using an image sensor to capture the data while using a light source to illuminate the at least one organism.Embodiment 22. The system of any of embodiments 12-21, wherein the image sensor and / or the light source is disposed on: a lifting device of the robotic container-handling vehicle, or a downward facing portion of the robotic container-handling vehicle, optionally a cantilever portion of the robotic container-handling vehicle.Also described herein are the following numbered examples:Example 1. A system including a robotic container-handling vehicle for use in an automated storage and retrieval system comprising a storage grid including a plurality of stacked containers, the robotic container-handling vehicle comprising: at least one sensor configured to capture data indicative of a status of a container of the plurality of stacked containers; wherein the system comprises: at least one controller arranged to determine the status of the container based on the data, and control the robotic container-handling vehicle based on the status of the container.Example 2. The system of example 1, wherein capturing the data comprises capturing data associated with a content of the container.Example 3. The system of example 2, wherein the one or more controllers are configured to determine, based at least in part on the data associated with the content of the container, that: the container is empty; the container is empty and a reference content associated with the container indicates that the container is not empty; the container is not empty and a reference content associated with the container indicates that the container is empty; or the container is not empty and the content of the container does not match a reference content associated with the container.Example 4. The system of example 3, wherein, responsive to determining that the container is empty, the one or more controllers are configured to: control the robotic container-handling vehicle to leave the container in its place within the storage grid; and / or control a second robotic container-handling vehicle to retrieve the container.Example 5. The system of example 3 or 4, wherein, responsive to determining that the container is not empty and the content of the container does not match the reference content, wherein the one or more controllers are configured to: notify the automated storage and retrieval system that there is a discrepancy in the content of the container; and / or control the robotic container-handling vehicle or a second robotic containerhandling vehicle to transport the container to an inspection area of the automated storage and retrieval system.Example 6. The system of any of examples 3-5, wherein, responsive to determining that the container is empty and the reference content associated with the container indicates that the container is not empty, or that the container is not empty and the reference content associated with the container indicates that the container is empty, or that the container is not empty and the content of the container does not match a reference content associated with the container, the one or more controllers are configured to: control the robotic container-handling vehicle or a second robotic containerhandling vehicle to transport the container to another location within the storage grid or to a port or access station of the automated storage and retrieval system.Example 7. The system of any of examples 2-6, wherein the one or more controllers are configured to: determine, based at least in part on the data indicative of the content of the container, that the content of the container exceeds a height or volume threshold of the container; and responsive to determining that the content of the container exceeds the threshold, control the robotic container-handling vehicle or a second robotic container-handling vehicle to retrieve the container.Example 8. The system of any preceding example, wherein capturing the data indicative of the status of the container comprises capturing data indicative of an orientation of the container, optionally wherein the orientation of the container is an abnormal orientation and / or an orientation that does not match a reference orientation of the container.Example 9. The system of example 8, wherein:responsive to determining the orientation of the container, the one or more controllers are configured to control the robotic container-handling vehicle or a second robotic container-handling vehicle to retrieve the container or correct the orientation of the container to match the reference orientation.Example to. The system of example 1 or 2, wherein capturing the data indicative of the status of the container comprises capturing data indicative of a growth, nutrition, temperature or climate status of one or more organisms stored in the container; and optionally, the one or more controllers are configured to determine whether a change in a nutrition, temperature or climate parameter is required; and optionally, responsive to determining that the change is required, wherein the one or more controllers are configured to notify the automated storage and retrieval system of the required change and / or control the robotic container-handling vehicle to move the container to another location within the automated storage and retrieval system or to an access port or station.Example 11. The system of any preceding example, wherein the sensor is at least one of: a LIDAR sensor; an image sensor; a distance sensor; a colour sensor; a contrast sensor; or a sensor arranged to determine a type, orientation and / or shape of an item in the container, a label or marker on an item in the container, an orientation of the container, and / or a height or volume of contents in the container relative to the internal height or volume of the container.Example 12. The system of any preceding example, wherein the sensor is arranged to image the container to capture the data indicative of the status of the container.Example 13. The system of example 12, wherein imaging of the container comprises capturing, using a camera, an image of the inside of the container, and / or the contents of the container and / or the exterior of the container; and optionally, the robotic containerhandling vehicle further comprises a light source arranged to illuminate the inside of the container, the contents of the container and / or the exterior of the container.Example 14. The system of example 13, wherein the camera and / or the light source is disposed on: a lifting device of the robotic container-handling vehicle, or a cantilever portion of the robotic container-handling vehicle, ora downward-facing portion of the robotic container-handling vehicle.Example 15. The system of any preceding example, wherein the sensor is a weight sensor arranged to capture the combined weight of the container and its contents, wherein the determined status of the container is location of the container within the storage grid, and the one or more controllers are arranged to control the robotic container-handling vehicle based on the location of the container.

Claims

CLAIMS1. A method for use in an automated storage and retrieval system comprising a storage grid including a plurality of stacked containers, the method comprising: capturing, by at least one sensor on a robotic container-handling vehicle, data indicative of a status of a container of the plurality of stacked containers; determining the status of the container based on the data; and controlling an operation of the robotic container-handling vehicle based on the status of the container.

2. The method of claim 1, wherein capturing the data comprises capturing data associated with a content of the container.

3. The method of claim 2, further comprising: determining, based at least in part on the data associated with the content of the container, that: the container is empty; the container is empty and a reference content associated with the container indicates that the container is not empty; the container is not empty and a reference content associated with the container indicates that the container is empty; or the container is not empty and the content of the container does not match a reference content associated with the container.

4. The method of claim 3, further comprising, responsive to determining that the container is empty: controlling the robotic container-handling vehicle to leave the container in its place within the storage grid; and controlling a second robotic container-handling vehicle to retrieve the container.

5. The method of claim 3 or 4, further comprising, responsive to determining that the container is not empty and the content of the container does not match the reference content: notifying the automated storage and retrieval system that there is a discrepancy in the content of the container; and / orcontrolling the robotic container-handling vehicle or a second robotic containerhandling vehicle to transport the container to an inspection area of the automated storage and retrieval system.

6. The method of any of claims 3-5, further comprising, responsive to determining that the container is empty and the reference content associated with the container indicates that the container is not empty, or that the container is not empty and the reference content associated with the container indicates that the container is empty, or that the container is not empty and the content of the container does not match a reference content associated with the container: controlling the robotic container-handling vehicle or a second robotic containerhandling vehicle to transport the container to another location within the storage grid or to a port or access station of the automated storage and retrieval system.

7. The method of any of claims 2-6, further comprising: determining, based at least in part on the data associated with the content of the container, that the content of the container exceeds a height or volume threshold of the container; and responsive to determining that the content of the container exceeds the threshold, controlling the robotic container-handling vehicle or a second robotic container-handling vehicle to retrieve the container.

8. The method of any preceding claim, wherein capturing the data indicative of the status of the container comprises capturing data indicative of an orientation of the container, optionally wherein the orientation of the container is an abnormal orientation and / or an orientation that does not match a reference orientation of the container.

9. The method of claim 8, further comprising: responsive to determining the orientation of the container, controlling the robotic container-handling vehicle or a second robotic container-handling vehicle to retrieve the container or correct the orientation of the container to match the reference orientation.

10. The method of claim 1 or 2, wherein capturing the data indicative of the status of the container comprises capturing data indicative of a growth, nutrition, temperature or climate status of one or more organisms stored in the container; and optionally, themethod further comprises determining whether a change in a nutrition, temperature or climate parameter is required; and optionally, responsive to determining that the change is required, notifying the automated storage and retrieval system of the required change and / or moving the container to another location within the automated storage and retrieval system or to an access port or station. it. The method of any preceding claim, wherein the sensor is at least one of: a LIDAR sensor; an image sensor; a distance sensor; a colour sensor; a contrast sensor; or a sensor arranged to determine a type, orientation and / or shape of an item in the container, a label or marker on an item in the container, an orientation of the container, and / or a height or volume of contents in the container relative to the internal height or volume of the container.

12. The method of any preceding claim, wherein the sensor is a weight sensor and the data captured is the combined weight of the container and its contents, wherein the status of the container indicates the location of the container within the storage grid, and the robotic container-handling vehicle is operated based on the location of the container.

13. The method of any preceding claim, wherein capturing the data comprises capturing an image of the inside of the container and / or the exterior of the container, and / or the contents of the container; and optionally, the method further comprises using a light source to illuminate the inside of the container and / or the exterior of the container.

14. The method of claim 13, wherein the camera and / or the light source is disposed on: a lifting device of the robotic container-handling vehicle, or a cantilever portion of the robotic container-handling vehicle, or a downward-facing portion of the robotic container-handling vehicle.

15. A system including a robotic container-handling vehicle for use in an automated storage and retrieval system comprising a storage grid including a plurality of stacked containers, the robotic container-handling vehicle comprising: at least one sensor configured to capture data indicative of a status of a container of the plurality of stacked containers; wherein the system comprises:at least one controller arranged to determine the status of the container based on the data, and control the robotic container-handling vehicle based on the status of the container.6o

Images