Storage and retrieval system

Abstract

A method of operating a storage and retrieval system including a plurality of storage containers. The method includes, for each storage container, storing a content record including an item type and bounding box for each item previously determined to be placed in the storage container. Upon receiving an indication that an item of a specified item type has been added to the content of the selected storage container: acquiring one or more images of the content; processing the image to detect a single item and generate a current bounding box for each detected item; identifying that one of the current bounding boxes is related to the added item; and updating the content record to include the specified item type of the added item and the identified current bounding box.

Description

Technical Field

[0001] The present disclosure relates to a storage and retrieval system and a method for operating such a system. Specifically, the present disclosure provides a method for operating such a system, improving the efficiency of accessing items, and an apparatus for implementing such a method. Background Art

[0002] Warehousing facilities typically include receiving operations for receiving and processing transport items from various vendors before they are stored in the warehouse. The received items can be dumped into containers and stored in a storage and retrieval system, such as the system disclosed in WO2015 / 185628A1, in which stacks of storage containers filled with items are arranged within a grid structure that provides the storage portion of the system. The storage containers are accessible by loading handling equipment that runs on tracks located on top of the grid structure. In order to fulfill customer orders, one or more storage containers are removed from the storage portion by the loading handling equipment and transported to a picking station, where the ordered products are picked out of the containers, for example by human pickers or robotic picking equipment.

[0003] In order to be able to retrieve items from such a storage and retrieval system, it is necessary to record the items stored in each storage container and track the location of the storage containers within the storage portion. To achieve this, a conventional approach is to ensure that each storage container only contains items of a single type (e.g., items of a single stock keeping unit (SKU)). This approach simplifies the unloading operation (in which received items are placed into a storage container at a unloading station) and simplifies the process of identifying and locating the storage container containing the desired type of item. Specifically, using this approach, only an identifier for each storage container, the location of each storage container within the storage portion of the system, and a single item identifier that identifies the type of item placed within the storage container need to be stored.

[0004] While this conventional approach simplifies some operations, it does not optimally utilize the available space within the storage containers, resulting in a greater number of storage containers being required to store several different types of items and thereby increasing the size of the storage section required to store these items.

[0005] The present invention is designed under such background. Summary of the Invention

[0006] According to a first aspect, a computer-implemented method for operating a storage and retrieval system comprising a plurality of storage containers is provided. The method comprises, for each storage container, storing a contents record for the storage container, the contents record comprising an item type and a bounding box for each item previously determined to be placed in the storage container. The method further comprises, upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container from the plurality of storage containers: (i) capturing one or more images of the contents of the selected storage container, (ii) processing the one or more images to detect individual items and generating a current bounding box for each detected item, (iii) identifying one of the current bounding boxes as being associated with the added item by comparing the current bounding box with the bounding boxes stored in the contents record, and (iv) updating the contents record for the selected storage container to include the specified item type and the identified current bounding box for the added item.

[0007] Identifying that one of the current bounding boxes is associated with the added item may include comparing the current bounding box with the bounding boxes stored in the content record to identify a current bounding box that does not match any bounding box stored in the content record. Identifying that one of the current bounding boxes is associated with the added item may then include, for each bounding box stored in the content record, matching the stored bounding boxes with one of the current bounding boxes, and identifying that the current bounding box that does not match any of the stored bounding boxes is associated with the added item. Matching the stored bounding boxes with one of the current bounding boxes may include identifying any current bounding boxes having dimensions corresponding to respective dimensions of the stored bounding boxes.

[0008] Matching the stored bounding boxes to one of the current bounding boxes may include identifying any current bounding boxes having dimensions within a predefined range of dimensions of the stored bounding boxes. The predefined range may be defined as a percentage of the corresponding dimensions of the stored bounding boxes. Matching the stored bounding boxes to one of the current bounding boxes may include, for each current bounding box, determining whether a dimension of the current bounding box is within the predefined range of dimensions of the stored bounding boxes.

[0009] The step of matching the stored bounding box with one of the current bounding boxes may further include: if only one of the current bounding boxes is identified as having a size corresponding to each size of the stored bounding boxes, matching the identified current bounding box with the stored bounding box. The step of matching the stored bounding box with one of the current bounding boxes may further include: if a plurality of current bounding boxes are identified as having sizes corresponding to each size of the stored bounding boxes, matching the stored bounding box with a current bounding box of the plurality of current bounding boxes having a center closest to the center of the stored bounding box.

[0010] The method may further include, after updating the contents record of the selected storage container, transporting the selected storage container to a storage portion of the storage system. The method may further include, upon receiving a request to remove an item of a specified item type, searching the contents records of a plurality of storage containers to identify a selected storage container containing an item of the specified item type, and removing the selected storage container from the storage portion of the storage system. The method may further include, upon removing the selected storage container from the storage portion of the storage system, delivering the storage container to a picking station of the storage system, the picking station being configured to enable items to be removed from the contents of the selected storage container.

[0011] An item may be added to a selected storage container at a dump station of a storage system, the dump station being configured to enable the item to be added to the contents of the selected storage container and including an image capture device configured to capture an image of the contents of the selected storage container. The method may then further include removing the storage container from a storage portion of the storage system and delivering the storage container to the dump station of the storage system. The dump station may include a scanner configured to scan a machine-readable element (e.g., a barcode, a QR code, a radio frequency identification (RFID) tag, a near field communication (NFC) tag, etc.) associated with the item to obtain item type information for the item.

[0012] Items may be added to the selected storage container at an inventory handling station of the storage system, the inventory handling station being configured to enable items to be added to and removed from the contents of the selected storage container and including an image capture device configured to capture an image of the contents of the selected storage container. The method may then further include removing the storage container from the storage portion of the storage system and delivering the storage container to the inventory handling station of the storage system.

[0013] According to a second aspect, a computer-implemented method for operating a storage and retrieval system comprising a plurality of storage containers is provided. The method includes, for each storage container, storing a contents record for the storage container, the contents record including an item type and a bounding box for each item previously determined to be placed in the storage container. The method further includes, upon receiving a request to retrieve items of a target item type from a selected storage container of the plurality of storage containers: (i) capturing one or more images of the contents of the selected storage container, (ii) processing the one or more images to detect individual items and generating a current bounding box for each detected item, (iii) identifying an item in the current bounding box as being associated with an item of the target item type by comparing the current bounding box with the bounding boxes stored in the contents record, and (iv) generating a pick instruction using the identified current bounding box, the pick instruction indicating a location of the item of the target item type in the selected storage container.

[0014] The method may further include: upon confirming that an item of the target item type has been removed from the contents of the storage container, updating a record of the contents of the storage container to reflect the removal of the item from the storage container. The method may further include: after generating the picking instruction, obtaining information identifying the item type of the item that has been removed from the contents of the storage container, and comparing the identified item type with the target item type, wherein: (i) if the identified item type matches the target item type, confirming that the item of the target item type has been removed, and (ii) if the identified item type does not match the target item type, generating a notification that the target item has not been removed.

[0015] Identifying one of the current bounding boxes associated with an item of the target item type may include comparing the current bounding box with bounding boxes stored in the content record to identify one of the current bounding boxes that matches a bounding box stored in the content record for an item of the target item type. Identifying one of the current bounding boxes associated with an item of the target item type may include, for each bounding box stored in the content record, matching the stored bounding boxes with one of the current bounding boxes and identifying the current bounding box that matches a bounding box stored in the content record for an item of the target item type.

[0016] Matching the stored bounding boxes to one of the current bounding boxes may include identifying any current bounding boxes having dimensions corresponding to the dimensions of the stored bounding boxes. Matching the stored bounding boxes to one of the current bounding boxes may include determining whether the dimensions of any current bounding box are within a predefined range of the dimensions of the stored bounding boxes. The predefined range may be 15% of the dimensions of the stored bounding boxes. Matching the stored bounding boxes to one of the current bounding boxes may include, for each current bounding box, determining whether the dimensions of the current bounding box are within the predefined range of the dimensions of the stored bounding boxes.

[0017] The step of matching the stored bounding box with one of the current bounding boxes may further include: if only one of the current bounding boxes is identified as having a size corresponding to each size of the stored bounding boxes, matching the identified current bounding box with the stored bounding box. The step of matching the stored bounding box with one of the current bounding boxes may further include: if a plurality of current bounding boxes are identified as having sizes corresponding to each size of the stored bounding boxes, matching the stored bounding box with a current bounding box of the plurality of current bounding boxes having a center closest to the center of the stored bounding box.

[0018] The method may further include, upon receiving a request to remove an item of a target item type, searching through the contents records of a plurality of storage containers, identifying a selected storage container containing an item of the target item type, and removing the selected storage container from the storage portion of the storage system. The method may further include, upon removing the storage container from the storage portion of the storage system, delivering the storage container to a picking station of the storage system, the picking station being configured to enable items to be removed from the contents of the selected storage container and including an image capture device configured to capture an image of the contents of the storage container. The picking station may include a scanner configured to scan a machine-readable element associated with the item removed from the contents of the storage container to obtain information identifying the item type of the removed item. The method may further include, after updating the contents record of the storage container to reflect the removal of the item from the storage container, transporting the storage container to the storage portion of the storage system.

[0019] The step of generating picking instructions indicating the location of items of the target type in the storage container may include one or more of the following steps: (i) generating a visual representation of the identified current bounding box on a captured image of the contents of the storage container and displaying a combined image including the captured image and the visual representation of the bounding box, (ii) using one or more light sources to illuminate the location of the items of the target type defined by the identified bounding box, and (iii) transmitting picking instruction information to a robotic picking device, the picking instruction information including the location of the items of the target type defined by the identified bounding box.

[0020] According to a third aspect, a computer-implemented method for operating a storage system comprising a plurality of storage containers is provided. The method comprises the method according to the first aspect and the method according to the second aspect. Items may be added to each of the plurality of storage containers using the method according to the first aspect, and items may be removed from each of the plurality of storage containers using the method according to the second aspect.

[0021] According to a fourth aspect, a computer-implemented method for operating a storage and retrieval system comprising a plurality of storage containers is provided. The method comprises, for each storage container, storing a contents record for the storage container, the contents record comprising an item type and a bounding box for each item previously determined to be placed in the storage container. The method further comprises, upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container from the plurality of storage containers: (i) capturing one or more first images of the contents of the selected storage container, (ii) processing the one or more first images to detect individual items and generating a first bounding box for each detected item, (iii) identifying one of the first bounding boxes as being associated with the added item by comparing the first bounding boxes with the bounding boxes stored in the contents record, and (iv) updating the contents record for the selected storage container to include the specified item type and the identified first bounding box for the added item. The method further includes, upon subsequently receiving a request to retrieve an item of the specified item type from the selected storage container: (v) capturing one or more second images of the selected storage container, (vi) processing the one or more second images to detect individual items and generate a second bounding box for each detected item, (vii) identifying one of the second bounding boxes as being associated with an item of the specified item type by comparing the second bounding boxes with bounding boxes stored in the content record, and (viii) generating a picking instruction using the identified second bounding box, the picking instruction indicating the location of the item of the specified item type in the selected storage container.

[0022] Receiving a request to remove an item of a specified item type from a selected storage container may include: upon receiving the request to remove an item of the specified item type, searching through the contents records of a plurality of storage containers and identifying a selected storage container containing an item of the specified item type. The method may further include: after updating the contents record of the selected storage container, transporting the selected storage container to a storage portion of the storage system. The method may further include: upon receiving the request to remove an item of the first item type from the selected storage container, removing the selected storage container from the storage portion of the storage system.

[0023] According to a fifth aspect, there is provided a storage and retrieval system arranged to store items in a plurality of storage containers. The storage and retrieval system comprises: (a) a memory storing a contents record for each storage container, each contents record comprising an item type and a bounding box for each item previously determined to be placed in the storage container, (b) an input device arranged to receive an indication that an item of a specified item type has been added to the contents of a selected storage container from the plurality of storage containers, (c) one or more image capture devices arranged to capture one or more images of the contents of the selected storage container, and (d) one or more processors. The one or more processors are configured to, upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container: (i) cause one or more image capture devices to capture one or more images of the selected storage container, (ii) process the one or more images to detect individual items and generate a current bounding box for each detected item, (iii) identify one of the current bounding boxes as being associated with the added item by comparing the current bounding boxes with bounding boxes stored in the contents record, and (iv) update the contents record of the selected storage container to include the specified item type of the added item and the identified current bounding box.

[0024] The storage and retrieval system according to the fifth aspect may be further configured to perform the method according to the first aspect.

[0025] According to a sixth aspect, there is provided a storage and retrieval system arranged to store items in a plurality of storage containers. The storage and retrieval system comprises: (a) a memory storing a content record for each storage container, each content record comprising an item type and a bounding box for each item previously determined to be placed in the storage container, (b) an input device configured to receive a request to retrieve an item of a target item type from a selected storage container of the plurality of storage containers, (c) one or more image capture devices configured to capture one or more images of the contents of the selected storage container, and (d) one or more processors. The one or more processors are configured to, upon receiving a request to retrieve an item of a target item type from a selected storage container: (i) cause one or more image capture devices to capture one or more images of the selected storage container, (ii) process the one or more images to detect individual items and generate a current bounding box for each detected item, (iii) identify one of the current bounding boxes as being associated with an item of the target item type by comparing the current bounding box with bounding boxes stored in a content record, and (iv) generate a picking instruction using the identified current bounding box, the picking instruction indicating the location of the item of the target item type in the selected storage container.

[0026] The storage and retrieval system according to the sixth aspect may be further configured to perform the method according to the second aspect.

[0027] According to a seventh aspect, there is provided a storage and retrieval system arranged to store items in a plurality of storage containers. The storage and retrieval system comprises: (a) a memory storing a contents record for each storage container, each contents record comprising an item type and a bounding box for each item previously determined to be placed in the storage container, (b) a first input device arranged to receive an indication that an item of a specified item type has been added to the contents of a selected storage container from the plurality of storage containers, (c) a second input device arranged to receive a request to remove an item of the specified item type from the selected storage container, (d) one or more image capture devices arranged to capture one or more images of the contents of the selected storage container, and (e) one or more processors. The one or more processors are configured to, upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container: (i) cause one or more image capture devices to capture one or more first images of the selected storage container, (ii) process the one or more first images to detect individual items and generate a first bounding box for each detected item, (iii) identify one of the first bounding boxes as being associated with the added item by comparing the first bounding boxes with bounding boxes stored in a contents record, and (iv) update the contents record of the selected storage container to include the specified item type of the added item and the identified first bounding box. The one or more processors are further configured to, upon subsequently receiving a request to remove an item of the specified item type from the selected storage container: (v) cause the one or more image capture devices to capture one or more second images of the selected storage container, (vi) process the one or more second images to detect individual items and generate a second bounding box for each detected item, (vii) identify one of the second bounding boxes as being associated with an item of the specified item type by comparing the second bounding boxes with bounding boxes stored in the content record, and (viii) generate a picking instruction using the identified second bounding box, the picking instruction indicating the location of the item of the specified item type in the selected storage container.

[0028] The storage and retrieval system according to the seventh aspect may be further configured to perform the method according to the first aspect and / or the method according to the second aspect.

[0029] In an eighth aspect, there is provided a computer program comprising instructions which, when executed by a computer, causes the computer to perform the method according to any one of the first, second, third or fourth aspects.

[0030] According to an eighth aspect, there is provided a data processing system comprising a processor configured to perform the method according to any one of the first, second, third or fourth aspects.

[0031] Other variations and advantages will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and other aspects of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 is a flow chart illustrating an embodiment of a method of operating a storage and retrieval system; Figure 2 It shows that it can be used in Figure 1 A flowchart of an embodiment of a method for processing a bounding box in a method; Figure 3 is a flow chart illustrating an embodiment of a further method of operating a storage and retrieval system; Figure 4 It shows that it can be used in Figure 3 A flowchart of an embodiment of a method for processing a bounding box in a method; Figure 5 is a schematic diagram of an embodiment of a storage and retrieval system suitable for implementing the methods described herein; Figure 6 is a schematic diagram of a further embodiment of a storage and retrieval system suitable for implementing the methods described herein; Figure 7 is a schematic diagram of yet another further embodiment of a storage and retrieval system suitable for implementing the methods described herein.

[0033] In the drawings, like features are indicated by like reference numerals where appropriate. DETAILED DESCRIPTION

[0034] A method of operating a storage and retrieval system is now described. The method is applicable to a storage and retrieval system comprising a plurality of storage containers.

[0035] Figure 1 is a flow chart illustrating an embodiment of a method 100 for operating a storage and retrieval system including a plurality of storage containers. Specifically, Figure 1 An embodiment of a method 100 of operating a system in adding an item to a selected storage container of a plurality of storage containers is presented.

[0036] At step 101, the system stores a contents record for each of a plurality of storage containers. For each storage container, the contents record includes an item type and a bounding box for each item previously determined to be placed in the storage container. The item type includes information specifying the item type of the item. For example, the item type information may include an item type identifier, such as the item's stock keeping unit (SKU), universal product code (UPC), or international article number (IAN). As described in more detail below, the bounding box stored in the contents record defines the last known location of the item within the storage container. For example, if the bounding box is two-dimensional, the bounding box data may include the x- and y-coordinates of the center of the bounding box, as well as the width and height of the bounding box. Alternatively, the bounding box data for a two-dimensional bounding box may include the x- and y-coordinates of a first corner of the bounding box, as well as the x- and y-coordinates of a second diagonally opposite corner of the bounding box. If the bounding box is three-dimensional, the bounding box data may also include the depth of the bounding box.

[0037] At step 102, the system receives an indication that an item of a specified item type has been added to the contents of a selected storage container among a plurality of storage containers. In an exemplary embodiment, step 102 may include receiving a specific item type of an item to be added to the contents of the selected storage container and then receiving an indication that the item has been added to the storage container.

[0038] For example, receiving the specific item type of the item to be added to the contents of the selected storage container can include receiving the item type identifier from a scanner, which has acquired the item type identifier by scanning a machine-readable element associated with the item. For example, such a scanner can include a reader device capable of reading any one of a barcode, QR code, RFID tag, or NFC tag attached to, integral with, or integrated into the item and configured to provide the item type identifier. The act of scanning the item can be performed immediately before the item is added to the contents of the storage container.

[0039] The step of receiving an indication that an item has been added to the storage container may then include receiving an indication from an input device of the system. For example, the input device may include a user input device, such as a graphical user interface or a user-operable switch or button configured to generate an indication upon operation by a user. Alternatively, the input device may include a sensor configured to detect, upon activation, that an item has been added to the storage container and generate a corresponding indication. For example, such a sensor may include a load sensor configured to detect a change in weight of the contents of the storage container and generate a corresponding indication when the weight increases significantly. Alternatively, such a sensor may include a motion sensor or a proximity sensor configured to detect when an item enters the storage container.

[0040] In an alternative embodiment, step 102 may include receiving a specific item type of item to be added to the contents of the selected storage container, and treating receipt of the specific item type as an indication that the item will have been added to the storage container after a short delay. The system may then be configured to implement a delay of a predefined time before the next step in the opening process.

[0041] At step 103 , receipt of the indication in step 102 initiates the acquisition of one or more images of the contents of the selected storage container.

[0042] At step 104, the one or more images captured at step 103 are processed to detect individual items within the image(s) and generate a current bounding box for each detected item. The image processing required to detect items and generate corresponding bounding boxes may be performed using any suitable object localization method. For example, neural network-based methods such as You Only Look Once (YOLO), Regional Convolutional Neural Network (R-CNN), Fast Regional Convolutional Neural Network, or Single Shot Detector (SSD) may be applicable. Each of these methods generates a bounding box, a class label, and a confidence score for an object detected in the image, where the bounding box indicates the predicted location and predicted size of the object, the class label indicates the predicted class or type of the object, and the confidence score indicates the probability that the bounding box contains the object.

[0043] In one exemplary embodiment, an image is processed by an object detection algorithm that is trained to identify only items within the image, without specifying class labels for different item types. Specifically, the object detection algorithm is trained using a training dataset that includes a plurality of images and, for each training image, a bounding box for a single item appearing in the training image and a class label for each item. The class label is the same for each individual item across the plurality of training images. In other words, in the training dataset, items are assigned a single class label (e.g., "item") regardless of their type. As described in more detail below, according to the methods described herein, identifying item types during the image processing step is unnecessary. Therefore, the object detection algorithm only needs to detect individual items within a storage container, without having to classify them individually. This simplifies the training process and increases the flexibility of object detection because the object detection algorithm does not need to be trained to specifically detect a wide range of potentially distinct item types, which can vary significantly and include items of different types that appear visually very similar. Additionally, this avoids the need to retrain the algorithm when items are changed or updated.

[0044] In step 105, the system identifies one of the current bounding boxes as being associated with the added item by comparing the current bounding box with the bounding boxes stored in the content record. Specifically, the current bounding box is compared with the bounding boxes stored in the content record to identify a current bounding box that does not match any bounding box stored in the content record. The unmatched current bounding box is then identified as being associated with the added item. In this application, the term "match" is used to refer to things that are identical or similar in some way. Specifically, things that correspond in size, shape, position, etc. can be considered a match.

[0045] In one exemplary embodiment, identifying that one of the current bounding boxes is associated with the added item includes, for each bounding box stored in the content record, matching the stored bounding box with one of the current bounding boxes. Subsequently, identifying the remaining current bounding boxes that do not match any of the stored bounding boxes as being associated with the added item.

[0046] At step 106 , the system updates the contents record of the selected storage container to include the specified item type of the added item and the identified current bounding box.

[0047] Reference above Figure 1The method may then further include, after updating the contents record of the selected storage container, transporting the selected storage container to a storage portion of the storage system. In a preferred embodiment, the selected storage container is transported to the storage portion by an automated load handling device or a robotic load handling device. The method may then further include instructing the automated load handling device or the robotic load handling device to transport the selected storage container to the storage portion.

[0048] Reference above Figure 1 The method may further include, upon receiving a request to retrieve an item of a specified item type, searching through the contents records of the plurality of storage containers to identify a selected storage container containing an item of the specified item type, and retrieving the selected storage container from a storage portion of the storage system. In a preferred embodiment, the selected storage container is retrieved from the storage portion by an automated loading and handling device or a robotic loading and handling device. The method may then further include instructing the automated loading and handling device or the robotic loading and handling device to retrieve the selected storage container from the storage portion.

[0049] In an optional embodiment, the above reference Figure 1 The described method may further include the step of confirming that an item has been added to the contents of the storage container. In such embodiments, this step is performed after receiving an indication that an item has been added to the storage container (step 102). For example, confirming that an item has been added to the contents of the storage container may include comparing a count (i.e., quantity) of a current bounding box with a count of a stored bounding box. If an item has been added to the contents of the storage container, the count of the current bounding box will be one greater than the count of the stored bounding box, thereby confirming that the item has been added to the contents. If the count of the current bounding box is greater than the count of the stored bounding box by more than one, it indicates that more than one item has been added. Similarly, if the count of the current bounding box is not greater than the count of the stored bounding box by more than one (i.e., equal to or less than the count of the stored bounding box), it indicates that no item has been added to the contents. The method may then further include generating an error indication or a warning indication if the count of the current bounding box is not one greater than the count of the stored bounding box. For example, the step of generating an error indication or a warning indication may include generating one or more visual, auditory and / or tactile indications using a user interface of the system, sending an error message or a warning message, etc.

[0050] In the context of storage and warehousing, the process of adding items to a storage container is referred to as a dumping operation, as the incoming items are removed from any transport containers and / or packaging and dumped into some separate storage device. The storage and retrieval system may therefore include one or more dumping stations at which dumping operations are performed, wherein the dumping stations are arranged to receive storage containers and enable items to be added to the contents of the storage containers. In such a system, at least steps 102 to 106 will be performed when a selected storage container is located at a dumping station of the system. Figure 1 The described method may then further include, before adding the item to the selected storage container, retrieving the selected storage container from the storage portion of the storage system and delivering the storage container to a dump station of the storage system.

[0051] In one exemplary embodiment, each content record is stored in association with an identifier of an associated storage container. The method may then further include, before adding an item to the storage container, obtaining an identifier for a selected storage container. The storage container identifier may then be used to obtain the content record for the selected storage container. For example, obtaining the identifier for the selected storage container may include scanning a machine-readable element associated with the selected storage container, the machine-readable element providing (e.g., encoded) the storage container identifier for the selected storage container. For example, this may include scanning a barcode, QR code, NFC tag, or RFID tag attached to or integral with the storage container. In an alternative embodiment, obtaining the identifier for the selected storage container may include receiving the storage container identifier from a control system that has selected the storage container. For example, the control system may query the content records of a plurality of storage containers to select a storage container suitable for storing the item to be added, and may then provide the storage container identifier for the selected storage container. Depending on this embodiment, performing a lookup may then include identifying an item of the target type in one of the content records and then using the associated storage container identifier to identify the associated storage container. The location of the storage container in the storage portion can then be stored either in association with the content record of the storage container so that it can be directly determined by the identified content record, or stored separately in association with the storage container identifier so that the location of the storage container can be determined by performing a further lookup using the storage container identifier from the identified content record.

[0052] Figure 2 It shows that according to the above reference Figure 1 A flow chart of an embodiment of the steps (step 105) of the described method, wherein the steps may be performed to identify that one of the current bounding boxes is associated with the added item.

[0053] In step 201, a content record of a selected storage container is obtained. In step 202, a first bounding box stored in the content record is selected for processing.

[0054] In step 203, the dimensions of each current bounding box are compared to the dimensions of the stored bounding boxes. Any current bounding box having dimensions corresponding to the dimensions of the stored bounding boxes is then identified as a possible match for the stored bounding box. In one exemplary embodiment, if the dimensions are within a predefined range of the dimensions of the stored bounding boxes, the current bounding box is identified as having dimensions corresponding to the dimensions of the stored bounding boxes. Step 203 may then include, for each current bounding box, determining whether the dimensions of the current bounding box are within a predefined range of the dimensions of the stored bounding boxes. For example, the predefined range may be 15% of the corresponding dimensions of the stored bounding boxes. Each dimension of the current bounding box must then be within 15% of the respective dimensions of the stored bounding boxes in order to be considered to have the corresponding dimensions and, therefore, to be considered a possible match.

[0055] The predefined range used when identifying possible matches for a stored bounding box can be selected based on the system's accuracy when generating the bounding box and / or the differences in size among the item types processed by the system. For example, a system that processes a wide variety of items (at least some of which have similar sizes) can use a smaller range to ensure greater accuracy when identifying possible matches. Conversely, if the system processes a wide variety of items in which different item types have significantly different sizes, a larger range can be used to ensure that possible matches are identified without increasing the frequency of false matches. Thus, the predefined range can be any value, but is preferably from 1% to 50%, and more preferably from 1% to 33%.

[0056] At step 204 , after each current bounding box has been compared to the stored bounding boxes, it is determined whether a plurality (ie, more than one) of the current bounding boxes have sizes that correspond to the respective sizes of the stored bounding boxes.

[0057] If only one of the current bounding boxes is identified as having sizes corresponding to the respective sizes of the stored bounding boxes, the process proceeds to step 205. At step 205, the single current bounding box of similar size is then identified as a match to the stored bounding box, and the process proceeds to step 207.

[0058] If multiple current bounding boxes are identified as having sizes corresponding to respective sizes of the stored bounding box, such that there are multiple possible matches to the stored bounding box, the process proceeds to step 206. In step 206, the stored bounding box is matched with the one of the multiple current bounding boxes of corresponding sizes (i.e., the possible matches identified in step 203) that has a center closest to the center of the stored bounding box. In other words, if multiple current bounding boxes are determined to have similar sizes to the stored bounding box, the center of each of these similarly sized current bounding boxes is compared to the center of the stored bounding box to identify which of the similarly sized current bounding boxes has a center closest to the center of the stored bounding box. The similarly sized current bounding box with a center closest to the center of the stored bounding box is then determined to be a match to the stored bounding box, and the process proceeds to step 207.

[0059] At step 207, a determination is made as to whether all bounding boxes in the contents record of the selected storage container have been processed. If not all stored bounding boxes have been processed, the process proceeds to step 208, where the next bounding box in the contents record is selected for processing, and then returns to step 203. If all stored bounding boxes have been processed, the process proceeds to step 209. At step 209, if each stored bounding box has been matched with one of the current bounding boxes, the remaining current bounding box that does not match any stored bounding box is identified as being associated with the added item.

[0060] Reference above Figure 1 The described method allows a storage and retrieval system to store multiple different types of items within each storage container because the contents of each storage container can be accurately recorded and tracked. In addition, as described in more detail below, the above reference Figure 1 The described method may also improve the efficiency and accuracy of the subsequent process of retrieving items from storage containers containing multiple different types of items.

[0061] Figure 3 is a flow chart illustrating an embodiment of a further method 300 of operating a storage and retrieval system including a plurality of storage containers. Specifically, Figure 3 An embodiment of a method 300 of operating a system in removing an item from a selected storage container among a plurality of storage containers is presented.

[0062] In step 301, the system stores a content record for each storage container in a plurality of storage containers. For each storage container, the content record includes the item type and bounding box of each item previously determined to be placed in the storage container. For example, the content record for each storage container can be stored using the reference above. Figure 1 Specifically, for each instance in which an item is added to a storage container, the specified item type of the item being added and the item type of the item being added can be used as described above. Figure 1 and / or Figure 2 The current bounding box identified by the described method updates the content record of the storage container.

[0063] In step 302, the system receives a request to remove an item of a target item type from a selected storage container among a plurality of storage containers. In an exemplary embodiment, step 302 may include receiving the request to remove an item of the target item type and then searching through the content records of the plurality of storage containers to select a storage container whose content record includes an item of the target item type. For example, receiving the request to remove an item of the target item type may include receiving an order from an ordering system, the order including at least one item of the target item type. The system may then be configured to fulfill the order by selecting a storage container whose content record indicates that the storage container contains an item of the target item type and retrieving the selected storage container from a storage portion of the storage system. In an exemplary embodiment, performing the search may include identifying an item of the target item type in one of the content records and then identifying the associated storage container using a storage container identifier stored in association with the content record.

[0064] At step 303, receipt of the request at step 302 initiates the acquisition of one or more images of the contents of the selected storage container.

[0065] In step 304, the one or more images acquired in step 303 are processed to detect individual items within the one or more images and generate a current bounding box for each detected item. Figure 1 As described above, any suitable object localization method can be used to perform the image processing required to detect items and generate corresponding bounding boxes. For example, neural network-based methods such as You Only Look Once (YOLO), Regional Convolutional Neural Network (R-CNN), Fast Regional Convolutional Neural Network, or Single Shot Detector (SSD) may be suitable.

[0066] At step 305, the system identifies one of the current bounding boxes that is associated with an item of the target item type by comparing the current bounding box with the bounding boxes stored in the content record. Specifically, the current bounding box is compared with the bounding boxes stored in the content record to identify one of the current bounding boxes that matches a bounding box of an item of the target item type stored in the content record.

[0067] In one exemplary embodiment, identifying that one of the current bounding boxes is associated with an item of the target item type includes, for each bounding box stored in the content record, matching the stored bounding box with one of the current bounding boxes. The current bounding box that matches a bounding box stored in the content record for an item of the target item type can then be readily identified.

[0068] In step 306, the system generates a picking instruction using the identified current bounding box, which indicates the location of items of the target type within the selected storage container. In an exemplary embodiment, the operation of picking items from the storage container may be performed manually (i.e., by a human picker or picker). In such an embodiment, step 307 may include generating a visual representation of the identified bounding box on a captured image of the contents of the storage container and displaying a combined image including the captured image and the visual representation of the bounding box. The combined image is displayed on a display device visible to the human picker, such that the picking instruction indicates not only the target item type but also the location of items of the target item type within the selected storage container. In this way, method 300 simultaneously improves the speed and accuracy of manually removing (i.e., picking) items from the storage container. Additionally or alternatively, step 307 may include using one or more light sources to at least partially illuminate the location of the target item type within the storage container defined by the identified bounding box. For example, a directable light source (e.g., a directable light source on a motorized mount) may be aligned to illuminate the location. As another example, groups of fixed light sources may be selectively activated to at least partially illuminate a location.

[0069] In an alternative embodiment, the operation of picking items from the storage container may be performed automatically (e.g., using a robot). In such an embodiment, step 307 may include sending a pick instruction message to the robotic picking device, the pick instruction message including the location of the item of the target type in the storage container defined by the identified bounding box.

[0070] Reference above Figure 3The described method may then further comprise, upon confirming that an item of the target item type has been removed from the contents of the storage container, updating a contents record of the storage container to reflect the removal of the item from the storage container. In an exemplary embodiment, updating the contents record to reflect the removal of the item from the storage container may comprise deleting information associated with the removed item from the contents record. Specifically, this may comprise deleting the item type and bounding box of the removed item from the contents record. In an alternative embodiment, updating the contents record to reflect the removal of the item from the storage container may comprise updating the contents record to include an indication that the removed item is no longer contained in the storage container. For example, the indication may comprise the date and time at which the item was removed, which date and time may then be used to track the contents of the storage container over time.

[0071] In an optional embodiment, the above reference Figure 3 The described method may further include, after matching each stored bounding box with one of the current bounding boxes, updating the contents record of the storage container to replace the stored bounding box with the current bounding box. In other words, for each item in the contents record, the bounding box stored in the contents record is updated with the current bounding box for each item in the contents record. This ensures that the stored bounding box data is refreshed each time a bounding box is generated for an item in the contents record. The updating of the contents record may be performed immediately after each stored bounding box is matched with one of the current bounding boxes, or alternatively, after determining whether the item has been removed from the contents of the storage container. If the latter is the case, the contents record may be updated to reflect both the removal of the item from the storage container and the replacement of the stored bounding box with the current bounding box for each item remaining in the storage container.

[0072] In an optional embodiment, the above reference Figure 3The described method may further include the step of confirming that the selected storage container contains the expected number of items. In such embodiments, this step is performed after generating a current bounding box for each detected item (step 304) and before identifying an item of the target item type within the current bounding box (step 305). For example, confirming that the selected storage container contains the expected number of items may include comparing the count (i.e., quantity) of the current bounding box with the count of stored bounding boxes. If the selected storage container contains the expected number of items, the count of the current bounding box will equal the count of the stored bounding box. If the count of the current bounding box does not equal the count of the stored bounding box, this indicates that the storage container does not contain the expected number of items, which may cause problems when completing the picking operation. The method may then further include generating an error or warning indication if the count of the current bounding box does not equal the count of the stored bounding box. For example, generating an error or warning indication may include generating one or more visual, audible, and / or tactile indications using the system's user interface, sending an error or warning message, etc.

[0073] Reference above Figure 3 The described method may further include, after generating the picking instruction, obtaining information identifying an item type of the item that has been picked from the contents of the storage container, and comparing the identified item type with a target item type. For example, obtaining information identifying the item type of the item that has been picked from the contents of the selected storage container may include receiving an item type identifier from a scanner, the scanner having obtained the item type identifier by scanning a machine-readable element associated with the item. For example, such a scanner may include a reader device capable of reading any one of a barcode, QR code, RFID tag, or NFC tag configured to provide an item type identifier, the barcode, QR code, RFID tag, or NFC tag being attached to, integral with, or integrated into the item. The method may then further include, if it is determined that the identified item type matches the target item type, confirming that an item of the target item type has been removed, and if it is determined that the identified item type does not match the target item type, generating a notification that the target item has not been removed. Specifically, if it is determined that the identified item type does not match the target item type, then an item of the wrong type has been removed from the selected storage container. The step of generating a notification that the target item has not been removed may then include generating a further picking instruction indicating that an item of the target type has not been removed from the storage container, and preferably placing the incorrectly picked item back into the storage container.

[0074] In addition, the above reference Figure 3The method may further include, before acquiring the one or more images of the selected storage container, removing the selected storage container from the storage portion of the storage system. In a preferred embodiment, the selected storage container is removed from the storage portion by an automated or robotic loader. The method may then further include instructing the automated or robotic loader to remove the selected storage container from the storage portion.

[0075] Reference Figure 3 The described method may further include, after updating the storage container's contents record to reflect the removal of the item from the storage container, transporting the storage container to a storage portion of the storage system. In a preferred embodiment, the selected storage container is transported to the storage portion by an automated load handler or a robotic load handler. The method may then further include instructing the automated load handler or the robotic load handler to transport the storage container to the storage portion of the storage system.

[0076] In the context of storage and warehousing, the process of removing items from storage containers is referred to as a picking operation, as outbound items are picked from storage containers and consolidated into orders. A storage and retrieval system may therefore comprise one or more picking stations at which picking operations are performed, wherein the picking stations are arranged to receive storage containers and enable items to be removed from the contents of the storage containers. In such a system, at least steps 303 to 306 will be performed when a selected storage container is located at a picking station of the system. Figure 3 The described method may then further include, prior to removing the item from the selected storage container, retrieving the selected storage container from the storage portion of the storage system and delivering the storage container to a picking station of the storage system.

[0077] Figure 4 It shows that according to the above reference Figure 3 A flow chart of an embodiment of the described method steps (step 305 ) that may be performed to identify an item in a current bounding box that is associated with a target item type.

[0078] Except for the final step (step 409), Figure 4 The steps of the method are Figure 2 Therefore, for the sake of brevity, the previous steps (steps 401 to 408) will not be repeated here. Figure 2 In the described method, the final step (step 209) comprises identifying the remaining one current bounding box that does not match any stored bounding box as being associated with the added item. Figure 4In the final step of the illustrated method (step 409), a current bounding box associated with an item of the target item type may be identified by using the content record to identify which stored bounding box is associated with an item of the target item type, and then selecting the current bounding box that has matched the identified stored bounding box.

[0079] In a preferred embodiment, the above reference Figure 1 The method described dumps any incoming items to be stored in the storage and retrieval system into a storage container. Figure 3 The described method picks any outbound items from a storage container to be retrieved / removed from the storage and retrieval system.

[0080] Figure 5 is a schematic diagram of an embodiment of a storage and retrieval system suitable for implementing the methods described herein. As a schematic diagram, Figure 5 Only the elements and functional entities necessary for understanding the arrangement are shown, other components are omitted for simplicity. Figure 5 Different as shown. Figure 5 The connections shown are logical connections, and the actual physical connections may be different. It will be apparent to those skilled in the art that the arrangement may also include other functions and structures.

[0081] exist Figure 5 In an embodiment of the present invention, storage and retrieval system 500 is implemented as a combination of computer hardware and software. Storage and retrieval system 500 is configured to store items in a plurality of storage containers 501 and includes a memory 502, one or more input devices 503, at least one image acquisition device 504, and one or more processors 505. System 500 may also include one or more output devices 506.

[0082] At least one of the one or more input devices 503 is configured to obtain item type information of an item to be added to and / or removed from the contents of the selected storage container 501. For example, and as described above, the one or more input devices 503 may include a scanner configured to scan a machine-readable element associated with an item to be added to the contents of the storage container 501 to obtain an item type identifier of the item. By way of further example, the one or more input devices 503 may include a receiver configured to receive item type information of an item to be removed from the contents of the selected storage container 501. Specifically, the receiver may be configured to receive a request to remove an item of a target item type from a storage container within the system 500, the request being received via wireless communication or wired communication.

[0083] At least one of the one or more input devices 503 may also be configured to receive an indication that an item has been added to and / or removed from the contents of a storage container. For example, and as described above, the one or more input devices 503 may include a user-operable switch or button that, upon operation by a user, is configured to generate an indication that an item has been added to or removed from a storage container. Alternatively, and as described above, the one or more input devices 503 may include a sensor that, upon activation of the sensor, is configured to detect that an item has been added to or removed from a storage container and generate a corresponding indication.

[0084] One or more image capture devices 504 are configured to capture images of the contents of a selected storage container. As will be described in more detail below, the image capture device 504 may be located at an inventory processing station, such as a picking station, a dumping station, or a combined picking and dumping station of the system 500. The image capture device 504 will then be configured to capture images of the interior of the storage container as it appears at the inventory processing station. In an optional embodiment, the one or more image capture devices 504 may include a depth camera (e.g., a stereo camera, a combination of a camera and a depth sensor, a structured light camera, etc.) configured to capture distance / depth information for the captured images. The distance / depth information may then be used to generate a three-dimensional bounding box for each detected item. Using a three-dimensional rather than a two-dimensional bounding box improves the ability to distinguish items based on size.

[0085] At least one of the one or more output devices 506 can be configured to provide feedback to a user of the system 500. For example, one of the one or more output devices 506 can include a display or graphical user interface configured to display information related to the progress, success, and / or failure of each step of the methods described herein.

[0086] At least one of the one or more output devices 506 can be configured to provide picking instructions associated with removing an item from a selected storage container, wherein the picking instructions are configured to indicate the location of the item of the target type within the selected storage container. For example, and as described above, one of the one or more output devices 506 can include a display or graphical user interface configured to display a combined image comprising a captured image of the storage container and a visual representation of a bounding box identified as associated with the item of the target type, thereby highlighting the intended location of the item within the storage container. By way of further example, and as described above, one of the one or more output devices 506 can include a directable light source that can be directed to illuminate at least a portion of the location of the item of the target type within the storage container. As yet another example, one of the one or more output devices 506 can include a transceiver configured to transmit a picking instruction message to an automated or robotic picking device, the picking instruction message including the location of the item of the target type within the storage container as defined by the identified bounding box.

[0087] Memory 502 stores various programs / computer-executable instructions for execution by processor(s) 505. Memory 502 also provides storage for any required data, such as a record of the contents of each storage container 501. The record of the contents of each storage container 501 includes the item type and a bounding box for each item determined to be placed in the storage container. As described above, the bounding box defines the last known location of the item within the storage container and is generated based on a captured image of the storage container.

[0088] The programs / computer-executable instructions stored in memory 502 and implemented by processor(s) 505 include, but are not limited to, an image processing unit 507, a bounding box processing unit 508, a record maintenance unit 509, and an output generation unit 510. Storage system processor(s) 505 are then configured to execute the programs / computer-executable instructions stored in memory 502 and, in doing so, control at least input device(s) 503, image acquisition device(s) 504, and output device(s) 506. Image processing unit 507 is configured to process images acquired by image acquisition device(s) 504 in order to detect individual items and generate a current bounding box for each detected item. Subsequently, the bounding box processing unit 508 is configured to compare the current bounding box generated by the image processing unit 507 with the bounding boxes stored in the content record of the selected storage container, and thereby identify the current bounding box associated with an item that has been added to the storage container (i.e., for a dumping operation) or an item to be removed from the storage container (i.e., for a picking operation). The record maintenance unit 509 is configured to maintain a content record for each storage container. Specifically, upon determining that an item has been added to the storage container, the record maintenance unit 509 is configured to update the content record for the storage container to include the specified item type of the added item and the identified current bounding box. Upon determining that an item has been removed from the storage container, the record maintenance unit 509 is configured to update the content record for the storage container to reflect the removal of the item from the storage container (e.g., by deleting details of the removed item from the content record or by adding an indication that the removed item is no longer in the storage container). The output generation unit 510 is then configured to generate any associated output to be issued by the one or more output devices 506. For example, the output generation unit 510 can be configured to generate the picking instructions described above (e.g., for display on a display device by at least partially illuminating the target item, or as an instruction message sent to an automated picking device).

[0089] Figure 6 FIG is a schematic diagram of a detailed embodiment of a storage and retrieval system 600 suitable for implementing the above-described method. Figure 6 In the embodiment of FIG. 6 , the storage and retrieval system 600 includes a storage portion / location 611 , one or more dump stations 612 , one or more picking stations 613 , and a control system 614 .

[0090] The storage portion 611 of the system 600 is arranged to store a plurality of storage containers 601. For example, the storage portion 611 may be the same as or similar to that disclosed in WO 2015 / 185628 A1, in which the storage containers are arranged in stacks within a grid structure. In the arrangement disclosed in WO 2015 / 185628 A1, the storage containers in the storage portion are accessed by a loading and handling device running on tracks located at the top of the grid structure, which then transports the storage containers between the storage portion and the rest of the system. As a further example, the storage portion 611 may include a plurality of racks on which the storage containers 601 may be placed when stored in the storage portion 611.

[0091] Each of the one or more dumping stations 612 is arranged to receive a storage container 601 and enable items to be added to the contents of the received storage container 601 . Figure 6 Thus, a storage container 601 containing a plurality of items (I) of various item types is shown and has been presented at a dump station 612 to receive one or more items. Each dump station 612 includes one or more image capture devices 604 configured to capture images of the contents of the storage container. Each dump station 612 may also include one or more input devices 603 and / or one or more output devices 606, such as the devices described above. For example, each dump station 612 may include an input device 603 comprising a scanner or reader for obtaining the item type via a machine-readable element associated with the item that has been added to the storage container, and a user-operable button for generating an indication that the item has been added to the storage container. Each dump station 612 may then also include an output device 606 in the form of a display device for displaying feedback. For example, such a display device may be integrated into the dump station 612, or may be provided on a portable device having a display (e.g., a smartphone, a smart wearable computer, or a tablet) that is configured to communicate with the dump station 612 to receive data for display. In alternative embodiments, one or more of the dump stations 612 may include robotic item handling equipment configured to add incoming items to storage containers present at the dump station 612.

[0092] Each of the one or more picking stations 613 is arranged to receive a storage container 601 and enable items to be removed from the contents of the received storage container 601 . Figure 6Thus, a storage container 601 containing a plurality of items (I) of various item types is shown, and has been presented to a picking station 613 for removal of one or more items. Each picking station 613 includes one or more image capture devices 604 configured to capture images of the contents of the storage container. Each picking station 613 may further include one or more input devices 603 and / or one or more output devices 606, such as those described above. For example, each picking station 613 may include an output device 606 in the form of a display device for displaying picking instructions (e.g., a captured image of the storage container and a visual representation of a bounding box identified as associated with an item of the target type) and for displaying any feedback. Such a display device may be integrated into the picking station 613, or may be provided on a portable device with a display (e.g., a smartphone, a smart wearable computer, or a tablet), which is configured to communicate with the picking station 613 to receive data for display. Each picking station 613 may then further include a scanner or reader for obtaining the item type via a machine-readable element associated with the item that has been removed from the storage container. In an alternative embodiment, one or more picking stations 613 may include robotic item handling equipment configured to remove outbound items from the storage container present at the picking station 613.

[0093] Subsequently, the control system 614 is configured to implement the computer processing required by the system 600 and thus provides the memory 602 and the processor 605, and optionally provides an external interface to the system 600. Figure 6 The control system 614 shown is centralized, having interfaces with the tipping station 612, the picking station 613, and (if necessary) the storage part 611, but the control system 614 can also be at least partially decentralized so that each tipping station 612 and the picking station 613 include a memory and a processor for collaboratively implementing the control system 614.

[0094] Subsequently, the system 600 further includes transport devices 615 to 620, each of which is configured to transport items or storage containers. Specifically, the system 600 includes one or more inbound item transport devices 615, which are configured to transport inbound items to the dumping station 612 so that these inbound items can be dumped into storage containers. The system 600 also includes one or more container transport devices 616, 617 configured to transport storage containers between the dumping station 612 and the storage part 611. Subsequently, the system 600 further includes one or more container transport devices 618, 619 configured to transport storage containers between the storage part 611 and the picking station 619. The system 600 also includes one or more outbound item transport devices 620 configured to transport outbound items away from the picking station 613. In this regard, the outbound items are either transported individually or picked into an outbound container capable of accommodating a plurality of items (e.g., at least a portion of an order), and the outbound items are then transported from the picking station 613 in such an outbound container. Each transport device 615 to 620 may include any suitable transport device. For example, each transport device 615 to 620 may include one or more conveyors (e.g., roller conveyors or belt conveyors) or automated load handling equipment capable of following a fixed route or autonomous load handling equipment (e.g., automated guided vehicles (AGVs) or autonomous mobile robots (AMRs) capable of autonomously planning their routes).

[0095] Figure 7 is a schematic diagram of a further detailed embodiment of a storage and retrieval system 700 suitable for implementing the methods described herein. Figure 7 The system 700 shown is substantially the same as that of the reference Figure 6 The system described is the same. Figure 7 The system 700 is similar to the Figure 6 The main differences between the described systems are: Figure 7 Rather than including separate dumping and picking stations, system 700 includes an inventory processing station 712 configured to perform both dumping and picking operations. In an exemplary embodiment, inventory processing station 712 is configured to perform only one of dumping and picking operations for each storage container that appears at inventory processing station 712. However, in alternative embodiments, inventory processing station 712 may be configured to perform both dumping and picking operations for each storage container that appears at inventory processing station 712. For example, when storage container 701 appears at inventory processing station 712 (e.g., having been transported from storage section 711), inventory items may first be picked from storage container 701, and then incoming items may be added to storage container 701 before storage container 701 is returned to storage section 711.

[0096] Since the inventory processing station 712 is used for both unloading and picking operations, the transport devices 715 to 718 are configured to transport items or storage containers to and from the inventory processing station 712. Specifically, the system 700 includes one or more inbound item transport devices 715, which are configured to transport inbound items to the inventory processing station 712 so that these inbound items can be unloaded into storage containers. The system 700 also includes one or more container transport devices 716, 717 configured to transport storage containers between the inventory processing station 712 and the storage section 611. The system 700 also includes one or more outbound item transport devices 718 configured to transport outbound items away from the inventory processing station 712.

[0097] It should be understood that the various features described herein may take the form of a computer program embodied as a computer-readable medium having computer-executable code for use with or in connection with a computer. For the purposes of this specification, a computer-readable medium is any tangible device that can contain, store, transmit, disseminate, or transport a program for use with or in connection with a computer. Furthermore, a computer-readable medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a propagation medium. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), hard disks, and optical disks. Currently, examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and DVD.

[0098] The flowchart in the figure shows the architecture, functionality and operation of possible implementations of the method according to various embodiments. In this regard, each block in the flowchart can represent a module, fragment or portion of code, which includes one or more executable instructions for implementing a specified logical function (one or more than one). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks may not be performed in the order mentioned in the figure. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or, depending on the functions involved, the blocks may sometimes be executed in a reversed order. It should also be noted that each block in the flowchart and the combination of blocks in the flowchart can be implemented by a dedicated hardware-based system (performing the specified function or behavior) or a combination of dedicated hardware and computer instructions.

[0099] It should be understood that the features described above can all be used together in a single system. In other embodiments of the present invention, some features may be omitted. These features may be used in any compatible arrangement. Many variations and modifications not explicitly described above are possible without departing from the scope of the present invention as defined by the appended claims.

[0100] It should be understood that the above description is made by way of example only and that various modifications may be made by those skilled in the art. Although various embodiments have been described above in some detail, or have been described with reference to one or more individual embodiments, those skilled in the art may make many changes to the disclosed embodiments without departing from the scope of the invention.

Claims

1. A computer-implemented method of operating a storage and retrieval system comprising a plurality of storage containers, the method comprising: For each storage container, storing a contents record for the storage container, the contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; Upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container of the plurality of storage containers: capturing one or more images of the contents of the selected storage container; processing the one or more images to detect individual items and generate a current bounding box for each detected item; identifying one of the current bounding boxes as being associated with the added item by comparing the current bounding box with the bounding boxes stored in the content record; The contents record of the selected storage container is updated to include the specified item type and the identified current bounding box of the added item.

2. The method according to claim 1, wherein Identifying one of the current bounding boxes associated with the added item includes comparing the current bounding box with the bounding boxes stored in the content record to identify one of the current bounding boxes that does not match any of the bounding boxes stored in the content record.

3. The method of claim 2, wherein: The step of identifying one of the current bounding boxes associated with the added item comprises: For each of the bounding boxes stored in the content record, matching the stored bounding box with one of the current bounding boxes; and The current bounding box that does not match any of the stored bounding boxes is identified as being associated with the added item.

4. The method according to claim 3, wherein: The step of matching the stored bounding box with one of the current bounding boxes comprises: Any of the current bounding boxes having dimensions corresponding to respective dimensions of the stored bounding boxes are identified.

5. The method according to any one of claims 3 or 4, wherein The step of matching the stored bounding box with one of the current bounding boxes comprises: Any of the current bounding boxes having a size within a predefined range of the respective sizes of the stored bounding boxes are identified.

6. The method according to claim 5, wherein: The step of matching the stored bounding box with one of the current bounding boxes comprises: For each of the current bounding boxes, it is determined whether the size of the current bounding box is within the predefined range of the respective sizes of the stored bounding boxes.

7. The method according to any one of claims 4 to 6, wherein The step of matching the stored bounding box with one of the current bounding boxes further comprises: If only one of the current bounding boxes is identified as having sizes corresponding to the respective sizes of the stored bounding boxes, the identified current bounding box is matched with the stored bounding boxes.

8. The method according to any one of claims 4 to 7, wherein The step of matching the stored bounding box with one of the current bounding boxes further comprises: If a plurality of the current bounding boxes are identified as having sizes corresponding to the respective sizes of the stored bounding box, the stored bounding box is matched with one of the plurality of the current bounding boxes having a center closest to a center of the stored bounding box.

9. A computer-implemented method of operating a storage and retrieval system comprising a plurality of storage containers, the method comprising: For each storage container, storing a contents record for the storage container, the contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; Upon receiving a request to retrieve an item of a target item type from a selected storage container among the plurality of storage containers: acquiring one or more images of the selected storage container; processing the one or more images to detect individual items and generate a current bounding box for each detected item; identifying one of the current bounding boxes as being associated with an item of the target item type by comparing the current bounding box to the bounding boxes stored in the content record; A picking instruction is generated using the identified current bounding box, the picking instruction indicating a location of items of the target type in the selected storage container.

10. The method of claim 9, further comprising: Upon confirming that an item of the target item type has been removed from the contents of the storage container, the contents record of the storage container is updated to reflect the removal of the item from the storage container.

11. The method according to any one of claims 9 or 10, wherein The step of identifying one of the current bounding boxes that is associated with an item of the target item type includes comparing the current bounding box with the bounding boxes stored in the content record to identify one of the current bounding boxes that matches the bounding box stored in the content record for an item of the target item type.

12. The method according to any one of claims 9 to 11, wherein The step of identifying an item in the current bounding box that is related to the target item type includes: For each of the bounding boxes stored in the content record, matching the stored bounding box with one of the current bounding boxes; and The current bounding box is identified that matches the bounding box stored in the content record for an item of the target item type.

13. The method of claim 12, wherein: The step of matching the stored bounding box with one of the current bounding boxes comprises: Any of the current bounding boxes having dimensions corresponding to respective dimensions of the stored bounding boxes are identified.

14. The method according to any one of claims 12 or 13, wherein The step of matching the stored bounding box with one of the current bounding boxes comprises: It is determined whether the size of any of the current bounding boxes is within a predefined range of the respective sizes of the stored bounding boxes.

15. The method of claim 14, wherein: The step of matching the stored bounding box with one of the current bounding boxes comprises: For each of the current bounding boxes, it is determined whether the size of the current bounding box is within the predefined range of the respective sizes of the stored bounding boxes.

16. The method according to any one of claims 12 to 15, wherein The step of matching the stored bounding box with one of the current bounding boxes further comprises: If only one of the current bounding boxes is identified as having sizes corresponding to the respective sizes of the stored bounding boxes, the identified current bounding box is matched with the stored bounding boxes.

17. The method according to any one of claims 12 to 16, wherein The step of matching the stored bounding box with one of the current bounding boxes further comprises: If a plurality of the current bounding boxes are identified as having sizes corresponding to the respective sizes of the stored bounding box, the stored bounding box is matched with one of the plurality of the current bounding boxes having a center closest to a center of the stored bounding box.

18. A computer-implemented method of operating a storage system comprising a plurality of storage containers, the method comprising: A method according to any one of claims 1 to 8 and a method according to any one of claims 9 to 17.

19. A computer-implemented method of operating a storage and retrieval system comprising a plurality of storage containers, the method comprising: For each storage container, storing a contents record for the storage container, the contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; Upon receiving an indication that an item of a specified item type has been added to the contents of a selected storage container of the plurality of storage containers: capturing one or more first images of the contents of the selected storage container; processing the one or more first images to detect individual items and generate a first current bounding box for each detected item; identifying one of the first current bounding boxes as being associated with the added item by comparing the first current bounding box to the bounding boxes stored in the content record; updating the contents record of the selected storage container to include the specified item type of the added item and the identified first current bounding box; Upon subsequently receiving a request to retrieve an item of the specified item type from the selected storage container: acquiring one or more second images of the selected storage container; processing the one or more second images to detect individual items and generate a second current bounding box for each detected item; identifying that one of the second current bounding boxes is associated with an item of the specified item type by comparing the second current bounding box with the bounding boxes stored in the content record, and Picking instructions are generated using the identified second current bounding box, the picking instructions indicating the location of the items of the specified type in the selected storage container.

20. The method of claim 19, wherein: The step of receiving a request to remove an item of the specified item type from the selected storage container comprises: Upon receiving a request to retrieve an item of the specified item type, a search is performed among the content records of the plurality of storage containers, and the selected storage container containing the item of the specified item type is identified.

21. A storage and retrieval system arranged to store items in a plurality of storage containers, the storage and retrieval system comprising: a memory storing a contents record for each storage container, each contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; an input device configured to receive an indication that an item of a specified item type has been added to the contents of a selected storage container of the plurality of storage containers; one or more image capture devices configured to capture one or more images of the contents of the selected storage container; as well as one or more processors configured, upon receiving the indication that an item of a specified item type has been added to the contents of the selected storage container: causing the one or more image capture devices to capture one or more images of the selected storage container; processing the one or more images to detect individual items and generate a current bounding box for each detected item; identifying one of the current bounding boxes as being associated with the added item by comparing the current bounding box with the bounding boxes stored in the content record; as well as The contents record of the selected storage container is updated to include the specified item type of the added item and the identified current bounding box.

22. A storage and retrieval system arranged to store items in a plurality of storage containers, the storage and retrieval system comprising: a memory storing a contents record for each storage container, each contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; an input device configured to receive a request to remove an item of a target item type from a selected storage container of the plurality of storage containers; one or more image capture devices configured to capture one or more images of the contents of the selected storage container; as well as One or more processors configured, upon receiving a request to retrieve an item of a target item type from a selected storage container: causing the one or more image capture devices to capture one or more images of the selected storage container; processing the one or more images to detect individual items and generate a current bounding box for each detected item; identifying that one of the current bounding boxes is associated with an item of the target item type by comparing the current bounding box with the bounding boxes stored in the content record, and A picking instruction is generated using the identified current bounding box, the picking instruction indicating a location of items of the target type in the selected storage container.

23. A storage and retrieval system arranged to store items in a plurality of storage containers, the storage and retrieval system comprising: a memory storing a contents record for each storage container, each contents record including an item type and a bounding box for each item previously determined to be placed in the storage container; a first input device configured to receive an indication that an item of a specified item type has been added to the contents of a selected storage container of the plurality of storage containers; a second input device configured to receive a request to remove an item of the specified item type from the selected storage container; one or more image capture devices configured to capture one or more images of the contents of the selected storage container; as well as One or more processors, the one or more processors configured to: Upon receiving the indication that an item of the specified item type has been added to the contents of the selected storage container: causing the one or more image capture devices to capture one or more first images of the selected storage container; processing the one or more first images to detect individual items and generate a first current bounding box for each detected item; identifying one of the first current bounding boxes as being associated with the added item by comparing the first current bounding box to the bounding boxes stored in the content record; as well as updating the contents record of the selected storage container to include the specified item type of the added item and the identified first current bounding box; as well as Upon subsequently receiving a request to retrieve an item of the specified item type from the selected storage container: causing the one or more image capture devices to capture one or more second images of the selected storage container; processing the one or more second images to detect individual items and generate a second current bounding box for each detected item; identifying that one of the second current bounding boxes is associated with an item of the specified item type by comparing the second current bounding box with the bounding boxes stored in the content record, and Picking instructions are generated using the identified second current bounding box, the picking instructions indicating the location of the items of the specified type in the selected storage container.

Images