Method of collecting data about contents of a container

Abstract

The present disclosure relates to a computer-implemented method for collecting data about contents of containers in an automated storage and retrieval system comprising a storage grid with a plurality of stacked containers, the method comprising: controlling a robotic vehicle on the storage grid to lift a first container of the plurality of stacked containers out of the storage grid; in response to the controlling, collecting, with a sensor of the robotic vehicle, data indicative of the contents of the first container; and storing the collected data in association with a unique identifier of the first container.

Description

Technical Field

[0001] This disclosure relates to a method for collecting data about the contents of a container. More specifically, the invention relates to a computer-implemented method for collecting data relating to the contents of a container in an automated storage and retrieval system, and to a system and computer-readable medium for performing the method. Background Technology

[0002] Traditional storage solutions typically involve arranging goods on rows of shelves within a warehouse. The shelf location of each item is recorded in inventory, and goods are retrieved from the shelves by a goods picker. The shelves are replenished and inventory is updated as needed when goods enter and leave the warehouse.

[0003] Robotic pickers and automated inventory management systems can assist warehouse workers. Automated transport systems can also be implemented in traditional warehouse setups to move goods from their storage locations to picking and / or packing stations.

[0004] An alternative to traditional warehouse setups is an automated storage and retrieval system (AS / RS), in which robots retrieve items from their recorded locations within the warehouse and transport them to packing stations or ports. Such systems can reduce or eliminate the space required to navigate between rows of shelves to access inventory, thus eliminating the need for wide aisles within the warehouse. One example of such a system involves arranging goods in boxes or containers configured to be stacked side-by-side within a three-dimensional grid. A track system is arranged on top of the grid, along which robotic container handling vehicles, configured to lift containers from the grid, can travel. The container handling vehicles are configured to transport containers from the grid to ports or stations at the periphery of the grid, allowing the goods inside the containers to be picked up and packed.

[0005] Given the wide variety of items that can be stored in an automated storage and retrieval system, monitoring the location and condition of these items can be challenging. Depending on the items stored in the grid of containers, their condition and quality can change or deteriorate over time. If the product is damaged, rotten, or unsuitable for picking when the container arrives at the port, this delays the picking and packaging process and can reduce efficiency.

[0006] One or more aspects of the invention described in this application are set forth in the claims. Attached Figure Description

[0007] This disclosure will now be described in more detail with reference to several exemplary embodiments shown in the accompanying drawings, in which: Figure 1 A perspective view of a storage system is shown, which includes a grid and multiple robotic container handling vehicles configured to retrieve and / or rearrange goods stored within the grid. Figure 2 It shows Figure 1 A top view of the system; Figure 3A It shows that it is suitable for use in Figure 1 Side view of the first robotic container handling vehicle used in the system; Figure 3B It shows that it is suitable for use in Figure 1 Side view of the second robotic container handling vehicle used in the system; Figure 3C yes Figure 3B A three-dimensional top-side view of the robot; Figure 4 A computing device for implementing the operations described herein is shown; Figure 5 It shows Figure 3B A three-dimensional view of the bottom of the robot shown. Figure 6 A flowchart is shown for a method of collecting data about the contents of a container; Figure 7 A flowchart is shown for a method of processing the collected data about the contents of the container. Detailed Implementation

[0008] In summary, this disclosure relates to a method for collecting data related to the contents of boxes in an automated storage and retrieval system without requiring the boxes to be removed from the system and manually inspected. As a robot moves boxes around a grid, it can lift the boxes off the grid to transport them to another location. This lifting can be used to trigger sensors that collect data about the box's contents. Therefore, data related to the box's contents can be collected regardless of whether the box is later brought to a pick-up port or remains in the grid. The robot continues to move the boxes as instructed, and the data can be processed and used to monitor the state of the box's contents. Therefore, it is unnecessary to present the container to a port to obtain an indication of the quality or state of its contents. For example, if groceries are stored in containers, they may spoil over time in ways that are not easily predictable. To avoid spoiled groceries being presented to the pick-up port, they can be monitored while still within the grid, as long as the boxes containing the groceries are moved by the robot.

[0009] Overview of Automated Storage and Retrieval Systems

[0010] refer to Figure 1In the embodiment shown, the grid 100 comprises a frame formed by a plurality of substantially linear adjacent vertical columns 102, which are formed between vertical frame members 104 and extend in the X and Y directions 108, 110. The grid elements can be made of any suitable material; for example, the frame members can be formed from extruded aluminum. Boxes (or “storage containers” or “containers”) 112 are stacked one on top of another in the storage columns 102 in the Z direction 114, preferably stacked in a self-supporting manner, thereby forming storage volumes for the storage units of the respective boxes 112 extending in the X, Y, and Z directions 108, 110, 114.

[0011] A track system or network 116 is formed on top of grid 100 and includes pairs of vehicle tracks or rails 118a, 118b and 120a, 120b extending in the X and Y directions 108, 110, respectively. A robotic container handling vehicle or robot (or “robot” or “robotic vehicle”) 122 is provided, which may have various sizes, shapes, and functions, and is configured to run on tracks 118, 120 and transport boxes 112 in both the X and Y directions 108, 110. Robot 122 is also configured to lift / lower boxes 112 from column 102 in the Z direction 114, with boxes 112 optionally guided by vertical frame members 104. Robot 122 approaches boxes 112 via access openings 124 formed above column 102 and between tracks 118, 120.

[0012] Some columns 102 can be used for alternative purposes beyond bin storage. For example, port columns 126, 128 include ports or access columns that allow bins 112 to be moved into and / or out of grid 100. Port columns 126, 128 provide vertical channels for raising bins 112 from or lowering bins 112 into one or more ports 130, 132. Ports 130, 132... Figure 1 The diagram shows the lowest level of the grid; however, the ports can be located at any vertical position along the columns. The corresponding port columns 126, 128 can be assigned for removing (“unloading”) box 112 from grid 100 and / or returning or conveying (“picking up”) the box to the grid. Therefore, ports 130, 132 are configured to allow box 112 to be removed and (horizontally) reintroduced into the associated port column. Thus, ports 130, 132 can include conveyors ( Figure 1 (Not shown in the image) Box 112 can be lowered onto a conveyor and transported horizontally out of the port column. Port columns 126, 128 include openings or access points through which box 112 enters and exits the column.

[0013] Box 112 can be transported by robot 122 along the top of grid 100 to and / or from port columns 126, 128, and from ports 130, 132 to a location outside grid 100, which may be an access station (not shown) for handling box 112 or its contents, such as a pick-up station for adding or removing contents from box 112. In alternative instances (not shown), box 112 can be transported to a port of another grid at the same or another level, or to an external facility. Transport of box 112 to and from ports 130, 132 can be carried out by any suitable means (not shown), including conveyors, transport vehicles, elevators, or robots.

[0014] refer to Figure 2 The illustrated embodiment provides a more detailed view of the XY configuration 200 of the track system 116 and the various types of robots 202, 204. The track system includes tracks 206 that define vertical column access openings 124 for accessing the bin 112. Tracks 206 can be of any suitable type for allowing robots 202, 204 to travel in the X and Y directions 108, 110, including (not shown) recessed tracks for receiving vehicle wheels, or raised tracks for engaging wheel recesses. Each track 206 may include a single track or multiple parallel tracks in each of the X and Y directions 108, 110.

[0015] The first "cantilever" type robot 202 Figure 3A The diagram is shown in more detail and includes a body 300, a set of wheels 302, and a lifting device 304. The body 300 contains operating equipment (not shown) for the robot 202, which includes a drive system, a power system, and a control system. The wheels 302 allow the robot 202 to move in one of the X and Y directions, and an additional set of wheels (not visible in this view) allows movement in the other direction, both along corresponding tracks or rails 206. One or both sets of wheels can be raised or lowered to allow selective engagement of tracks for movement in the desired direction. The lifting device 304 includes a cantilever element 306 extending in the XY plane from the top of the body 300, and a gripping device 308 capable of being raised and lowered from the cantilever element 306. The gripping device 308 is configured to grip or engage a box 112; for example, by gripping a portion of the box 112, or by passively or actively engaging a suitable configuration portion of the box 112.

[0016] The second "internal cavity" type robot 204 is in Figure 3BAs shown in more detail below, and as an alternative to a cantilever-type lifting system, it includes an inner cavity 310 within the main body 300, and a lifting device 312 including a clamping device (not shown) is located within this inner cavity. In this case, the main body 300 includes the robot's operating equipment and storage space for one or more boxes 112 for use, for example, when transporting the boxes 112.

[0017] Figure 3C It shows Figure 3B A stereoscopic side view of the robot, in which one can see Figure 3B The first set of wheels, 302. Mentioned above but not mentioned in... Figure 3B The additional set of wheels shown in Figure 3C The wheel 303 is shown in the middle. The additional set of wheels 303 is arranged perpendicular to the first set of wheels 302 to allow the robot 204 to move in the X and Y directions via the first set of wheels and the second set of wheels 302 and 303, respectively. Figure 3C The first set of wheels and the second set of wheels 302, 303 shown can be configured to independently lower to engage with the track (and conversely raise to disengage from the track), thereby allowing the robot 202 to traverse... Figure 2 The track arrangement shown moves in both the X and Y directions. Although Figure 3C The 3D diagram shown has Figure 3B Robot 204, but it should be understood that a similar vertical wheel arrangement can be applied to Figure 3A Robot 202.

[0018] Control and monitoring system

[0019] The control and monitoring of the automated storage and retrieval system (including monitoring and storing box positions, controlling box delivery, retrieval and transport, and robot path planning and collision avoidance) is handled by, for example, Figure 4 The control system shown communicates with the robot and / or other controllable system components. Control can be performed locally or remotely and can be implemented by a processing system, such as a computing device. Therefore, the methods described herein can form all or part of a computer-implemented method, or a system configured to perform the methods described herein.

[0020] refer to Figure 4 A processing system 400 suitable for performing the methods described herein will now be described. Figure 4A block diagram of one implementation of a processing system 400 is shown, which takes the form of a computing device within which a set of instructions can be executed to cause the computing device to perform any or more of the methods described herein. In some implementations, the computing device may be connected (e.g., networked) to other machines in a local area network, intranet, extranet, or the Internet. The computing device may operate at the capacity of a server or client machine in a client-server network environment, or at the capacity of a peer-to-peer machine in a peer-to-peer (or distributed) network environment. The computing device may be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), cellular phone, networked home appliance, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) specifying the actions to be taken by the machine. Furthermore, although only a single computing device is illustrated, the term "computing device" should also be understood to include any set of machines (e.g., computers) that individually or collectively execute one or more sets of instructions to perform any or more of the methods described herein.

[0021] The instance processing system 400 includes a processor 402, a main memory 404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 406 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 418), which communicate with each other via a bus 430.

[0022] Processor 402 represents one or more general-purpose processors, such as microprocessors, central processing units, etc. More specifically, processor 402 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing combinations of instruction sets. Processor 402 may also be one or more special-purpose processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processor 402 is configured to execute processing logic (instruction 422) to perform the operations and steps described herein.

[0023] The processing system 400 may also include a network interface device 408. The processing system 400 may also include any of the following devices: a video display unit 410 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 412 (e.g., a keyboard or a touch screen), a cursor control device 414 (e.g., a mouse or a touch screen), and an audio device 416 (e.g., a speaker).

[0024] Obviously, Figure 4 Some features of the processing system 400 shown may be absent. For example, the processing system 400 may not require a display device 410 (or any associated adapter). This may be the case, for example, for a particular server-side computer device used only for its processing capabilities and not required to display information to a user. Similarly, a user input device 412 may not be necessary. In its simplest form, the processing system 400 includes a processor 402 and main memory 404.

[0025] Data storage device 418 may include one or more machine-readable storage media (or more specifically one or more non-transitory computer-readable storage media) 428 storing one or more instruction sets 422 embodying any or more of the methods or functions described herein. The instructions 422 may also reside wholly or at least partially in main memory 404 and / or processor 402 during execution by processing system 400, which also constitute computer-readable storage media 428.

[0026] The various methods described herein can be implemented by a computer program. A computer program may include computer code arranged to instruct a computer to perform one or more of the various methods described herein. The computer program and / or code for performing such methods may be provided to a device, such as a computer, on one or more computer-readable media or more generally on a computer program product. The computer-readable media may be transient or non-transient. One or more computer-readable media may be, for example, an electronic system, a magnetic system, an optical system, an electromagnetic system, an infrared system, or a semiconductor system, or a propagation medium for data transmission, such as for downloading code via the Internet. Alternatively, one or more computer-readable media may take the form of one or more physical computer-readable media, such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid disk, or optical disk, such as CD-ROM, CD-R / W, or DVD.

[0027] A computer program can be executed by processor 402 to perform the functions of the system and methods described herein.

[0028] In implementation, the modules, components and other features described herein may be implemented as discrete components or integrally formed within the functionality of hardware components such as ASICs, FPGAs, DSPs or similar devices.

[0029] A "hardware component" is a tangible (e.g., non-transitory) physical component (e.g., a group or more processors) capable of performing certain operations and which can be configured or arranged in a particular physical manner. A hardware component may include a dedicated circuit system or logic permanently configured to perform certain operations. A hardware component may be or include dedicated processors, such as field-programmable gate arrays (FPGAs) or ASICs. A hardware component may also include programmable logic or circuit systems temporarily configured by software to perform specific operations.

[0030] Therefore, the phrase “hardware component” should be understood to encompass tangible entities that can be physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform the specific operations described herein.

[0031] Furthermore, modules and components can be implemented as firmware or functional circuitry systems within a hardware device. Additionally, modules and components can be implemented as any combination of hardware devices and software components, or solely as software (e.g., code stored in or transmitted on a machine-readable medium or otherwise embodied in such media).

[0032] Operation of the automatic storage and retrieval system

[0033] In operation, each box 112 is assigned a unique identifier, which can be marked on the box 112 using a computer-readable identifier (e.g., a barcode, quick-response code, or RFID tag) for easy identification. The database of the processing system 400 stores the location of each box 112 associated with the unique identifier and optionally stores the contents of that box. When a box 112 is moved (e.g., when it is removed from grid 100), the database is updated to record the change in its location.

[0034] When it is desired to retrieve box 112 from grid 100, under the control of processing system 400, robots 202 and 204 are guided via track system 116 to vertical column 102, which includes storage units. Box 112 is positioned in the storage unit according to a database, and lifting devices 304 and 312 (depending on robot type) are positioned above the corresponding access opening 124, which is adjacent to or below robots 202 and 204. Robots 202 and 204 lower gripping device 308, which engages, grips, and lifts box 112. Robots 202 and 204 then transport box 112 to, for example, unloading port columns 126 and 128 for delivery to ports 130 and 132 for further processing outside grid 100. If the target or designated box 112 is located below other boxes in the stack, then robots 202, 204, or multiple robots possibly dedicated to this task, are controlled in a "digging" operation to temporarily or permanently lift and reposition the boxes above the target box 112 in sequence to retrieve the target box. It should be understood that other operations related to box 112 can be performed in a similar manner. For example, box 112 can be conveyed at ports 130, 132 of pick-up port columns 126, 128 for storage in grid 100, gripped and lifted by robots 202, 204, and conveyed to the desired storage unit, whereby, if necessary, the boxes above the desired location are repositioned as discussed above.

[0035] Control container lifting

[0036] The first "cantilever" type robot 202 and the second "cavity" type robot 204 each include lifting devices 304 and 312. Figure 5 The image shows a lifting device 312 for a second type of robot 204. The robot, moving on top of the grid 100, can move above the vertical column 102 of the box 112. The lifting devices 304, 312 can then descend to grip the box 112 and then retract to lift the box 112 outside the grid 100, for example, into the cavity 310 of the robot 204 or into the space between the cantilever element 306 of the robot 202 and the top of the grid 100. See below for reference. Figure 6 As described, sensors attached to robots 202, 204 can be triggered to collect data via the retraction of lifting devices 304, 312. This could be a proximity-based trigger configured to detect when the sensing box 112 is raised outside the grid 100.

[0037] Data on the contents of the collection box

[0038] Figure 6A flowchart illustrates a computer-implemented method for collecting data about the contents of boxes in an automated storage and retrieval system comprising a storage grid 100 of multiple stacked boxes 112.

[0039] As a method of computer implementation, Figure 7 The method can be executed by any data processing system, including the data processing system on processing system 400 and / or robots 202, 204. Therefore, each step of the method can be executed by any such data processing system.

[0040] In step S100, an instruction to transport the first box to its destination may be received. This step may be executed by the data processing system of robots 202 and 204. The instruction may be received from robots 202 and 204 (e.g., from processing system 400), or it may have been pre-programmed into a list of tasks to be performed by robots 202 and 204. The destination may be a storage unit within grid 100, a location at the top of grid 100, or a destination outside the grid, such as a port 300 (e.g., a pickup station) at the edge of grid 100.

[0041] Then, in step S105, robots 202 and 204 can be controlled to move to the first box. For example, the exact route traveled by robots 202 and 204 may have been calculated individually based on the geometry of grid 100. This allows the robots to be positioned for lifting the box and collecting data about its contents. Control may include a data processing system (e.g., processing system 400) that sends instructions to robots 202 and 204 and / or a data processing system (e.g., a data processing system on robots 202 and 204) that controls the drive systems of robots 202 and 204.

[0042] In step S110, robots 202 and 204 can be controlled on storage grid 100 to lift the first (“first box”) of a plurality of stacked boxes 112 out of storage grid 100. Lifting can be achieved through methods such as... Figure 3A The cantilever robot 202 shown is used, or by means of, Figure 3B The lifting is performed by the cavity robot 204 shown. This lifting can also be performed by any other robot configured to transport the box 112 from one location in the grid 100 to another. Control may include a data processing system (e.g., processing system 400) that sends instructions to robots 202, 204 and / or a data processing system (e.g., a data processing system on robots 202, 204) that controls the lifting mechanisms of robots 202, 204.

[0043] As the first box is lifted out of storage grid 100 by robots 202, 204 (e.g., from vertical column 102), the first box approaches robots 202, 204. The appropriate or optimal time to collect data about its contents will depend on what data is to be collected and what type of sensor is used to collect that data. Therefore, data collection can be triggered in many ways. Typically, the appropriate or optimal time to collect data can be some point in time after robots 202, 204 are controlled to lift the first box out of storage grid 100 and before robots 202, 204 are controlled to lift another box out of storage grid 100. Therefore, the collected data can be directly associated with the box to which it belongs, since the specific box 112 being lifted has been tracked (e.g., by processing system 400).

[0044] For example, when acquiring image data, the appropriate time to acquire the data could be when the first box is at a distance that causes the image to focus. As another example, when acquiring image data, the appropriate time to acquire the data could be when the first box has not yet been fully lifted by robots 202, 204, i.e., when the first box has not yet reached its minimum distance relative to robots 202, 204 (e.g., when robot 204 is not yet fully inside cavity 310) making ambient lighting available. As yet another example, the appropriate time to acquire the data could be when the first box has been fully lifted by robots 202, 204, i.e., when the first box has reached its minimum distance relative to robots 202, 204; this allows data acquisition to continue even when robots 202, 204 have reached their destination, and can allow for the acquisition of more accurate or consistent data.

[0045] In step S115, it can be determined that the first box is close to robots 202 and 204. More specifically, this determination can be made by determining that the position of the first box meets a (predetermined) proximity criterion. For example, the proximity criterion can be that the position of the first box is within a predetermined threshold distance of robots 202 and 204.

[0046] The determination can be made using sensors of robots 202 and 204 (e.g., in, on, or mounted to robots 202 and 204) or based on a predetermined amount of time elapsed since the (first) self-control of the robots to lift the first container out of storage grid 100.

[0047] This determination could be made by determining that the lifting devices 304 and 312 of robots 202 and 204 are in a retracted configuration. The retracted configuration could correspond to the first box being fully or at least partially lifted by the robot. For example, Figure 5 The retraction configuration of the lifting device 312 shown can be when the gripping device has gripped the box 112 and has fully returned upward within the cavity 310 of the robots 202, 204.

[0048] In step S120, data indicating the contents of the first container is collected. Sensors from robots 202 and 204 can be used to collect the data. As mentioned above, collection can be performed at any time, although generally, collection is in response to control (or lifting itself). Collection can be in response to any determination made in step S115, such as determining that the first container is close to robots 202 and 204, that the position of the first container relative to robots 202 and 204 meets a proximity criterion, that a predetermined amount of time has elapsed since the control robot lifted the first container out of storage grid 100, or that the lifting devices 304 and 312 of robots 202 and 204 are in a retracted configuration, etc. The types of data that can be collected will be described in more detail below. The sensors of robots 202 and 204 can be any sensors in, on, or mounted to robots 202 and 204. These sensors can be the same as those used in step S115, or they can be other sensors.

[0049] In step S125, the collected data is stored in association with a unique identifier for the first box. The method of storing the collected data can depend on the type of data being collected. The collected data can be stored in a database, or anywhere in the memory (e.g., temporary or transient memory) of any device performing the storage, or anywhere in the memory (e.g., temporary or transient memory) communicatively connected to any device performing the storage. The collected data can be stored in the memory of robots 202 and 204.

[0050] The unique identifier can be any identifier capable of uniquely identifying the first box. In some instances, the unique identifier is a numeric identifier, an alphabetic identifier, or a combination thereof. In some instances, the unique identifier identifies the position of the first box in or on grid 100 (e.g., in the form of three-dimensional coordinates); in this case, the unique identifier of the first box changes as the first box moves around grid 100. The unique identifier cannot directly identify the first box; for example, the unique identifier may indicate a sequence of collected data, and then the first box can be identified when associated with a sequence of boxes 112 that have been lifted by robots 202, 204. The unique identifier can be the same identifier used to track boxes in a storage and retrieval system using processing system 400. The unique identifier can also be used in warehouse management or inventory management systems. Data and unique identifiers may be sent for processing in step S135, which is described in more detail below.

[0051] In step S130, robots 202 and 204 can be controlled to move the first box 112 to the destination contained in the instructions of step S100. The control of step S130 may occur after or following the acquisition in step S120 and / or the storage in step S125, but more generally, it will occur after the control of step S110 (e.g., while acquisition continues, even when robots 202 and 204 have reached their destination). Therefore, the operation of the storage and retrieval system is unaffected by data acquisition. In some instances, robots 202 and 204 may be controlled to move the first box 112 to a different destination in response to the processing of the acquired data, for example, if it is determined that the contents of the box are, for example, of low quality. This is referred to below. Figure 7 To describe in more detail.

[0052] Collecting data about the contents of box 112 as it moves around grid 100 allows for indication of the status of items inside box 112 during normal operation and transfer, without removing box 112 from the system (e.g., taking them to a pickup station). This makes it possible to monitor the contents of box 112 more efficiently and continuously. This is particularly useful when the nature of the stored items is highly time-dependent, for example, because the items are perishable or because items are frequently added to or removed from box 112.

[0053] Depending on where the collected data is processed, it may need to be sent to a (separate) data processing system (e.g., processing system 400). Therefore, in step S135, the collected data can be transmitted to the data processing system associated with a unique identifier. Alternatively, the collected data can be processed by the data processing system on the robots 202 and 204 themselves.

[0054] The collected data can also be transmitted to a warehouse management system for storage. This allows the collected data to be tracked over time and stored in association with the container's management data. For example, the warehouse management system can store one or more recent images of the container's contents, one or more recent temperature readings, or one or more recent weight readings.

[0055] In one example, robots 202 and 204 are controlled by instructions transmitted via a first network, while collected data is transmitted via a second network, and the first and second networks are different. Therefore, the transmission of collected data does not interfere with communication traffic on the network controlling the movement of robots 202 and 204, and vice versa. This can be a particular consideration, for example, when many vehicles are operating on grid 100, or when large amounts of data are collected and transmitted. In another example, robots are controlled by instructions transmitted with a first network traffic priority (e.g., a first Quality of Service QoS), while collected data is transmitted with a second network traffic priority (e.g., a second Quality of Service QoS), and the second network traffic priority is lower than the first network traffic priority. That is, instructions controlling the movement of robots on grid 100 are transmitted before collected data. Therefore, unnecessary data collection does not interrupt instruction transmission and does not hinder the movement of robots 202 and 204 on grid 100.

[0056] Data type of contents of the box

[0057] The type of data collected by the sensor in step S120 can depend on the type of items (or "products") stored in the bin 112 of grid 100. For example, different data can be used to monitor fresh produce or clothing and apparel. Therefore, the type of data collected can be selected based on the expected contents of the bin.

[0058] As an example, a sensor can be a camera that acquires one or more optical images. Therefore, image recognition and processing algorithms can be applied to analyze the data. See below for reference. Figure 7 To describe these in more detail.

[0059] As another example, the sensor can be an infrared camera that acquires one or more infrared images.

[0060] As yet another example, the sensor can collect air composition data. Similarly, it can collect humidity or temperature data. This could be useful, for example, when the container stores fresh produce or plants. In another example, the sensor can collect a measurement of the mass or weight of the container, including its contents (the mass or weight of the contents can be obtained by subtracting a predetermined container weight), or a measurement of the mass or weight of the contents of the container.

[0061] These types of sensors and the data they collect can be used independently or in any combination.

[0062] Figure 7 A flowchart is shown illustrating a computer-implemented method for processing data about the contents of a box, which can be used to process data about the contents of a box. Figure 6 The method shown is used to collect data or any other data about the contents of the box. Figure 7 The steps can therefore be performed independently or in conjunction with... Figure 6 The steps are combined and executed to produce a method for collecting and processing data about the contents of the box.

[0063] As a method of computer implementation, Figure 7 The method can be executed by any data processing system, including the data processing system on processing system 400 and / or robots 202, 204. Therefore, each step of the method can be executed by any such data processing system.

[0064] Determining the contents of the box

[0065] At step S200, data indicating the contents of the first box is received. Data can be received from any source. For example, data can be received at a data processing system (e.g., processing system 400) from any device performing step S120 or (in the case of) the device apparatus performing step S135. As another example, data can be received internally at the data processing system on robots 202, 204 (e.g., when step S135 is omitted).

[0066] At step S205, which can be performed in parallel with step S200, warehouse management information may also be received, for example, from the inventory database of the automated storage and retrieval system. This may include, for example, information about inventory levels and the expected contents of the first box. It may also include information about the expected (but unverified) type of the contents of the first box. In another instance, the warehouse management information may include images of the box's contents. In yet another instance, the warehouse management information may include data from earlier (e.g., previous) data collection for comparison (e.g., earlier images of the box's contents, or earlier weights of the container and its contents, or the contents of the container).

[0067] In step S210, at least one type of item and / or the quantity (or “quantity”) of items in the first box can be determined. In some instances, this determination is made by reading from warehouse management information, thus indicating what is expected in the first box. In some instances, this determination is additionally or alternatively based on collected data, thus indicating what might actually be in the first box. This is described in more detail below.

[0068] Depending on the contents of box 112, the type of items can be determined at different levels of specificity. For example, in a grocery setting, the type could include "fruits and vegetables," "meat and fish," etc., or it could be more specific and include "carrots," "peaches," "cod," "beef," etc.

[0069] The type of product can also be determined from an image of the contents of a box using character recognition. For example, this can be used to read barcodes or other product information from item packaging.

[0070] When box 112 stores multiple types of items, the multiple item types in the first box can be determined. When box 112 stores only one type of item, the (single) item type in the first box can be determined.

[0071] When the collected data is in the form of images, image classification can be used to determine the type of item. For example, a machine learning model can be trained to identify the contents of a box from a predetermined set of options.

[0072] When there are multiple types of items in the first box, determining the quantity of items in the first box can determine the quantity of each type of item present in the first box.

[0073] Any of the above methods for determining the type or quantity of items in the first box may be used alone or in any combination.

[0074] Figure 7 The method may also include identifying differences between the expected quantity of items in the first box and the determined quantity of items, or differences between at least one expected type of items in the first box and at least one type of items determined. For example, there may be a different number of items in the box than expected based on warehouse management information.

[0075] Similarly, Figure 7 The method may additionally or alternatively include comparing the collected data with the expected contents data of the first box, and in response to the comparison, identifying the differences between the expected contents data and the collected data.

[0076] The expected content data can be previous images or other previously acquired data. Therefore, identifying differences may include determining whether changes have occurred in the content data since the last data acquisition, or whether changes meet predetermined importance criteria.

[0077] If a discrepancy is identified in any way, it can be flagged and prompted for (manual) confirmation by the operator, or a message can be sent back to the warehouse management system to update the inventory database.

[0078] Determining the quality of the contents of the box

[0079] In automated storage and retrieval systems, the quality of stored items can change over time—and in particular deteriorate. For example, perishable items may rot, thus causing other adjacent items to rot. Therefore, Figure 7 The method can determine the quality of the items in the first box.

[0080] The method of determining quality can depend on determining the type of items in the first box. In this case, in step S215, a quality determination algorithm can be selected based on the determined type of items in the first box. Then, in step S220, the selected quality determination algorithm is used to determine quality indicators. For example, if the type of items in the first box is determined to be fresh produce in the first stage, then the selected algorithm could be an algorithm for determining whether the product is spoiled. Otherwise, if the first box is determined to contain shelf-stable items, then the selected algorithm could be an algorithm for reading the “earliest” or expiration date marked on the items (e.g., using optical character recognition). As another example, a machine learning model can be trained to determine when fresh produce spoils or rots, or when glass breaks, and the selected algorithm can apply the machine learning model. As yet another example, fruits and vegetables release various gases (e.g., ethylene) as they ripen and begin to rot, and therefore sensors can be used to measure one or more of these gas levels, and the selected algorithm can compare these levels with corresponding thresholds to determine a quality indicator regarding the freshness of the produce.

[0081] In step S220, a quality index of the contents of the first box can be determined based on the collected data. The quality index can be “good” or “bad,” or “low,” “medium,” and / or “high.” Alternatively, the quality index can be a numerical value that falls within a predetermined range. For example, a quality index can be an integer having values ​​between 0 and 10 or 100, where higher values ​​indicate higher quality, and vice versa. When “low” quality is mentioned herein, this may mean that a “poor” or “low” quality index has been determined, or that the quality index has been determined to be within a predetermined low quality range. Similarly, when “high” quality is mentioned herein, this may mean that a “good” or “high” quality index has been determined, or that the quality index has been determined to be within a predetermined high quality range.

[0082] Determining quality indicators may also include comparing the collected data with at least one of the following: expected contents data for acceptable quality items; expected contents data for low quality items; or expected contents data for high quality items. For example, where temperature readings are an indication of quality, expected temperature ranges for low, acceptable, and high quality items can be defined, and the quality indicator can be determined based on which range the collected data falls into.

[0083] Quality metrics can be determined using image classification. This can be useful if, for example, low-quality items clearly appear to be high-quality items.

[0084] Quality metrics can be stored and monitored over time. This allows for the prediction of how quickly items will become low-quality. This information can then be used, for example, to prioritize items for picking before they become low-quality or to promote future sales. Therefore, quality metrics can be sent to a warehouse management system for storage.

[0085] Processing of low-quality contents

[0086] Once the quality indicators have been determined, they can be used as a guide for humans to examine the contents of the box. Therefore, in step S225, in response to determining that the quality indicators indicate the contents of the first box are of low quality, at least a portion of the collected data can be sent to the operator for review. Thus, the quality indicators can be used to trigger a more thorough, reliable, or accurate inspection of the collected data (e.g., image data), but when the contents of the first box have been determined to be of high quality, such an inspection is unnecessary, thereby reducing the need for operator review. Whether step S225 is performed can depend on the collected data and the likelihood that the operator will be able to accurately assess the quality from the data. For example, checking the “earliest” date may not be useful or unnecessary for the operator, but it could provide a more accurate assessment of whether the produce has spoiled. Alternatively, in response to determining that the quality indicators indicate the contents of the first box are of low quality, in step S245, robots 202 and 204 can be controlled to transport the first box to the processing station.

[0087] In step S230, user input regarding the quality of the contents of the first box can be received from the operator. This can take several forms. User input may include simple confirmation or rejection of quality indicators. User input may additionally or alternatively include a request for manual inspection of the first box (e.g., at an inspection station), which may be performed at that time or when the first box is moved to the inspection station. User input may additionally or alternatively include the degree of the operator's determination regarding the quality of the contents.

[0088] In cases where the user input indicates low certainty regarding the quality of the contents of the first box, or where the user input includes a request to inspect the first box, robots 202 and 204 can be controlled to transport the first box to the inspection station in step S235. The format and timing of the request will depend on the configuration of the storage and retrieval system. In some instances, the inspection station may be a dedicated port 300, to which the box is presented and inspected by the operator. In other instances, a pick-up port may be used, and the box is marked and provided to the operator to indicate that the item does not need to be picked up from the box but is simply to be inspected.

[0089] If the user input in step S230 or the confirmation in step S220 indicates that the contents of the first box have acceptable quality, the control robots 202 and 204 can be prevented from transporting the first box to the inspection station. Therefore, after confirming that the quality of the items inside the box is acceptable to remain within grid 100, the system continues normal operation. This is in... Figure 7 Step S240 is shown. Therefore, data can be collected as the box moves from one location to another within the grid 100, and it is not always necessary to present the box 112 to the operator or inspection station to determine the quality of the item.

[0090] When a user inputs an indication that the contents of the first bin are of unacceptable quality, or when the user inputs a request to discard the contents of the first bin, in step S245, robots 202 and 204 can be controlled to transport the first bin to the processing station. Thus, rotten items can be automatically discarded without otherwise disrupting the operation of bins around grid 100. The processing station can be a dedicated waste port where the contents of the bins are discarded. Alternatively, it can be a pick-up port where an operator receives instructions to remove low-quality items from the bin. All contents of the bin can be discarded, or alternatively, only items determined to be of low quality can be discarded, while other items remain in the bin.

[0091] Warehouse management information can be updated to reflect quality determinations through the results of quality indicator determinations and / or confirmations from operators. Warehouse management information or inventory can also be updated if a box is moved to a processing station and its contents are removed.

[0092] Once completed, it can be repeated. Figure 6 and Figure 7 Any of the methods in. Figure 6 and Figure 7 The method can be repeated using the same first box (at a later time), or with another box from box 112. Alternatively, Figure 6 and Figure 7 The method can be repeated if the box is handled (lifted, transported, etc.) by another robot in robots 202 and 204. These methods can be repeated at a frequency that depends on the type of item determined in step S210; therefore, more perishable or easily decayed items can be inspected more frequently.

[0093] Hardware and software implementation

[0094] Figure 6 and Figure 7 The method shown can be executed by a data processing system configured to perform the method. The data processing system may include... Figure 4Any component of the processing system 400 shown, such as one or more processors 402 configured to execute the method. The data processing system may be on robots 202, 204. The method may be separated such that different steps are executed by different data processing systems. Any step in steps S200-S225 (which may be relatively computationally intensive) may be executed, for example, in a cloud computing environment.

[0095] The method can be executed by a data processing system configured to execute instructions stored on a computer-readable medium, which may be temporary or non-temporary.

[0096] The data processing system may form part of an automated storage and retrieval system, which in turn may include any of the aforementioned robots; a first box; multiple stacked boxes; and / or a storage grid.

[0097] Final Comments

[0098] The example described herein refers to a vertical stacking system in which boxes 112 are arranged in a vertical stack and boxes are lifted from the top of each stack. However, it will be understood that this method can be applied to a system in which boxes 112 are arranged in a horizontal stack and that digging is performed by moving boxes 112 horizontally to approach boxes 112 stored behind them.

[0099] The examples described herein refer to data acquisition sensors on robots 202, 204. However, it will be understood that the sensors may alternatively be arranged above grid 100 and thus in positions to acquire more data from bins 112 and robots 202, 204 and / or within each bin 112, thereby allowing data acquisition at any time without being triggered as in step S115.

[0100] Unless otherwise specified, the steps of the method described herein need not be performed in the order stated above, and can be performed in any order. For example, step S130 may be performed before step S120 (i.e., data may be collected while robots 202 and 204 are moving), or before step S125 (i.e., data may be stored while robots 202 and 204 are moving). Figure 7 The steps of the method do not need to be in Figure 6 The steps are performed after the first box. For example, steps S200 to S220 can be performed before the first box is controlled to move to its destination in step S130.

[0101] Similarly, unless otherwise specified, the steps of the method described herein may be omitted, combined, or performed in parallel. For example, data can be collected even when the robot is not moving on grid 100, and therefore steps S100, S105, and S130 can therefore be omitted. As another example, data can be collected and processed on the same data processing system, and therefore step S135 can therefore be omitted. As yet another example, warehouse management information is not required to process the collected data, so step S205 can be omitted. As yet another example, the steps can be omitted by simply executing... Figure 7 Some determinations are made to obtain useful information about the contents of the box, so any one of steps S210, S215, and S220 can be omitted.

[0102] Examples of this disclosure are described below.

[0103] A computer-implemented method is provided for collecting data about the contents of containers in an automated storage and retrieval system comprising a storage grid of multiple stacked containers.

[0104] The method may optionally include controlling a robotic vehicle on a storage grid to lift a first container from a plurality of stacked containers from the storage grid.

[0105] The method includes acquiring data indicating the contents of the first container.

[0106] Alternatively, sensors from the robotic vehicle can be used to collect data.

[0107] Optionally, the data is collected in response to control.

[0108] The method also includes storing the collected data in association with a unique identifier of the first container.

[0109] Optionally, data is collected in response to determining that the position of the first container relative to the robotic vehicle meets an proximity criterion.

[0110] Optionally, data is collected in response to determining that the lifting device of the robot vehicle is in a retracted configuration.

[0111] Optionally, it also includes: Before taking control: Receive instructions to transport the first container to its destination, and Control the automated vehicle to move to the first container; and after data collection: Control the robotic vehicle to move the first container to its destination.

[0112] Optionally, it also includes transmitting the collected data to a data processing device in association with a unique identifier.

[0113] Optionally, at least one of the following: The robotic vehicle is controlled by instructions transmitted via a first network, and the collected data is transmitted via a second network; or The robot vehicle is controlled by transmitting instructions with a first network traffic priority and the collected data is transmitted with a second network traffic priority, wherein the second network traffic priority is lower than the first network traffic priority.

[0114] Optionally, the collected data includes at least one of the following: One or more optical images, One or more infrared images, Air composition data, Humidity data, Weight data, or Temperature data.

[0115] Optionally, it also includes determining at least one type of item in the first container.

[0116] Optionally, the determination of at least one type of item in the first container is based on at least one of the following: The system automatically stores and retrieves inventory databases; and The data collected.

[0117] Optionally, image classification can be used to determine at least one type of item in the first container.

[0118] Optionally, it also includes determining the quantity of each item in at least one type of item in the first container based on the collected data.

[0119] Optionally, it also includes determining the number of items in the first container based on the collected data.

[0120] Optionally, it also includes identifying the difference between the expected quantity of items in the first container and the determined quantity of items.

[0121] Optionally, it also includes determining quality indicators of the contents of the first container based on the collected data.

[0122] Optionally, image classification can be used to determine quality metrics.

[0123] Optionally, the quality indicators include: Based on at least one type of item determined in the first container, a quality determination algorithm is selected from multiple algorithms; and The selected quality determination algorithm is applied to determine the quality indicators.

[0124] Optionally, it also includes sending at least a portion of the collected data to the operator for review in response to determining that the quality indicator indicates that the contents of the first container are of low quality.

[0125] Optionally, it also includes user input from the operator regarding the quality of the contents of the first container.

[0126] Optionally, the user input indicates a low degree of confidence regarding the quality of the contents of the first container, or the user input includes a request to inspect the first container, and the method further includes, in response to the user input, controlling a robotic vehicle to transport the first container to an inspection station.

[0127] Optionally, the user input indicates that the contents of the first container have acceptable quality, and the method further includes, in response to the user input, stopping the control of the robotic vehicle to transport the first container to the inspection station.

[0128] Optionally, the user input indicates that the contents of the first container are not of acceptable quality, or the user input includes a request to discard the contents of the first container, and the method further includes controlling a robotic vehicle to transport the first container to a processing station.

[0129] Optionally, it also includes: Compare the collected data with the expected contents data of the first container; and In response to comparison, it identifies the differences between the expected content data and the collected data.

[0130] Optionally, determining the quality indicators includes comparing the collected data with at least one of the following: Data on the expected contents of acceptable quality goods; Data on the expected contents of low-quality goods; and Data on the expected contents of high-quality items.

[0131] A data processing system is provided that can be configured to perform any of the methods described herein.

[0132] A computer-readable medium including instructions is provided, which, when executed by a data processing system, cause the data processing system to perform any of the methods described herein.

[0133] An automated storage and retrieval system is provided, comprising at least one data processing system and a robotic vehicle, the automated storage and retrieval system being configured to perform any of the methods described herein.

[0134] Optionally, the automated storage and retrieval system includes at least one of the following: First container; Multiple stacked containers; and Storage grid.

[0135] It should be understood that the above description is intended to be illustrative and not restrictive. Many other implementations will be apparent to those skilled in the art after reading and understanding the above description. Although this disclosure has been described with reference to specific exemplary embodiments, it should be recognized that this disclosure is not limited to the described embodiments, but can be implemented using variations and modifications within the spirit and scope of the appended claims. Therefore, the specification and drawings are to be regarded as illustrative and not restrictive. Consequently, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A computer-implemented method for collecting data about the contents of containers in an automated storage and retrieval system comprising a storage grid having multiple stacked containers, the method comprising: Control the robotic vehicle on the storage grid to lift the first container of the plurality of stacked containers out of the storage grid; In response to the control, the sensors of the robotic vehicle are used to collect data indicating the contents of the first container; as well as The collected data is stored in association with the unique identifier of the first container.

2. The method according to claim 1, wherein, The data is acquired in response to determining that the position of the first container relative to the robotic vehicle meets an proximity criterion.

3. The method according to any one of the preceding claims further comprises: Before the control: Receive instructions to transport the first container to its destination, and Control the automated vehicle to move to the first container; and After the data collection: The robot vehicle is controlled to move the first container to the destination.

4. The method according to any one of the preceding claims further includes transmitting the collected data in association with the unique identifier to a data processing device.

5. The method according to any one of the preceding claims, wherein the collected data includes at least one of the following: One or more optical images, One or more infrared images, Air composition data, Humidity data, Weight data, and Temperature data.

6. The method according to any one of the preceding claims further includes determining at least one type of article in the first container.

7. The method according to claim 6, wherein, The determination of at least one type of item in the first container is based on at least one of the following: The inventory database of the automated storage and retrieval system; and The collected data.

8. The method according to any one of the preceding claims further includes determining the quantity of items in the first container based on the collected data.

9. The method according to claim 8, further comprising: Identify the difference between the expected quantity of items in the first container and the determined quantity of items.

10. The method according to any one of the preceding claims further includes determining a quality index of the contents of the first container based on the collected data.

11. The method of claim 10 when dependent on claim 6, wherein determining the quality index comprises: Based on at least one type of item in the first container, a quality determination algorithm is selected from multiple algorithms; as well as The selected quality determination algorithm is applied to determine the quality index.

12. The method according to any one of claims 10 to 11, further comprising, in response to determining that the quality indicator indicates that the contents of the first container are of low quality, sending at least a portion of the collected data to an operator for viewing.

13. The method according to any one of the preceding claims, further comprising: The collected data is compared with the expected contents data of the first container; as well as In response to the comparison, differences between the expected content data and the collected data are identified.

14. A data processing system configured to perform the method according to any one of the preceding claims, or A computer-readable medium comprising, when executed by a data processing system, instructions that cause the data processing system to perform the method according to any one of the preceding claims.

15. An automated storage and retrieval system comprising at least one data processing system and a robotic vehicle, said automated storage and retrieval system being configured to perform the method according to any one of claims 1 to 13, and said system optionally further comprising at least one of the following: The first container; The plurality of stacked containers; and The storage grid.

Images