Apparatus and method for automated palette builder calibration - Patents.com

Abstract

A pallet building apparatus for automatically building pallet loads of pallet luggage units on pallet supports, the pallet building apparatus including a frame defining a pallet station at a pallet building base for the pallet support, an articulated robot connected to the frame, at least one three-dimensional camera connected to the frame separate and distinct from the articulated robot and positioned to generate a three-dimensional image of the pallet support on the pallet building base and the pallet load build on the pallet support, a controller operably connected to the at least one articulated robot and the at least one three-dimensional camera, and an automatic calibration system for calibrating the articulated robot, the controller communicatively connected to the automatic calibration system, the automatic calibration system configured to automatically calibrate the articulated robot from a substantially uncalibrated state and automatically teach the articulated robot locations of the pallet stations.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application is a nonprovisional application and claims the benefit of U.S. Provisional Application No. 63 / 310,450, filed February 15, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] [Technical field] The disclosed embodiments and aspects thereof relate generally to storage and retrieval systems, and more specifically to palletizing / depalletizing cells in storage and retrieval systems. [Background technology]

[0003] A brief description of related developments Retail distribution of products (whether traditional "brick and mortar" stores, online stores, or mixed retail channels) demands improved storage, sorting, and transportation efficiency, especially for the distribution of what are known as mixed or heterogeneous cases (within a given shipment), whether for store replenishment or individual orders. The application of intelligent / adaptive automation to these is increasingly facilitating efficiency improvements at many levels of distribution, including storage, sorting, and transportation. Still, difficult problems remain, such as the mixed-case pallet load (or truck load) efficiency problem, which creates bottlenecks that impede or negatively impact the efficiency of the overall system. As can be easily understood, the difficulty of the pallet load (or truck load) efficiency problem is not simply due to the desirability of high packing density, but rather the efficiency of the pallet load depends on both packing density and building the pallet load in an optimal time (i.e., the build puzzle of packing a pallet load at greater than 90% density can be easily solved given the required time and the required selection of mixed cases, but if the pallet load build time is not optimal, such a pallet load will not be efficient).

[0004] Calibration of conventional pallet loaders (e.g., robotic palletizers) can affect pallet building efficiency and create a bottleneck in system throughput efficiency. Pallet loaders are typically calibrated with respect to the locations of features of the pallet loader workspace with which the pallet loader interfaces (e.g., pallet location, conveyor location, etc.). Traditionally, calibration of pallet loaders is performed manually by pallet loader experts in a time-consuming and complicated process. Because each pallet loader is calibrated by a human, the calibration can be subjective and inconsistent, depending heavily on the thoroughness and skill of the human performing the calibration. Additionally, the subjective nature of the manual calibration process makes it difficult to identify the root cause of palletization inaccuracies. Summary of the Invention

[0005] The foregoing aspects and other features of the disclosed embodiments are explained in the following description, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram of a flow installation in accordance with aspects of the disclosed embodiment; [Diagram 2] FIG. 1 is a schematic diagram of a palletized load in accordance with aspects of the disclosed embodiment; [Figure 3A] FIG. 2 is a top perspective view of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment, where the system is shown with an empty pallet elevator in its upper position and a palletized elevator in its lower position. [Figure 3B] FIG. 3B is another top perspective view of the palletizer cell of FIG. 3A in accordance with aspects of the disclosed embodiment, where the system is shown with an empty pallet elevator in its lower position and a palletized elevator in its upper position. [Figure 3C] FIG. 3B is a side perspective view of the palletizer cell of FIG. 3A in accordance with aspects of the disclosed embodiment, where the system is shown with its telescoping forks extended towards the palletization elevator; [Figure 3D] FIG. 3B is a perspective view of a palletizing robot of the palletizing system of FIG. 3A in accordance with aspects of the disclosed embodiment; [Figure 4] FIG. 3B is a schematic isometric view of the palletizer cell of FIG. 3A in accordance with aspects of the disclosed embodiment; [Figure 4A] 5 is a schematic exploded isometric view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment; [Figure 4B] FIG. 5 is a schematic plan or top view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4C] FIG. 5 is a schematic right side view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4D] FIG. 5 is a schematic front view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4E] FIG. 5 is a schematic left side view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4F] 5 is a schematic rear or back view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4G] FIG. 5 is a schematic isometric view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4H] FIG. 5 is a schematic left side view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4I] FIG. 5 is a schematic front view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4J] FIG. 5 is a schematic plan or top view of the palletizer cell of FIG. 4 in accordance with aspects of the disclosed embodiment. [Figure 4K] 5 is a schematic isometric view of the palletizer cell of FIG. 4 highlighting the field of view of the cameras of the palletizer cell's vision system, in accordance with aspects of the disclosed embodiment. [Figure 4L] 5 is a schematic isometric view of the palletizer cell of FIG. 4 highlighting the field of view of the cameras of the palletizer cell's vision system, in accordance with aspects of the disclosed embodiment. [Figure 4M]5 is a schematic isometric view of the palletizer cell of FIG. 4 highlighting the field of view of the cameras of the palletizer cell's vision system, in accordance with aspects of the disclosed embodiment. [Figure 4N] 5 is a schematic isometric view of the palletizer cell of FIG. 4 highlighting the field of view of the cameras of the palletizer cell's vision system, in accordance with aspects of the disclosed embodiment. [Figure 5A] 2 is a schematic diagram of a portion of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment. [Figure 5B] 2 is a schematic perspective view of a point cloud of a portion of the palletizer cell and vision system of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 6] 2 is a schematic perspective view of a portion of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 7] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 8] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 9] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 10] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 11] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 12] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 13] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 14] FIG. 13 is an exemplary flow diagram of a method according to an aspect of the disclosed embodiment; [Figure 15] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 16] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 17] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 18A] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 18B] 2 is an exemplary point cloud obtained with a vision system of the palletizer of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 19A] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 19B] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 20] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 21A] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 21B] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 22] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 23A] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 23B] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 24A] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 24B]2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 24C] 2 is an exemplary image obtained with a vision system of the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Diagram 25] 2 is exemplary sensor data obtained for the palletizer cell of FIG. 1 in accordance with aspects of the disclosed embodiment; [Figure 26] FIG. 13 is an exemplary flow diagram of a method according to an aspect of the disclosed embodiment; [Figure 27] FIG. 13 is an exemplary flow diagram of a method according to an aspect of the disclosed embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] FIG. 1 is a schematic diagram of a warehouse system or distribution facility 100WS (herein referred to as warehouse system 100WS) according to aspects of the disclosed embodiment. Although aspects of the disclosed embodiment are described with reference to the drawings, it should be understood that aspects of the disclosed embodiment may be embodied in many forms. Furthermore, any suitable size, shape, or type of elements or materials may be used. Although the distribution facility 100WS is described herein as an automated distribution facility, it should be understood that aspects of the disclosed embodiment are applicable to distribution facilities having any suitable conveying system, such as both automated and manual conveying systems, or even to fully manual conveying systems.

[0008] 1 and 2, according to aspects of the disclosed embodiment, a warehouse system 100WS includes at least one real-time adaptive palletizer / depalletizer cell 10A, 10B (collectively referred to herein as palletizer cell 10). The palletizer cell 10 has one or more robotic case manipulators 14 (also referred to herein as articulated robots or robots) that place (individually or with fabricated pick faces) article units CU (also referred to herein as case units or cases) of the mixed pallet load into stacks SL1-Sn and / or layers PL1-PL4 to build a mixed case pallet load PAL with vision system assistance, such as in the manner described in U.S. Pat. No. 10,894,676, issued Jan. 19, 2021, the entire disclosure of which is incorporated herein by reference.

[0009] The palletizer cell 10 (or pallet building apparatus) is provided with a vision system 310 (collectively referred to herein as vision system 310 - see Figures 3A-4N and 5A-5B) including three-dimensional (3D) time-of-flight (TOF) cameras 310C1-310C4 and an electromagnetic sensor 500 separate and distinct from the 3D cameras 310C1-310C4, which are utilized for fully automatic calibration (and maintaining that calibration) and verification of the palletizer cell 10 (e.g., from a substantially uncalibrated state) in the manner described herein.

[0010] Three-dimensional (3D) time-of-flight (TOF) camera(s) and electromagnetic sensor 500 (e.g., two-dimensional (2D) color camera 500C) are mounted on the frame 24 of the palletizer cell 10 separate and distinct from the robot(s) 14. The three-dimensional camera(s) may be included in an automatic calibration system ACS, connected to the frame and configured to three-dimensionally image at least a portion of the robot(s) 14 and the pallet support SPAL or pallet load build BPAL on the pallet build base 301. For example, the vision system 310 forms at least a part of the automatic calibration system ACS for calibrating the robot(s) 14 of the palletizer cell 10. As described herein, the controller 10C is communicatively connected to the auto-calibration system ACS to automatically calibrate the robot(s) 14 from a substantially uncalibrated state, automatically teach the robot(s) a location or datum of the palletizer cell 10 pallet station (i.e., a surface or feature of the palletizer cell pallet station to which other features of the palletizer cell pallet refer) (e.g., a location of the pallet load support SPAL on the elevator 12 and / or empty pallet transport mechanism 20) that resolves the location of the end effector(s) of the robot(s) 14 or end-of-arm tool(s) 410 relative to the pallet station, and build a pallet load (e.g., FIG. 2) by placing articles or case units CU (also referred to herein as pallet cargo item units) on the pallet load support SPAL. The controller 10C characterizes the placement location of each case unit CU placed with the end effector 410 that builds the pallet load PAL on the pallet support SPAL. The controller 10C is configured to calibrate at least the three-dimensional cameras 310C1-310C4 of the vision system 310 relative to the end-of-arm tool 410, for example based on at least data obtained from the automatic calibration system ACS.As described herein, the controller 10C and auto-calibration system ACS are configured to provide auto-calibration and auto-teaching of the robot(s) 14 in substantial agreement with the auto-calibration of each of the three-dimensional cameras 310C1-310C4 that image each case unit CU placed by the robot(s) 14 building the pallet load PAL.

[0011] As used herein, calibration refers to automatic calibration for the movement (i.e., lift direction) of the pallet lift or elevator 12 (e.g., see FIGS. 3A-3C, 4, and 4B—e.g., automatically calculating the zero position of the lift 12), automatic calibration of the push axis PAX of the end-of-arm tool 410 (see FIG. 4) of the robot 14 (e.g., automatically verifying / calculating the zero position of the end-of-arm tool 410), automatic teaching of the reference frame (e.g., coordinate system and work envelope) of the robot 14, automatic teaching of the work reference frame (also called pallet offset), and automatic calibration of calibration errors. / Automatic detection of mechanical problems (e.g., abnormal robot motion, abnormal encoder operation, bent transfer forks 20, bent fingers on end-of-arm tool 410, tilted lift 12, off-center pallet, etc.), automatically defining a picking position on the in-feed conveyor 16, and automatically defining (one or more) common references (e.g., to each of the robots 14) of the palletizer cell 10 (e.g., the reference frame of the pallet transfer forks 20, the reference frame of the pallet on the forks 20, and the reference frame of the pallet on the lift 12).

[0012] The vision system 310 may also be utilized to verify the calibration of the robot 14 at any suitable time, such as after a robot movement error and / or periodically. As can be appreciated, automatic calibration of the robot 14 may simultaneously address the quality / control of pallets and their construction by the robot 14. The vision system 310 incorporated in the automated palletizer cell 10 informs and enables the cell controller 10C to provide real-time command inputs (to automated devices such as the robot(s) 14) responsive to the position / movement of the calibration fixtures to effect automatic calibration of the robot 14 in the manner described herein.

[0013] Aspects of the disclosed embodiments result in simplified and accelerated calibration and teaching of the palletizer cell robot 14. For illustrative purposes, the robot 14 described herein is a four-degree-of-freedom (four-axis) robot, but aspects of the disclosed embodiments may be equally applied to a six-degree-of-freedom (six-axis) robot or a robot having any suitable degree of freedom. Because the calibration and teaching of the robot 14 is fully automated (e.g., does not rely on human intervention), the calibration / teaching of the robot 14 is consistent across robots 14 of the same palletizer cell 10 and between robots 14 of different palletizer cells 10 in the same facility (and between robots / palletizer cells in different facilities). With the calibration / recalibration of the cell 10 being fully automated, the calibration may be periodically verified to substantially prevent and detect misalignments between components of the palletizer cell 10 (e.g., robot 14 axis misalignment, lift 12 axis misalignment, etc.). Data from the calibration / teaching can also be stored in the controller 10C to provide a quantitative assessment of the accuracy of the calibration / teaching.

[0014] Referring again to FIG. 1 , according to aspects of the disclosed embodiment, the distribution facility 100WS includes a storage and retrieval system 100 that may operate, for example, at a retail distribution center or warehouse to fulfill orders received from retailers for case units. In one example, a case unit may be a case or unit of goods that is not stored (e.g., not contained) on a tray, on a tote, or on a pallet. In other examples, a case unit may be a case or unit of goods that is contained in any suitable manner, such as on a tray, on a tote, or on a pallet. It is noted that a case unit may include cased units of goods (e.g., cases of soup cans, boxes of cereal, etc.) or individual goods that are adapted to be removed from or placed on a pallet. According to aspects of the disclosed embodiment, a transport case for the case unit (e.g., a carton, barrel, box, crate, jug, or any other suitable device for holding the case unit) may have a variable size, may be used to hold the case unit during transport, and may be configured to be palletized for transport. For example, it is noted that when a stack or pallet of case units arrives at the storage and retrieval system, the contents of each pallet may be uniform (e.g., each pallet holds a predetermined number of the same items, i.e., one pallet holds soup and another pallet holds cereal), and when the pallets leave the storage and retrieval system, the pallets may contain any suitable number and combination of different case units (e.g., each pallet may hold different types of case units, i.e., a pallet holds a combination of soup and cereal). In aspects of the disclosed embodiments, the storage and retrieval systems described herein may be applied to any environment in which units are stored and retrieved.

[0015] The storage and retrieval system 100 may be configured to be installed in an existing warehouse structure, for example, or may be adapted to a new warehouse structure. In aspects of the disclosed embodiment, the storage and retrieval system may include one or more infeed transfer stations 170 and one or more outfeed transfer stations 160, in / out case conveyors 150A, 150B, 150C (collectively referred to as in / out case conveyors 150), a storage structure array 130, and a number of autonomous vehicle-transporting robots 110 (referred to herein as "bots"). In aspects of the disclosed embodiment, the storage and retrieval system may also include a robot or bot transfer station, as described in U.S. Patent No. 9,096,375, issued August 4, 2015, the entire disclosure of which is incorporated herein by reference. In aspects of the disclosed embodiment, the bot transfer station may provide an interface between the bot 110 and the in / out case conveyor 150 such that case units may be indirectly transferred between the bot 110 and the in / out case conveyor 150 via the bot transfer station. In accordance with aspects of the disclosed embodiment, case units may be transferred directly between the bot 110 and the in / out case conveyor 150.

[0016] The storage structure array 130 may include multiple levels of storage rack modules forming a storage array of storage locations 130SL for case units, each of which is arranged to store at least one case unit at each storage location 130SL. In one embodiment, each level of the storage structure array 130 includes a respective storage / picking aisle 130A and a transfer deck 130B for transferring case units between any of the storage areas of the storage structure array 130 and any shelf of any of the in / out case conveyors 150. The storage aisles 130A and the transfer deck 130B are also configured to allow the bot 110 to traverse the storage aisles 130A and the transfer deck 130B and retrieve an ordered case unit to place the case unit in a picking stock, where the case unit is stored or otherwise held within the storage aisle 130A and / or on the transfer deck 130B at the storage location 130SL. The bot 110 may be any suitable bot capable of carrying and transporting case units throughout the storage and retrieval system 100. Suitable examples of bots can be found in U.S. Pat. No. 8,425,173, issued April 23, 2013; U.S. Pat. No. 9,561,905, issued February 7, 2017; U.S. Pat. No. 8,965,619, issued February 24, 2015; U.S. Pat. No. 8,696,010, issued April 15, 2014; U.S. Pat. No. 9,187,244, issued November; U.S. Pat. No. 113 / 326,952, entitled "Automated Bot with Transfer Arm," filed December 15, 2011 (non-provisional patent application having U.S. Serial No. 61 / 423,365, filed December 15, 2010); and U.S. Pat. No. 9,499,338, issued November 22, 2016, the entire disclosures of which are incorporated herein by reference for illustrative purposes only.The bot 110 may be configured to place case units, such as the retail items described above, in a picking stock at one or more levels of the storage structure array 130, and then selectively retrieve the ordered case units and ship the ordered case units, for example, to a store or other suitable location.

[0017] The infeed transfer station 170 and the outfeed transfer station 160 operate in conjunction with respective in / out case conveyors 150A, 150B to transfer case units bidirectionally between one or more levels of the array of storage structures 130, resulting in the infeed of case units to and the output of case units from the array of storage structures 130. Although the infeed transfer station 170 and the outfeed transfer station 160 (and their respective in / out case conveyors 150A, 150B and palletizer / depalletizer cells 10A, 10B) are described as being dedicated inbound (e.g., in-feed) transfer station 170 and dedicated outbound (e.g., out-feed) transfer station 160, it is noted that in aspects of the disclosed embodiment, each of the transfer stations 170, 160 may be used for both inbound and outbound transfer of case units from the storage and retrieval system. It is noted that although an in / out conveyor is described herein, the conveyor may be any suitable conveyor (including any suitable conveying path orientation, such as vertical and / or horizontal conveyor paths) or a transfer / picking device having any suitable conveying path orientation.

[0018] In one embodiment, as noted above, the infeed transfer station 170 and the outfeed transfer station 160 each include a respective in / out case conveyor 150A, 150B and a respective palletizer / depalletizer cell 10A, 10B (collectively referred to herein as palletizer cells 10). In one embodiment, the palletizer / depalletizer cells 10 are automated cells, each of which is configured to receive loaded pallets (such as uniform or mixed case units or products) from an area within the pallet load 175, which may include, for example, an in-out loaded pallet conveyor 175C (illustrated in FIG. 1 as an input conveyor), and / or to build loaded pallets (such as uniform or mixed case units or products) for transport to an area outside the pallet load 180, which may include, for example, an in-out loaded pallet conveyor 180C (illustrated in FIG. 1 as an output conveyor). In one embodiment, the conveyors 175C, 180C are each connected to the storage structure array 130 and configured to transport loaded pallets bidirectionally in an input direction toward the storage structure array 130 and in different output directions away from the storage structure array 130. In one embodiment, the conveyors 175C, 180C may each include a conveyor arrangement with distributed conveyor beds arranged to form a transport path, or in other embodiments, the conveyors 175C, 180C may be individual transport units, such as, for example, forklifts / pallet trucks. Suitable examples of automated palletizer / depalletizer cells 10A, 10B may be found in U.S. Pat. No. 10,370,199, issued Aug. 6, 2019, and U.S. Pat. No. 8,965,559, issued Feb. 24, 2015, the entire disclosures of which are incorporated herein by reference. Each palletizer cell includes one or more robotic case manipulators 14, which may also be referred to as articulated robots or robots.The one or more robotic case manipulators 14 are configured to sequentially transport and place the item units CU (also referred to herein as cases or case units) of the pallet load onto the pallet supports to build the pallet load 250 on the pallet build base 301 (see FIG. 3 ), as described herein.

[0019] When the palletizer cell 10 functions in an output role as a palletizer, the palletized article units CU, which may be of various sizes, arrive at the palletizer cell 10 via the in / out case conveyor 150B and are picked by one of the robotic case manipulators 14 and placed on the pallet PAL, as described herein. When the palletizer cell 10 functions in an output role as a palletizer, a full pallet PAL (see FIG. 2) created from the various case units is ready to be picked up by a forklift from the palletizer cell 10 for transport to an outside pallet load 180 area. When the palletizer / depalletizer cell 10 functions in an input role as a depalletizer, a full pallet (similar to a pallet PAL, which may be formed of homogenous or mixed cases) created from the various palletized article units CU is transferred from an inside pallet load 175 area to the palletizer cell 10 in any suitable manner, such as by a forklift. One or more robotic case manipulators 14 pick the palletized item units CU from the pallets PAL and transfer them to the storage structure array 130.

[0020] In one aspect, each in-feed transfer station 170 forms a case input path Ip along which the palletizer / depalletizer cell 10A depalletizes case units layer by layer or otherwise depalletizes case units from standard pallets (e.g., homogenous pallets having suitable stability for automatic engagement of pallet layers by an automated layer interface unit, such as product picking device 14) into single case units. The palletizer / depalletizer cell 10A is in communication with a transport system of the automated storage and retrieval system 100, such as an in / out case conveyor 150A, to form an integrated input system (e.g., in-feed transfer station 170) that supplies case units to the automated storage and retrieval system 100. Each infeed transfer station 170 defines a case input path Ip that is integrated with the automated storage and retrieval system 100 and the warehouse management system 199, where the warehouse management system 199 includes any suitable controller 199C configured using any suitable non-transitory program code and memory to manage at least the input of case units into the storage structure array 130B, the storage distribution of case units within and the retrieval of case units from the storage structure array 130B, the inventory / replenishment of case units, and the output of case units.

[0021] In one aspect, each case unit input path Ip includes at least one corresponding case unit inspection cell 142 in communication with the warehouse management system 199. In one aspect, the at least one corresponding case unit inspection cell 142 may be any suitable inspection cell including any suitable volumetric inspection with a multi-dimensional light curtain, an imaging system, and / or any other suitable sensing / sensor arrangement configured to detect defects in the case units and identify the case units for, for example, inventory, transport sequencing, storage distribution, and sequencing of the case units for output from the storage structure array 130B.

[0022] In one embodiment, as mentioned above, the palletizer / depalletizer cell 10A may be fully automatic to disassemble or remove layer(s) from pallets unloaded by the palletizer / depalletizer cell 10A. With reference to Figure 2, it is noted that the term decommissioning refers to the removal of pallet layers PL1, PL2, PL3, PL4 (full or partial) from the pallet PAL such that the article units CU of each pallet load are removed from layers PL1, PL2, PL3, PL4 at a given level 200 (which may correspond to a commissioning level or transfer surface) of the pallet PAL, whereby the pallet PAL is indexed (e.g., with a pallet lift PLFT (see Figure 4)) to the next level of the pallet PAL for removal of the next layer PL2, PL3 (full or partial) corresponding to the next level of the pallet PAL.

[0023] In one embodiment, the palletizer / depalletizer cell 10A is configured to release layers PL1, PL2, PL3, PL4 such that the release is synchronized or otherwise coordinated (e.g., matches) by the warehouse control system 199 with a predetermined percentage of the supply amount or feed rate of case units established by the warehouse control system 199 in the automated storage and retrieval system 100. For example, in one embodiment, the warehouse control system 199 is configured to set and / or monitor the predetermined percentage of the supply amount of case units in the automated storage and retrieval system 100. For example, the warehouse management system 199 monitors and manages the automated systems of the automated storage and retrieval system 100 (e.g., in / out case conveyors 150A, 150B, bot 110, and palletizer / depalletizer cells 10A, 10B, etc.), where each of the automated systems, or one or more of the automated systems, actually, alone or in combination, under the control of the warehouse management system 199 or any other suitable controller (e.g., bot controller, conveyor controller, palletizer / depalletizer controller, etc.) of the automated storage and retrieval system 100, has a given transaction time (e.g., the time / period that results in a basic unit of case transport or transfer to transfer a case unit on / off the in / out case conveyor to a picking / placement station or to lift the case unit a predetermined distance, or picking / placement of a bot transport at a storage location, the time to transfer a pallet layer between pallets, etc.) that defines a predetermined percentage of the supply of case units in the automated storage and retrieval system 100 established by the warehouse management system 199. For example, the controller 199C of the warehouse management system 199 is communicatively connected to the in / out case conveyors 150A, 150B so that the in / out case conveyors 150A, 150B transport case units in both directions between the storage structure arrays 130 at a predetermined case feed rate.The controller 199C may also be communicatively connected to the palletizer / depalletizer cells 10A, 10B corresponding to the in / out case conveyors 150A, 150B such that the loading and unloading of layers of the palletizer / depalletizer cells 10A, 10B, respectively, which are substantially consecutive, is matched to a predetermined case feed rate. Although aspects of the disclosed embodiments are described herein with respect to a distribution facility 100WS having an automated storage and retrieval system 100 with an automated conveying system, aspects of the disclosed embodiments are also applicable to a distribution facility having any suitable conveying system, such as both an automated conveying system and a manual conveying system, or a fully manual conveying system, where both the automated and manual conveying transactions each have a respective transaction time, and where the loading and unloading of case units between pallets may be matched to the transaction times in a manner generally similar to that described herein.

[0024] In one embodiment, each outfeed transfer station 160 forms a case output path Op where the palletizer / depalletizer cell 10B pallets the case units layer by layer onto the pallet PAL using an automated layer interface unit such as one or more robotic case manipulators 14. In one embodiment, the pallet PAL may be formed as a standard pallet (e.g., homogenous case units) or as a mixed pallet, such as those described in U.S. Pat. No. 9,856,083, issued Jan. 2, 2018, the disclosure of which is incorporated herein by reference in its entirety. In one embodiment, the warehouse management system 199 is configured to establish a pallet solution using mixed case units that provides a stable pallet load stack suitable for transfer as a layer by the end effectors of one or more robotic case manipulators 14. As noted above, a suitable example of a palletizer / depalletizer cell 10B may be found in U.S. Pat. No. 10,370,199, issued Aug. 6, 2018, the disclosure of which was previously incorporated herein by reference in its entirety.

[0025] In one aspect, the palletizer / depalletizer cell 10B is in communication with a transport system of the automated storage and retrieval system 100, such as an in / out case conveyor 150B, to form an integrated output system (e.g., outfeed transfer station 160) that receives case units from the automated storage and retrieval system 100 for placement on a pallet according to any suitable case out order sequence. For example, as described above, the item units CU of a pallet load routed to one or more robotic case manipulators 14 are transferred by end effectors of the one or more robotic case manipulators 14 to a pallet PAL, and the item units CU of the pallet load (output case units) are arranged in a predetermined sequence established by the warehouse management system 199, layer by layer (it being noted that a layer may cover all or a portion of a pallet) to form a standard output pallet load.

[0026] Each outfeed transfer station 160 defines a case output path Op that is integrated with the automated storage and retrieval system 100 and the warehouse control system 199, where the warehouse control system 199 includes any suitable controller 199C configured to manage operation of the distribution facility 100WS including the case units output from the storage structure array 130B using any suitable non-transitory program code and memory as described herein. In one embodiment, each case unit output path Op includes at least one corresponding case unit inspection cell 142 (as described above) in communication with the warehouse control system 199. In one embodiment, as described above, the palletizer / depalletizer cell 10B may be fully automated to build or load the layer(s) onto pallets for loading into the palletizer / depalletizer cell 10B. With reference to FIG. 2, it is noted that the term commission refers to the construction of (full or partial) pallet layers PL1, PL2, PL3, PL4 onto the pallet PAL, such that item units CU of each pallet load are inserted into layers PL1, PL2, PL3, PL4 at a given level 200 (which may correspond to a removal / attachment level or a transfer surface) of the pallet PAL until pallet layers PL1, PL2, PL3, PL4 are formed, whereby the pallet PAL is indexed to the next level of the pallet PAL (e.g. using a pallet lift PLFT (see FIG. 4)) for the construction of the next layer PL1, PL2 (full or partial) corresponding to the next level of the pallet PAL.In one aspect, the palletizer / depalletizer cell 10B is configured to load layers PL1, PL2, PL3, PL4 in a manner generally similar to that described above with respect to the removal of layers PL1, PL2, PL3, PL4, such that the loading is synchronized or otherwise coordinated (e.g., matched) by the warehouse management system 199 with a predetermined percentage of the supply amount or rate of case units established by the warehouse management system 199 in the automated storage and retrieval system 100, where the warehouse management system 199 manages other related aspects of the output such as the pick-up sequence of case units, the sequence of mixed case units output to the load-out sequence of palletized loads of mixed case units, and inventory adjustments.

[0027] 1, 3A, and 4, palletizer cell(s) 10 (the term "palletizer" is used for convenience, but as noted above, the features of a palletizer may also be enabled in a depalletizer as otherwise applicable) are coupled to a storage and retrieval system 100 so as to communicate a flow (see case output path(s) Op and case input path(s) Ip) of case units CU (see FIG. 2) with the storage and retrieval system 100. The palletizer 10, according to aspects of the disclosed embodiment, is an adaptive palletizer system 300 that may provide time-optimal pallet load building, thus complementing and leveraging the case order flow throughput of the storage and retrieval system 100, in a manner similar to that described in U.S. Pat. No. 10,894,676, issued January 19, 2021, the entire disclosure of which is incorporated herein by reference (although in other aspects the adaptive palletizer 300 may be coupled to any suitable storage and retrieval system, including conventional, manual, or semi-automated retrieval systems with manually loaded feed stations for the palletizer 10).

[0028] 2, the palletizer cell(s) 10 are configured to build a pallet load PAL, where the pallet load PAL has a pallet load build structure RPAL (the pallet build feature can be applied to truck loads as well) that is a three-dimensional array of mixed case or pallet load item units CU (one or more) that contain fabricated / built item units (pick faces) structured into stacks S1-Sn and layers PL1-PL5, where each of the multiple cases / items is placed on a pallet / pallet support SPAL (case unit / pallet load item unit means case, tote, pack, shrink wrap, etc.). The pallet load build structure RPAL is determined by the control from the ordered case unit(s) CU (e.g., case unit CU output from the storage and retrieval system 100). For example, in one embodiment, the palletizer controller 10C may be coupled to a controller 199C of a warehouse management system 199, while in other embodiments, the palletizer controller 10C forms an integrated warehouse management controller module that manages the transportation of components of the storage and retrieval system 100, including the palletizer / depalletizer cells 10, thereby receiving information defining a pallet load build structure RPAL including corresponding datum reference boundaries, case poses, and difference thresholds from reference to the pallet load build provided by the palletizer 10. The sequence of case poses by which the robot(s) 14 of the palletizer 10 build the pallet load PAL is effected by the storage and retrieval system 100, such that the cases output by the storage and retrieval system 10 which feed the bot pick station 350 of the palletizer 10 arrive (just in time or appropriately buffered) in a predetermined picking sequence for building the pallet load PAL, enabling higher picking / placing rates of the robot(s) 14 (e.g., the case flow output from the storage and retrieval system 100 substantially eliminates or reduces sorting of case units CU by the robot(s) 14).Suitable examples of classifications of case flows output from the storage and retrieval system 100 can be found, for example, in U.S. Patent Application Publication No. 2016 / 0214808, published July 28, 2016, U.S. Patent Application Publication No. 2016 / 0207709, published July 21, 2016, U.S. Patent Application Publication No. 2016 / 0207711, published July 21, 2016, U.S. Patent Application Publication No. 2016 / 0214797, published July 28, 2016, U.S. Patent Application Publication No. 2016 / 0167880, published June 16, 2016, and U.S. Patent Application Publication No. 2016 / 0207710, published July 21, 2016, the entire disclosures of which are incorporated herein by reference. The pick / place speed of the robot 14 has a pick / place cycle from picking at the input station (e.g., bot picking station 350) to placing on the palletized load build structure BPAL and return of, for example, about 5 seconds (with two robots the pick / place cycle is about 2.5 seconds), and the adaptive feedback loop FBL (see FIG. 3) of the vision system 310 is effected in real time within the pick / place cycle of the robot(s) 14 to substantially effect continuous construction of the palletized load build structure RPAL in the manner described in U.S. Pat. No. 10,894,676, issued on January 19, 2021, the entire disclosure of which has previously been incorporated by reference herein.

[0029] 3A-3D and 4, each palletizer cell 10 generally includes a frame 24, at least one robot 14 with an end-of-arm tool 410, a controller 10C, and a vision system 310 including at least one three-dimensional time-of-flight camera 310C. A suitable example of a palletizer cell 10 can be found in U.S. Pat. No. 10,370,199, issued Aug. 6, 2019, the entire disclosure of which is incorporated herein by reference. The frame 24 defines a pallet construction base 301 (see FIG. 3) for the pallet support SPAL (FIG. 2). The at least one robot 14 is connected to the frame 24 and is configured to transport and place pallet load units CU (see also FIG. 2) on the pallet support SPAL (see FIG. 2) in a sequential manner so as to construct the pallet load PAL (see FIG. 2) on the pallet construction base 301. The controller 10C is operatively connected to the at least one robot 14 and is configured (using any suitable hardware and non-transitory computer program code) to control the movement of the articulated robot relative to the pallet building base 301, thereby effecting calibration of the robot 14 and pallet load building BPAL of the pallet load PAL as described herein. The controller 10C is arranged to generate a three-dimensional image of the pallet support (as described herein) on the pallet building base 301 and a three-dimensional image of the pallet load building BPAL on the pallet support SPAL.

[0030] The pallet build base 301 may be a palletizing elevator 12 as described herein or any other suitable pallet support. The palletizing elevator 12 gradually lowers the pallets from the second level 11 to the first level 13 as products are moved onto the pallets by, for example, two product picking devices or robots 14. Each robot 14 picks products from an adjacent infeed conveyor 16 and places them on a pallet on the palletizing elevator 12. An empty pallet elevator 18 supplies empty pallets to the second level 11, and an empty pallet transport mechanism 20 is disposed between the empty pallet elevator 18 and the palletizing elevator 12 to move empty pallets 21 from the second level of the empty pallet elevator 18 to the palletizing elevator 12 in a manner described in U.S. Pat. No. 10,370,199, issued Aug. 6, 2019, the entire disclosure of which was previously incorporated herein by reference.

[0031] As input to the palletizing cell 10, case units or products CU (see FIG. 2), which may be of different sizes, arrive at the palletizing cell 10 via the infeed conveyor 16, are picked by one of the robots 14 and placed in a palletizing area 23 corresponding to a portion of the palletizing elevator 12 on the upper level 11. Substantially simultaneously with this palletizing operation, the empty pallet elevator 18 carries an empty pallet 21 to the empty pallet transport mechanism 20. The mechanism 20 transfers the empty pallet 21 from the empty pallet elevator 18 to the palletizing area 23. The elevator 12 picks up the pallet 21 and lowers it as the products are palletized. As output from the palletizing cell, full pallets (as shown in FIG. 2) made of different products are ready to be picked by a forklift or other transport means in the lower portion of the palletizing elevator 12.

[0032] As mentioned above, the palletizer cell 10 includes an upper level 11 and a lower level 13 defined by an elevated platform 22. This platform 22 and the equipment thereon are supported on a frame 24. The frame 24 may be formed by an integrated frame element or by adjacent independent sub-platforms, each of which supports one or more components of the palletizer cell 10. The robots 14 are, for example, in the form of standard industrial articulated robot arms 26 suitable for palletizing operations, each of which is operatively positioned adjacent the infeed conveyor 16 and equipped with an end-of-arm tool 410 for picking products from the infeed conveyor 16. Suitable examples of robots 14 include ABB's IRB460 or 4600, FANUC's M410 or R2000, or any similar robot arms proposed by other manufacturers such as Kuka or Motoman. The robotic arm 14 is coupled in any suitable manner to a controller 10C that controls the operation of the robotic arm 14A and the end-of-arm tool 410.

[0033] The end-of-arm tool 410 includes a frame 424 for mounting the end-of-arm tool 410 to a respective robot arm 14, a support assembly 450 for supporting a case unit CU thereon, a gripper assembly 26, and a pusher assembly 28, all of which are mounted to the frame 424. The frame 424 receives a robot mounting bracket or wedge 30 that allows the tool 10A to be conventionally mounted to a flange 14F of the robot arm 14A, where the robot mounting bracket 30 is coupled to the frame 424 in any suitable manner, such as with mechanical and / or chemical fasteners. Suitable examples of the end-of-arm tool 410 can be found in U.S. Pat. No. 9,701,490, issued July 11, 2017, and U.S. Provisional Patent Application No. 63 / 208,023, filed June 8, 2021, entitled "Tool for Palletizing Mixed Load Products, Palletizing Robot Including the Tool, and Method Therefor," the entire disclosures of which are incorporated herein by reference. As described further herein, the end of arm tool also includes a distance sensor 411 that provides for determining the picking position of the robot relative to the in-feed conveyor 16 (see FIG. 4).

[0034] Still referring to Figures 3A-3C, the elevator 18 comprises a pallet receiving platform 30 mounted to a wall 32 for slidable movement along the wall 32. The wall 32 extends from the first level 13 to the second level 11 and is secured to the frame 24 at its perimeter. The platform 30 includes a bottom 34 fixedly mounted between two side walls 36. The side walls 36 are slidably mounted to the wall 32 via sliding elements such as, but not limited to, rollers (not shown) in tracks 38. The up and down movement of the platform 30 along the wall 32 is actuated by a drive 40 operatively coupled to the platform 30 via a belt assembly secured to the wall 32 and the platform 30. The platform 30 is configured and sized to receive a pallet 21 thereon and to slidably receive a portion of the empty pallet transport mechanism 20, as shown in Figure 3A. The platform 30 is not limited to any size and shape and is adapted to a predetermined size and configuration of pallets. According to another aspect (not shown), the platform 30 is defined by a fork element. The empty pallet elevator 18 is not limited to the illustrated embodiment and may comprise any other mechanism for selectively lowering and raising the empty pallet receiving platform 30.

[0035] The palletizing elevator 12 is spaced apart from the empty pallet elevator 18 by the empty pallet transport mechanism 20 and extends between the first level 13 and the second level 11 like the empty pallet elevator 18. The platform 22 includes a rectangular opening 44 that allows the passage of an empty pallet 21 from the empty pallet transport mechanism 20 onto the elevator 12. The rectangular opening 44 provides for the placement of products on the pallet 21 (transferred from the empty pallet elevator 18 to the elevator 12 by the empty pallet transport mechanism 20) by the end-of-arm tool 28. The opening 44 is located in the palletizing area 23, and the robot 14 on the opposite side of the opening 44 carries the products picked from the respective infeed conveyor 16.

[0036] As described herein, the palletizing elevator 12 gradually descends as products are loaded onto the pallet 21 (supported by the palletizing elevator 21) such that the top level of the highest product on the pallet 21 remains approximately level with the platform 22. Still referring to FIGS. 3A-3C, the palletizing elevator 12 generally includes a rectangular pallet receiving element 46 supported by a frame 48. The frame 48 is slidably mounted to the frame 24 via an actuation mechanism 45 for reciprocating movement of the frame 48 and the pallet receiving element 46 generally between the upper level 11 and the lower level 13. The frame 48 includes two first parallel beams 47 (only one shown) secured to two second parallel beams 49 transversely thereof. The frame 48 receives three transverse chain assemblies 50.

[0037] Each of the chain assemblies 50 comprises an endless chain 52 operably attached to the frame structure 48 via a sprocket assembly 54 for rotation along an axis defined by the transport direction of the pallets on the empty pallet transport mechanism 20. The chain assemblies 50 together define the movable pallet receiving element 46. As the construction and operation of chain assemblies are believed to be well known in the art, the assemblies 50 will not be described in greater detail herein for the sake of brevity. The number and configuration of the chain assemblies 50 are also not limited to the illustrated embodiment. Also, the pallet receiving elements 46 are not limited to being movable, and other mechanisms may be provided to move the pallets 21 into and out of the palletizing elevator 12.

[0038] The actuation mechanism 45 comprises four chain assemblies 56, which are fixed between them to both the frame 24 and the frame 48 and are operated by a drive 58. More specifically, each chain assembly 56 includes a chain 60 which is rotatably attached to the frame 24 in an endless manner via two sprocket wheels 62 or 63 (only the lower wheel is shown). The chain 60 generally extends between the platform 22 and the lower level 13 along a direction generally perpendicular to the platform 22. The chain assemblies 56 are interconnected to rotate in unison, firstly by two shafts 64 coaxially attached to each of two opposing pairs of sprockets 62 and 63, and secondly by a gear assembly 66 operatively linking two adjacent sprockets 63, one from each of the aforementioned pairs. The drive 58 includes a motor 68 coupled to the gear assembly 66. The motor 68 includes a brake system. Each adjacent sprocket 63 is connected to a disc brake 70 to ensure safe descent of the palletizing elevator.

[0039] The frame 48 is secured to each of the four chains 60 for movement in unison via four brackets 72 (only two shown) secured to the frame 48. During operation of the palletizing elevator 12, controlled actuation of the drive 58 causes the four lower sprocket wheels 62 and 63 to rotate in the same direction, which causes the chains 60 to rotate in unison in an endless manner, and accordingly, the pallet receiving element 46 to descend or ascend via the frame 48 between the upper level 11 and the lower level 13. The drive 58 operates to cause the pallet receiving element 46 to descend incrementally a step distance corresponding to the height of each level of product added to the pallet 21. However, the upward movement of the pallet receiving element 46 is continuous to return the pallet receiving element 46 to the upper level 11 after a full pallet 21 is removed at the lower level 13, ready for filling with new product.

[0040] The palletizing elevator 12 includes an additional actuator (not shown) coupled to the chain assembly 50 for causing selective translational movement of the pallet receiving element 46 to move the product-filled pallets out of the palletizing elevator 12 when the pallet receiving element 46 is at the lower level 13. The palletizing elevator 12 is not limited to that illustrated herein and may include any other mechanism for lowering and raising the palletizing element 46. The palletizing element 46 is also not limited to that described herein. For example, the chain assembly 50 may be replaced with another pallet transport mechanism or may be omitted.

[0041] Still referring to Figures 3A-3C, the empty pallet transport mechanism 20 will now be described. In one embodiment, the mechanism 20 is in the form of a telescoping fork. The telescoping fork 20 comprises a drive 74, a first stage 76 fixedly mounted to the frame 24, a second stage 78 slidably mounted to the first stage 76 for relative translational movement between two opposing longitudinal directions, and a third stage 80 slidably mounted to the second stage 78 for relative translational movement along the same two opposing directions (see arrows 82 and 84 in Figure 3C). The telescoping fork 20 mechanism includes a three stage system that uses carriages and belts to extend the telescoping fork from one end to the other.

[0042] The position and reach of the telescoping forks 20 are such that when the forks 20 are fully extended in direction 84, the third stage 80 overlaps the pallet receiving platform 30 (see FIG. 3A ) of the empty pallet elevator 18, and when the third stage 80 is fully extended in direction 82, the third stage 80 overlaps the palletizing area 44 to transfer pallets from the third stage forks 20 to the endless chains 52 of the elevator 12.

[0043] 4, 4A-4N, 5A, and 5B, at least one three-dimensional time-of-flight camera 310C of vision system 310 is positioned on frame 24 and one or more of robot(s) 14 to generate three-dimensional images (e.g., the 3D images of FIGS. 7, 9, 11, 13, 15, etc.) of at least a portion of palletizer cell 10 to effect calibration of robot 14 as described herein. Although at least one three-dimensional camera 310C is described herein as a time-of-flight camera, any suitable three-dimensional sensor / imager may be used, including a laser scanner, sonar, or other suitable machine vision system. As described herein, the at least one three-dimensional camera 310C is communicatively connected to the controller 10C, such that the controller 10C records real-time three-dimensional image data (e.g., point clouds as illustrated in Figures 7, 9, 11, 13, 15, etc., and / or any suitable data obtained from the point clouds) from the at least one three-dimensional camera 310C embodying different corresponding three-dimensional images of the palletizer cell 10.

[0044] In one embodiment, the at least one three-dimensional camera 310C is configured to provide three-dimensional imaging of the pallet support SPAL on the pallet building base 301 and three-dimensional imaging of the pallet load build BPAL on the pallet support SPAL of the end-of-arm tool 410 at one or more predetermined positions of the palletizer cell 10. In one embodiment, the at least one three-dimensional camera 310C includes four cameras 310C1, 310C2, 310C3, 310C4 (see, e.g., Figures 5A and 5B), each of which is coupled to the frame 24 in any suitable position to have a respective field of view FOV1-FOV4 (see, e.g., Figures 4 and 4A-4N) for imaging at least two sides of the pallet support SPAL and the pallet load build BPAL / pallet load build structure RPAL, e.g., a top side (see Figure 2) and one of a front side, a back side, and a vertical side (extending between the front and back) (see Figure 2). At least one camera 310C may be oriented such that the top and at least one side (e.g., front, back, or vertical) of the pallet support SPAL and each case unit CU disposed on the pallet support SPAL are visible within a field of view FOV1-FOV4 that covers a corresponding portion of the pallet support SPAL / pallet load build structure RPAL. Also, referring to Figures 4G-4J, in one embodiment, the cameras 310C1, 310C2, 310C3, 310C4 may have any suitable focal length for a given image strength and may be positioned at an angle of, for example, 45° (see Figure 4H) relative to the frame 24 (e.g., the horizontal plane of the frame 24, defined by the pallet build base 301, for example) and / or relative to each other such that at least two sides are imaged by at least one camera, although in other embodiments, the angle between the cameras 310C1, 310C2, 310C3, 310C4 and / or the frame 24 may be greater or less than 45°.In one embodiment, each field of view FOV1-FOV4 (collectively referred to as the field of view FOV) of cameras 310C1, 310C2, 310C3, 310C4 (see Figures 4H and 4K-4N which illustrate each of the fields of view highlighted relative to the others) may be a 45° field of view, however in other embodiments the field of view FOV may be greater or less than 45° so long as at least two sides of the pallet support part SPAL, and of the pallet support part SPAL and the pallet load build structure BPAL / pallet load build structure RPAL, are imaged.

[0045] In one embodiment, at least one camera 310C resolves three-dimensional definition of case unit features (e.g., edges of the case unit) and calibration fixture features from two or more orthogonal planes such that maximum certainty of feature pose (e.g., X, Y, Z, θ, α, μ position of feature (see FIG. 3G)) is obtained from a single image of the item in each of the fields of view FOV1-FOV4 of the at least one camera 310C. Here, resolution of three-dimensional definition of case unit features and / or calibration fixture features is independent of the placement of the camera 310C (as long as the top and one side of the feature are imaged) and is performed in real-time (e.g., within the pick / place cycle of at least one robot 14 or within the calibration procedure of at least one robot 14).

[0046] Although four cameras 310C1-310C4 are described, it should be understood that more or less than four cameras 310C may be used and positioned such that the field of view of the camera(s) 310C of the vision system 310 covers the entirety (or at least a predetermined portion) of the pallet building base 301 of the frame 24, the pallet support SPAL seated thereon, and the anticipated pallet load building structure RPAL, at any suitable desired resolution to capture three-dimensional time-of-flight images of the object(s) at all locations on the pallet support SPAL, and at all locations on the pallet load building structure RPAL. The combined fields of view FOV1-FOV4 result in a near-complete 360° coverage of the pallet load building structure RPAL with overlap of the fields of view FOV1-FOV4. For example, the combined fields of view FOV1-FOV4 may cover a standard pallet support SPAL (e.g., having dimensions of 48 inches by 48 inches, 48 ​​inches by 40 inches, and / or 36 inches by 36 inches), and it should be understood that the cameras 310Ca-310C4 and associated fields of view FOV1-FOV4 may cover (e.g., image) a larger field of view as desired (e.g., including a truck bed or any field size). Additionally, the fields of view FOV1-FOV4 may cover any suitable height PH of the palletized building structure RPAL (see FIG. 3H), such as, for example, heights of 60 inches, 70 inches, and 80 inches, although in other embodiments the fields of view FOV1-FOV4 may cover heights less than 60 inches or greater than 80 inches.

[0047] In one embodiment, each of the cameras 310C1-310C4 may have a resolution of 176 pixels by 132 pixels, while in other embodiments, each, or one or more, of the cameras 310C1-310C4 may have a higher resolution (e.g., a resolution of 320 pixels by 240 pixels or more) as needed to provide a desired minimum depth map accuracy of about 0.5 inches at the outermost boundary of the three-dimensional space 3DS of the pallet build (so that the depth map accuracy across the captured images of the entirety or a given portion of the pallet support / pallet build is no less than about 0.5 inches). In that way, sufficient resolution is provided to resolve the grid features of the pallet support SPAL to an accuracy that is well established such that the planarity of the entire pallet is determined for positioning the stable first layer PL1 of the case units CU on the pallet support SPAL by the vision system 300 as described herein. Also, sufficient resolution may be provided to resolve case unit features (e.g., edges of the case, etc.) such that planarity across the top of each layer PL1-PL4 (see FIG. 3H) may be determined and sufficiently established for placing stable layers PL2-PL5 on previously placed layers PL1-PL4. The resolution of cameras 310C1-310C4 may be such that minimal processing is required to resolve case unit features (e.g., edges of the case units) such that the case unit features are resolved substantially in real time from images received by controller 10C.

[0048] As described herein, the vision system 310 also includes an electromagnetic sensor 500 configured to sense calibration features (e.g., of various jig(s) or fixture(s) as described herein, such as the calibration box 810 and / or integral features of the robot 14 / end-of-arm tool 410). The controller 10C is communicatively coupled to the electromagnetic sensor 500 and records the sensor signal substantially coincident with the electromagnetic sensor 500 sensing the calibration feature and generating a sensor signal that embodies predetermined characteristics of the calibration feature relative to a corresponding pose of the robot 14 (and a corresponding pose of the calibration feature due to movement of the robot 14).

[0049] The electromagnetic sensor 500 is configured to be robust to variable lighting conditions. In one or more embodiments, the electromagnetic sensor 500 is a two-dimensional (2D) color camera 500C (also referred to herein as a "2D camera") disposed on the frame 24 and one or more of the robot(s) 14 to generate a two-dimensional image (e.g., 2D images of Figures 8, 10, 12, 14, 16, etc.) of at least a portion of the palletizer cell 10 to provide calibration of the robot 14 as described herein (see, e.g., Figures 3C, 4G, 5A, and 5B). Here, the two-dimensional camera 500C is a color camera that identifies contrast patterns (e.g., grid patterns 800, 815, etc.) with a desired accuracy commensurate with the motion accuracy of the robot 14 (e.g., approximately equal to or less than the motion encoder error of the robot 14). The camera 500C may be any suitable two-dimensional camera, such as a camera available from Basler Inc. (Exton, PA, USA), having a field of view FOV2D (see FIG. 12) configured to image at least the pallet building area of ​​the palletizer cell 10. The two-dimensional camera 500C may be a high-resolution camera, for example, having a resolution of about 2.3 megapixels to about 24 megapixels (or more). Any suitable illumination (such as a ring light 500L or any other suitable illumination source) may be provided to illuminate the object being imaged by the two-dimensional camera 500C. The two-dimensional camera 500 is communicatively connected to the controller 10C, whereby the controller 10C records real-time two-dimensional image data from the two-dimensional camera 500C embodying different corresponding two-dimensional images of the palletizer cell 10. In one embodiment, the two-dimensional camera 310C is configured to provide two-dimensional imaging of the pallet support SPAL on the pallet building base 301 and two-dimensional imaging of the pallet load build BPAL on the pallet support SPAL of the arm end tool 410 at one or more predetermined positions of the palletizer cell 10.In other aspects, the camera 500C may be any suitable imaging device, such as any suitable electromagnetic sensor (i.e., a sensor for detecting the electromagnetic spectrum / frequency range of the electromagnetic radiation of an object(s)), including, but not limited to, a camera, a laser scanner, and the like. In other aspects, the camera 500C may be a three-dimensional camera having sufficient resolution to extract data as described herein for two-dimensional cameras from a three-dimensional image acquired thereby. In yet other aspects, the electromagnetic sensor 500 may be a laser scanner / reader 500LSR configured to scan / read features / characteristics of a target object (e.g., a three-dimensional structure, etc.) that may determine the pose of the target object (such as those described herein).

[0050] It is noted that alignment of the fields of view of the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C is performed by aligning the image from each camera 310C1-310C4, 500C with the image from the other camera 310C1-310C4, 500C by utilizing image overlay annotations. In other aspects, the cameras 310CA1-310C4, 500C of the vision system 310 may be synchronized using a calibration fixture placed on the lift 12 in a manner generally similar to that described in U.S. Patent Application No. 17 / 070,753, filed October 14, 2020, entitled "Vision-Assisted Robotized Depalletizer," the entire disclosure of which is incorporated herein by reference.

[0051] 3A-3D, 5A, 5B, palletizer cell references are described, where these references are utilized by the controller 10C to effect a fully automated calibration of the palletizer cell 10 from an uncalibrated state of the palletizer cell 10. The references described herein are defined for illustrative purposes (i.e., exemplary in nature) and convenience of explanation. In other aspects of the disclosed embodiments, any suitable references, such as those described herein, may be selected and / or defined in any suitable manner, where the references are located at any suitable locations of the palletizer cell and / or palletizer cell components.

[0052] It is noted that the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C are pre-calibrated such that each of the cameras 310C1-310C4, 500C has a respective origin reference COR (see FIG. 5A, i.e., the origin of the coordinate system (X, Y, Z)) located at the center of the camera lens. The three-dimensional cameras 310C1-310C4 are configured to provide a three-dimensional point cloud of a portion of the palletizer cell 10, where the imaged positions are determined with reference to the respective origin reference. The two-dimensional camera 500C may also be configured for three-dimensional vision by intrinsic parameters in the calibration of the two-dimensional camera 500C (i.e., given a plane in the 3D world of the palletizer cell 10, the pixels in the pixel coordinates of the two-dimensional camera 500C can be transformed to the world coordinates of the palletizer cell 10). Calibration of the two-dimensional camera 500 is performed in a manner similar to that described in Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000, the entire disclosure of which is incorporated herein by reference.

[0053] Each of the robots 14 has a base 14B and an arm or manipulator 14A. The end-of-arm tool 410 is coupled to the arm 14A at a flange 14F. Each of the robots 14 includes at least a base reference RBR, a flange reference RFR, and a tool reference RTR. For example, as illustrated in FIGS. 3A-3C, the two robots 14 are arranged on the frame 24 juxtaposed to each other. The bases 14B are arranged on substantially the same (or common) plane CP (see FIG. 3A) defined by the frame 24 of the palletizer cell 10. The base reference RBR is arranged in the center of the base 14B of the respective robot 14 with the X-axis extending in the direction of the flow of the cases on the picking conveyor 16 and the origin of the Z-axis being arranged on the plane CP. It is again noted that the references described herein are defined for illustrative purposes (i.e., are exemplary in nature) and for convenience of explanation. In other aspects of the disclosed embodiments, any suitable reference feature, such as those described herein, may be selected and / or defined in any suitable manner, where the reference feature is located at any suitable location on the palletizer cell and / or palletizer cell components.

[0054] The robot flange reference portion RFR is positioned in the center of the flange 14F (see also FIG. 4C ) with the Z axis pointing downward (and having its origin at the flange's end-of-arm tooling coupling plane) and the X axis extending toward the finger 450F (support assembly 450) of the end-of-arm tooling 410. Here, the robot flange rotates about the Z axis of the flange reference portion RFR (e.g., such that the flange reference portion RFR rotates with its X axis aligned with the finger 450F due to the arm tooling orientation), although in other embodiments the flange 14F may not rotate.

[0055] The Tool Reference RTR (also called Tool Center Point or TCP) defines the reference frame of the tool from which picking and placement are calculated. The Tool Reference RTR has an origin located at the tip of the finger furthest from the camera 310C1-310C4, 500C during picking from the respective picking conveyor 16 (see also Figures 3A and 3D, and 16), with the Z axis pointing downwards (and having its origin at the intersection of the top surface of the finger 450F with the tip surface of the finger) and the X axis extending in the direction of the finger 450F.

[0056] 3A-3C and 17, the empty pallet transport mechanism 20 (e.g., telescopic fork) has a telescopic fork origin reference TFRP, which in one or more embodiments is located substantially on the centerline CL of the opening 44 of the elevator 12. The location of the telescopic fork origin reference (e.g., datum) TFRP may be defined by the nominal dimensions of the palletizing cell 10 (e.g., from a computer-aided drafting (CAD) file) or in any other suitable manner. The telescopic fork origin reference TFRP is common to each of the robots 14, resulting in synchronization or otherwise connection of the workspaces of the robots 14. The origin of the telescopic fork origin reference TFRP is located a predetermined distance from the robot base reference RBR in the X direction, and the Y origin is located substantially at the midpoint between the robot base references RBR. The telescopic fork origin reference TFRP is projected onto the plane of the empty pallet transport mechanism 20 (see FIG. 17--so that the Z-axis origin is on the top surface of the empty pallet transport mechanism 20). The telescopic fork origin reference part TFRP is used for building the pallet and positioning the pallet support part SPAL in the workspace of the palletizer cell 10.

[0057] 18A and 18B, a fork reference (e.g., datum) TFR is also defined in each of the three-dimensional cameras 310C1-310C4. The fork reference TFR is utilized by the controller 10C to segment the point cloud of each of the three-dimensional cameras 310C1-310C4. Here, the fork reference TFR locates the transfer fork 20F1, 20F2 of the empty pallet transport mechanism 20 within the workspace of the palletizer cell 10 relative to each of the three-dimensional cameras 310C1-310C4. The fork reference TFR is defined with a Z-axis pointing upward from a plane defined by the top surfaces of the forks 20F1, 20F2 and an X-axis pointing away from the three-dimensional cameras 310C1-310C4 in line with the centerline between the forks 20F1, 20F2 (see FIGS. 18A and 18B). The origin of the fork reference TFR for each respective three-dimensional camera 310C1-310C4 is at the root point of the X and Z plane when the camera 310C1-310C4 projects onto the plane. The Y origin is then aligned with the center line CL of the forks 20F1, 20F2. For example, exemplary images from the three-dimensional cameras 310C1 and 310C3 are illustrated in FIGS. 18A and 18B, showing the point clouds of the forks 20F1, 20F2 detected by the respective cameras and the fork reference TFR for the respective cameras. Here, the selection of a separate common reference (such as the fork reference TFR) for each of the three-dimensional cameras may, in some embodiments, function in the calibration of the three-dimensional cameras relative to each other by solving the pose of the reference and the characteristics from the detection of the pose, given the predetermined positions and known characteristics of the three-dimensional cameras. In such a case, the calibration of the three-dimensional cameras corresponds to the placement accuracy of the robot 14 / end effector 410.

[0058] Additional references (e.g., datums) TFRA1, TFRA2 may be determined for each individual fork 20F1, 20F2 for one or more of the three-dimensional cameras 310C1-310C4. These additional references TFRA1, TFRA2 result in detection of anomalies such as bent / twisted forks and height differences between the forks 20F1, 20F2 along the longitudinal lengths of the forks 20F1, 20F2. Here, the controller 10C, for example, using the three-dimensional camera 310C1 (see FIG. 18B), results in image sampling (see sampling boxes S1-S6) of a portion of each fork 20F1, 20F2 to generate individual fork references TFRA1 (for fork 20F1), TFRA2 (for fork 20F2) based on a planar fit of the respective fork surface with the respective sampling boxes S1-S6. The origin of each fork reference portion TFRA1, TFRA2 is located on each fork 20F1, 20F2 with the Z axis perpendicular to the plane of the respective fork 20F1, 20F2 (as established by the sampling boxes S1-S6) and pointing upwards, and the X axis pointing away from the camera imaging the fork (camera 310CA in this example). In one embodiment, anomalies are recorded from fork features identified based on imaging of one three-dimensional camera 310C1-310C4. In another embodiment, various indications of anomalies are detected by comparison of images of the fork captured from more than one of the three-dimensional cameras 310C1-310C4 (where the cameras are calibrated to each other using a common electromagnetic sensor 500).

[0059] 3D, the distance sensor 411 may also include a reference (e.g., datum) DSF. The distance sensor reference DSF is defined as an intersection of the laser line and a plane PL located in the flange reference (e.g., datum) RFR (e.g., at a height of about 1000 mm in the flange reference RFR, but the height of the plane PL may be any suitable height). The laser line is imaged by one or more of the cameras (e.g., two-dimensional camera 500C), and the controller 10C is configured to determine the position of the distance sensor 411 on the end-of-arm tool 410 based on the camera images and to determine the position of the distance sensor reference.

[0060] 8, in accordance with aspects of the disclosed embodiment, the automatic calibration system ACS includes structures and / or indicia having calibration features connected to the end-of-arm tool 410 (or end effector), where the features have predetermined properties that define a predetermined or known shape or pattern. For example, a fully automatic calibration may be effected by the controller 10C and the automatic calibration system ACS with reference to a target object held on or integral with the end-of-arm tool 410, or positioned in a predetermined geometric relationship to the end-of-arm tool 410 and features of the end-of-arm tool 410. The target object may be one or more target objects (e.g., one or more jigs, fixtures, or other suitable structures / objects), where the one or more target objects include suitable structures and / or indicia, where the structures and / or indicia have predetermined properties that are determining the three-dimensional orientation / location of the target object. For example, a given characteristic of a structure and / or indicia of a target object may have four edges that are not collinear but are coplanar, where the distances and / or positions between the edges are known to result in a homography between images of different cameras 310C1-310C4, 500C of the vision system 310.

[0061] Different separate target objects may be utilized for locating / calibrating the robot 14 / end effector 410 in the workspace of the palletizing cell and for establishing references (e.g., datums). In one aspect, the calibration features of the (e.g., each of) the target objects are dependent on the end-of-arm tool 410 or the calibration features and the end-of-arm tool 410 are commonly dependent on a common member of the robot 410 (e.g., flange 14F, etc.), and the calibration features and the end-of-arm tool 410 move as a unit relative to the frame 24 with each articulation of the robot 14. For example, one of the one or more target objects is a grid pattern 800 of known characteristics (such as the four non-collinear but coplanar edges described above) integrally formed on the frame 424 of the end-of-arm tool 410 (see FIG. 8), where the grid pattern 800 has four edges in a known relationship to one another. Here, the grid pattern 800 may be integrated with any suitable portion of the structure of the respective robot case manipulator 14 and / or any suitable portion of the structure of the end-of-arm tool 410 to effect localization of the respective robot 14. One or more of the target objects may also include a jig(s) or fixture(s) that is distinct from the integrally formed features of the end-of-arm tool 410 (and robot 14), where the jig(s) or fixture(s) are utilized to set the references described herein. The jig(s) or fixture(s) may be a calibration box 810 (or other suitable object) carried by the end-of-arm tool 410, where the fixture includes a grid pattern 815 of known features, such as the four non-collinear but coplanar edges described above. In one or more embodiments, the jig(s) or fixture(s) may be utilized for both localization of the robot 14 / end effector 410 and setting the references.

[0062] For illustrative purposes only, calibration box 810 has a width W of approximately 12 inches, a height H of approximately 12 inches, and a length L of approximately 24 inches, although in other embodiments, calibration box 810 may have any suitable dimensions, including, but not limited to, dimensions corresponding to case units expected to be handled by end-of-arm tool 410. Grid pattern 815 is formed by a number of quadrilaterals arranged in a two-dimensional array (i.e., on at least two planes of calibration box 810). For illustrative purposes, the two-dimensional array of quadrilaterals on each side of calibration box 810 forms a checkerboard pattern with squares measuring approximately 2 inches by 2 inches, although in other embodiments, the quadrilaterals on each side of the calibration box may have any suitable dimensions for forming a two-dimensional array.

[0063] The calibration box 810 (or other target object) is optimized to take advantage of the imaging characteristics of both the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C. For example, the squares of the checkerboard pattern are such that the colors of the squares are such that the two-dimensional array is not detected by the three-dimensional cameras 310C1-310C2 (e.g., such that the checkerboard pattern does not appear in the three-dimensional image, resulting in a localization of the plane formed by the sides of the calibration box), but have similar luminance values ​​with hue values ​​that are far enough apart to facilitate imaging of the two-dimensional array by the two-dimensional camera 500C. Suitable examples of contrasting colors for the two-dimensional array include, but are not limited to, green and tan.

[0064] 3A-3D, 5A-15, and 19A-25, an auto-calibration system ACS with controller 10C is configured to image the calibration box 810 and other portions of the palletizer cell to provide fully automatic calibration (including localization) of each of the robots 14 within the palletizer cell reference frame and automatic teaching of each station (e.g., conveyor, pallet load surface, forks, etc.) of the palletizer cell, as described further herein. To facilitate the calibration procedure for each degree of freedom of the drive(s) of the robots 14, the position of each robot 14 is constrained relative to the palletizer cell reference frame such that measurements obtained by the vision (in other words, auto-calibration and teaching) system 310 do not affect (i.e., are modified) the relative position of the robot 14 within the mathematical model of the palletizer cell 10 generated from the calibration procedure. In particular, one or more bounds / correction constraints (referred to herein as constraints) are set on the position of the robot 14 such that the robot base 14B rests on a common plane (e.g., defined by the frame 24). It is also noted that the robots 14 are "zeroed" or moved to their home positions to reset the encoders 14DE (FIG. 3D) of the robots 14 prior to the auto-calibration procedure such that the position of the robot flange 14F is known relative to the base 14B.

[0065] A calibration procedure for calibrating the robot 14 is initiated in any suitable manner, such as by an operator interface having a user interface UI of the controller 10C (or controller 199C) (FIG. 14, block 1400).

[0066] As described above, one or more constraints are identified for each robot 14 (FIG. 14, block 1405). The constraints may be predetermined / known from the design of the palletizer cell, and one non-limiting example of a boundary / modification constraint is a plane CP formed by the frame 24 (see above-FIG. 3A) in which the robot is positioned. The plane CP may determine, at least in part, the origin and pose of the base 14B of the robot 14 (relative to at least the electromagnetic sensors 500 and / or the three-dimensional cameras 310C1-310C4). The plane CP may define the orientation / tilt (e.g., in the Z direction) of each robot 14 in the X-axis and Y-axis relative to the reference frame of at least the electromagnetic sensors 500 and / or the three-dimensional cameras 310C1-310C4.

[0067] The controller 10C provides for the determination of a common fork origin reference TFRP for each of the three-dimensional cameras 310C1-310C4 in a manner described herein (FIG. 14, block 1420). As described previously, the forks 20F1, 20F2 form the base of the pallet support on which the pallet load is built by each of the robots 14, either together or alone. The pallet support (and the forks 20F1, 20F2) are the palletizer cell stations, and the origin reference positions or references of the forks (also the boundaries of the pallet support) are the location(s) at which the robot(s) 14 are calibrated and taught. The determination of the common fork reference frame / reference TFRP is described here first, but may be provided at any step throughout the calibration / teaching process. Unless otherwise specified, the order of the process is not constrained.

[0068] One of the robots 14 is commanded by the controller 10C to position the end-of-arm tool 410 so that an operator can place the calibration box 810 on the end-of-arm tool 410 (FIG. 14, block 1410), as illustrated in FIGS. 8 and 16. In other embodiments, as described herein, a target may be integrated into the end-of-arm tool 410, in which case the calibration box 810 may not be placed on the end-of-arm tool 410. The placement of the calibration box 810 on the end-of-arm tool 410 may be random, so long as the end-of-arm tool 410 can grasp the calibration box 810. Once the calibration box 810 is grasped by the end-of-arm tool 410, the position of the calibration box 810 relative to the end-of-arm tool 410 of the robot 14 does not change (i.e., is invariant) throughout the calibration procedure. Here, a calibration feature (e.g., of the calibration box 810) is positioned relative to the robot 14 such that predetermined characteristics of the calibration feature determine a predetermined relationship between the calibration feature and the robot reference frame (such as the flange reference portion RFR and / or the base reference portion RBR).

[0069] Now, the calibration box is grasped by the end-of-arm tool 410 for a calibration procedure (FIG. 14, block 1430), thereby implementing camera synchronization in the calibration process. Now, the robot 14 is moved (as commanded by the controller 10C) via the drive(s) 14D (see FIG. 3D) of the robot 14, and the calibration box 810 is moved to a series of, for example, four known positions (the robot may be moved to more or less than four known positions). The position in the robot reference frame is known from the encoder data of the drive(s) 14D (i.e., the drive section 14DS (see FIG. 3D) of the robot). Each position in the series of positions positions the target object such that the calibration system / camera images the target object (e.g., senses a predetermined characteristic that determines the pose, with the accuracy identified herein). At each position in the sequence of positions, at least one different one of the three-dimensional cameras 310C1-310C4 and electromagnetic sensors 500 (e.g., two-dimensional camera 500C) images the target object. For example, each camera 310C1-310C4 and electromagnetic sensor 500 images the robot's end-of-arm tool 410 at each of the four positions. The position of the robot determined by the robot encoder 14DE (FIG. 3D) is associated with the position of the robot in each of the four images of each camera 310C1-310C4 and electromagnetic sensor 500. In camera synchronization, the controller 10C utilizes a three-dimensional localization of the calibration box 810, and then a two-dimensional localization of the calibration box 810, where the box is placed in a position within the palletized volume that is known to be "easy" for the camera pre-synchronization algorithm of the controller 10C to detect (e.g., there are few features other than the calibration box in the image).

[0070] Once the position of the robot 14 is (e.g., approximately) located relative to the camera positions (e.g., the transformation of the robot's position into the camera's frame of reference is approximated), the robot 14 is automatically calibrated / synchronized with the cameras 310C1-310C4 and the electromagnetic sensor 500. An algorithm utilized by the controller 10C for automatic calibration / localization of the robot 14 detects a calibration box 810 in both three and two dimensions at various positions in a series of positions and at each pose of each position (to optimize the transformation between the robot reference and the camera reference), and outputs, for example, corners, planes, and inliers of the detected box. Calibration / localization of the robot 14 results in detection of a three-dimensional calibration box 810 (e.g., box corners) and a two-dimensional calibration box 810 (e.g., corners and grid 815) in the world (i.e., reference frame) of each of the cameras 310C1-310C4 and electromagnetic sensors 500 for each position in a sequence of positions, thereby bringing all references together (or synchronizing) in a combined space.

[0071] As described herein, the controller 10C moves the robot 14 (by commanding actuation of one or more degrees of freedom of the robot drive(s)) to, for example, detect calibration features of the calibration box 810 and / or integral features of the end-of-arm tool 410 at known or predetermined positions within the robot reference frame defined by the degrees of freedom ΔX, ΔY, ΔZ, Δμ (e.g., see FIGS. 3D and 4G) from the initial / home position of the robot 14, such that the electromagnetic sensor (two-dimensional camera) 500 senses the calibration features (at known or predetermined positions within the robot reference frame defined by the degrees of freedom ΔX, ΔY, ΔZ, Δμ (e.g., see FIGS. 3D and 4G) from the initial / home position of the robot 14). The controller 10C is configured to position the electromagnetic sensor 500 at an initial position (at an initial position in the vicinity of the sensor 500), and the controller 10C is configured to solve the pose of the calibration feature relative to a sensor reference frame (e.g., origin reference frame COR) and determine from the solved pose a transformation between the origin reference frame COR of the electromagnetic sensor 500 (here the two-dimensional camera 500C) and the selected three-dimensional camera reference frame (e.g., origin reference frame COR of one or more of the three-dimensional cameras 310C1-310C4) to calibrate the three-dimensional cameras 310C1-310C4. As can be understood, the calibration (solving the pose of the origin reference frame COR) of the electromagnetic sensor 500 and the selected three-dimensional cameras 310C1-310C4 is thus effected substantially in agreement. The localization and teaching of the robot 14 and the end-of-arm tool 410 in the palletizer cell space is also initiated and effected substantially in agreement with the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4 as described further herein. For example, an algorithm utilized by the controller 10C for automatic calibration of the robot 14, in one embodiment, utilizes detection of a three-dimensional calibration box 810 from three-dimensional cameras 310C1-310C4 and subsequent detection of a two-dimensional calibration box 810 by the electromagnetic sensor 500, where the three-dimensional detection facilitates segmentation of the two-dimensional image as described herein and faster pose solution via the electromagnetic sensor 500 (here the two-dimensional camera 500C).Three-dimensional detection provides for easy detection of the calibration box 810 to facilitate resolution of the two-dimensional image based on the corners of the front and top surfaces (sides may also be detected) of the calibration box 810, for example, found by the controller 10C based on the planes of the front and top surfaces. Thus, two-dimensional localization of the calibration box 810 is performed based on the corners found in the detection of the three-dimensional camera calibration box 810 to facilitate segmentation of the two-dimensional image of the calibration box 810 (wherein a mask is generated and applied to the two-dimensional image by the controller 10C to blank out extraneous features in the two-dimensional image). In other aspects, if the electromagnetic sensor 500 is other than an imaging sensor or camera (e.g., a raster scanner, etc.), visualization by the three-dimensional cameras 310C1-310C4 of the calibration box 810 complements the pose calibration of the electromagnetic sensor 500 such that the calibrations of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4 are brought about in substantial agreement, but the three-dimensional detection from the three-dimensional cameras 310C1-310C4 may not be used to assist in the calibration and pose determination of the electromagnetic sensor 500 (and in some aspects, if the electromagnetic sensor 500 is a two-dimensional camera 500C and other suitable image processing of the controller 10C is utilized to segment the two-dimensional image, the three-dimensional visualization / detection by the three-dimensional cameras 310C1-310C4 may not assist in the pose determination of the electromagnetic sensor 500).

[0072] In the calibration procedure, the references of the robot 14, the cameras 310C1-310C4 and the electromagnetic sensor 500 are set (FIG. 14, block 1434). By way of example, the references of the electromagnetic sensor 500, which define the pose of the electromagnetic sensor 500 (also called the calibration sensor, which in this example is the two-dimensional camera 500C), are calibrated against the pose of the target object (calibration box 810 in this example), where the pose of the target object is substantially unconstrained within the acuity of the sensor. Here, a predefined positioning of the target object relative to the field of view of the electromagnetic sensor 500 and vice versa is not required. As described herein, the target object includes predefined characteristics that are determining the pose solution of the target object, whereby the calibration of the electromagnetic sensor 500 (and the three-dimensional cameras 310C1-310C4) solves the pose of the target object and the respective electromagnetic sensor 500 / three-dimensional camera 310C1, 310C4 for each (or one or more) of the robot 14 and the end-of-arm tool 410. For example, a grid 815 of calibration boxes 810 is determined by the controller 10C from which corners of the calibration box 810 are defined in the world of the electromagnetic sensor 500 (here, for purposes of illustration, the two-dimensional camera 500C). The controller 10C determines the pose of the calibration box 810 in the world of the electromagnetic sensor 500 and determines the calibration of the electromagnetic sensor 500 via the transformation between the electromagnetic sensor 500 and the selected three-dimensional camera 310C1-310C4 of the respective three-dimensional camera 310C1-310C4.

[0073] In automatic calibration of the robot 14, the robot 14 moves the calibration box 810 (e.g., within the field of view of the cameras 310C1-310C2 and the electromagnetic sensor 500) in one or more degrees of freedom of the robot drive section (as each pose is indicated by at least one degree of freedom of the drive, e.g., by a respective drive position sensor of that degree of freedom) to multiple predetermined / known positions and angles within the palletization volume (whereby the positions and angles are defined by the movement in each degree of freedom of the drive section and communicated to all stations within the palletizer cell space) (FIG. 14, block 1431), such that each of the three-dimensional cameras 310C1-310C4 (and the electromagnetic sensor 500) images at least the calibration box 810 (FIG. 14, block 1432), and the image of each three-dimensional camera is consistent with the resolution of the electromagnetic sensor 500 of the calibration box. Here, movement of the calibration box 810 by the robot 14 resolves the pose of the calibration box 810 (and the position of the robot 14) for each degree of freedom (X, Y, Z, μ, and in some embodiments rotation of the end-of-arm tool about axis θ), such that calibration of the electromagnetic sensor 500 and three-dimensional cameras 310C1-310C4 is performed by the controller 10C approximately simultaneously for each robot (as described herein), approximately simultaneous bot localization determinations are made by the controller 10C (as described herein), and approximately simultaneous teaching of the palletizer cell stations for the robots 14 is performed by the controller 10C (as described herein).

[0074] Movement of the calibration box 810 (or other target object as described herein) by the robot 14, for example by movement of the flange 14F along a known path (e.g., determined by a position sensor in the drive section of the robot 14), provides an indication of the individual position / pose of the calibration box 810 relative to each of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4 along the known path. As described herein, the calibration box 810 is moved (to any suitable number of predetermined / known positions) along any suitable number of predetermined / known paths to resolve parameters of the bot pose relative to the references described herein. Given the known bounds / rectification constraints, the known path in the robot reference frame, and the pose of the calibration box 810, the controller 10C determines pose parameters of each robot / sensor (the term sensor includes the three-dimensional cameras 310C-310C4 and the electromagnetic sensor 500) in the sensor reference frame and the palletizer cell reference frame in a manner described herein.

[0075] In one or more embodiments, imaging of the calibration box 810 by the three-dimensional cameras 310C1-310C4 and the electromagnetic sensor 500 (e.g., at each position / pose of the calibration box 810) is performed substantially simultaneously in order to calibrate the three-dimensional cameras 310C1-310C4 to the pose of the electromagnetic sensor 500 and to resolve the calibration relative to each (or one or more) other three-dimensional cameras 310C1-310C4 (FIG. 14, block 1435) (however, in other embodiments, imaging of the calibration box 810 by the three-dimensional cameras 310C1-310C4 and the electromagnetic sensor 500 may be performed at different times). Example positions and angles of the calibration box 810 are illustrated in FIG. 15, and although three positions / angles are illustrated, it is noted that the robot 14 may move the calibration box 810 to any suitable number of positions / angles such that the placement of the calibration box 810 on the end-of-arm tool 410 is automatically resolved and sufficient data is acquired by the vision system 310 to locate the end-of-arm tool within the reference frame / portion of the palletizer cell 10.

[0076] It is noted that the location of the calibration box 810 on the end-of-arm tool 410 is not pre-set to any datum or reference location on the end-of-arm tool 410, other than that the calibration box 810 is seated on the fingers 450F of the end-of-arm tool 410. Here, the location of the calibration box 810 on the end-of-arm tool 410 (and robot 14) is common / constant throughout all images such that the relative pose of the calibration box 810 with respect to the robot 14 is cancelled out from the algorithms of the controller 10C which determine the pose and position of the robot 14, since the position of the calibration box 810 with respect to the robot 14 / end-of-arm tool 410 is consistent / constant throughout the entire calibration process and can be ignored. As an example, by moving the calibration box 810 to as few as three positions (each position having a different box angle / pose where an image is generated by each camera 310C1-310C4, 500C at each of the three positions), the placement of the calibration box 810 on the end-of-arm tool 410 can be resolved and sufficient data can be acquired to localize the robot 14. In one embodiment, 16 images are captured by each camera at 16 different calibration box positions / poses. In yet other embodiments, the robot 14 may move the calibration box 810 to more than 16 positions / poses for image capture. The movement of the calibration box 810 to various positions allows the controller 10C to monitor the dynamics of the robot 14 for consistency, and if the dynamics are not consistent (e.g., due to an anomaly in the robot encoder), the controller 10C alerts the operator via the user interface GUI that the robot 14 may need attention.

[0077] At each position / pose of the calibration box 810, each of the three-dimensional cameras 310CA-310C4 detects the calibration box 810 to define a box face reference. As mentioned above and as can be seen in Figures 8 and 9 (see also Figures 10 and 11, and Figures 12 and 13), the data from the three-dimensional cameras 310Ca1-310C4 is in the form of a respective point cloud (Figure 9 is a point cloud representation of the two-dimensional image of Figure 8, Figure 11 is a point cloud representation of Figure 10, and Figure 13 is a point cloud representation of Figure 12). As with the three-dimensional box reference, the fork reference TFRP is utilized in one embodiment by the controller 10C to "mask" extraneous areas of the point cloud that are not of the calibration box 810 or the end-of-arm tool 410 and that do not determine the calibration of the robot 14. For example, points in the point cloud below the fork reference portion TFRP are removed, points in the point cloud to the left and right of the lift zone (i.e., the zone in which the lift operates) are removed, and points further away (i.e., further from the imaging camera) than the furthest edge (i.e., the edge furthest from the imaging camera) of the lift opening 44 are removed. The result of the masking is a point cloud that includes substantially only the calibration box 810 and the end-of-arm tool 410.

[0078] One box reference is defined on the front and top surfaces of the calibration box 810 and is utilized to create a sampling box (similar to that described herein with respect to FIG. 18B) on the surface of the calibration box 810, which is utilized to fine-tune the corner positions of the calibration box 810 as described herein. With reference to FIGS. 8, 9, 19B, and 19B, the box references BRI, BRF, BRT defined on the surface of the calibration box 810 define the X-axis of the box references BRI, BRF, BRT using a target orientation based on the orientation of the Y-axis of the robot flange 14F. The Z-axis of the box references BRI, BRF, BRT extends in the same direction as the Z-axis of the fork origin reference TFRP (see FIGS. 18A and 18B) (it is noted that the directional conventions described herein and throughout the description are used for convenience, and that in other embodiments, any suitable orientation may be used). The (initial) box reference parts BRIF, BRIT (FIG. 19A) are determined from the positions of the respective planes 900, 901 of the front and top surfaces as determined by the point cloud data (see FIG. 9). The controller 10C is configured to refine the initial box reference parts BRIF, BRIT using any suitable corner detection algorithm, where a sampling box S7 is utilized to position / fit the planes 900, 901 to the surfaces of the respective calibration boxes 810 to define the refined box reference parts BRF, BRT (see FIG. 19B) (where the position of the sampling box is determined in the region of the surface having a consistent normal direction, as described herein with respect to FIG. 18B). It is noted that the origin of the box reference parts BRF, BRT may not be updated from the initial box reference parts BRIF, BRIT, and the origin may not be at the center of the box surface of interest. Here, the rotations of the reference axes X, Y, Z are fixed in a precise manner (e.g., to define a calibration precision that is lower than the precision of the robot (e.g., the robot has a precision of between about 1 mm and about 2 mm) as described herein).

[0079] Once the refined box surface reference parts BRF, BRT are defined, a box surface reference part BFR (see FIG. 19B) is determined, where the front surface normal defines the Z axis of the box surface reference part BFR, the intersection between the top surface and the front surface defines the Y axis of the box surface reference part BFR, and the cross product of the Y and Z axes defines the X axis. The X and Z origins of the box surface reference part BFR are placed on the intersection between the front surface and the top surface, and the Y origin is aligned with the side of the top or front surface plane (the side where the camera has an optimal viewpoint relative to the surface).

[0080] Once the box surface reference BFR is defined for each of the three-dimensional cameras 310C1-310C4, the box surface reference BFR2D is determined for the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C). It is noted that the box surface reference BFR2D may be calculated prior to the box surface reference BFR. Referring to FIG. 22, the edges of the calibration box 810 are extracted in the two-dimensional image by the controller 10C using any suitable edge detection algorithm. For example, the edges are split by the controller 10C at sharp curvature points and polarity changes, such that each split edge segment represents one side of a square in the grid pattern 800. For each edge segment tip (e.g., one side of a square in the grid pattern 800), the controller 10C is configured to find three other local edge tips in the local region to form a quartet (i.e., one square of the grid pattern 800). For the controller 10C to determine the presence of a quartet, the four edge tips should have two pairs of parallel opposing segments within a predetermined tolerance, the two pairs of parallel segments intersect perpendicularly within a predetermined tolerance, the opposing segments of each pair have opposite polarity, and the color / hue of the squares surrounding the edge segments should be close to, e.g., green and close to, e.g., yellow, in opposite diagonal directions (it is noted that yellow and green are exemplary colors for the grid patterns described herein).

[0081] In one or more embodiments, the point clouds of one or more of the three-dimensional cameras 310Cal-310C4 are utilized to minimize the effects of variable image intensity, non-uniform lighting, shadows, complex scenery, perspective projection and distortion (as well as other variations) on the segmentation of the two-dimensional image acquired by the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C). Here, the image of the calibration box 810 is positioned in front of and close to the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C) such that there is no perspective view of the calibration box 810 in the two-dimensional image of the calibration box taken by the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C). The box references BFR, BFR2D are determined in the manner described herein. An approximate transformation between the three-dimensional cameras 310Ca1-310C4 and the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C) is determined by the cross product of the reference parts BFR, BFR2D (e.g., Camera 3D to Camera 2D transformation = BFR x BFR2D) (it is noted that the above transformation is provided for illustrative purposes only, and there are multiple ways in which the three-dimensional to two-dimensional transformation may be determined). The surface of the calibration box 810 is found in the point cloud for each of the images taken by the three-dimensional cameras 310C1-310C4, and then mapped in the two-dimensional image to a given calibration box position and pose (such as those described herein) using the approximate transformation and the intrinsic parameters of the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C) to project the surface of the calibration box 810 onto the two-dimensional image plane. This projection from the point cloud onto the two-dimensional image plane creates a two-dimensional image mask that is utilized to locate the box faces within the two-dimensional image (removing substantially all remaining image complexity seen in the two-dimensional image - e.g., with reference to Figures 10 and 12, the portions of the two-dimensional image outside the mask boundary are masked out for segmentation of the two-dimensional image). Thus, the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4 is provided substantially in line with or simultaneously with one another. The position and teaching of the robot 14 is also provided automatically in substantially line with the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 410C1-410C4.

[0082] With respect to the box plane reference BFR2D, the intersections (as described above) are ordered by the controller 10C, e.g., from top to bottom, left to right (or in any other suitable order). The ordered intersections are associated with their corresponding coordinates in a world calibration grid (the origin of the world calibration grid is the box plane reference BFR of the 3D point cloud for each camera 310C1-310C4). The pose defined for the box front face is such that the X axis runs from the top left corner to the top right corner of the front face, the Y axis runs from the top left corner to the bottom left corner of the front face, and the Z axis is the cross product of the X and Y axes. The box plane reference BFR2D is defined in the same way for each robot 14. The pose of the 2D camera 500C with respect to the box plane reference BFR2D is determined by the controller 10C, e.g., using a homography fit between the intersections found in the 2D image(s) and their corresponding world coordinates. The homography fit is performed by the controller 10C in a manner generally similar to that described in Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000, the entire disclosure of which is incorporated herein by reference.

[0083] Once the calibration box 810 is detected in all of the two-dimensional images acquired by the electromagnetic sensor (e.g., the two-dimensional camera 500C), the pose of the calibration box 810 relative to the reference part RBR of the robot base 14B and the flange 14F relative to the electromagnetic sensor is optimized. Now, referring also to FIG. 20, a flange-to-box-center pose reference part FBCPR is defined at the center of the calibration box 810 in the flange reference part RFR. The rotation axis is in the same general position as the flange axis Fθ (see FIG. 3D), but is optimized independently. To define the flange-to-box-center pose reference part FBCPR, the pose of the calibration box 810 is nominally defined and then refined during optimization. As described herein, the calibration box 810 is invariant with respect to the robot 14 holding the calibration box 810, with the Y axis always pointing toward the front of the calibration box 810, the Z axis pointing downwards, and the X axis pointing toward the right side (as viewed in the two-dimensional images) of the calibration box 810. The X, Y, and Z axes are aligned with the edges of the calibration box 810 and are therefore not in the same orientation as the flange reference portion RFR on which the flange-to-box center pose reference portion FBCPR is defined.

[0084] 21A and 21B, a relationship between the pose of grid 815 and flange-to-box-center pose reference FBCPR (e.g., pose grid-to-box reference PGBR) is established for at least the top and front faces of calibration box 810. In some aspects, a relationship may also be established for the sides of calibration box 810. With reference to FIG. 21A, as an example, for the front face, the pose grid-to-box reference PGBR(front face) is defined at the first top-left intersection (as seen in the two-dimensional image) of grid 815 (see FIG. 21A), where the X-axis points toward the right face of calibration box 810, the Y-axis points downward, and the Z-axis points into calibration box 810, as shown in FIG. 21A. With reference to FIG. 21B, for the top face, the pose grid-to-box reference PGBR(top face) is defined at the intersection of the top grid 815 closest to the box center with the left face of calibration box 810. The axis rotation of the pose grid pair box reference portion PGBR (top surface) is the same as the flange pair box center pose reference portion FBCPR, where the Z axis points down, the X axis points toward the right surface of the calibration box 810, and the Y axis points toward the front surface.

[0085] 22, in a fully automated calibration of the robot(s) 14, a box-to-2D camera pose or reference part BCP is also defined. The box-to-2D camera reference part BCP is defined such that the X-axis runs from the top left corner to the top right corner of the calibration box (as seen in the 2D image), the Y-axis runs from the top left corner to the bottom left corner of the calibration box 810, and the Z-axis points as the cross product of the X- and Y-axes. The box-to-2D camera pose or reference part BCP is defined in the same way for each robot 14 gripping the calibration box 810.

[0086] As can be appreciated, fully automated robot calibration (i.e., localization and teaching) is performed (substantially in line with the calibration of the electromagnetic sensors 500 and the three-dimensional cameras 310C1-310C4) such that the controller 10C (and the robot 14 it controls) "knows" or is precisely taught the features of the palletizer cell 10, where the resolution of the two-dimensional camera 500C is greater than the resolution of the three-dimensional cameras 310C1-310C4, and it is desired to know the location of the physical items in the two-dimensional images in the three-dimensional world and the transformations between them. Once the above-mentioned references are determined, the controller 10C (or any suitable controller, such as the controller 199C, as described above) is configured to refine one or more of the above-mentioned references. By way of example, the reference optimization algorithm of the controller 10C may utilize, for example, a Levenberg-Marquadt algorithm (or any other suitable optimization algorithm) to create an optimized solution. In the optimization process, the position of the robot 14 is optimized (i.e., located) within the electromagnetic sensor 500 world and simultaneously within the palletizer cell space for each palletizer cell station using collected information (from the reference determination above) including, but not limited to, the box face on which the grid 815 is located, the grid intersections within the pixel, the grid intersections within the grid world, and the position of the robot flange within the base reference RBR.

[0087] The Z-axis position of the robot base origin may also be optimized. For example, if the robot 14 is a four-degree-of-freedom (e.g., 4-axis, X, Y, Z, μ (see FIG. 3D)) robot, the calibration box 810 cannot rotate about the X (e.g., roll) and Y (e.g., pitch) axes, so that the robot base Z-axis origin is not fixed to a specific location. The robot base 14B is not in the field of view of the cameras 310C1-310C4, 500C, but the flange 14F is in the field of view and is therefore detectable. Here, the flange 14F is detected by the cameras 310C1-310C4 and / or the electromagnetic sensor 500, and the height of the flange 14F at the flange reference portion RFR (determined by the controller 10C from the images acquired by the cameras 310C1-310C4 and / or the electromagnetic sensor 500) is compared to a nominal CAD value to fix the robot base Z-axis origin. As described herein, in some embodiments, the flange 14F may form part of a calibration fixture / jig (also referred to herein as the target object).

[0088] The above description of FIG. 14, block 1431, which automatically calibrates the robot 14 and synchronizes the axes of the robot 14 with the cameras 310C1-310C4, is repeated for each robot 14 in the palletizer cell 10.

[0089] Still referring to Figures 14 and 3D, each robot 14 includes a distance sensor 411, as described herein. The distance sensor 411 is calibrated so that it can be utilized to scan and calibrate the picking position of the conveyor 16 (Figure 14, block 1432). Here, as described herein, the fork reference part TFRP is common to the robot 14 and the three-dimensional cameras 310C1-310C4. The controller 10C registers and calibrates the fork reference part TFRP in the palletizer cell space, the three-dimensional cameras 310Ca-310C4 space, and the electromagnetic sensor 500 space (e.g., utilizing images of the transport mechanism 20 acquired by one or more of the three-dimensional cameras 310C1-310C4). The controller 10C teaches the robot 14 the position of the reference part TFRP and the position of the pallet support part SPAL (see Figure 14, block 1436). The pose and position of the distance sensor 411 are registered and calibrated via a calibration movement of the respective robot 14 that moves the distance sensor 411 to a picking position. For example, the end-of-arm tool 410 is moved to a position such that the light 411L emitted from the distance sensor 411 is projected onto a flat surface (e.g., as a "spot" of light), such as the top surface of the calibration box 810 (see also FIG. 5B) or the top surface of the transfer forks 20F1, 20F2 with the calibration box 810 seated on one or more of the transfer forks 20F1, 20F2 (assuming that the fineness of the planar feature surface is or can be determined in advance, such as by homography via the electromagnetic sensor 500, for precise positioning of the spot of light projected by the emitter of the distance sensor 411 onto the calibration box 810 or the forks 20F1, 20F2 commensurate with the positioning accuracy of the robot flange 14F provided by the drive section encoder). As can be seen, the calibration box 810 (or the forks 20F1, 20F2) presents / defines a plane PLN (on which the light 411L is projected) that can be determined via homography using the electromagnetic sensor 500. The position of the plane PLN is determined, and the distance (e.g., ΔZ) from the plane PLN to the flange reference portion RFR (or from the plane PLN to the fork reference portion TFR) is determined.The distance ΔZ from the plane PLN to the flange reference part RFR (or fork reference part TFR) determined from the distance sensor 411 is compared to a distance determined (e.g., via homography) between the plane PLN and the flange reference part RFR (or fork reference part TFR) to correct the distance sensor 411 relative to the robot flange reference part RFR / robot flange 14F (or fork reference part TFR). When the light 411L is projected onto the calibration box 810, the end-of-arm tool 410 is rotated about the flange axis Fθ and the end-of-arm tool 410 is moved to a different height relative to the transfer forks 20F1, 20F2. The two-dimensional camera 500C images the projection of the light 411L on the calibration box 810 to find the position of the distance sensor 411 on the end-of-arm tool 410 relative to the flange 14F (e.g., based on the change in the position of the projection of the light 411L on the calibration box 810) and correlates the distance measured by the distance sensor 411 with the height of the flange 14F to calibrate the distance sensor 411 relative to the flange 14F (and the tool reference point RTR). This calibration of the distance sensor 411 is performed for each robot 14 of the palletizer cell 10. The corrected distance sensor 411 (e.g., relative to the flange reference point RFR) can then be utilized to scan the forks 20F1, 20F2 to calibrate the distance from the seating surface(s) of the forks 20F1, 20F2 to the flange reference point RFR and fine-tune the pose of the forks 20F1, 20F2 (Z-axis) relative to the robot(s) 14. To address bias in the pose of the robot 14, multiple scans of the forks 20F1, 20F2 with the distance sensor 411 at different heights and different orientations (eg, ±90° end-of-arm tool rotation) may be utilized.

[0090] 6 and 25, the distance sensor 411 is utilized to scan each picking conveyor 16 and calibrate the picking position of the picking conveyor 16 (FIG. 14, block 1433). The controller 10C commands the movement of the robot 14 to position the end-of-arm tool 410 over the picking conveyor 16 such that the distance sensor 411 scans, for example, the roller 16R of the picking conveyor 16 along the longitudinal axis of the picking conveyor 16. Here, the distance sensor 411 determines the position of the roller 16R (see FIG. 25). The squareness of the conveyor 16 can also be determined by scanning the roller 16R at two or more positions along the length of the roller 16R (such as scanning the end of the roller 16R adjacent to the side of the conveyor 16 (as shown by the arrow in FIG. 6)). Multiple scans of roller 16R can be compared and correlated with robot base 14B to determine the angle of roller 16R, and therefore conveyor 16, relative to robot base reference RBR. Multiple scans of roller 16R can be performed with distance sensor 411 at different heights and different orientations (e.g., ±90° end-of-arm tool rotation) to resolve biases in the pose of robot 14.

[0091] Once the position (and angle) of the roller 16R is known, the calibration box 810 is placed on the conveyor 16 and the robot 14 is commanded to move the end-of-arm tool 410 so that the distance sensor 411 scans the top surface of the calibration box. The distance sensor 411 data acquired from scanning the top surface of the calibration box 810 on the conveyor is communicated to the vision system 310, which fits a plane to the distance sensor data. Knowing the dimensions of the calibration box 810, the determined position of the plane is used to determine the plane of the roller 16R (and to verify the plane of the roller, which may be determined from the scanning of the roller by the distance sensor 411 (see FIG. 25)). Once the plane of the conveyor 16 and the position of the roller are known, the controller 10C determines the picking position of the conveyor 16. A calibration of the picking position of the conveyor 16 is performed for each robot 14 of the palletizer cell 10.

[0092] In the above calibration, the robot 14 is positioned in the palletizer cell space or reference frame (see FIGS. 3A and 14, block 1438). Now, for each robot 14, the controller 10C maps the path of the robot 14 to the boundaries of the palletizer cell space (determined from the known design configuration of the palletizer cell). The controller 10C calibrates the coordinates of the robot 14 based on the pose of the palletizer cell station and the teaching of the palletizer cell space to the robot 14, as described herein.

[0093] 14 and 2, 5B, and 7, in a fully automatic calibration of the palletizer cell 10, a pallet reference is determined such that the positions of the pallet supports SPAL on the elevator 12 and on the transfer forks 20F1, 20F2 of the empty pallet transport mechanism 20 are known. Here, the pallet supports SPAL are circulated by the forks 20F1, 20F2 of the empty pallet transport mechanism 20 through the palletizer cell 10 to the elevator 12. By way of example, 3-5 pallets are circulated, although in other embodiments less than 3 or more than 5 pallets may be circulated. The pallet supports SPAL are imaged by the vision system 310 with each pallet support SPAL positioned on the forks 20F1, 20F2 and with the pallet support SPAL positioned on the elevator 12. The vision system 310 determines the zero position of the lift 12 and the offset of the pallet support SPAL from an image (e.g., a three-dimensional and / or two-dimensional image) of the pallet support SPAL to result in the placement of case units CU on the pallet support SPAL to build a pallet load PAL.

[0094] The distance 555 between the transfer forks 20F1, 20F2 of the palletizer cell 10 and the floor (see also FIG. 3A ) may also be determined from images of the transfer forks 20F1, 20F2 and the floor 700 acquired by the vision system 310. For example, a plane may be fitted to the point cloud data points corresponding to the floor 700 and the forks 20F1, 20F2 (as described herein) and the distance 555 may be determined from the two planes.

[0095] 14, block 1450, and also with reference to FIGS. 23A-24C, the fingers 450F of the end-of-arm tool 410 are imaged, for example, to optimize the tool center point or tool reference RTR, to account for bending of the fingers 450F under a tool load as may result from gripping a calibration load / fixture CLD (see FIG. 8 - e.g., in one embodiment, defined during calibration of the vision system with the calibration box 810), and to verify the tool reference RTR in an unloaded state (i.e., empty end-of-arm tool 410, a gripped box in a fully retracted position on the fingers (see FIG. 23A), or an ungripsed box held by the end-of-arm tool 410). FIG. 14, block 1450 may be included in a calibration of the robot 14 (e.g., FIG. 14, block 1430) and / or may be performed prior to cycling of the pallet (e.g., FIG. 14, block 1440). To optimize the tool reference RTR, a set / series of images is used that place the calibration load CLD in a series of positions within the field of view of the electromagnetic sensor that best represent the bending caused by gripping the calibration fixture CLD by the end-of-arm tooling 410, allowing optimization of the tool reference RTR in an unloaded state, where the series of images are acquired by the electromagnetic sensor 500 (e.g., two-dimensional images acquired by the two-dimensional camera 500C) and include, but are not limited to, a side view (side view) image of the retracted calibration box 810 (see FIG. 23A), a side view (side view) image of the pushed calibration box 810 (see FIG. 23B where the calibration box 810 has been pushed to the tip of the finger 450F by the pusher assembly 28 of the end-of-arm tooling 410), and a front view of the finger 450F such that only the top of the finger 450F is visible in the image, where the orientation indicator (e.g., side view, front view, etc.) of said images is relative to the field of view of the electromagnetic sensor 500.

[0096] The bending algorithm of the controller 10C utilized to optimize the tool reference RTR calculates a pose transformation between the box 810 held on the seating surface of the "unbent" load bearing member(s) of the end-of-arm tool 410 (here the fingers 450F, see FIG. 23A) and the box 810 held on the seating surface of the "bent" load bearing member (here again the fingers 450F, see FIG. 23B) of the end-of-arm tool 410. As can be appreciated, the "unbent" fingers may not actually be bent, such as when the calibration box 810 is gripped in a fully retracted position (see FIG. 23A) and the fingers 450F are minimally bent (e.g., the tips of the fingers are about 2 mm or less lower than the base of the fingers). The minimal bending of the fingers is within the error of the positioning of the robot 14 (e.g., a positioning error between about 1 mm and about 2 mm) and may be ignored in the calculation of the "unbent" tool reference RTR. Here, the controller 10C uses a flexion algorithm to detect the calibration box 810 in three dimensions (using images taken by cameras 310C1-310C4, 500C) and in the imaged space of the electromagnetic sensor 500 (e.g., in two dimensions for the two-dimensional camera 500C in a manner generally similar to that described herein) and assigns references of the calibration box 810 to the three-dimensional world of the two-dimensional camera 500C in the unflexed and flexed images, where these references are the finger flex box center to electromagnetic sensor referential and the finger not flexed box center to electromagnetic sensor referential, respectively.The finger flexed box center to electromagnetic reference pose and finger unflexed box center to electromagnetic sensor reference pose are such that the Y axis points towards the front of the calibration box 810, the X axis points towards the right side of the calibration box 810, and the Z axis points down with the origin located at the center of the calibration box 810 (see FIGS. 23A and 23B, respectively). A box center flex to box center not flex pose referential is calculated from the finger flexed box center to electromagnetic sensor reference and the finger unflexed box center to electromagnetic sensor reference, resulting in a transformed pose that links the positions of the two calibration boxes.

[0097] Once the finger flexed box center to electromagnetic sensor reference and finger non-flexed box center to electromagnetic sensor reference are assigned, the tool reference RTR is detected in another image where the calibration box 810 is placed as shown in FIG. 23A, but the end-of-arm tool 410 is moved such that the finger 450F is positioned allowing easy image segmentation and detection (as described herein) such that the tool reference RTR is detected in the pixels of the two-dimensional image and projected onto the plane of the Box not flex to electromagnetic sensor, where: Box not flex to electromagnetic sensor = robot base referential RBR to electromagnetic sensor * robot flange to robot base * box center flex to robot flange * box center flex to box center not flex

[0098] In the unbent box-to-electromagnetic sensor pose, the pixel points are projected onto a plane below the finger 450F as it is perceived in the three-dimensional world of the electromagnetic sensor 500 (e.g., a two-dimensional camera). The pixel points are then reprojected onto the top of the finger 450F and then described in flange coordinates before being sent to the robot 14. The bending algorithm may also recalculate the X-axis of the tool reference RTR to measure the squared-ness of the tool reference RTR relative to the flange reference RFR. In this way, the pitch of the grip of the end-of-arm tool 410 to the flange 14F may be resolved.

[0099] 24A-24C, to calibrate the tool reference RTR, the TCP calibration algorithm of the controller 10C commands the movement of the robot 14 to an appropriate detection position while holding the calibration box 810 above it. The detection position is at a position and distance from the cameras 310C1-310C4, 500C, and at a height in the workspace that represents the working height of the tool load (and thus the load bearing surface / member) being built into the pallet (see FIG. 24A). At this position, the top of the calibration box 810 is detected to create a plane 2400 in the world of the electromagnetic sensor 500. Using the created plane 2400, a parallel plane 2401 is at a predetermined distance DP (e.g., corresponding to the placement of the load-bearing surface of the end-of-arm tool 410 relative to the tool load, here about 12 inches, or in other embodiments the distance may be greater or less than about 12 inches) below the created plane, corresponding to the load-bearing (here, finger 450F) plane (it is noted that the calibration box 810 has a typical height of about 12 inches). From this image (such as the image in FIG. 24A of the two-dimensional camera 500C), a difference image is acquired by moving the calibration box 810 left / right ( FIG. 24B ) and / or up / down ( FIG. 24C ), where the difference image is utilized by the controller 10C to detect the edge of the finger 450F. In FIG. 24B , the calibration box 810 (and finger 450F) is moved left / right by a distance of about 1 mm (in other embodiments the distance may be greater or less than about 1 mm) and a difference image is acquired. In FIG. 24C, the calibration box 810 (and fingers 450F) are moved up / down a distance of about 2 mm (in other embodiments, the distance may be greater or less than about 2 mm) and a difference image is taken.

[0100] The controller 10C analyzes the difference images using any suitable image analysis algorithm to detect the tips of the fingers 450F as well as the orientation of the fingers. For example, the left / right difference images are analyzed to find the fingers via geometry (e.g., the fingers are the first object encountered in the image depending on the robot, and the images are analyzed from bottom to top). Once the fingers are identified, both edges of each finger 450F are found and the centerline of each finder is determined. Using the centerline, a projection area is set in the top / bottom different images and the edges of each finger are found and projected to the centerline. Once the pixels of the finger edges are found, the pixels are projected onto the plane 2401 to obtain a point in the three-dimensional world of the electromagnetic sensor 500 (here, for example, the two-dimensional camera 500C), where the reference point is transformed through the following chain: TCPInFlangeRef = PoseRobotBaseToFlange × PoseCam2DToRobotBase × TCPInCam2DRef

[0101] When the Tool Reference RTR is written to the Flange Reference RFR, the Tool Reference RTR is valid for the entire working area. Axes for the Tool Reference RTR are obtained as well. For example, the centerline of the finger is projected onto the plane 2401 and serves as the X-axis for the end-of-arm tool 410. The Z-axis is normal to the plane of the finger 2401 except that it points down, and the Y-axis is perpendicular to both the X-axis and the Z-axis.

[0102] 3A-3C, 5B, 7, and 17, the elevator 12 has an elevator origin reference EOR located substantially in the center of the opening 44 of the elevator 12. The location of the elevator origin reference EOR may be defined by the nominal dimensions of the palletizing cell 10 (e.g., from a computer-aided drafting (CAD) file) or in any other suitable manner. The elevator origin reference EOR is common to each of the robots 14 and provides for the synchronization or otherwise connection of the workspaces of the robots 14 (in addition to or instead of the telescopic fork origin reference TFRP). The origin of the elevator origin reference EOR is located at a predetermined distance from the robot base reference RBR in the X direction, and the Y origin is located substantially at the midpoint between the robot base references RBR. The elevator origin reference EOR is projected onto the plane of the empty pallet transport mechanism 20 (see FIG. 17 - such that the Z-axis origin is on the top surface of the empty pallet transport mechanism 20). The elevator origin reference EOR is utilized to map the workspace of the palletizer cell 10 .

[0103] 18A and 18B, an elevator origin reference EOR is also defined in each of the three-dimensional cameras 310C1-310C4. The elevator origin reference EOR is utilized by the controller 10C to segment the point cloud of each three-dimensional camera 310C1-310C4. Here, the elevator origin reference EOR locates the opening 44 of the elevator 12 (and its sides 44S1-44S4 / boundaries) within the workspace of the palletizer cell 10 relative to each of the three-dimensional cameras 310C1-310C4. As mentioned above, the elevator origin reference EOR is defined with the Z axis pointing upward from the plane defined by the upper surfaces of the forks 20F1, 20F2, and the X axis aligned with the centerline of the opening 44 (see FIGS. 18A and 18B, the centerline of the opening 44 may coincide with the centerline of the forks 20F1, 20F2) pointing away from the three-dimensional cameras 310C1-310C4. The origin of the elevator origin reference EOR for each respective three-dimensional camera 310C1-310C4 is at the root point of the X and Z planes where the cameras 310C1-310C4 project onto the plane. The Y origin is then aligned with the centerline CL of the opening 44. For example, exemplary images from three-dimensional cameras 310C1 and 310C3 are illustrated in FIGS. 18A and 18B, showing the point clouds of the opening 44 (and its sides 44S1-44S4) detected by the respective cameras and elevator origin reference EOR. Here, the selection of a separate common reference (such as elevator origin reference EOR) for each of the three-dimensional cameras 310C1-310C4 can serve in some embodiments to calibrate the three-dimensional cameras 310C1-310C4 relative to one another by solving the pose of the elevator origin reference EOR and the characteristics from the detection of the pose, given the predetermined positions and known characteristics of the three-dimensional cameras 310C1-310C4. In such a case, the calibration of the three-dimensional cameras corresponds to the placement accuracy of the robot 14 / end effector 410.

[0104] 3A-3D, 5A-13, 15, and 19A-26, a method is provided for automatically building a pallet load PAL of palletized cargo units CU on a pallet support SPAL using a pallet building device (such as a palletizer cell 10). In the method, a pallet building device is provided (FIG. 26, block 2600), the pallet building device including a frame 24 defining a pallet station with a pallet building base 301 for the pallet load support SPAL. An articulated robot 14 is connected to the frame 24 and has a drive section 14DS with multiple independent drive axes (e.g., of the drive unit 14D) arranged in a robot reference frame, and an end effector (e.g., end-of-arm tool 410) configured to hold the palletized cargo units CU during transport. At least one three-dimensional camera 310C1-310C4 is connected to the frame 24 separately and differently from the articulated robot 14. The controller 10C is operatively connected to at least one articulated robot 14 and at least one three-dimensional camera 310C1-310C4, and the controller 10C is configured to control the movement of the articulated robot relative to the pallet building base 301, thereby resulting in a pallet load build of the pallet load PAL. The palletizer cell 10 also includes an automatic calibration system ACS for calibrating the articulated robot 14, where the controller 10C is communicatively connected to the automatic calibration system ACS. In the method, the articulated robot 14 transports and places a pallet load unit CU on a pallet support SPAL to build the pallet load PAL on the pallet building base 301 (FIG. 26, block 2610). The at least one three-dimensional camera 310C1-310C4 generates a three-dimensional image of the pallet support SPAL on the pallet building base 301 and the pallet load build on the pallet support SPAL (FIG. 26, block 2620). The controller 10C automatically calibrates the articulated robot 14 from a substantially uncalibrated state (FIG. 26, block 2630).The controller 10C automatically teaches the articulated robot 14 the position of the pallet station which resolves the position of the end effector (e.g., the end-of-arm tool 410) relative to the pallet station (FIG. 26, block 2640), and places the pallet load unit CU on the pallet support SPAL to build the pallet load PAL. The controller 10C calibrates at least one three-dimensional camera 310C1-310C4 relative to the end effector (e.g., the end-of-arm tool 410) (FIG. 26, block 2650).

[0105] 3A-3D, 5A-13, 15, 19A-25, and 27, a method is provided for automatically building a pallet load PAL of pallet load units CU on a pallet support SPAL using a pallet building device (such as a palletizer cell 10). In the method, a pallet building device is provided (FIG. 27, block 2700), which includes a frame 24 defining a pallet station with a pallet building base 301 for the pallet support SPAL. An articulated robot 14 is connected to the frame 24 and has a drive section 14DS with multiple independent drive axes (e.g., of the drive unit 14D) arranged in a robot reference frame, and an end effector (e.g., an end-of-arm tool 410) configured to hold the pallet load units CU during transport. A controller 10C is operably connected to the at least one articulated robot 14, and an automatic calibration system ACS for calibrating the articulated robot includes at least one three-dimensional camera 310C1-310C4. At least one three-dimensional camera 310C1-310C4 is connected to the frame 24 and configured to three-dimensionally image at least one of the parts of the articulated robot 14 and the pallet support SPAL or the pallet load build on the pallet building base 301. The articulated robot 14 transports and places the pallet load unit CU on the pallet support SPAL so as to build the pallet load PAL on the pallet building base 301 (FIG. 27, block 2710). The controller controls the movement of the articulated robot relative to the pallet building base, thereby resulting in the pallet load build of the pallet load PAL (FIG. 27, block 2720). The controller 10C generates a three-dimensional image of the pallet support SPAL on the pallet building base 301 and the pallet load build on the pallet support SPAL (FIG. 27, block 2730).The controller 10C is communicatively connected to an automatic calibration system ACS and uses the three-dimensional image to automatically calibrate the articulated robot 14 from a substantially uncalibrated state (FIG. 27, block 2740) and automatically teaches the articulated robot 14 datums (as described herein) of the pallet station that characterize the placement positions for each of the pallet load units CU placed by the end effectors that build the pallet load PAL on the pallet support SPAL (FIG. 27, block 2750).

[0106] According to one or more aspects of the present disclosure, there is provided a pallet building device for automatically building a pallet load of pallet cargo item units on a pallet support, the pallet building device comprising: a frame defining a pallet station with a pallet construction base for the pallet support; an articulated robot having a drive section with multiple independent drive axes connected to a frame and disposed in a robot reference frame, the articulated robot configured to transport and place palletized load units on a pallet support and build palletized loads on a pallet build base, the articulated robot having an end effector configured to hold the palletized load units during transport; at least one three-dimensional camera connected to the frame separate and distinct from the articulated robot and positioned to generate a three-dimensional image of the pallet support on the pallet build base and of the pallet load build on the pallet support; a controller operatively connected to the at least one articulated robot and the at least one three-dimensional camera, the controller configured to control movement of the articulated robot relative to the pallet build base, thereby resulting in a pallet load build of the pallet load; An auto-calibration system for calibrating an articulated robot, the auto-calibration system including a controller communicatively connected to the auto-calibration system, the controller configured to automatically calibrate the articulated robot from a substantially uncalibrated state, automatically teach the articulated robot positions of pallet stations that resolve positions of an end effector relative to the pallet stations, and build a pallet load by placing pallet load item units on a pallet support, the controller configured to calibrate at least one three-dimensional camera relative to the end effector. Equipped with.

[0107] According to one or more aspects of the present disclosure, the controller and auto-calibration system are configured to provide auto-calibration and auto-teaching of the articulated robot generally coincident with auto-calibration of each of at least one three-dimensional camera positioned by the articulated robot to image each palletized cargo item unit that constitutes the pallet load.

[0108] In accordance with one or more aspects of the present disclosure, an automated calibration system includes one or more structures and indicia having calibration features connected to an end effector, the calibration features having predetermined characteristics that define a predetermined or known shape or pattern.

[0109] In accordance with one or more aspects of the present disclosure, the calibration feature is dependent on the end effector, or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

[0110] According to one or more aspects of the present disclosure, a calibration feature is positioned relative to an articulated robot such that a predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

[0111] In accordance with one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera and configured to sense the calibration feature, and a controller is communicatively coupled to the electromagnetic sensor and records a sensor signal substantially coincident with the electromagnetic sensor sensing the calibration feature and generating a sensor signal, the sensor signal embodying a predetermined characteristic for a corresponding pose of the articulated robot (and a corresponding pose of the calibration feature due to movement of the articulated robot).

[0112] According to one or more aspects of the present disclosure, a controller is configured to move the articulated robot and position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to solve a pose of the calibration feature relative to the electromagnetic sensor reference frame and determine from the solved pose a transformation between the electromagnetic sensor reference frame and the three-dimensional camera reference frame to calibrate at least one three-dimensional camera.

[0113] According to one or more aspects of the present disclosure, a pallet building apparatus is provided for automatically building a pallet load of pallet cargo item units on a pallet support, the pallet building apparatus comprising: a frame defining a pallet station with a pallet building base for the pallet support; an articulated robot having a drive section connected to the frame and with a plurality of independent drive axes disposed within a robot reference frame, the articulated robot configured to transport and place the pallet cargo item units on the pallet support to build the pallet load on the pallet building base, the articulated robot having an end effector configured to hold the pallet cargo item units during transport; and a controller operatively connected to the at least one articulated robot, the controller configured to control movement of the articulated robot relative to the pallet building base, thereby resulting in a pallet load build of the pallet load, the controller configured to control the ... and an auto-calibration system for calibrating an articulated robot, the auto-calibration system including a controller arranged to generate a three-dimensional image of a pallet support on the frame and of a pallet load build on the pallet support, the at least one three-dimensional camera being connected to the frame and configured to three-dimensionally image at least one portion of the articulated robot and the pallet support or the pallet load build on the pallet build base, the controller being communicatively connected to the auto-calibration system to automatically calibrate the articulated robot from a substantially uncalibrated state using the three-dimensional image and to automatically teach the articulated robot datums of the pallet stations to characterize placement positions for each of the pallet load item units placed with the end effector that builds the pallet load on the pallet support.

[0114] According to one or more aspects of the present disclosure, at least one three-dimensional camera is connected to the frame separate and distinct from the articulated robot.

[0115] According to one or more aspects of the present disclosure, the controller is configured to resolve a position of the end effector relative to the pallet station and build a pallet load by placing pallet cargo item units on the pallet supports.

[0116] According to one or more aspects of the present disclosure, the controller is configured to calibrate the at least one three-dimensional camera relative to the end effector.

[0117] According to one or more aspects of the present disclosure, the controller and auto-calibration system are configured to provide auto-calibration and auto-teaching of the articulated robot generally coincident with auto-calibration of each of at least one three-dimensional camera positioned by the articulated robot to image each palletized cargo item unit that constitutes the pallet load.

[0118] In accordance with one or more aspects of the present disclosure, an automated calibration system includes one or more structures and indicia having calibration features connected to an end effector, the calibration features having predetermined characteristics that define a predetermined or known shape or pattern.

[0119] In accordance with one or more aspects of the present disclosure, the calibration feature is dependent on the end effector, or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

[0120] According to one or more aspects of the present disclosure, a calibration feature is positioned relative to an articulated robot such that a predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

[0121] In accordance with one or more aspects of the present disclosure, an automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera and configured to sense a calibration feature, and a controller is communicatively coupled to the electromagnetic sensor and records a sensor signal substantially coincident with the electromagnetic sensor sensing the calibration feature and generating a sensor signal, the sensor signal embodying a predetermined characteristic for a corresponding pose of the articulated robot.

[0122] According to one or more aspects of the present disclosure, a controller is configured to move the articulated robot and position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to solve a pose of the calibration feature relative to the electromagnetic sensor reference frame and determine from the solved pose a transformation between the electromagnetic sensor reference frame and the three-dimensional camera reference frame to calibrate at least one three-dimensional camera.

[0123] According to one or more aspects of the present disclosure, a method is provided for automatically building a pallet load of pallet cargo items units on a pallet support using a pallet builder, the method including the steps of providing a pallet builder comprising: a frame defining a pallet station with a pallet build base for the pallet support; an articulated robot connected to the frame, the articulated robot having a drive section with a plurality of independent drive axes disposed in a robot reference frame and an end effector configured to hold the pallet cargo items units during transport; at least one three-dimensional camera separate and distinct from the articulated robot and connected to the frame; a controller operatively connected to the at least one articulated robot and the at least one three-dimensional camera, the controller configured to control movement of the articulated robot relative to the pallet build base to thereby effect a pallet load build of the pallet load; and an automatic calibration system for calibrating the articulated robot. the automatic calibration system including a controller communicatively connected to the automatic calibration system; using an articulated robot to transport and place pallet cargo units on a pallet support such that a pallet load is built on the pallet build base; generating three-dimensional images of the pallet support on the pallet build base and of the pallet load build on the pallet support using at least one three-dimensional camera; automatically calibrating the articulated robot from a substantially uncalibrated state using the controller; automatically teaching the articulated robot a position of a pallet station that resolves a position of an end effector relative to the pallet station using the controller, and building the pallet load by placing the pallet cargo units on the pallet support; and calibrating the at least one three-dimensional camera relative to the end effector using the controller.

[0124] According to one or more aspects of the present disclosure, the controller and auto-calibration system provide auto-calibration and auto-teaching of the articulated robot generally in concert with auto-calibration of each of at least one three-dimensional camera positioned by the articulated robot that images each palletized cargo item unit that constitutes the pallet load.

[0125] In accordance with one or more aspects of the present disclosure, an automated calibration system includes one or more structures and indicia having calibration features connected to an end effector, the calibration features having predetermined characteristics that define a predetermined or known shape or pattern.

[0126] In accordance with one or more aspects of the present disclosure, the calibration feature is dependent on the end effector, or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

[0127] According to one or more aspects of the present disclosure, a calibration feature is positioned relative to an articulated robot such that a predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

[0128] In accordance with one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera, and the method further includes using the electromagnetic sensor to sense a calibration feature and using a controller communicatively coupled to the electromagnetic sensor to record a sensor signal that embodies a predetermined characteristic for a corresponding pose of the articulated robot generally coincident with the electromagnetic sensor sensing the calibration feature and generating a sensor signal.

[0129] According to one or more aspects of the present disclosure, the method further includes using a controller to move the articulated robot to position a calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and resolving a pose of the calibration feature relative to the electromagnetic sensor reference frame and determining from the resolved pose a transformation between the electromagnetic sensor reference frame and the three-dimensional camera reference frame to calibrate at least one three-dimensional camera.

[0130] According to one or more aspects of the present disclosure, a method is provided for automatically building a pallet load of pallet cargo units on a pallet support using a pallet builder, the method including the steps of providing a pallet builder comprising: a frame defining a pallet station at a pallet build base for the pallet support; an articulated robot connected to the frame, the articulated robot having a drive section with a plurality of independent drive axes disposed in a robot reference frame and an end effector configured to hold the pallet cargo units during transport; a controller operatively connected to the at least one articulated robot; and an automatic calibration system for calibrating the articulated robot, the automatic calibration system including at least one three-dimensional camera connected to the frame and configured to three-dimensionally image at least one portion of the articulated robot and the pallet support or pallet load build on the pallet build base. The method includes the steps of: transporting and placing pallet load units on a pallet support using an articulated robot to build the pallet load on the pallet build base; controlling, using a controller, the movement of the articulated robot relative to the pallet build base, thereby resulting in a pallet load build of the pallet load; and generating, using the controller, a three-dimensional image of the pallet support on the pallet build base and of the pallet load build on the pallet support, the controller being communicatively connected to an automatic calibration system, the controller using the three-dimensional image to automatically calibrate the articulated robot from a substantially uncalibrated state, and automatically teaching the articulated robot a pallet station datum characterizing a placement position for each of the pallet load units placed with the end effector that builds the pallet load on the pallet support.

[0131] According to one or more aspects of the present disclosure, at least one three-dimensional camera is connected to the frame separate and distinct from the articulated robot.

[0132] According to one or more aspects of the present disclosure, the method further includes using the controller to resolve a position of the end effector relative to the pallet station and build the pallet load by placing pallet cargo item units on the pallet supports.

[0133] According to one or more aspects of the present disclosure, the controller calibrates at least one three-dimensional camera relative to the end effector.

[0134] According to one or more aspects of the present disclosure, the controller and auto-calibration system provide auto-calibration and auto-teaching of the articulated robot generally in concert with auto-calibration of each of at least one three-dimensional camera positioned by the articulated robot that images each palletized cargo item unit that constitutes the pallet load.

[0135] In accordance with one or more aspects of the present disclosure, an automated calibration system includes one or more structures and indicia having calibration features connected to an end effector, the calibration features having predetermined characteristics that define a predetermined or known shape or pattern.

[0136] In accordance with one or more aspects of the present disclosure, the calibration feature is dependent on the end effector, or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

[0137] According to one or more aspects of the present disclosure, a calibration feature is positioned relative to an articulated robot such that a predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

[0138] In accordance with one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera, and the method further includes sensing a calibration feature using the electromagnetic sensor, and a controller is communicatively coupled to the electromagnetic sensor and records a sensor signal that embodies a predetermined characteristic for a corresponding pose of the articulated robot substantially coincident with the electromagnetic sensor sensing the calibration feature and generating a sensor signal.

[0139] According to one or more aspects of the present disclosure, the method further includes using a controller to move the articulated robot to position a calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and resolving a pose of the calibration feature relative to the electromagnetic sensor reference frame and determining from the resolved pose a transformation between the electromagnetic sensor reference frame and the three-dimensional camera reference frame to calibrate at least one three-dimensional camera.

[0140] It should be understood that the foregoing description is merely illustrative of aspects of the present disclosure. Various alternatives and modifications may be contemplated by those skilled in the art without departing from the aspects of the present disclosure. Accordingly, the aspects of the present disclosure are intended to embrace all such alternatives, modifications, and variations that are within the scope of any claims appended hereto. Moreover, the mere fact that different features are recited in mutually different dependent or independent claims does not indicate that a combination of these features cannot be used to advantage and that such combination remains within the scope of the aspects of the present disclosure.

Claims

1. 1. A pallet building apparatus for automatically building a pallet load of palletized luggage units on a pallet support, said pallet building apparatus comprising: a frame defining a pallet station with a pallet construction base for said pallet supports; an articulated robot having a drive section with multiple independent drive axes connected to the frame and positioned in a robot reference frame, the articulated robot configured to transport and place the pallet load unit on the pallet support and build the pallet load on the pallet build base, the articulated robot having an end effector configured to hold the pallet load unit during transport; at least one three-dimensional camera connected to the frame separate and distinct from the articulated robot and arranged to generate a three-dimensional image of the pallet support on the pallet building base and of the pallet load build on the pallet support; a controller operatively connected to at least one articulated robot and to the at least one three-dimensional camera, the controller configured to control movement of the articulated robot relative to the pallet building base, thereby resulting in the pallet load building of the pallet load; an automatic calibration system for calibrating the articulated robot, the controller being communicatively connected to the automatic calibration system, the controller being configured to automatically calibrate the articulated robot from a substantially uncalibrated state, automatically teach the articulated robot positions of the pallet stations that resolve positions of the end effector relative to the pallet stations, and build the pallet load by placing the pallet cargo item units on the pallet supports; and the controller being configured to calibrate the at least one three-dimensional camera relative to the end effector. A pallet building device comprising:

2. 2. The pallet building apparatus of claim 1, wherein the controller and the automatic calibration system are configured to provide automatic calibration and automatic teaching of the articulated robot in substantial coincidence with automatic calibration of each of the at least one three-dimensional camera that images each pallet cargo unit positioned by the articulated robot that builds the pallet load.

3. The pallet construction device of claim 1 , wherein the automatic calibration system includes one or more of structures and indicia having calibration features connected to the end effector, the calibration features having predetermined characteristics defining a predetermined or known shape or pattern.

4. 4. The pallet construction apparatus of claim 3, wherein the calibration feature is dependent on the end effector or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

5. The pallet construction apparatus of claim 3 , wherein the calibration features are positioned relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration features and the robot reference frame.

6. 4. The pallet construction apparatus of claim 3, wherein the automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera and configured to sense the calibration features, and the controller is communicatively coupled to the electromagnetic sensor and records the sensor signal embodying the predetermined characteristic for a corresponding pose of the articulated robot substantially coincident with the electromagnetic sensor sensing the calibration features and generating a sensor signal.

7. 7. The pallet construction apparatus of claim 6, wherein the controller is configured to move the articulated robot to position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to solve a pose of the calibration feature relative to an electromagnetic sensor reference frame and determine from the solved pose a transformation between the electromagnetic sensor reference frame and a three-dimensional camera reference frame to calibrate the at least one three-dimensional camera.

8. 1. A pallet building apparatus for automatically building a pallet load of palletized luggage units on a pallet support, said pallet building apparatus comprising: a frame defining a pallet station with a pallet construction base for said pallet supports; an articulated robot having a drive section with multiple independent drive axes connected to the frame and positioned in a robot reference frame, the articulated robot configured to transport and place the pallet load unit on the pallet support and build the pallet load on the pallet build base, the articulated robot having an end effector configured to hold the pallet load unit during transport; A controller, the controller is operatively connected to at least one articulated robot, the controller being configured to control movement of the articulated robot relative to the pallet building base, thereby resulting in a pallet load build of the pallet load; the controller is arranged to generate a three-dimensional image of the pallet support on the pallet build base and of the pallet load build on the pallet support. A controller; an automatic calibration system for calibrating the articulated robot, the automatic calibration system including at least one three-dimensional camera connected to the frame and configured to three-dimensionally image at least one portion of the articulated robot and the pallet support or the pallet load build on the pallet build base; Equipped with The controller is communicatively connected to the automatic calibration system, and uses the three-dimensional image to automatically calibrate the articulated robot from a substantially uncalibrated state, and automatically teach the articulated robot datums of the pallet stations to characterize placement positions for each of the pallet cargo item units placed by the end effector that builds the pallet load on the pallet support.

9. The pallet construction apparatus of claim 8 , wherein said at least one three-dimensional camera is connected to said frame separate and distinct from said articulated robot.

10. 9. The pallet building apparatus of claim 8, wherein the controller is configured to resolve a position of the end effector relative to the pallet station to build the pallet load by placing the pallet cargo item units on the pallet supports.

11. The pallet construction device of claim 8 , wherein the controller is configured to calibrate the at least one three-dimensional camera relative to the end effector.

12. 9. The pallet building apparatus of claim 8, wherein the controller and the automatic calibration system are configured to provide automatic calibration and automatic teaching of the articulated robot in substantial coincidence with automatic calibration of each of the at least one three-dimensional camera that images each pallet cargo unit positioned by the articulated robot that builds the pallet load.

13. 9. The pallet construction apparatus of claim 8, wherein the automatic calibration system includes one or more of structures and indicia having calibration features connected to the end effector, the calibration features having predetermined characteristics defining a predetermined or known shape or pattern.

14. The pallet construction apparatus of claim 13, wherein the calibration feature is dependent on the end effector or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

15. The pallet construction apparatus of claim 13 , wherein the calibration features are positioned relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration features and the robot reference frame.

16. 14. The pallet construction apparatus of claim 13, wherein the automatic calibration system includes an electromagnetic sensor separate and distinct from the three-dimensional camera and configured to sense the calibration features, and the controller is communicatively coupled to the electromagnetic sensor and records the sensor signal embodying the predetermined characteristic for a corresponding pose of the articulated robot substantially coincident with the electromagnetic sensor sensing the calibration features and generating a sensor signal.

17. 17. The pallet construction apparatus of claim 16, wherein the controller is configured to move the articulated robot to position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to solve a pose of the calibration feature relative to an electromagnetic sensor reference frame and determine from the solved pose a transformation between the electromagnetic sensor reference frame and a three-dimensional camera reference frame to calibrate the at least one three-dimensional camera.

18. 1. A method for automatically building a pallet load of palletized cargo units on a pallet support using a pallet building device, the method comprising: Providing a pallet construction device, the pallet construction device comprising: a frame defining a pallet station with a pallet construction base for said pallet supports; an articulated robot connected to the frame, the articulated robot having a drive section with multiple independent drive axes disposed in a robot reference frame, and an end effector configured to hold the palletized cargo item unit during transport; At least one three-dimensional camera connected to the frame, separate and distinct from the articulated robot; a controller operatively connected to at least one articulated robot and to the at least one three-dimensional camera, the controller configured to control movement of the articulated robot relative to the pallet building base, thereby resulting in a pallet load build of the pallet load; an automatic calibration system for calibrating the articulated robot, the controller being communicatively connected to the automatic calibration system; A process comprising: using the articulated robot to transport and place the pallet load unit onto the pallet support so as to build the pallet load on the pallet build base; generating, using the at least one three-dimensional camera, a three-dimensional image of the pallet support on the pallet build base and of a pallet load build on the pallet support; automatically calibrating the articulated robot from a substantially uncalibrated state using the controller; using the controller to automatically teach the articulated robot the location of the pallet station which resolves the location of the end effector relative to the pallet station and builds the pallet load by placing the pallet cargo item units on the pallet supports; calibrating the at least one three-dimensional camera with the controller relative to the end effector; A method comprising:

19. 20. The method of claim 18, wherein the controller and the auto-calibration system effect auto-calibration and auto-teaching of the articulated robot substantially coincident with auto-calibration of each of the at least one three-dimensional camera that images each palletized cargo unit positioned by the articulated robot that constitutes the pallet load.

20. 20. The method of claim 18, wherein the automated calibration system includes one or more of structures and indicia having calibration features connected to the end effector, the calibration features having predetermined characteristics defining a predetermined or known shape or pattern.

21. 21. The method of claim 20, wherein the calibration feature is dependent on the end effector or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

22. The method of claim 20 , wherein the calibration feature is positioned relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

23. The automatic calibration system includes an electromagnetic sensor that is separate and distinct from the three-dimensional camera, and the method further comprises: sensing the calibration features with the electromagnetic sensor; using the controller communicatively coupled to the electromagnetic sensor, substantially coincident with the electromagnetic sensor sensing the calibration feature to generate a sensor signal, the sensor signal embodying the predetermined characteristic for a corresponding pose of the articulated robot; 21. The method of claim 20, further comprising:

24. Using the controller, moving the articulated robot to position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature; resolving a pose of the calibration features relative to an electromagnetic sensor reference frame and determining from the resolved pose a transformation between the electromagnetic sensor reference frame and a three-dimensional camera reference frame to calibrate the at least one three-dimensional camera; 24. The method of claim 23, further comprising:

25. 1. A method for automatically building a pallet load of palletized cargo units on a pallet support using a pallet building device, the method comprising: Providing a pallet construction device, the pallet construction device comprising: a frame defining a pallet station with a pallet construction base for said pallet supports; an articulated robot connected to the frame, the articulated robot having a drive section with multiple independent drive axes disposed in a robot reference frame, and an end effector configured to hold the palletized cargo item unit during transport; a controller operatively connected to the at least one articulated robot; an automatic calibration system for calibrating the articulated robot, the automatic calibration system including at least one three-dimensional camera connected to the frame and configured to three-dimensionally image at least one portion of the articulated robot and the pallet support or pallet load build on the pallet build base; A process comprising: using the articulated robot to transport and place the pallet load unit onto the pallet support so as to build the pallet load on the pallet build base; using said controller to control movement of an articulated robot relative to said pallet building base, thereby resulting in a pallet load build of said pallet load; generating, using the controller, a three-dimensional image of the pallet support on the pallet build base and of the pallet load build on the pallet support; Including, The method includes a controller communicatively connected to the automatic calibration system, the controller using the three-dimensional image to automatically calibrate the articulated robot from a substantially uncalibrated state, and automatically teaching the articulated robot a pallet station datum characterizing a placement position for each of the pallet cargo item units placed by the end effector that builds the pallet load on the pallet support.

26. 26. The method of claim 25, wherein the at least one 3D camera is connected to the frame separate and distinct from the articulated robot.

27. 26. The method of claim 25, further comprising using the controller to resolve a position of the end effector relative to the pallet station to build the pallet load by placing the pallet cargo unit on the pallet support.

28. The method of claim 25 , wherein the controller calibrates the at least one three-dimensional camera relative to the end effector.

29. 26. The method of claim 25, wherein the controller and the auto-calibration system effect auto-calibration and auto-teaching of the articulated robot substantially coincident with auto-calibration of each of the at least one three-dimensional camera that images each palletized cargo unit positioned by the articulated robot that constitutes the pallet load.

30. 26. The method of claim 25, wherein the automated calibration system includes one or more of structures and indicia having calibration features connected to the end effector, the calibration features having predetermined characteristics defining a predetermined or known shape or pattern.

31. 31. The method of claim 30, wherein the calibration feature is dependent on the end effector or the calibration feature and the end effector are commonly dependent on a common member of the articulated robot, and the calibration feature and the end effector move as a unit relative to the frame with each joint movement of the articulated robot.

32. The method of claim 30 , wherein the calibration feature is positioned relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

33. The automatic calibration system includes an electromagnetic sensor that is separate and distinct from the three-dimensional camera, and the method further comprises: further comprising sensing the calibration features with the electromagnetic sensor; 31. The method of claim 30, wherein the controller is communicatively coupled to the electromagnetic sensor and records the sensor signal substantially coincident with the electromagnetic sensor sensing the calibration feature and generating a sensor signal, the sensor signal embodying the predetermined characteristic for a corresponding pose of the articulated robot.

34. Using the controller, moving the articulated robot to position the calibration feature at an initial position where the electromagnetic sensor senses the calibration feature; resolving a pose of the calibration features relative to an electromagnetic sensor reference frame and determining from the resolved pose a transformation between the electromagnetic sensor reference frame and a three-dimensional camera reference frame to calibrate the at least one three-dimensional camera; 34. The method of claim 33, further comprising:

Images