Apparatus and method for automatic pallet builder calibration

Abstract

A pallet building apparatus for automatically building a pallet load of a pallet load item unit onto a pallet support is disclosed, comprising: a frame defining a pallet station having a pallet building base for the pallet support; an articulated robot connected to the frame; at least one three-dimensional camera connected to the frame, separate and distinct from the articulated robot, and arranged to generate a three-dimensional image of a pallet support on a pallet building base and a pallet loading build on the pallet support; a controller operably connected to the at least one articulated robot and the at least one three-dimensional camera; and an automatic calibration system for calibrating the articulated robot, the controller communicatively connected to the automatic calibration system and configured to automatically calibrate the articulated robot from a substantially uncalibrated state and to automatically teach a position of the articulated robot pallet station.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of and claims the benefit of U.S. Provisional Application No. 63 / 310,450, filed on February 15, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed embodiments and aspects thereof relate generally to storage and retrieval systems (storage and retrieval systems), and more particularly to palletizing / depalletizing cells of storage and retrieval systems. Background Art

[0004] Product retail distribution (whether conventional "brick and mortar" stores, online stores or hybrid retail channels) requires improved storage, sorting and transportation efficiency, especially for the distribution of so-called mixed boxes or heterogeneous boxes (within a given transportation range), whether for store replenishment or personal orders. The application of intelligent / adaptive automation to it has increasingly promoted the improvement of efficiency at multiple distribution levels including storage, sorting and transportation. However, the persistent problems still produce bottlenecks that destroy or adversely affect the overall system efficiency, such as mixed box pallet loading (or truck loading) efficiency problems. As can be easily recognized, the difficulty of pallet loading (or truck loading) efficiency problems does not only stem from the expectation of high packaging density, but the pallet loading efficiency depends on both packaging density and building pallet loading (pallet load) in a time-optimal manner (i.e., given any necessary time and necessary mixed box selection, the construction problem of packing pallet loading to a density of more than 90% can be easily solved, but if the pallet loading construction time is not time-optimal, such pallet loading will not be efficient).

[0005] Calibration of conventional pallet loaders (e.g., robotic palletizers) may affect pallet building efficiency, thereby creating a bottleneck in terms of system throughput efficiency. Pallet loaders are typically calibrated with respect to the location of features of the pallet loader workspace to which the pallet loader interfaces (e.g., pallet position, conveyor position, etc.). Conventionally, calibration of pallet loaders is manually performed by pallet loader experts in a time-consuming and complex process. Since each pallet loader is calibrated by a person, calibration is subjective and may be inconsistent, which depends largely on the thoroughness and skill of the person performing the calibration. In addition, due to the subjective nature of the manual calibration process, it is difficult to determine the root cause of palletizing inaccuracies. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing aspects and other features of the disclosed embodiments are explained in the following description in conjunction with the accompanying drawings, in which:

[0007] Figure 1 is a schematic illustration of a distribution facility according to aspects of the disclosed embodiments;

[0008] Figure 2 is a schematic illustration of a pallet load according to aspects of the disclosed embodiments;

[0009] Figure 3A According to aspects of the disclosed embodiments Figure 1 A top perspective illustration of a palletizer unit of FIG. 1 , wherein the system is shown with the empty palletizer in its upper position and the palletizer in its lower position;

[0010] Figure 3B According to aspects of the disclosed embodiments Figure 3A Another top perspective illustration of a palletizer unit of FIG. 1 , wherein the system is shown with the empty palletizer in its lower position and the palletizer in its upper position;

[0011] Figure 3C According to aspects of the disclosed embodiments Figure 3A A side perspective illustration of a palletizer unit of FIG. 1 , wherein the system is shown with its telescopic forks extended toward the palletizer unloader;

[0012] Figure 3D According to aspects of the disclosed embodiments Figure 3A A perspective diagram of a palletizing robot for a palletizing system;

[0013] Figure 4 According to aspects of the disclosed embodiments Figure 3A A schematic isometric view of a palletizer unit;

[0014] Figure 4A According to aspects of the disclosed embodiments Figure 4 A schematic exploded isometric view of a palletizer unit;

[0015] Figure 4B According to aspects of the disclosed embodiments Figure 4 A schematic plan view or top view of a palletizer unit;

[0016] Figure 4C According to aspects of the disclosed embodiments Figure 4 A schematic right side view of a palletizer unit;

[0017] Figure 4D According to aspects of the disclosed embodiments Figure 4 A schematic front view of a palletizer unit;

[0018] Figure 4E According to aspects of the disclosed embodiments Figure 4A schematic left side view of a palletizer unit;

[0019] Figure 4F According to aspects of the disclosed embodiments Figure 4 A schematic rear or back view of a palletizer unit;

[0020] Figure 4G According to aspects of the disclosed embodiments Figure 4 A schematic isometric view of a palletizer unit;

[0021] Figure 4H According to aspects of the disclosed embodiments Figure 4 A schematic left side view of a palletizer unit;

[0022] Fig. 4I According to aspects of the disclosed embodiments Figure 4 A schematic front view of a palletizer unit;

[0023] Figure 4J According to aspects of the disclosed embodiments Figure 4 A schematic plan view or top view of a palletizer unit;

[0024] Figure 4K According to aspects of the disclosed embodiments Figure 4 A schematic isometric view of a palletizer unit, which focuses on the field of view of the camera of the vision system of the palletizer unit;

[0025] Figure 4L According to aspects of the disclosed embodiments Figure 4 A schematic isometric view of a palletizer unit, which focuses on the field of view of the camera of the vision system of the palletizer unit;

[0026] Figure 4M According to aspects of the disclosed embodiments Figure 4 A schematic isometric view of a palletizer unit, which focuses on the field of view of the camera of the vision system of the palletizer unit;

[0027] Figure 4N According to aspects of the disclosed embodiments Figure 4 A schematic isometric view of a palletizer unit, which focuses on the field of view of the camera of the vision system of the palletizer unit;

[0028] Figure 5A According to aspects of the disclosed embodiments Figure 1 A schematic illustration of a portion of a palletizer unit;

[0029] Figure 5B According to aspects of the disclosed embodiments Figure 1Schematic perspective illustration of a portion of a palletizer unit and a point cloud of the vision system;

[0030] Figure 6 According to aspects of the disclosed embodiments Figure 1 A schematic perspective illustration of a portion of a palletizer unit;

[0031] Figure 7 According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0032] Figure 8 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0033] Fig. 9 According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0034] Fig.10 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0035] Fig.11 According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0036] Fig.12 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0037] Fig.13 According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0038] Fig.14 is an exemplary flow chart of a method according to aspects of the disclosed embodiments;

[0039] Fig.15 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0040] Fig.16 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0041] Fig.17 According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0042] Fig.18A According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0043] Fig.18B According to the disclosed embodiments, Figure 1 Example point cloud obtained by the vision system of a palletizer;

[0044] Fig.19A According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0045] Fig.19B According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0046] Fig. 20 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0047] Fig.21A According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0048] Fig. 21B According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0049] Fig. 22 According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0050] Fig.23A According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0051] Fig. 23B According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0052] Fig.24A According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0053] Fig. 24B According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0054] Fig.24C According to the disclosed embodiments, Figure 1 An exemplary image obtained by a vision system of a palletizer unit;

[0055] Fig.25 According to the disclosed embodiments, Figure 1 Exemplary sensor data obtained from a palletizer unit;

[0056] Fig.26 is an exemplary flow chart of a method according to aspects of the disclosed embodiments; and

[0057] Fig. 27 is an exemplary flow chart of a method according to aspects of the disclosed embodiments. DETAILED DESCRIPTION

[0058] Figure 1 100WS is a schematic illustration of a warehouse system or distribution facility 100WS (referred to herein as warehouse system 100WS) according to aspects of the disclosed embodiments. Although aspects of the disclosed embodiments will be described with reference to the accompanying drawings, it should be understood that aspects of the disclosed embodiments may be embodied in a variety of forms. In addition, elements or materials of any suitable size, shape or type may be used. It should be understood that although the distribution facility 100WS is described herein as an automatic distribution facility, aspects of the disclosed embodiments are also applicable to distribution facilities having any suitable transportation system, such as both automatic and manual transportation systems or a completely manual transportation system.

[0059] refer to Figure 1 and Figure 2According to aspects of the disclosed embodiment, the warehouse system 100WS includes at least one real-time adaptive palletizer / depalletizer unit 10A, 10B (generally referred to herein as a palletizer unit 10). The palletizer unit 10 has one or more robotic case manipulators 14 (also referred to herein as articulated robots or robots), which, with the assistance of a vision system, place (individually or as manufactured picking faces) mixed pallet load item units CU (also referred to herein as case units or cases) into stacks SL1-Sn and / or layers PL1-PL4 to build mixed case pallet loads PAL, such as in the manner described in U.S. Patent No. 10,894,676, issued on January 19, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0060] The palletizer unit 10 (or pallet building device) is provided with a vision system 310, which includes three-dimensional (3D) time-of-flight (TOF) cameras 310C1-310C4 and an electromagnetic sensor 500, which is separate and distinct from the 3D cameras 310C1-310C4 (collectively referred to herein as the vision system 310 - see Figure 3A-4N as well as Figure 5A-5B ) for fully automatic calibration (and maintenance of that calibration) and confirmation of a palletizer unit 10 (eg, from a substantially uncalibrated state) in the manner described herein.

[0061] A three-dimensional (3D) time-of-flight (TOF) camera and an electromagnetic sensor 500 (e.g. a two-dimensional (2D) color camera 500C) are mounted to the frame 24 of the palletizer unit 10, separate and distinct from the robot 14. These three-dimensional cameras may be included in an automatic calibration system ACS and connected to the frame and are configured to perform three-dimensional imaging of at least a portion of the robot 14 and a pallet support SPAL or a pallet load build BPAL on a pallet building base 301. For example, the vision system 310 forms at least a part of the automatic calibration system ACS for calibrating the robot 14 of the palletizer unit 10. As will be described herein, the controller 10C is communicatively connected to the automatic calibration system ACS so as to automatically calibrate the robot 14 from a substantially uncalibrated state and automatically teach the robot the position or reference of the palletizer unit 10 pallet station (i.e., the surface or feature of the palletizer unit pallet station and the position or reference of other features of the palletizer unit pallet in relation to said surface or feature) (e.g., the position of the pallet load support SPAL on the lift 12 and / or the empty pallet conveying mechanism 20) so as to resolve the position of the end effector or end-of-arm tool 410 of the robot 14 to the pallet station to build a pallet load (such as on the pallet load support SPAL) by placing object or case units CU (also referred to herein as pallet load object units) on the pallet load support SPAL. Figure 2). The controller 10C characterizes the placement position of each case unit CU placed by means of the end effector 410, which builds the pallet load PAL on the pallet support SPAL. The controller 10C is configured to calibrate at least the three-dimensional cameras 310C1-310C4 of the vision system 310 relative to the end-of-arm tool 410 based on data obtained, for example, at least from the automatic calibration system ACS. As will be described herein, the controller 10C and the automatic calibration system ACS are configured to enable automatic calibration and automatic teaching of the robot 14, which occurs substantially simultaneously with the automatic calibration of each three-dimensional camera 310C1-310C4 that images each case unit CU that builds the pallet load PAL placed by the robot 14.

[0062] As used herein, calibration includes one or more of the following: automatically calibrating the travel (ie, in the lifting direction) of the pallet lift or unloader 12 (see, e.g., Figures 3A-3C , Figure 4 and Figure 4B - For example, automatically calculating the zero position of the lift 12), automatically calibrating the end-of-arm tool 410 of the robot 14 (see Figure 4 ), automatically determine the push axis PAX of the end-of-arm tool 410 (e.g., automatically confirm / calculate the zero position of the end-of-arm tool 410), automatically teach the reference frame of the robot 14 (e.g., coordinate system and work envelope), automatically teach the work reference frame (also known as pallet offset), and automatically detect calibration errors / mechanical problems (e.g., abnormal robot kinematics, abnormal encoder operation, bent transfer fork 20, bent fingers on the end-of-arm tool 410, skewed elevator 12, uncentered pallet, etc.), automatically define the pick position of the feed conveyor 16, and automatically define a common reference (e.g., for each robot 14) / multiple references of the palletizer unit 10 (e.g., the pallet transfer fork 20 reference frame, the pallet on the fork 20 reference frame, and the pallet on the elevator 12 reference frame).

[0063] The vision system 310 may also be used to confirm the calibration of the robot 14 at any suitable time, such as after a robot movement error and / or periodically. As can be appreciated, automatic calibration of the robot 14 may simultaneously address pallet quality / control and builds achieved with the robot 14. The vision system 310 incorporated into the automated palletizer cell 10 informs and enables the cell controller 10C to provide real-time command inputs (to automated devices such as the robot 14) that are responsive to calibration fixture positions / movements to achieve automatic calibration of the robot 14 in the manner described herein.

[0064] Aspects of the disclosed embodiments enable simplified and accelerated palletizer cell robot 14 calibration and teaching. For exemplary purposes, the robot 14 described herein is a four-degree-of-freedom (4-axis) robot, but aspects of the disclosed embodiments are equally applicable to a six-degree-of-freedom (6-axis) robot, or a robot having any suitable degrees of freedom. Since the calibration and teaching of the robot 14 is fully automatic (e.g., not dependent on human-machine interaction), the calibration / teaching of the robot 14 is consistent between several robots 14 of the same palletizer cell 10 and between robots 14 of different palletizer cells 10 in the same facility (and between robots / palletizer cells of different facilities). Since the calibration / recalibration of the cell 10 is fully automatic, the calibration can be periodically verified to substantially prevent and detect misalignment between palletizer cell 10 components (e.g., misalignment of the robot 14 axis, misalignment of the lift 12 axis, etc.). Data from the calibration / teaching can also be stored in the controller 10C to enable quantitative assessment of the calibration / teaching accuracy.

[0065] Reference again Figure 1 According to aspects of the disclosed embodiments, the distribution facility 100WS includes a storage and retrieval system 100 that can be operated in a retail distribution center or warehouse to, for example, fulfill orders for case units received from retail stores. In one example, the case unit can be a case or cargo unit that is not stored in a pallet, on a tote, or on a pallet (e.g., unboxed). In other examples, the case unit can be a case or cargo unit that is contained in any suitable manner, such as in a pallet, on a tote, or on a pallet. It should be noted that the case unit can include a box-type cargo unit (e.g., a soup can box, a cereal box, etc.) or individual goods suitable for being removed from a pallet or placed on a pallet. According to aspects of the disclosed embodiments, a shipping box for the case unit (e.g., a carton, a barrel, a box, a crate, a pot, or any other suitable device for containing the case unit) can have a variable size and can be used to contain the case unit when shipped, and can be configured so that it can be palletized for shipping. It should be noted that, for example, when bundles or pallets of case units arrive at the storage and retrieval system, the contents of each pallet may be uniform (e.g., each pallet holds a predetermined number of the same items—one pallet holds soup and another pallet holds cereal), and when the pallets leave the storage and retrieval system, the pallets may hold any suitable number and combination of different case units (e.g., each pallet may hold a different type of case unit—a pallet holds a combination of soup and cereal). In aspects of the disclosed embodiments, the storage and retrieval system described herein may be applied to any environment in which case units are stored and retrieved.

[0066] The storage and retrieval system 100 can be configured to be installed in, for example, an existing warehouse structure or to adapt to a new warehouse structure. In aspects of the disclosed embodiments, the storage and retrieval system can include one or more infeed transfer stations 170 and one or more outfeed transfer stations 160, in / out box conveyors 150A, 150B, 150C (commonly referred to as in / out box conveyors 150), a storage structure array 130, and a plurality of autonomous vehicle transport robots 110 (referred to herein as "robots"). In aspects of the disclosed embodiments, the storage and retrieval system can also include a robot or a robot transfer station, as described in U.S. Patent No. 9,096,375, issued on August 4, 2015, the disclosure of which is incorporated herein by reference in its entirety. In aspects of the disclosed embodiments, the robot transfer station can provide an interface between the robot 110 and the in / out box conveyor 150, so that box units can be indirectly transferred between the robot 110 and the in / out box conveyor 150 through the robot transfer station. In aspects of the disclosed embodiment, case units may be transferred directly between the robot 110 and the in / out case conveyor 150 .

[0067] The storage structure array 130 may include multi-level storage rack modules that form a storage array of storage locations 130SL for case units, each of which is arranged to store at least one case unit at each storage location 130SL. In one aspect, each level of the storage structure array 130 includes a corresponding storage / picking channel 130A and a transfer platform 130B for transferring case units between any storage area of ​​the storage structure array 130 and any shelf of any in / out case conveyor 150. The storage channel 130A and the transfer platform 130B are also configured to allow the robot 110 to pass through the storage channel 130A and the transfer platform 130B to place the case unit into the picking inventory and retrieve the ordered case unit, wherein the case unit is stored or otherwise maintained in the storage channel 130A and / or the storage location 130SL on the transfer platform 130B. The robot 110 can be any suitable robot capable of carrying and transferring case units throughout the storage and retrieval system 100. For exemplary purposes only, suitable examples of robots may be found in the following U.S. patents: U.S. Patent No. 8,425,173 issued on April 23, 2013, U.S. Patent No. 9,561,905 issued on February 7, 2017, U.S. Patent No. 8,965,619 issued on February 24, 2015, U.S. Patent No. 8,696,010 issued on April 15, 2014, and U.S. Patent No. 14,836,979 entitled “Automated Bot with Transfer Function” filed on December 15, 2011. No. 9,187,244, issued in November as 113 / 326,952 to "Arm" (which is a non-provisional application of U.S. Serial No. 61 / 423,365 filed on December 15, 2010), and U.S. Patent No. 9,499,338, issued on November 22, 2016, the disclosures of which are incorporated herein by reference in their entirety. The robot 110 can be configured to place case units (such as the above-mentioned retail items) into the picking inventory in one or more levels of the storage structure array 130, and then selectively pick up units of the ordered cases so that the ordered case units can be shipped to, for example, a store or other suitable location.

[0068] The in-feed transfer station 170 and the out-feed transfer station 160 can operate with their respective in / out case conveyors 150A, 150B to bi-directionally transfer case units to and from one or more levels of the storage structure array 130, thereby enabling the infeeding of case units into the storage structure array 130 and the output of case units from the storage structure array 130. It should be noted that while the in-feed transfer station 170 and the out-feed transfer station 160 (and their respective in / out case conveyors 150A, 150B and palletizer / depalletizer units 10A, 10B) are described as dedicated in-station (e.g., in-feed) transfer stations 170 and dedicated out-station (e.g., out-feed) transfer stations 160, in aspects of the disclosed embodiments, each of the transfer stations 170, 160 can be used for both in-station and out-station transfer of case units from the storage and retrieval system. It should be noted that although an in / out box conveyor is described herein, the conveyor may be any suitable conveyor (including any suitable transport path orientation, such as vertical and / or horizontal conveyor paths) or a transfer / picking device having any suitable transport path orientation.

[0069] In one aspect, as described above, each of the infeed transfer station 170 and the outfeed transfer station 160 includes a respective in / out case conveyor 150A, 150B and a respective palletizer / depalletizer unit 10A, 10B (generally referred to herein as a palletizer unit 10). In one aspect, the palletizer / depalletizer unit 10 is an automated unit that is each configured to receive a loaded pallet (such as with uniform or mixed case units or products) from, for example, a pallet loading 175 area that may include an in / out loaded pallet conveyor 175C (at Figure 1 and / or constructing loaded pallets (such as with uniform or mixed case units or products) for transport to, for example, a pallet out 180 area, which may include an in-and-out loaded pallet conveyor 180C (in Figure 1175C, 180C are each connected to the storage structure array 130 and are configured to transport cargo pallets bidirectionally in an input direction toward the storage structure array 130 and in different output directions away from the storage structure array 130. In one aspect, the conveyors 175C, 180C can each include a conveyor device having a distributed conveyor bed that is arranged to form a conveying path; or in other aspects, the conveyors 175C, 180C can be discrete transport units, such as, for example, forklifts / pallet trucks. Suitable examples of automated palletizer / depalletizer units 10A, 10B can be found in U.S. Patent No. 10,370,199, issued on August 6, 2019, and U.S. Patent No. 8,965,559, issued on February 24, 2015, the disclosures of which are incorporated herein by reference in their entirety. Each palletizer unit includes one or more robotic case manipulators 14, which may also be referred to as articulated robots or robots. As described herein, the one or more robotic case manipulators 14 are configured to continuously transport and place pallet load item units CU (also referred to herein as cases or case units) onto the pallet supports so as to build the pallet load 250 on the pallet building base 301 (see FIG. 3 ).

[0070] In the case where the palletizer unit 10 is used for the output role as a palletizer, palletized item units CU, which may have various sizes, arrive at the palletizer unit 10 via the in / out case conveyor 150B, are picked up by a robotic case manipulator 14 and placed on the pallet PAL, as will be described herein. In the case where the palletizer unit 10 is used for the output role as a palletizer, a full pallet PAL made of various case units (see Figure 2 ) are ready to be picked up by a forklift from the palletizer unit 10 for transport to the pallet load-out 180 area. In the case of the palletizer / depalletizer unit 10 used for input function as a depalletizer, full pallets made of a variety of palletized item units CU (which may be similar to a pallet PAL and formed of homogeneous or mixed boxes) are transferred from the pallet load-in 175 area to the palletizer unit 10 by any suitable means (such as a forklift). One or more robotic case manipulators 14 pick up the palletized item units CU from the pallet PAL for transfer to the storage structure array 130.

[0071] In one aspect, each infeed transfer station 170 forms a case input path Ip, where the palletizer / depalletizer unit 10A depalletizes case units layer by layer, or otherwise depalletizes case units into single case units from a standard pallet (e.g., a homogeneous pallet having stability suitable for automatically engaging pallet layers by an automatic layer interface unit, such as a product picking device 14). The palletizer / depalletizer unit 10A communicates with a transport system (such as an in / out case conveyor 150A) of the automated storage and retrieval system 100 to form an integrated input system (e.g., an infeed transfer station 170) that feeds case units into the automated storage and retrieval system 100. Each feed transfer station 170 defines a case input path Ip that is integrated with the automated storage and retrieval system 100 and the warehouse management system 199, wherein the warehouse management system 199 includes any suitable controller 199C, which is configured with any suitable non-transitory program code and memory to manage at least case unit input to the storage structure array 130B, case unit storage distribution within the storage structure array 130B, and case unit retrieval, case unit inventory / replenishment and case unit output from the storage structure array 130B.

[0072] In one aspect, each case unit input path Ip includes at least one corresponding case unit inspection unit 142 in communication with the warehouse management system 199. In one aspect, the at least one corresponding case unit inspection unit 142 can be any suitable inspection unit, including any suitable volumetric inspection, such as by means of a multi-dimensional light curtain, an imaging system, and / or any other suitable sensing / sensor device configured to detect case unit defects and identify case units, such as inventory, shipping sequencing, storage distribution, and sequencing of case units output from the storage structure array 130B.

[0073] In one aspect, as described above, the palletizer / depalletizer unit 10A may be fully automated to remove or deactivate a layer(s) from a pallet being unloaded at the palletizer / depalletizer unit 10A. Figure 2 , the term decommissioning refers to the removal of a pallet layer PL1, PL2, PL3, PL4 (in whole or in part) from the pallet PAL, such that at a predetermined level 200 of the pallet PAL (which may correspond to a decommissioning / commissioning level or transfer plane) each pallet load unit CU is removed from the layers PL1, PL2, PL3, PL4, such that the pallet PAL is directed to the next level of the pallet PAL (e.g. by means of a pallet lift PLFT – see Figure 4 ) in order to remove the next layer PL2, PL3 (in whole or in part) corresponding to the next level of the pallet PAL.

[0074] In one aspect, the palletizer / depalletizer unit 10A is configured to deactivate the levels PL1, PL2, PL3, PL4 such that the deactivation is synchronized or otherwise coordinated (e.g., matched) by the warehouse management system 199 with a predetermined case unit continuous supply rate or feed rate in the automated storage and retrieval system 100 established by the warehouse management system 199. For example, in one aspect, the warehouse management system 199 is configured to set and / or monitor a predetermined case unit continuous supply rate within the automated storage and retrieval system 100. For example, the warehouse management system 199 monitors and manages the automated systems of the automated storage and retrieval system 100 (such as, for example, the in / out case conveyors 150A, 150B, the robots 110, and the palletizer / depalletizer units 10A, 10B), wherein each automated system or one or more automated systems has a given handling time (such as the time / cycle to achieve a basic case transport or transfer unit (e.g., transfer a case unit from the in / out case conveyor to a pick / place station, or lift a case unit a predetermined distance, or robotic transfer pick / place at a storage location), the time to transfer a pallet layer to a pallet or transfer a pallet layer from a pallet, etc.), which actually defines a predetermined continuous supply rate of case units established by the warehouse management system 199 in the automated storage and retrieval system 100, either individually or in combination, under the control of the warehouse management system 199 or any other suitable controller of the automated storage and retrieval system 100 (e.g., a robot controller, a conveyor controller, a palletizer / depalletizer controller, etc.). For example, the controller 199C of the warehouse management system 199 is communicatively connected to the in-and-out case conveyors 150A, 150B so that the in-and-out case conveyors 150A, 150B bidirectionally transport case units to and from the storage structure array 130 at a predetermined case feed rate. The controller 199C may also be communicatively connected to the palletizer-depalletizer units 10A, 10B corresponding to the in-and-out case conveyors 150A, 150B so that the layer activation and deactivation of the palletizer / depalletizer units 10A, 10B, which are respectively substantially continuous, matches the predetermined case feed rate. Although aspects of the disclosed embodiments are described herein with respect to a distribution facility 100WS having an automated storage and retrieval system 100 with an automated transport system, aspects of the disclosed embodiments are also applicable to distribution facilities having any suitable transport system, such as both automated and manual transport systems or a fully manual transport system, wherein both automated transport handling and manual transport handling each have corresponding handling times, wherein the activation and deactivation of case units to and from pallets can be matched to the handling times in a manner substantially similar to that described herein.

[0075] In one aspect, each outfeed transfer station 160 forms a case output path Op, wherein the palletizer / depalletizer unit 10B palletizes case units layer by layer onto a pallet PAL, such as with the aid of an automated layer interface unit (such as one or more robotic case manipulators 14). In one aspect, the pallet PAL may be formed as a standard pallet (e.g., homogeneous case units) or a mixed pallet, such as described in U.S. Pat. No. 9,856,083, issued on January 2, 2018, the disclosure of which is incorporated herein by reference in its entirety. In one aspect, the warehouse management system 199 is configured to establish a pallet solution for mixed case units that provides a stable pallet loading stack suitable for transfer in layers by the end effector of one or more robotic case manipulators 14. As described above, a suitable example of a palletizer / depalletizer unit 10B may be found in U.S. Pat. No. 10,370,199, issued on August 6, 2018, the disclosure of which was previously incorporated herein by reference in its entirety.

[0076] In one aspect, the palletizer / depalletizer unit 10B communicates with the transport system (such as the in / out case conveyor 150B) of the automated storage and retrieval system 100 to form an integrated output system (e.g., an outbound transfer station 160) that receives case units from the automated storage and retrieval system 100 for placement on a pallet according to any suitable case outbound order sequence. For example, as described above, a pallet load item unit CU sent to one or more robotic case manipulators 14 is transferred to a pallet PAL by the end effector of one or more robotic case manipulators 14, wherein the pallet load item units CU (output case units) are arranged layer by layer in a predetermined sequence established by the warehouse management system 199 (note that the layer may fully or partially cover the pallet) to form a standard output pallet load.

[0077] Each outbound transfer station 160 defines a case output path Op that is integrated with the automated storage and retrieval system 100 and the warehouse management system 199, wherein the warehouse management system 199 includes any suitable controller 199C configured with any suitable non-transitory program code and memory to manage the operation of the distribution facility 100WS, including the output of case units from the storage structure array 130B, as described herein. In one aspect, each case unit output path Op includes at least one corresponding case unit inspection unit 142 (as described above) that communicates with the warehouse management system 199. In one aspect, as described above, the palletizer / depalletizer unit body 10B can be fully automated to build or enable layers to a pallet load at the palletizer / depalletizer unit body 10B. It should be noted that reference Figure 2, the term activation refers to the construction of the pallet layers PL1, PL2, PL3, PL4 (completely or partially) to the pallet PAL, so that each pallet load unit CU is inserted into the layer PL1, PL2, PL3, PL4 at a predetermined level 200 of the pallet PAL (which may correspond to an inactive / active level or a transfer plane) until the pallet layers PL1, PL2, PL3, PL4 are formed so that the pallet PAL is directed to the next level of the pallet PAL (for example, by means of a pallet lift PLFT – see Figure 4 ) in order to construct the next layer PL1, PL2 (in whole or in part) of the next level corresponding to the pallet PAL. In one aspect, the palletizer / depalletizer unit body 10B is configured to enable the layers PL1, PL2, PL3, PL4 so that the activation is synchronized or otherwise coordinated (e.g., matched) with a predetermined case unit continuous supply rate or feed rate established by the warehouse management system 199 in the automated storage and retrieval system 100 through the warehouse management system 199, in a manner substantially similar to that described above with respect to the deactivation of the layers PL1, PL2, PL3, PL4, wherein the warehouse management system 199 manages the case unit retrieval order and the mixed case unit output to load out the sequence of the mixed case unit pallet loading sequence, as well as other associated output aspects such as inventory reconciliation.

[0078] Reference now Figure 1 , Figure 3A and Figure 4 The palletizer unit 10 (it should be noted that the term "palletizer" is used for convenience and, as mentioned above, the features of the palletizer can also be implemented in the depalletizer as originally applicable) is coupled to the storage and retrieval system 100 so as to communicate with the storage and retrieval system 100 the case units CU (see Figure 2 ) flow (see case output path Op and case input path Ip). According to aspects of the disclosed embodiment, the palletizer 10 is an adaptive palletizer system 300 that achieves time-optimal pallet load construction and can therefore complement and utilize the storage and retrieval system 100 case order flow throughput (although in other aspects, the adaptive palletizer 300 can be coupled to any suitable storage and retrieval system, including a conventional, manual or semi-automated retrieval system with a feed station for manual loading of the palletizer 10), in a manner similar to that described in U.S. Patent No. 10,894,676 issued on January 19, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0079] Also refer to Figure 2, the palletizer unit 10 is configured to build a pallet load PAL, wherein the pallet load PAL has a pallet load building structure RPAL (pallet building features can also be applied similarly to truck loads), the pallet load building structure RPAL is a three-dimensional array of mixed cases or pallet load item units CU structured into several stacks S1-Sn and layers PL1-PL5, including manufactured / constructed item units (picking faces), each of the multiple cases / items is placed on a pallet / pallet support SPAL (case unit / pallet load item unit means a case, a tote, a package, a shrink wrap, etc.). The pallet load building structure RPAL is determined by the control from the ordered case units CU (e.g. the case units CU output from the storage and retrieval system 100). For example, in one aspect, the palletizer controller 10C can be connected to the controller 199C of the warehouse management system 199; while in other aspects, the palletizer controller 10C can form a module of an integrated warehouse management controller that manages the delivery of storage and retrieval system 100 components (including the palletizer / depalletizer unit 10) so as to receive information defining the pallet load build structure RPAL, including the corresponding baseline reference limits, box postures, and change thresholds from the reference of the pallet load build implemented by the palletizer 10. The sequence of case poses by which the robot 14 of the palletizer 10 builds the pallet load PAL may be implemented by the storage and retrieval system 100 so that the cases output by the storage and retrieval system 10 that feed the robotic picking station 350 of the palletizer 10 arrive in a predetermined picking sequence (either just in time or appropriately buffered) to build the pallet load PAL, thereby achieving a higher pick / place rate by the robot 14 (e.g., the output case flow from the storage and retrieval system 100 substantially eliminates or reduces case unit CU sorting by the robot 14). Suitable examples of sorting of the output box stream from the storage and retrieval system 100 can be found, for example, in the following U.S. patents: U.S. Publication Nos. US2016 / 0214808, published on July 28, 2016; US2016 / 0207709, published on July 21, 2016; US2016 / 0207711, published on July 21, 2016; US2016 / 0214797, published on July 28, 2016; US2016 / 0167880, published on June 16, 2016; and US2016 / 0207710, published on July 21, 2016, the disclosures of which are incorporated herein by reference in their entirety.The robot 14 pick / place rate has, for example, a pick / place cycle of approximately 5 seconds from picking up at an input station (e.g., the robot pick station 350) to placing on the pallet load building structure BPAL and returning (for 2 robots, the pick / place cycle is approximately 2.5 seconds), and an adaptive feedback loop of the vision system 310 is implemented in real time within the pick / place cycle of the robot 14 (see Figure 3) to achieve substantially continuous construction of the pallet load building structure RPAL in the manner described in U.S. Patent No. 10,894,676, issued on January 19, 2021, the disclosure of which was previously incorporated herein by reference in its entirety.

[0080] Reference now Figure 3A -D and Figure 4 Each palletizer unit 10 generally includes a frame 24, at least one robot 14 with an end-of-arm tool 410, a controller 10C, and a vision system 310 including at least one three-dimensional time-of-flight camera 310C. A suitable example of a palletizer unit 10 can be found in U.S. Patent No. 10,370,199, issued on August 6, 2019, the disclosure of which is incorporated herein by reference in its entirety. The frame 24 defines a pallet support SPAL ( Figure 2 ) of the pallet building base 301 (see, for example, FIG. 3 ). At least one robot 14 is connected to the frame 24 and is configured to load the pallet with item units CU (see also Figure 2 ) are transported and placed continuously on the pallet supports SPAL (see Figure 2 ) to place the pallet load PAL (see Figure 2 ) is built on the pallet build base 301. The controller 10C is operably connected to at least one robot 14 and is configured (by means of any suitable hardware and non-transitory computer program code) to control the articulated robot movement relative to the pallet build base 301 and thereby achieve calibration of the robot 14 and the pallet load build BPAL of the pallet load PAL as described herein. The controller 10C is arranged to generate a three-dimensional imaging of the pallet supports (as described herein) on the pallet build base 301 and the pallet load build BPAL on the pallet supports SPAL.

[0081] The pallet building base 301 may be a palletizer 12 as described herein or any other suitable pallet support. The palletizer 12 gradually lowers the pallet from the second level 11 to the first level 13 as the product moves thereon, by, for example, two product picking devices or robots 14. Each robot 14 picks up a product from an adjacent feed conveyor 16 for placement onto a pallet on the palletizer 12. An empty palletizer 18 supplies empty pallets to the second level 11, and an empty pallet conveying mechanism 20 is disposed between the empty palletizer 18 and the palletizer 12 for moving an empty pallet 21 from the second level of the empty palletizer 18 to the palletizer 12 in a manner described in U.S. Patent No. 10,370,199, issued on August 6, 2019, the disclosure of which was previously incorporated herein by reference in its entirety.

[0082] As input to the palletizing unit 10, there can be box units or products CU of various sizes (see Figure 2 ) arrives at the palletizing unit body 10 via the infeed conveyor 16 and is picked up and placed by one robot 14 at the palletizing area 23, which corresponds to the portion of the palletizer 12 on the upper level 11. Substantially simultaneously with this palletizing operation, the empty palletizer 18 brings the empty pallet 21 to the empty pallet conveying mechanism 20. The mechanism 20 transfers the empty pallet 21 from the empty palletizer 18 to the palletizing area 23. The loader 12 takes over the pallet 21 and lowers the pallet 21 while palletizing the products. As output from the palletizing unit body, a full pallet made of a plurality of products (such as shown in Figure 2 ) ready to be picked up by a forklift or other transport device at the lower portion of the palletizer 12.

[0083] As described above, the palletizer unit 10 includes a lower level 13 and an upper level 11, defined by an elevated platform 22. This platform 22 and the equipment thereon are supported on a frame 24. The frame 24 may be formed by an integrated frame element or adjacent independent sub-platforms, each of which supports one or more components of the palletizer unit 10. The robot 14 is in the form of a standard industrial articulated robot arm 26 suitable for palletizing operations, each robot arm 26 can be operatively positioned near the feed conveyor 16 and is equipped with an end-of-arm tool 410 to pick up products from the feed conveyor 16. Suitable examples of the robot 14 include the IRB 460 or 4600 of ABB, the M410 or R2000 of FANUC, or any similar robot arm provided by other manufacturers such as Kuka or Motoman. The robot arm 14 is connected to the controller 10C in any suitable manner, and the controller 10C controls the operation of the robot arm 14A and the end-of-arm tool 410.

[0084] The end-of-arm tool 410 includes a frame 424 for mounting the end-of-arm tool 410 to the corresponding robot arm 14, a support assembly 450 for supporting the box unit CU thereon, a clamping assembly 26, and a pusher assembly 28, all of which are mounted to the frame 424. The frame 424 receives a robot mounting bracket or wedge 30 that allows the tool 10A to be attached to the flange 14F of the robot arm 14A in a conventional manner, wherein the robot mounting bracket 30 is coupled to the frame 424 in any suitable manner (such as by means of mechanical and / or chemical fasteners). Suitable examples of the end-of-arm tool 410 can be found in U.S. Patent No. 9,701,490, issued on July 11, 2017, and U.S. Provisional Patent Application No. 63 / 208,023, entitled “Tool for Palletizing Mixed Load Products, Palletizing Robot Including the Tool, and Method Therefor” filed on June 8, 2021, the disclosures of which are incorporated herein by reference in their entirety. As will be further described herein, the end-of-arm tooling also includes a distance sensor 411 that enables positioning of the robot pick position relative to the feed conveyor 16 (see Figure 4 ).

[0085] Still refer to Figures 3A-3C , the lift truck 18 includes a pallet receiving platform 30 mounted to a wall 32 so as to be slidably movable therealong. The wall 32 extends from the first level 13 to the second level 11 and is secured to the frame 24 on its peripheral sides. The platform 30 includes a bottom 34 fixedly mounted between two side walls 36. The side walls 36 are slidably mounted to the wall 32 via sliding elements, such as, but not limited to, rollers (not shown) in tracks 38. The up and down movement of the platform 30 along the wall 32 is actuated by a drive 40, which is operably coupled to the platform 30 via a belt assembly secured to the wall 32 and the platform 30. The platform 30 is configured and sized for receiving a pallet 21 thereon, and for slidably receiving a portion of an empty pallet conveying mechanism 20, such as Figure 3A . The platform 30 is not limited to any size and shape and is adapted to pallets of predetermined sizes and configurations. According to another aspect (not shown), the platform 30 is defined by fork elements. The empty pallet lifter 18 is not limited to the illustrated embodiment and may be provided with any other mechanism to selectively lower and raise the empty pallet receiving platform 30.

[0086] The palletizer 12 is kept at a distance from the empty pallet handler 18 by the empty pallet conveyor 20 and extends between the first level 13 and the second level 11 similar to the empty pallet handler 18. The platform 22 includes a rectangular aperture 44 that allows the empty pallet 21 to be transferred from the empty pallet conveyor 20 to the handler 12. The rectangular aperture 44 also enables the placement of products on the pallet 21 (transferred from the empty pallet handler 18 to the handler 12 by the empty pallet conveyor 20) by the end-of-arm tool 28. The aperture 44 is located in the palletizing area 23, where the robot 14 on the opposite side of the aperture 44 picks up the product from the corresponding feed conveyor 16.

[0087] As described herein, as products are placed onto the pallet 21 (supported by the pallet lift 21), the pallet lift 12 is gradually lowered so that the top level of the tallest product on the pallet 21 remains approximately flush with the platform 22. Figures 3A-3C , the palletizer 12 generally includes a rectangular pallet receiving element 46 supported by a frame 48. The frame 48 is slidably mounted to the frame 24 via an actuation mechanism 45 so that the frame 48 and the pallet receiving element 46 generally reciprocate between the upper level 11 and the lower level 13. The frame 48 includes two (2) first parallel beams 47 (only one shown) transversely fastened to two (2) second parallel beams 49. The frame 48 receives three (3) transverse chain assemblies 50 thereon.

[0088] Each chain assembly 50 includes an endless chain 52, which is operably mounted to the frame structure 48 via a sprocket assembly 54 so as to rotate along an axis defined by the conveying direction of the pallets on the empty pallet conveying mechanism 20. The chain assemblies 50 collectively define a movable pallet receiving element 46. Since the configuration and operation of the chain assemblies are considered to be well known in the art, the assemblies 50 will not be described in more detail herein for the sake of brevity. The number and configuration of the chain assemblies 50 are also not limited to the illustrated embodiment. In addition, the pallet receiving element 46 is not limited to being movable, and other mechanisms may be provided to move the pallets 21 into or out of the stacker 12.

[0089] The actuation mechanism 45 includes four (4) chain assemblies 56, which are fastened between the frames 24 and 48 and are operated by a driver 58. More specifically, each chain assembly 56 includes a chain 60, which is rotatably mounted to the frame 24 in an endless manner via two sprockets 62 or 63 (only the lower sprocket is shown). The chain 60 generally extends between the platform 22 and the lower level 13 in a direction generally perpendicular to the platform 22. The chain assemblies 56 are interconnected for simultaneous rotation, firstly by two shafts 64 coaxially mounted to each of two pairs of facing (opposing) sprockets 62 and 63, and then by a gear assembly 66 operatively linking two adjacent sprockets 63, one sprocket from each of the aforementioned sprocket pairs. The driver 58 includes a motor 68 coupled to the gear assembly 66. The motor 68 has a built-in braking system. Each of the adjacent sprockets 63 is coupled to a disc brake 70 to ensure safe descent of the stacker unloader.

[0090] The frame 48 is fixedly mounted to each of the four chains 60 to move simultaneously therewith via four (4) brackets 72 (only two are shown) fastened to the frame 48. In operation of the palletizer 12, controlled actuation of the drive 58 causes the four lower sprockets 62 and 63 to rotate in the same direction, thereby causing the chains 60 to rotate synchronously in an endless manner, thereby causing the pallet receiving element 46 to be lowered or raised accordingly between the upper level 11 and the lower level 13 via the frame 48. The drive 58 operates in such a way that the pallet receiving element 46 is gradually lowered by a step distance corresponding to the height of each product level added to the pallet 21. However, the raising movement of the pallet receiving element 46 is continuous so that, after a full pallet 21 has been removed at the lower level 13, the element 46 is brought back to the upper level 11, ready for filling with new products.

[0091] The palletizer 12 includes an additional actuator (not shown) coupled to the chain assembly 50 to cause selective translation of the pallet receiving element 46 to move the pallet filled with products out of the palletizer 12 when the pallet receiving element 46 is at the lower level 13. The palletizer 12 is not limited to the palletizer described herein, and may be provided with any other mechanism to lower and raise the palletizing element 46. The palletizing element 46 is also not limited to the palletizing element described herein. For example, the chain assembly 50 may be replaced with another pallet conveying mechanism, or omitted.

[0092] Still refer to Figures 3A-3C, the empty pallet conveying mechanism 20 will now be described. In one aspect, the mechanism 20 is in the form of a telescopic fork. The telescopic fork 20 includes a drive 74, a first stage 76 fixedly mounted to the frame 24, a second stage 78 slidably mounted to the first stage 76 for relative translation between two opposite longitudinal directions, and a second stage 78 slidably mounted to the second stage 78 for relative translation along the same two opposite directions (see Figure 3C The mechanism of the telescopic fork 20 includes a three-stage system using a carriage and a belt to extend the telescopic fork from one end to the other.

[0093] The position and range of the telescopic forks 20 are such that when the forks 20 are fully extended in direction 84, the third stage 80 overlies the pallet receiving platform 30 of the empty pallet lift 18 (see Figure 3A and when the third stage 80 is fully extended in the direction 82 so as to transfer the pallet from the third stage fork 20 to the endless chain 52 of the loader 12, the third stage 80 overlies the palletizing area 44.

[0094] refer to Figure 4 , Figures 4A-4N , Figure 5A and Figure 5B , at least one 3D time-of-flight camera 310C of the vision system 310 is disposed on one or more of the frame 24 and the robot 14 to generate a 3D image of at least a portion of the palletizer unit 10 (e.g., such as Figure 7 , Fig. 9 , Fig.11 , Fig.13 , Fig.15 , etc.) to enable calibration of the robot 14, as described herein. Although the at least one 3D camera 310C is described herein as a time-of-flight camera, any suitable 3D sensor / imager may be used, including a laser scanner, sonar, or other suitable machine vision system. As described herein, the at least one 3D camera 310C is communicatively coupled to the controller 10C, so that the controller 10C records real-time 3D imaging data (such as, for example, in the 3D images of the different corresponding 3D images of the palletizer unit 10) from the at least one 3D camera 310C. Figure 7 , Fig. 9 , Fig.11 , Fig.13 , Fig.15 etc., and / or any suitable data obtained from the point cloud).

[0095] In one aspect, the at least one 3D camera 310C is configured to enable 3D imaging of the pallet support SPAL on the pallet building base 301, the end-of-arm tooling 410 at one or more predetermined positions of the palletizer unit 10, and the pallet load building BPAL on the pallet support SPAL. In one aspect, the at least one 3D camera 310C comprises four (4) cameras 310C1, 310C2, 310C3, 310C4 (see e.g. Figure 5A and Figure 5B ) which are coupled to the frame 24 at any suitable location so that the cameras 310C1, 310C2, 310C3, 310C4 each have a corresponding field of view FOV1-FOV4 (see, e.g., Figure 4 as well as Figures 4A-4N ) for imaging at least two sides of the pallet support SPAL and the pallet load building BPAL / pallet load building structure RPAL, for example one of the front side surface, the rear side surface and the vertical side surface (extending between the front and the rear) (see Figure 2 ) and top (see Figure 2 ). The at least one camera 310C may be oriented such that the top and at least one side surface (e.g. the front, rear or vertical side) of the pallet support SPAL and each case unit CU placed on the pallet support SPAL are visible within the field of view FOV1-FOV4 covering the corresponding part of the pallet support SPAL / pallet load building structure RPAL. See also Figure 4G-4J In one aspect, cameras 310C1, 310C2, 310C3, 310C4 may have any suitable focal length for a predetermined image intensity and may be positioned relative to frame 24 (e.g., a horizontal plane of frame 24 such as defined by, for example, pallet build base 301) and / or relative to each other at an angle of, for example, 45° (see Figure 4H ) are placed so that at least one camera images at least two sides; while in other aspects, the angle between the cameras 310C1, 310C2, 310C3, 310C4 and / or the frame 24 can be greater or less than 45°. In one aspect, each field of view FOV1-FOV4 (commonly referred to as the field of view FOV (see Figure 4H as well as Figure 4K-4N , which figures emphasize each field of view relative to the other fields of view) can be a 45° field of view; in other aspects, the field of view FOV can be larger or smaller than 45°, as long as the pallet support SPAL and at least two sides of the pallet support SPAL and the pallet loading structure BPAL / pallet loading building structure RPAL are imaged.

[0096] In one aspect, at least one camera 310C resolves the three-dimensional definition of case unit features (e.g., edges of case units) and calibration fixture features from two or more orthogonal planes such that maximum certainty of feature pose (e.g., X, Y, Z, θ, α, μ position of the feature, see FIG. 3G ) is obtained from a single item image in the corresponding field of view FOV1-FOV4 of at least one camera 310C. Here, the resolution of the three-dimensional definition of case unit and / or calibration fixture features is independent of camera 310C placement (as long as the top and one side of the feature are imaged) and is implemented in real time (e.g., within a pick / place cycle of at least one robot 14, or within a calibration routine of at least one robot 14).

[0097] Although four (4) cameras 310C1-310C4 are described, it will be appreciated that more or less than four (4) cameras 310C may be used and positioned such that the field of view of the cameras 310C of the vision system 310 covers the entirety (or at least a predetermined portion) of the pallet build base 301 of the frame 24, the pallet support SPAL mounted thereon and the intended pallet load build structure RPAL, so as to capture three-dimensional time-of-flight images of the object at any suitable desired resolution, desirably at any location on the pallet support SPAL and at any location on the pallet load build structure RPAL. The combined fields of view FOV1-FOV4 result in substantially complete 360° coverage of the pallet load build structure RPAL, with the fields of view FOV1-FOV4 overlapping. For example, the combined fields of view FOV1-FOV4 may cover a standard pallet support SPAL (e.g., 48 inches by 48 inches, 48 ​​inches by 40 inches, and / or 36 inches by 36 inches in size), it being understood that the cameras 30Ca-300C4 and associated fields of view FOV1-FOV4 may cover (e.g., image) a larger field of view (including, e.g., a truck bed or any desired field of view size) as appropriate. Furthermore, the fields of view FOV1-FOV4 may cover any suitable pallet load building structure RPAL height PH (see FIG. 3H ), such as, for example, 60 inches, 70 inches, and 80 inches in height; while in other aspects, the fields of view FOV1-FOV4 may cover heights less than 60 inches or greater than 80 inches.

[0098] In one aspect, each camera 310C1-310C4 may have a 176 pixel by 132 pixel resolution; while in other aspects, each or one or more of the cameras 310C1-310C4 may have a higher resolution (e.g., 320 pixel by 240 pixel resolution or higher) as desired to provide a desired minimum depth map defining about 0.5 inches at the outermost limits of the pallet build three-dimensional space 3DS (such that the depth map clarity of the captured image of the whole or a predetermined portion of the pallet support / pallet build is not less than about 0.5 inches). Thus, the vision system 300 provides sufficient resolution to resolve the lattice features of the pallet supports SPAL into clarity such that planarity across the pallet is determined and fully established for placing a stable first layer PL1 of case units CU on the pallet supports SPAL as will be described herein. Sufficient resolution may also be provided to resolve case unit features (e.g., such as case edges) such that planarity across the top of each layer PL1-PL4 (see FIG. 3H ) is determined and fully established to place stabilization layers PL2-PL5 on top of previously placed layers PL1-PL4. The resolution of cameras 310C1-310C4 may be such that case unit features (e.g., case unit edges) may be resolved with only minimal processing required such that case unit features are resolved substantially in real time from images received by controller 10C.

[0099] As described herein, the vision system 310 also includes an electromagnetic sensor 500 configured to sense a calibration feature (e.g., a calibration feature of a different jig or fixture as described herein, such as a calibration feature of the calibration box 810 and / or a calibration feature of an integral feature of the robot 14 / end-of-arm tool 410). The controller 10C is communicatively coupled to the electromagnetic sensor 500 and records a sensor signal embodying a predetermined characteristic of the calibration feature for a corresponding pose of the robot 14 (and a corresponding pose of the calibration feature due to movement of the robot 14), the recording occurring substantially simultaneously with the electromagnetic sensor 500 sensing the calibration feature and generating the sensor signal.

[0100] The electromagnetic sensor 500 is configured to be robust to variable lighting conditions. In one or more aspects, the electromagnetic sensor 500 is a two-dimensional (2D) color camera 500C (also referred to herein as a “2D camera”) disposed on one or more of the frame 24 and the robot 14 (see, e.g., Figure 3C , Figure 4G , Figure 5A and Figure 5B ) in order to generate a two-dimensional image of at least a portion of the palletizer unit 10 (e.g., such as Figure 8 , Fig.10 , Fig.12 , Fig.14 , Fig.16 , etc.) to enable calibration of the robot 14 as described herein. Here, the two-dimensional camera 500C is a color camera that distinguishes contrast patterns (such as the grid patterns 800, 815) with a desired clarity commensurate with the accuracy of the robot 14 motion (e.g., substantially the same as or less than the motion encoder error of the robot 14). The camera 500C may be any suitable two-dimensional camera, such as those available from Basler Corporation (located in Exton, Pennsylvania, USA), having a field of view FOV2D configured to image at least the pallet building area of ​​the palletizer unit 10 (see Fig.12 ). The two-dimensional camera 500C may be a high-resolution camera having a resolution of, for example, about 2.3 megapixels to about 24 megapixels (or higher). Any suitable illumination (such as a ring light 500L or any other suitable illumination source) may be provided to illuminate the object imaged by the two-dimensional camera 500C. The two-dimensional camera 500 is communicatively coupled to the controller 10C, so that the controller 10C records real-time two-dimensional imaging data representing different corresponding two-dimensional images of the palletizer unit 10 from the two-dimensional camera 500C. In one aspect, the two-dimensional camera 500C is configured to enable two-dimensional imaging of the pallet support SPAL on the pallet building base 301, the end-of-arm tool 410 at one or more predetermined positions of the palletizer unit 10, and the pallet load build BPAL on the pallet support SPAL. In other aspects, the camera 500C may be any suitable imaging device, such as any suitable electromagnetic sensor (i.e., a sensor of the electromagnetic spectrum / frequency range for detecting electromagnetic radiation of an object, including but not limited to a camera, a laser scanner, etc. In other aspects, the camera 500C may be a three-dimensional camera having sufficient resolution to extract data described herein with respect to a two-dimensional camera from the three-dimensional image obtained thereby. In still other aspects, the electromagnetic sensor 500 may be a laser scanner / reader 500LSR configured to scan / read features / characteristics of a target object (e.g., such as a three-dimensional structure), which features / characteristics may determine the pose of the target object (such as those described herein).

[0101] It should be noted that the alignment of the fields of view of the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C is implemented by aligning the images from each camera 310C1-310C4, 500C with the images from each other camera 310C1-310C4, 500C using image overlay annotations. In other aspects, the cameras 310CA1-310C4, 500C of the vision system 310 can be synchronized using a calibration fixture placed on the lift 12 in a manner substantially similar to that described in U.S. Patent Application No. 17 / 070,753, filed on October 14, 2020 and entitled “Vision-Assisted Robotized Depalletizer,” the disclosure of which is incorporated herein by reference in its entirety.

[0102] refer to Figures 3A-3D , Figure 5A and Figure 5B , palletizer unit references will be described, wherein the controller 10C employs these references to achieve fully automatic calibration of the palletizer unit 10 from an uncalibrated state of the palletizer unit. The references described herein are defined herein for exemplary purposes (i.e., exemplary in nature) and for ease of description. In other aspects of the disclosed embodiments, any suitable references may be selected and / or defined in any suitable manner such as described herein, wherein the references are located at any suitable location on the palletizer unit and / or palletizer unit components.

[0103] It should be noted that the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C are pre-calibrated so that each camera 310C1-310C4, 500C uses the corresponding origin reference COR (see Figure 5A – that is, the coordinate system (X, Y, Z) origin) is located at the center of the camera lens. The three-dimensional cameras 310C1-310C4 are configured to provide a three-dimensional point cloud of portions of the palletizer unit 10, wherein the positions of the imaged objects are generated with reference to the respective origins. Due to the inherent parameters in the calibration of the two-dimensional camera 500C, the two-dimensional camera 500C can also be configured for three-dimensional vision (that is, given a plane in the 3D world of the palletizer unit 10, the pixel coordinates of the two-dimensional camera 500C can be converted into the world coordinates of the palletizer unit 10). The calibration of the two-dimensional camera 500 is implemented similarly to the method described in the article A flexible new technique for camera calibration by Z. Zhang, published in the IEEE Journal of Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000, the disclosure of which is incorporated herein by reference in its entirety.

[0104] Each robot 14 has a base 14B and an arm or manipulator 14A. An end-of-arm tool 410 is coupled to the arm 14A at a flange 14F. Each robot 14 includes at least a base reference RBR, a flange reference RFR, and a tool reference RTR. Figures 3A-3C As shown in FIG. 1 , the two robots 14 are placed side by side on the frame 24. The base 14B is substantially located in the same (or common) plane CP defined by the frame 24 of the palletizer unit 10 (see FIG. 1 ). Figure 3A ). The base reference RBR is located at the center of the base 14B of the corresponding robot 14, wherein the X-axis extends in the direction of the case flow on the picking conveyor 16, and the origin of the Z-axis is located on the plane CP. It should be noted again that the references described herein are defined herein for exemplary purposes (i.e., exemplary in nature) and for convenience of description. In other aspects of the disclosed embodiments, any suitable reference may be selected and / or defined in any suitable manner such as described herein, wherein the reference is located at any suitable location of the palletizer unit and / or palletizer unit component.

[0105] The robot flange reference RFR is located at the center of the flange 14F (see also Figure 4C ), with the Z-axis pointing downward (and having the origin at the end of the end-of-arm tooling coupling face of the flange), and the X-axis extending in the direction of the end-of-arm tooling 410 finger 450F (of the support assembly 450). Here, the robot flange is rotated about the Z-axis of the flange reference RFR (e.g., such that the flange reference RFR rotates with the end-of-arm tooling orientation, thereby maintaining the X-axis aligned with the finger 450F); however, in other aspects, the flange 14F may not rotate.

[0106] The tool reference RTR (also called tool center point or TCP) describes the reference system of the tool from which the pick and place are calculated. Figure 3A and 3D as well as Fig.16 ) During picking, the tool reference RTR has its origin located at the tip of the finger farthest from the camera 310C1-310C4, 500C, with the Z axis pointing downward (and having its origin at the intersection of the top surface of finger 450F and the tip surface of the finger), and the X axis extends in the direction of finger 450F.

[0107] refer to Figures 3A-3C as well as Fig.17, the empty pallet conveying mechanism 20 (e.g., telescopic fork) has a telescopic fork origin reference TFRP, which in one or more aspects is substantially located at the centerline CL of the orifice 44 of the loader 12. The location of the telescopic fork origin reference (e.g., datum) TFRP can be defined by the nominal dimensions of the palletizing unit body 10 (e.g., from a computer-aided drawing (CAD) file) or in any other suitable manner. The telescopic fork origin reference TFRP is shared by each robot 14 and enables synchronization or otherwise connecting the workspaces of the robots 14. The origin of the telescopic fork origin reference TFRP is located at a predetermined distance from the robot base reference RBR in the X direction, and the Y origin is substantially located at the midpoint between the robot base reference RBR. The telescopic fork origin reference TFRP is projected onto the plane of the empty pallet conveying mechanism 20 (see Fig.17 - so that the Z axis origin is at the top surface of the empty pallet conveying mechanism 20). The telescopic fork origin reference TFRP is used for pallet building and positioning of the pallet support SPAL in the palletizer unit 10 working space.

[0108] Also refer to Fig.18A and Fig.18B , a fork reference (e.g., a reference) TFR is also defined in each of the three-dimensional cameras 310C1-310C4. The controller 10C uses the fork reference TFR to segment the point cloud of the corresponding three-dimensional camera 310C1-310C4. Here, the fork reference TFR positions the transfer forks 20F1, 20F2 of the empty pallet conveying mechanism 20 in the workspace of the palletizer unit 10 relative to each of the three-dimensional cameras 310C1-310C4. The fork reference TFR is defined so that the Z axis points upward from the plane defined by the top surfaces of the forks 20F1, 20F2, and the x axis is aligned with the forks 20F1, 20F2 (see Fig.18A and Fig.18B ) and points away from the 3D cameras 310C1-310C4. The origin of the fork reference TFR of each respective 3D camera 310C1-310C4 is at the root of the X and Z planes where the cameras 310C1-310C4 are projected onto. The Y origin is in line with the center line CL of the forks 20F1, 20F2. Fig.18A and Fig.18B , for example, from 3D cameras 310C1 and 310C3, which show point clouds of forks 20F1, 20F2 detected by the respective cameras and the fork reference TFRs of the respective cameras. Here, in some aspects, given the predetermined positions and known characteristics of the 3D cameras, selecting a separate common reference (such as the fork reference TFR) for each 3D camera can be used to calibrate the 3D cameras to each other by resolving the pose of the reference and the characteristics of the pose detection. In this case, the 3D camera calibration is commensurate with the robot 14 / end effector 410 placement accuracy.

[0109] Additional references (e.g., benchmarks) TFRA1, TFRA2 may be determined for each individual fork 20F1, 20F2 for one or more of the three-dimensional cameras 310C1-310C4. These additional references TFRA1, TFRA2 enable detection of abnormalities, such as bent / twisted forks and height differences between the forks 20F1, 20F2 along the longitudinal length of the forks 20F1, 20F2. Here, the controller 10C uses, for example, the three-dimensional camera 310C1 (see Fig.18B ) implements image sampling of portions of each fork 20F1, 20F2 (see sampling blocks S1-S6) to generate separate fork references TFRA1 (for fork 20F1), TFRA2 (for fork 20F2) based on planar fits of the respective fork surfaces to the respective sampling blocks S1-S6. The origin of the separate fork references TFRA1, TFRA2 is placed on the respective fork 20F1, 20F2, with the Z axis normal to the plane of the respective fork 20F1, 20F2 (established by sampling blocks S1-S6), with the Z axis pointing upward and the X axis pointing away from the camera imaging the fork (in this example, camera 310CA). In one aspect, anomalies are recorded from fork features identified based on imaging by only one three-dimensional camera 310C1-310C4. In other aspects, different signs of anomalies are detected by comparing fork images captured from more than one three-dimensional camera 310C1-310C4 (where the cameras are calibrated relative to each other by means of a common electromagnetic sensor 500).

[0110] See also Figure 3D , the distance sensor 411 may also include a reference (e.g., datum) DSF. The distance sensor reference DSF is defined as the intersection between the laser line and a plane PL located in a flange reference (e.g., datum) RFR (e.g., such as, for example, a height of about 1000 mm in the flange reference RFR, however, the height of the plane PL may be any suitable height). The laser line is imaged by one or more cameras (such as a two-dimensional camera 500C), and the controller 10C is configured to determine the position of the distance sensor 411 on the end-of-arm tool 410 based on the camera image, and determine the position of the distance sensor reference.

[0111] refer to Figure 8According to aspects of the disclosed embodiments, the automatic calibration system ACS includes structures and / or markings having calibration features connected to the end-of-arm tool 410 (or end effector), wherein the features have predetermined characteristics describing predetermined or known shapes or patterns. For example, the controller 10C and the automatic calibration system ACS implement fully automatic calibration with reference to a target object, which is maintained on or integral with the end-of-arm tool 410, or can be set in a predetermined geometric relationship relative to the end-of-arm tool 410 and the features of the end-of-arm tool 410. The target object can be one or more target objects (e.g., one or more clamps, fixtures, or other suitable structures / objects), wherein the one or more target objects include suitable structures and / or markings, wherein the structures and / or markings have predetermined characteristics that determine the three-dimensional orientation / positioning of the target object. For example, predetermined characteristics of the structure and / or markings of the target object may have four non-collinear but coplanar edges, where the distances and / or positions between these edges are known in order to achieve homography between images of different cameras 310C1-310C4, 500C of the vision system 310.

[0112] Different and distinct target objects may be used for positioning / calibrating the robot 14 / end effector 410 in the palletizing unit workspace and for setting a reference (e.g., a datum). In one aspect, a calibration feature of a target object (e.g., each of one or more target objects) is dependent on the end-of-arm tooling 410, or the calibration feature and the end-of-arm tooling 410 are co-dependent on a common component of the robot 410 (e.g., such as flange 14F), and the calibration feature and the end-of-arm tooling 410 move as a unit relative to the frame 24 with each articulation of the robot 14. For example, one of the one or more target objects is a target object having a calibration feature integrally formed in the frame 424 of the end-of-arm tooling 410 (see Figure 8) with known characteristics (such as the four non-collinear, but coplanar edges described above), wherein the grid pattern 800 has four edges in a known relationship to each other. Here, the grid pattern 800 can be integrated with any suitable portion of the corresponding robot box manipulator 14 structure and / or any suitable portion of the end-of-arm tool 410 structure to achieve positioning of the corresponding robot 14. One or more target objects may also include fixtures or fixtures with characteristics that are different and distinct from the integrally formed characteristics of the end-of-arm tool 410 (and the robot 14), wherein these fixtures or fixtures are used to set the references described herein. These fixtures or fixtures can be a calibration box 810 (or other suitable object) carried by the end-of-arm tool 410, wherein the fixture includes a grid pattern 815 with known characteristics (such as the four non-collinear, but coplanar edges described above). In one or more aspects, these fixtures or fixtures can be used for both robot 14 / end effector 410 positioning and reference setting.

[0113] For exemplary purposes only, the calibration box 810 has a width W of approximately 12 inches, a height H of approximately 12 inches, and a length L of approximately 24 inches; however, in other aspects, the calibration box 810 may have any suitable dimensions, including but not limited to dimensions corresponding to the case units expected to be handled by the end-of-arm tooling 410. The grid pattern 815 is formed by a plurality of quadrilaterals arranged in a two-dimensional array (i.e., arranged on at least two planar surfaces of the calibration box 810). For exemplary purposes, the two-dimensional array of quadrilaterals on respective sides of the calibration box 810 forms a checkerboard pattern having squares that measure approximately 2 inches by 2 inches; however, in other aspects, the quadrilaterals on respective sides of the calibration box may have any suitable dimensions for forming a two-dimensional array.

[0114] The calibration box 810 (or other target object) is optimized to take advantage of the imaging characteristics of both the three-dimensional cameras 310C1-310C4 and the two-dimensional camera 500C. For example, a checkerboard pattern of squares causes the colors of the squares to have similar brightness values ​​and widely separated hue values, which facilitates imaging of the two-dimensional array by the two-dimensional camera 500C while reproducing the two-dimensional array that is not detectable by the three-dimensional cameras 310C1-310C2 (e.g., so that the checkerboard pattern does not appear in the three-dimensional image and enables position determination of the plane formed by the sides of the calibration box). Suitable examples of contrasting colors for the two-dimensional array include, but are not limited to, green and tan.

[0115] refer to Figures 3A-3D , Figure 5A-15 as well as Figure 19A-25, the automatic calibration system ACS with the controller 10C is configured to image the calibration box 810 and other parts of the palletizer unit body to achieve fully automatic calibration (including positioning) of each robot 14 within the palletizer unit body reference system, as well as automatic teaching of each station (e.g., conveyor, pallet loading surface, fork, etc.) of the palletizer unit body, as will be further described herein. In order to facilitate the calibration procedure for each degree of freedom of the robot 14 actuator, the position of each robot 14 is constrained relative to the palletizer unit body reference system so that the measurements obtained by the vision (in other words, automatic calibration and teaching) system 310 do not affect (i.e., are corrected to) the relative position of the robot 14 in the mathematical model of the palletizer unit body 10 generated by the calibration procedure. In particular, one or more boundary / correction constraints (referred to herein as constraints) are set for the position of the robot 14 so that the robot base 14B is located on a common plane (e.g., defined by the frame 24). It should also be noted that prior to the automatic calibration procedure, the robot 14 is “zeroed” or moved to its starting position to reset the encoder 14DE ( Figure 3D ) so that the position of the robot flange 14F is known relative to the base 14B.

[0116] Initiate ( Fig.14 , box 1400) is used to calibrate the calibration procedure of the robot 14.

[0117] As described above, for each robot 14, one or more constraints are identified ( Fig.14 , box 1405). These constraints may be predetermined / known from the palletizer unit design; where a non-limiting example of a boundary / corrective constraint is the plane CP formed by the frame 24 on which the robot is positioned (as described above - see Figure 3A ). Plane CP may at least partially determine the origin (at least relative to the electromagnetic sensor 500 and / or the 3D cameras 310C1-310C4) and posture of the base 14B of the robot 14. Plane CP may define the orientation / tilt (e.g., in the Z direction) of each robot 14 in the X-axis and Y-axis relative to at least the electromagnetic sensor 500 and / or the 3D cameras 310C1-310C4 reference system.

[0118] The controller 10C implements the determination of a common fork origin reference TFRP (TFRP) in the manner described herein for each of the three-dimensional cameras 310C1-310C4. Fig.14, block 1420). As previously described, the forks 20F1, 20F2 form the base of the pallet support on which each robot 14 builds the pallet load, either collaboratively or individually. The pallet support (and therefore the forks 20F1, 20F2) is the palletizer unit station, and the fork origin reference position or reference (also the limit of the pallet support) is the position relative to which the robot 14 is calibrated and taught. Although described first here, the determination of the common fork reference frame / reference TFRP can be achieved at any step in the overall calibration / teaching process. Unless otherwise specified, the process order is not constrained.

[0119] The controller 10C commands one robot 14 to position the end-of-arm tool 410 so that the operator can place the calibration box 810 in the end-of-arm tool 410 ( Fig.14 , block 1410), such as Figure 8 and Fig.16 As illustrated in . In other aspects, as described herein, the target can be integral to the end-of-arm tool 410, in which case the calibration box 810 may not be placed on the end-of-arm tool 410. The placement of the calibration box 810 on the end-of-arm tool 410 can be random as long as the end-of-arm tool 410 is able to clamp the calibration box 810. With the calibration box 810 clamped by the end-of-arm tool 410, the position of the calibration box 810 relative to the end-of-arm tool 410 of the robot 14 does not change (i.e., is an invariant) throughout the calibration procedure. Here, the calibration features (e.g., of the calibration box 810) are arranged relative to the robot 14 so that predetermined characteristics of these calibration features determine a predetermined relationship between the calibration features and a robot reference system (such as a flange reference RFR and / or a base reference RBR).

[0120] Here, the calibration box is clamped by the end-of-arm tool 410 to perform the calibration procedure ( Fig.14 , block 1430), so that camera synchronization is implemented during the calibration process. Here, the robot 14 is driven by the driver 14D of the robot 14 (see Figure 3D ) (commanded by the controller 10C) so that the calibration box 810 moves to a series of (e.g., four) known positions (the robot may move to more or less than four known positions). Figure 3D) encoder data. Each position in the series of positions positions the target object so that the calibration system / camera images the target object (e.g., senses a predetermined characteristic that determines the pose with the clarity identified herein). In each position in the series of positions, at least one different 3D camera among the 3D cameras 310C1-310C4 and the electromagnetic sensor 500 (such as the 2D camera 500C) images the target object. For example, each camera 310C1-310C4 and the electromagnetic sensor 500 images the robot end-of-arm tool 410 in each of the four positions. The robot encoder 14DE ( Figure 3D ) is associated with the robot position in each of the cameras 310C1-310C4 and in each of the four images of the electromagnetic sensor 500. In camera synchronization, the controller 10C employs a three-dimensional positioning of the calibration box 810, and then a two-dimensional positioning of the calibration box 810, where the box is placed in a known "easy" location within the palletizing volume (e.g., a few features in the image other than the calibration box) for detection by the camera pre-sync algorithm of the controller 10C.

[0121] With the robot 14 position (e.g., approximately) positioned relative to the camera position (e.g., the robot position to camera reference frame transport is approximate), the robot 14 is automatically calibrated / synchronized with the cameras 310C1-310C4 and the electromagnetic sensor 500. The controller 10C algorithm for automatic calibration / localization of the robot 14 detects the calibration box 810 in three dimensions and two dimensions at various positions in a series of positions and corresponding poses at each position (for optimizing the transformation between the robot reference and the camera reference), and outputs, for example, detected box corners, planes, and inliers. The robot 14 calibration / localization implements three-dimensional calibration box 810 detection (e.g., box corners) and two-dimensional calibration box 810 detection (e.g., corners and grid 815) in the corresponding world (i.e., reference frame) of each camera 310C1-310C4 and electromagnetic sensor 500 for each position in the series of positions so as to bind (or synchronize) all references together into a cohesive space.

[0122] As described herein, the controller 10C is configured to move the robot 14 (by commanding actuation of the degrees of freedom of the robot actuators) and position calibration features, such as the calibration box 810 and / or integral features of the end-of-arm tooling 410, to an initial position (i.e., from an initial / starting position of the robot 14, by the degrees of freedom ΔX, ΔY, ΔZ, Δμ (see, e.g., Figure 3D and Figure 4G) in a robot reference system defined by a known or predetermined position in which the electromagnetic sensor (the two-dimensional camera 500) senses a calibration feature, the controller 10C being configured to resolve the pose of the calibration feature relative to the sensor reference system (e.g., the origin reference COR), and determine the transformation between the origin reference system COR of the electromagnetic sensor 500 (here, the two-dimensional camera 500C) and the selected three-dimensional camera reference system (e.g., the origin reference system COR of one or more three-dimensional cameras 310C1-310C4) based on the resolved pose, so as to calibrate the three-dimensional cameras 310C1-310C4. As can be appreciated, the calibration of the electromagnetic sensor 500 and the selected three-dimensional cameras 310C1-310C4 (resolution of the pose of the origin reference system COR) is thus substantially simultaneously achieved. In addition, the positioning and teaching of the robot 14 and the end-of-arm tool 410 in the palletizer unit space is also initiated and achieved, which occurs substantially simultaneously with the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4, as will be further described herein. For example, in one aspect, the algorithm used by the controller 10C for automatic calibration of the robot 14 employs three-dimensional calibration box 810 detection from the three-dimensional cameras 310C1-310C4, followed by two-dimensional calibration box 810 detection by the electromagnetic sensor 500, wherein here, the three-dimensional detection facilitates segmentation of the two-dimensional image as described herein, and accelerates pose resolution via the electromagnetic sensor 500 (here, the two-dimensional camera 500C). The three-dimensional detection enables easy detection of the calibration box 810 so that the two-dimensional image can be easily resolved based on the corners of the front and top faces of the calibration box 810 (the side faces can also be detected) (e.g., discovered by the controller 10C based on the front and top face planes). Therefore, two-dimensional positioning of the calibration box 810 is implemented based on the corners discovered in the three-dimensional camera calibration box 810 detection (where the controller 10C generates a mask and applies it to the two-dimensional image to eliminate redundant features in the two-dimensional image) to facilitate segmentation of the two-dimensional image of the calibration box 810. In other aspects, where the electromagnetic sensor 500 is not an imaging sensor or camera (e.g., a raster scanner, etc.), the three-dimensional camera 310C1-310C4 visualization of the calibration box 810 supplements the pose calibration of the electromagnetic sensor 500, so that calibration of the electromagnetic sensor 500 and the three-dimensional camera 310C1-310C4 is essentially achieved simultaneously, but the three-dimensional detection from the three-dimensional camera 310C1-310C4 cannot be used to assist in electromagnetic sensor 500 calibration and pose determination (and in some aspects, the three-dimensional visualization / detection of the three-dimensional camera 310C1-310C4 may not assist in pose determination of the electromagnetic sensor 500, where the electromagnetic sensor 500 is a two-dimensional camera 500C and other suitable image processing of the controller 10C can be used to segment the two-dimensional image).

[0123] In the calibration procedure, the reference ( Fig.14 , box 1434). As an example, the electromagnetic sensor 500 (also referred to as the calibration sensor, which in this example is the two-dimensional camera 500C) is calibrated relative to the pose of the target object (in this example, the calibration box 810) to define a reference of the pose of the electromagnetic sensor 500, wherein the pose of the target object is substantially unconstrained within the sensor sharpness). Here, a predetermined positioning of the target object relative to the field of view of the electromagnetic sensor 500 and vice versa is not required. As described herein, the target object includes predetermined characteristics that determine the pose resolution of the target object, wherein the calibration of the electromagnetic sensor 500 (and the three-dimensional cameras 310C1-310C4) resolves the pose of the target object and the corresponding electromagnetic sensor 500 / three-dimensional camera 310C1, 310C4 relative to each (or one or more of) the robot 14 and the end-of-arm tool 410. For example, the controller 10C determines the grid 815 of the calibration box 810, thereby defining the corners of the calibration box 810 in the world of the electromagnetic sensor 500 (here, for exemplary purposes, the two-dimensional camera 500C). The controller 10C determines the pose of the calibration box 810 in the world of the electromagnetic sensor 500, calibrates the electromagnetic sensor 500 via the transformation between the electromagnetic sensor 500 and the selected 3D cameras 310C1-310C4, and calibrates the corresponding 3D cameras 310C1-310C4.

[0124] In the automatic calibration of the robot 14, the robot 14 moves the calibration box 810 to a plurality of predetermined / known positions and angles within the palletizing volume (such that these positions and angles are described by the actions in each degree of freedom of the drive part and are communicated to all stations in the palletizer unit volume) by means of one or more degrees of freedom of the robot drive part (such that each posture is informed by at least one degree of freedom of the drive (e.g., by the corresponding drive position sensor of the degree of freedom)) Fig.14 , block 1431) (e.g., within the field of view of cameras 310C1-310C2 and electromagnetic sensor 500), such that each of three-dimensional cameras 310C1-310C4 (and electromagnetic sensor 500) images at least calibration box 810 ( Fig.14, block 1432), and each 3D camera image is consistent with the electromagnetic sensor 500 resolution of the calibration box. Here, the robot 14's movement of the calibration box 810 resolves the pose of the calibration box 810 (and the robot 14's position) relative to each degree of freedom (X, Y, Z, μ, and in some aspects, the rotation of the end-of-arm tooling about the axis θ), so that for each robot, the controller 10C performs substantially simultaneous electromagnetic sensor 500 and 3D camera 310C1-310C4 calibration (as described herein); the controller 10C performs substantially simultaneous robot positioning determination (as described herein); and the controller 10C performs substantially simultaneous teaching of the palletizer unit station to the robot 14 (as described herein).

[0125] The movement of the calibration box 810 (or other target object as described herein) by the robot 14 due to, for example, the movement of the flange 14F along a known path (e.g., determined by a position sensor of a driven portion of the robot 14), achieves the display of the discrete position / pose of the calibration box 810 along the known path to each of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4. As described herein, the calibration box 810 moves along any suitable number of predetermined / known paths (to any suitable number of predetermined / known positions) to reference the reference resolved robot pose parameters described herein. Given the known boundary / corrective constraints, the known path in the robot reference frame, and the pose of the calibration box 810, the controller 10C determines the pose parameters of each robot / sensor (the term sensor includes the three-dimensional cameras 310C-310C4 and the electromagnetic sensor 500) in the sensor reference frame and the palletizer unit body reference frame in the manner described herein.

[0126] In one or more aspects, the imaging of the calibration box 810 by each of the three-dimensional cameras 310C1-310C4 and the electromagnetic sensor 500 (e.g., at each position / pose of the calibration box 810) occurs substantially simultaneously (although in other aspects, the imaging of the calibration box 810 by the three-dimensional cameras 310C1-310C4 and the electromagnetic sensor 500 may occur at different times) to resolve the calibration of the three-dimensional cameras 310C1-310C4 to the pose of the electromagnetic sensor 500 and relative to each (or one or more) other three-dimensional cameras 310C1-310C4 ( Fig.14 , box 1435). Fig.15 Examples of calibration box 810 positions and angles are illustrated in FIG. 1 , noting that while three positions / angles are illustrated, the robot 14 may move the calibration box 810 to any suitable number of positions / angles such that the placement of the calibration box 810 on the end-of-arm tooling 410 is automatically resolved and the vision system 310 obtains sufficient data to position the end-of-arm tooling in the palletizer unit 10 reference frame / reference.

[0127] It should be noted that, except for the calibration box 810 being placed on the finger 450F of the end-of-arm tooling 410, the placement of the calibration box 810 on the end-of-arm tooling 410 is not preset to any datum or reference position on the end-of-arm tooling 410. Here, the placement of the calibration box 810 on the end-of-arm tooling 410 (and the robot 14) is common / constant in all images, so that the relative pose of the calibration box 810 to the robot 14 is cancelled from the controller 10C algorithm that determines the pose and position of the robot 14, because the position of the calibration box 810 relative to the robot 14 / end-of-arm tooling 410 is consistent / constant throughout the calibration process and can be ignored. As an example, the placement of the calibration box 810 on the end-of-arm tooling 410 can be resolved, and sufficient data for positioning the robot 14 can be obtained by moving the calibration box 810 to as few as three positions, each position having a different box angle / pose, with each camera 310C1-310C4, 500C generating an image at each of the three positions. In one aspect, each camera captures 16 images at 16 different calibration box positions / poses. In still other aspects, the robot 14 may move the calibration box 810 to more than 16 positions / poses for image capture. It should be noted that as the calibration box 810 moves to various positions, the controller 10C monitors the consistency of the robot 14 kinematics, and in the event of kinematic inconsistencies (such as due to an abnormal robot encoder), the controller 10C warns the operator through the user interface GUI that attention to the robot 14 may be required.

[0128] At each position / pose of the calibration box 810, each 3D camera 310CA-310C4 detects the calibration box 810 to define a box surface reference. Figure 8 and Fig. 9 (See also Fig.10 and Fig.11 as well as Fig.12 and Fig.13 ), the data from the 3D cameras 310Ca1-310C4 are in the form of corresponding point clouds ( Fig. 9 yes Figure 8 Point cloud representation of a two-dimensional image; Fig.11 yes Fig.10 A point cloud representation of ; and Fig.13 yes Fig.12). Similar to the 3D box reference, in one aspect, the controller 10C employs the fork reference TFRP to "mask" redundant areas of the point cloud that are neither part of the calibration box 810 or the end-of-arm tooling 410, nor otherwise decisive for the robot 14 calibration. For example, points in the point cloud that are below the fork reference TFRP are removed, points in the point cloud that are to the left and right of the lift zone (i.e., the area in which the lift operates) are removed, and points that are farther (i.e., from the imaging camera) than the farthest edge of the lift aperture 44 are removed. The result of the masking is that the point cloud includes substantially only the calibration box 810 and the end-of-arm tooling 410.

[0129] A box reference is defined on the front and top faces of the calibration box 810, and a sampling box is created on the face of the calibration box 810 using the anomaly (similar to the reference box in this article). Fig.18B These sampling boxes are used to refine the corner positions of the calibration box 810 as described herein. Figure 8 , Fig. 9 , Fig.19B and Fig.19B The box reference BRI, BRF, BRT defined on the face of the calibration box 810 uses the direction of interest based on the direction of the Y axis of the robot flange 14F to define the X axis of the box reference BRI, BRF, BRT. The Z axis of the box reference BRI, BRF, BRT is along the Z axis of the fork origin reference TFRP (see Fig.18A and Fig.18B ) in the same direction (note that for convenience, the directional conventions described here and throughout this specification are used; and in other aspects, any suitable direction may be used). Fig. 9 ) determines the positions of the corresponding planes 900, 901 of the front face and the top face to determine the (initial) box reference BRIF, BRIT ( Fig.19A The controller 10C is configured to refine the initial box references BRIF, BRIT using any suitable corner detection algorithm, wherein the sampling block S7 (as referred to herein) is used. Fig.18B The planes 900, 901 are positioned / fitted on the surfaces of the corresponding calibration boxes 810 to define the refined box references BRF, BRT (see Fig.19B ). It should be noted that the origin of the box reference BRF, BRT may not be updated from the initial box reference BRIF, BRIT and may not be at the center of the box face of interest. Here, the rotation of the reference axes X, Y, Z is fixed in a precise manner (e.g., in order to define a calibration accuracy that is less than the accuracy of the robot (e.g., the accuracy of the robot is between about 1 mm and about 2 mm) as described herein).

[0130] In the case of defining the refined box surface reference BRF, BRT, determine the box surface reference BFR (see Fig.19B ), where the front face normal defines the Z axis of the box face reference BFR, the intersection between the top plane and the front plane defines the Y axis of the box face reference BFR, and the vector product of the Y axis and the Z axis defines the X axis. The X and Z origins of the box face reference BFR are located on the intersection between the front plane and the top plane, and the Y origin is aligned with the side of the top plane or the front plane (the side where the camera has the best view of that face).

[0131] In the case where a box surface reference BFR is defined for each of the three-dimensional cameras 310C1-310C4, a box surface reference BFR2D is determined for the electromagnetic sensor 500 (eg, the two-dimensional camera 500C). It should be noted that the box surface reference BFR2D may be calculated before the box surface reference BFR. Fig. 22 , the controller 10C extracts the edge of the calibration box 810 in the two-dimensional image with the aid of any suitable edge detection algorithm. For example, the controller 10C segments the edge at sharp curvature points and polarity changes, so that each segmented edge segment represents one side of a square on the grid pattern 800. For each edge segment end (e.g., one side of a square in the grid pattern 800), the controller 10C is configured to find three other local edge ends in the local area to form a quadrilateral (i.e., one square of the grid pattern 800). For the controller 10C to determine that a quadrilateral exists, the four edge ends should have two pairs of parallel facing segments within a predetermined tolerance, two pairs of parallel segments intersecting perpendicularly within a predetermined tolerance, each pair of facing segments having opposite polarities, and the color / hue of the square surrounding the edge segment must be close to, for example, green in opposite diagonal directions, and close to, for example, yellow (note that yellow and green are exemplary colors of the grid pattern described herein).

[0132] In one or more aspects, the point cloud of one or more three-dimensional cameras 310C1-310C4 is used to minimize the effects of variable image intensity, non-uniform lighting, shadows, complex scenes, perspective projections, and distortions (among other variations) on the segmentation of a two-dimensional image obtained by an electromagnetic sensor 500 (e.g., a two-dimensional camera 500C). Here, the image of the calibration box 810 is positioned in front of and close to the electromagnetic sensor 500 (e.g., a two-dimensional camera 500C) so that there is no perspective of the calibration box 810 in the two-dimensional image of the calibration box taken by the electromagnetic sensor 500 (e.g., a two-dimensional camera 500C). The box reference BFR, BFR2D is determined in the manner described herein. An approximate transformation between the three-dimensional cameras 310C1-310C4 and the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C) is found by referencing the cross product of BFR, BFR2D (e.g., camera3D_to_Camera2D transformation = BFR x BFR2D) (it should be noted that the above transformation is for exemplary purposes only, and there are multiple ways in which the three-dimensional to two-dimensional transformation can be determined). The faces of the calibration box 810 are found in the point cloud of each image captured by the three-dimensional cameras 310C1-310C4, and then mapped into the two-dimensional image using the approximate transformation and the intrinsic parameters of the electromagnetic sensor 500 (e.g., the two-dimensional camera 500C) for a given calibration box position and pose (such as those described herein) to project the faces of the calibration box 810 into the two-dimensional image plane. This projection of the faces from the point cloud to the two-dimensional image plane creates a two-dimensional image mask that is used to identify the locations of the box faces in the two-dimensional image (thereby substantially eliminating all remaining image complexity found in the two-dimensional image - for example, referring to Fig.10 and Fig.12 , masking the portion of the two-dimensional image outside the mask boundary to segment the two-dimensional image). Therefore, the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4 is substantially performed simultaneously or simultaneously with each other. The positioning and teaching of the robot 14 is also substantially performed simultaneously and automatically with the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 410C1-410C4.

[0133] With respect to the box surface reference BFR2D, the controller 10C sorts the intersections (described above), for example, from top to bottom, from left to right (or in any other suitable order). The sorted intersections are associated with their corresponding coordinates in the world calibration grid (the origin of the world calibration grid is the box surface reference BFR of the three-dimensional point cloud of the corresponding camera 310C1-310C4). The pose is defined for the box front face so that the X-axis extends from the upper left corner of the front face to the upper right corner, the Y-axis extends from the upper left corner of the front face to the lower left corner, and the Z-axis is the vector product of the X-axis and the Y-axis. The box surface reference BFR2D is defined in the same way for each robot 14. The controller 10C determines the pose of the two-dimensional camera 500C relative to the box surface reference BFR2D, for example, by means of a homography fit between the intersections found in the two-dimensional image and their corresponding world coordinates. The controller 10C implements homography fitting in a manner substantially similar to that described in Z. Zhang's article A flexible new technique for camera calibration, IEEE Journal of Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000, the disclosure of which is incorporated herein by reference in its entirety.

[0134] In the case where the calibration box 810 is detected in all two-dimensional images obtained by means of an electromagnetic sensor (e.g., the two-dimensional camera 500C), the robot base 14B to the electromagnetic sensor reference RBR and the calibration box 810 to the flange 14F posture are optimized. Fig. 20 , a flange to box center pose reference FBCPR is defined at the center of the calibration box 810 in the flange reference RFR. The general arrangement of the rotation axis is the same as that of the flange axis Fθ (see Figure 3D ), but are optimized independently. To define the flange to box center pose reference FBCPR, the pose of the calibration box 810 is nominally calibrated and then refined during optimization. As described herein, the calibration box 810 is an invariant relative to the robot 14 holding the calibration box 810, wherein the Y axis always points to the front face of the calibration box 810, the Z axis points downward, and the X axis points to the right side face of the calibration box 810 (as seen in the two-dimensional image). The X-axis, Y-axis, and Z-axis are in line with the edge of the calibration box 810 and, therefore, are oriented differently from the flange reference RFR that defines the flange to box center pose reference FBCPR.

[0135] refer to Fig.21A and Fig. 21B , a relationship between the pose of the grid 815 and the flange to box center pose reference FBCPR (e.g., pose grid to box reference PGBR) is established at least for the top and front faces of the calibration box 810. In some aspects, the relationship can also be established for the side faces of the calibration box 810. Reference Fig.21AAs an example, for the front face, the pose grid to box reference PGBR(front face) is defined at the first top left (as seen in the two-dimensional image) intersection of the grid 815 (see Fig.21A ), wherein the X-axis points to the right side of the calibration box 810, the Y-axis points downward, and the Z-axis points to the calibration box 810, as shown in FIG. Fig.21A Reference Fig. 21B , for the top face, the pose grid to box reference PGBR(top face) is defined at the intersection of the top grid 815 closest to the box center and the left face of the calibration box 810. The rotation of the axis of the pose grid to box reference PGBR(top face) is the same as the flange to box center pose reference FBCPR, with the Z axis pointing downward, the X axis pointing to the right face of the calibration box 810, and the Y axis pointing to the front face.

[0136] refer to Fig. 22 , in the fully automatic calibration of the robot 14, a box-to-2D camera pose or reference BCP is also defined. The box-to-2D camera reference BCP is defined such that the X-axis extends from the upper left corner to the upper right corner of the calibration box (as seen in the 2D image), the Y-axis extends from the upper left corner to the lower left corner of the calibration box 810, and the Z-axis points to the vector product between the X-axis and the Y-axis. The box-to-2D camera pose or reference BCP is defined in the same manner for each robot 14 that clamps the calibration box 810.

[0137] As can be appreciated, fully automatic robot calibration (i.e., positioning and teaching) is performed (substantially simultaneously with the calibration of the electromagnetic sensor 500 and the three-dimensional cameras 310C1-310C4) so ​​that the controller 10C (and the robot 14 it controls) "learns" or is accurately taught the features of the palletizer unit 10. Here, the resolution of the two-dimensional camera 500C is greater than the resolution of the three-dimensional cameras 310C1-310C4, and it is desired to learn the location of the physical item in the two-dimensional image in the three-dimensional world and the transformation therebetween. With the above references determined, the controller 10C (or, as described above, any suitable controller, such as the controller 199C) is configured to refine one or more of the above references. As an example, the reference optimization algorithm of the controller 10C creates an optimized solution, for example, using the Levenberg-Marquardt algorithm (or any other suitable optimization algorithm). During the optimization process, the position of the robot 14 is optimized (i.e., positioned) in the electromagnetic sensor 500 world and simultaneously in the palletizer cell space relative to each palletizer cell station using the collected information (from the reference determination above), including but not limited to, for example, the box surface on which the grid 815 is located, the grid intersections in pixels, the grid intersections in the grid world, and the position of the robot flange in the base reference RBR.

[0138] The Z-axis position of the robot base origin can also be optimized. For example, in a robot 14 having four degrees of freedom (eg, 4 axes - X, Y, Z, μ-see Figure 3D ) robot, the calibration box 810 cannot rotate about the X-axis (e.g., roll) and Y-axis (e.g., pitch) so that the origin of the Z-axis of the robot base is not locked in a particular position. The robot base 14B is not within the field of view of the cameras 310C1-310C4, 500C, but the flange 14F is within the field of view and can thus be detected. Here, the cameras 310C1-310C4 and / or the electromagnetic sensor 500 detect the flange 14F, and the height of the flange 14F at the flange reference RFR (as determined by the controller 10C from the images obtained by the cameras 310C1-310C4 and / or the electromagnetic sensor 500) is compared to the nominal CAD value to fix the origin of the robot base Z-axis. As described herein, in some aspects, the flange 14F can form part of a calibration fixture / jig (also referred to herein as a target object).

[0139] References Fig.14 As described above, block 1431 automatically calibrates the robot 14 and synchronizes the robot 14 axes with the cameras 310C1 - 310C4 , and this operation is repeated for each robot 14 of the palletizer unit 10 .

[0140] Still refer to Fig.14 , and also refer to Figure 3D As described herein, each robot 14 includes a distance sensor 411. The distance sensor 411 is calibrated ( Fig.14 , box 1432), so that the distance sensor 411 can be used to scan and calibrate the pick-up position of the conveyor 16. Here, as described herein, the fork reference TFRP is common to the robot 14 and the three-dimensional cameras 310C1-310C4. The controller 10C records and calibrates (e.g., using images of the conveyor mechanism 20 obtained by one or more three-dimensional cameras 310C1-310C4) the fork reference TFRP to the palletizer unit space, the three-dimensional camera 310Ca-310C4 space, and the electromagnetic sensor 500 space. The controller 10C teaches the robot 14 the reference TFRP position and the position of the pallet support SPAL (see Fig.14 , block 1436). The pose and position of the distance sensor 411 is recorded and calibrated via a calibration movement of the corresponding robot 14 that moves the distance sensor 411 to a pick-up position. For example, the end-of-arm tool 410 moves to a position such that the light 411L emitted from the distance sensor 411 is projected onto a flat surface (e.g., as a light "spot"), such as the top surface of the calibration box 810 (where the calibration box 810 is placed on one or more transport forks 20F1, 20F2 (see also Figure 5B)) or the top surface of the transport fork 20F1, 20F2 (assuming that the definition of the characteristic surface plane is predetermined or can be determined, such as by homography via the electromagnetic sensor 500, so as to accurately locate the light spot projected by the emitter of the distance sensor 411 on the calibration box 810 or the fork 20F1, 20F2 with comparable accuracy to the robot flange 14F positioning accuracy provided by the drive part encoder). As can be appreciated, the calibration box 810 (or fork 20F1, 20F2) provides / defines the flat surface plane PLN (on which the light 411L is projected), which can be determined by means of the electromagnetic sensor 500 via homography. The position of the flat plane PLN is determined, and the distance (e.g., ΔZ) from the plane PLN to the flange reference RFR (or from the plane PLN to the fork reference TFR) is determined. The distance ΔZ of the plane PLN to the flange reference RFR (or fork reference TFR) determined by the distance sensor 411 is compared with the determined (e.g., via homography) distance between the plane PLN and the flange reference RFR (or fork reference TFR) to correct the distance sensor 411 relative to the robot flange reference RFR / robot flange 14F (or fork reference TFR). With light 411L projected onto the calibration box 810, the end-of-arm tool 410 is rotated about the flange axis Fθ and the end-of-arm tool 410 is moved to different heights relative to the transport forks 20F1, 20F2. The 2D camera 500C images the projection of light 411L on the calibration box 810 to find (e.g., based on the position of the change in the projection of light 411L on the calibration box 810) the position of the distance sensor 411 on the end-of-arm tool 410 relative to the flange 14F, and correlates the distance measured by the distance sensor 411 with the height of the flange 14F to calibrate the distance sensor 411 relative to the flange 14F (and the tool reference RTR). This distance sensor 411 calibration is performed for each robot 14 of the palletizer unit 10. The forks 20F1, 20F2 can then be scanned with the corrected (e.g., relative to the flange reference RFR) distance sensor 411 to calibrate the distance from the fork 20F1, 20F2 resting surface to the flange reference RFR to refine the fork 20F1, 20F2 (Z-axis) posture relative to the robot 14. Deviations in the posture of the robot 14 may be resolved by multiple scans of the forks 20F1 and 20F2 at different heights and different orientations (eg, rotation of the end-of-arm tool by ±90°) using the distance sensor 411 .

[0141] See also Figure 6 and Fig.25 , using the distance sensor 411 to scan the corresponding pick-up conveyor 16 and calibrate the pick-up position of the pick-up conveyor 16 ( Fig.14, block 1433). The controller 10C commands the movement of the robot 14 to place the end-of-arm tool 410 above the pick-up conveyor 16 so that the distance sensor 411 scans, for example, the roller 16R of the pick-up conveyor 16 along the longitudinal axis of the pick-up conveyor 16. Here, the distance sensor 411 determines the position of the roller 16R (see Fig.25 ). It is also possible to scan the roller 16R at two or more locations along the length of the roller 16R (such as scanning (such as by Figure 6 The squareness of the conveyor 16 can be determined by scanning the roller 16R (indicated by the arrow in FIG. 4 ) at the end of the roller 16R near the lateral side of the conveyor 16. Multiple scans of the roller 16R can be compared and correlated to the robot base 14B to determine the angle of the roller 16R, and therefore the conveyor 16, relative to the robot base reference RBR. Multiple scans of the roller 16R can be performed at different heights and different orientations (e.g., ±90° rotation of the end-of-arm tool) with the help of the distance sensor 411 to resolve deviations in the posture of the robot 14.

[0142] With the position (and angle) of roller 16R known, the calibration box 810 is placed on the conveyor 16 and the robot 14 is commanded to move the end-of-arm tooling 410 so that the distance sensor 411 scans the top surface of the calibration box. The data from the distance sensor 411 obtained by scanning the top surface of the calibration box 810 on the conveyor is communicated to the vision system 310 so that a plane is fitted to the distance sensor data. With the dimensions of the calibration box 810 known, the position of the determined plane is used to determine the plane of roller 16R (and verify the plane of the roller, as can be determined by scanning the roller with the distance sensor 411 - see Fig.25 ). Knowing the plane of the conveyor 16 and the position of the rollers, the controller 10C determines the pick-up position of the conveyor 16. Calibration of the pick-up position of the conveyor 16 is performed for each robot 14 of the palletizer unit 10.

[0143] In the above calibration, the robot 14 is positioned in the palletizer unit space or reference frame (see Figure 3A and Fig.14 , block 1438). Here, for each robot 14, the controller 10C maps the path of the robot 14 to the limits of the palletizer cell space (as determined by the known design configuration of the palletizer cell). The controller 10C calibrates the robot 14 coordinates based on the posture of the palletizer cell station and the teach of the palletizer cell space to the robot 14, as described herein.

[0144] Still refer to Fig.14 And also refer to Figure 2 , Figure 5B and Figure 7In the fully automatic calibration of the palletizer unit 10, a pallet reference can be determined so that the position of the transfer forks 20F1, 20F2 on the empty pallet conveying mechanism 20 and the pallet support SPAL on the lift 12 is known. Here, the pallet support SPAL is circulated through the palletizer unit 10 to the lift 12 by the forks 20F1, 20F2 of the empty pallet conveying mechanism 20. As an example, 3 to 5 pallets are circulated; however, in other aspects, less than 3 or more than 5 pallets may be circulated. The vision system 310 images the pallet support SPAL, wherein each pallet support SPAL is located on the forks 20F1, 20F2 and wherein the pallet support SPAL is located on the lift 12. The vision system 310 determines the zero position of the lift 12 and the offset of the pallet supports SPAL based on images (e.g. 3D and / or 2D images) of the pallet supports SPAL to enable placing of case units CU on the pallet supports SPAL to build a pallet load PAL.

[0145] It is also possible to determine the position of the transfer forks 20F1, 20F2 of the palletizer unit 10 and the floor 700 based on the images of the transfer forks 20F1, 20F2 and the floor 700 obtained by the vision system 310 (see also Figure 3A ) between the distance 555. For example, a plane can be fitted to the point cloud data points corresponding to the floor 700 and the forks 20F1, 20F2 (as described herein), and the distance 555 can be determined by the two planes.

[0146] refer to Fig.14 , block 1450, and also with reference to Figures 23A-24C , imaging the finger 450F of the end-of-arm tool 410, for example, to optimize the tool center point or tool reference RTR, and to account for deflection of the finger 450F under tool loads, such as may be caused by clamping calibration loads / fixture CLDs (see Figure 8 - For example, in one aspect, during calibration of the vision system by the calibration box 810), and under no-load conditions (i.e., empty end-of-arm tool 410, clamping box in fully retracted position on the fingers (see Fig.23A ), or an unclamped box held by the end-of-arm tool 410) confirms the tool reference RTR. Fig.14 , Block 1450 may include calibrating the robot 14 (e.g., Fig.14 , block 1430), and / or may be in the circulation of pallets (e.g., Fig.14, box 1440). To optimize the tool reference RTR, a set / series of images are needed that set the calibration load CLD in a series of positions that best describe the deflection caused by the clamping end-of-arm tool 410 of the calibration fixture CLD within the electromagnetic sensor field of view and achieve optimization of the tool reference RTR under no-load conditions. Here, this series of images is obtained by the electromagnetic sensor 500 (e.g., two-dimensional images obtained by the two-dimensional camera 500C) and includes but is not limited to side (side view) images of the retracted calibration box 810 (see Fig.23A ), a side (side view) image of the calibration box 810 being pushed (see Fig. 23B , where the calibration box 810 is pushed to the tip of finger 450F by the pusher assembly 28 of the end-of-arm tool 410) and a frontal image of finger 450F such that only the top of finger 450F is visible in the image, where the direction indicators of the above images (e.g., side view, facing forward, etc.) are relative to the field of view of the electromagnetic sensor 500.

[0147] The deflection algorithm of the controller 10C for optimizing the tool reference RTR calculates the load-bearing member (herein 450F, see FIG. 4 ) that maintains the “undeflected” load-bearing member of the end-of-arm tool 410. Fig.23A ) and the box 810 on the mounting surface of the end-of-arm tool 410 to hold the "flex" load-bearing member (also referred to here as 450F, see Fig. 23B ) between the posture changes of the box 810. As can be appreciated, "unflexed" means that it may not be truly unflexed, such as when the calibration box 810 is clamped in the fully retracted position (see Fig.23A ) and the finger 450F is minimally deflected (e.g., the tip of the finger is about 2 mm or less below the base of the finger). This minimal deflection of the finger is within the positioning error of the robot 14 (e.g., a positioning error between about 1 mm and about 2 mm) and can be ignored when calculating the "undeflected" tool reference RTR. Here, the controller 10C detects (with images taken by cameras 310C1-310C4, 500C) the calibration box 810 in three dimensions and in the imaging space of the electromagnetic sensor 500 (e.g., two dimensions of the two-dimensional camera 500C, in a manner substantially similar to that described herein) with the help of a deflection algorithm, and assigns calibration box 810 references in the undeflected and deflected images, in the three-dimensional world of the two-dimensional camera 500C, where these references are the deflected finger box center to the electromagnetic sensor reference and the undeflected finger box center to the electromagnetic sensor reference, respectively. The postures of the flexed finger box center to the electromagnetic reference and the unflexed finger box center to the electromagnetic sensor reference are such that the Y axis points to the front face of the calibration box 810, the X axis points to the right side face of the calibration box 810, and the Z axis points downward, with the origin at the center of the calibration box 810 (see Fig.23A and Fig. 23B). The flexed box center to unflexed box center pose reference is calculated based on the flexed finger box center to electromagnetic sensor reference and the unflexed finger box center to electromagnetic sensor reference, thereby obtaining a transformed pose linking the two calibration box positions.

[0148] With the assigned flexed finger box center to the electromagnetic sensor reference and the unflexed finger box center to the electromagnetic sensor reference, by means of Fig.23A The tool reference RTR is detected in another image by the calibration box 810 positioned as shown in FIG; but the end-of-arm tool 410 is moved so that the finger 450F is positioned to enable easy (as described herein) image segmentation and detection, so that the tool reference RTR is detected in the pixels of the two-dimensional image, and the tool reference RTR is projected to the plane "undeflected box to electromagnetic sensor", where:

[0149] Box notflex toelectromagnetic sensor=robot base referentialRBR to electromagnetic sensor*robot flange to robot base*flex box center to robot flange*flex box center to unflex box center

[0150] Without flexing the box to the EM sensor pose, the pixel points are projected to the plane below the finger 450F, thereby becoming known in the three-dimensional world of the EM sensor 500 (e.g., a two-dimensional camera). The pixel points are then reprojected to the top of the finger 450F and then described in terms of flange coordinates before being sent to the robot 14. The flex algorithm can also recalculate the X axis of the tool reference RTR to measure the squareness of the tool reference RTR relative to the flange reference RFR. Therefore, the pitch of the end-of-arm tool 410 clamped to the flange 14F can be resolved.

[0151] refer to Figures 24A-24C To calibrate the tool reference RTR, the TCP calibration algorithm of the controller 10C commands the robot 14 to move to a suitable detection position, where the calibration box 810 is held thereon. The detection position is at a distance from the cameras 310C1-310C4, 500C and at a height within the workspace that represents the working height of the tool load (and therefore, the load supporting surface / member) during pallet construction (see Fig.24A). At this position, the top of the calibration box 810 is detected to form a plane 2400 in the electromagnetic sensor 500 world. Using the formed plane 2400, a parallel plane 2401 is formed below the formed plane at a predetermined distance DP (e.g., corresponding to the setting of the load support surface of the end-of-arm tool 410 relative to the tool load, here, 12 inches, or in other aspects, the distance can be greater or less than about 12 inches), which corresponds to the load support (here, 450F) plane (note that the calibration box 810 has an exemplary height of about 12 inches). Based on this image (such as Fig.24A ), by moving left / right ( Fig. 24B ) and / or up / down( Fig.24C ) moves the calibration box 810 to obtain a difference image, wherein the controller 10C uses the difference image to detect the edge of the finger 450F. Fig. 24B In FIG. 4 , the calibration box 810 (and finger 450F) is moved left / right by a distance of about 1 mm (in other aspects, the distance may be greater or less than about 1 mm) to obtain a difference image. Fig.24C , the calibration box 810 (and finger 450F) is moved up / down a distance of about 2 mm (in other aspects, the distance may be greater or less than about 2 mm) to obtain a difference image.

[0152] The controller 10C analyzes the difference images with the aid of any suitable image analysis algorithm to detect the tip of the finger 450F and the finger orientation. For example, the left / right difference images are analyzed to find the finger by geometry (e.g., the finger is the first object encountered by the robot in the image, where the images are analyzed from bottom to top. In the case of identifying the finger, the two edges of each finger 450F are found so that the center line of each viewfinder is determined. Using the center line, a projection area is established in the upper and lower different images, where the edge of each finger is found and projected to the center line. In the case of finding the pixel of the finger edge, the pixel is projected to the plane 2401 to obtain a point in the three-dimensional world of the electromagnetic sensor 500 (here, for example, the two-dimensional camera 500C), where the reference point is transformed by the following chain:

[0153] TCPInFlangeRef = PoseRobotBaseToFlange * PoseCam2DToRobotBase * TCPInCam2Dref (TCP in flange ref = PoseRobotBaseToFlange * PoseCam2DToRobotBase * TCP in 2DCam2Dref)

[0154] Once the tool reference RTR is described in the flange reference RFR, the tool reference RTR is valid for the entire work area. The axes of the tool reference RTR are obtained in a similar manner. For example, the center line of the finger is projected onto the plane 2401 and serves as the X axis of the end-of-arm tool 410. The Z axis is normal to the plane of the finger 2401, but points downward, and the Y axis is perpendicular to both the X and Z axes.

[0155] refer to Figures 3A-3C , Figure 5B , Figure 7 and Fig.17 , the lifter 12 has a lifter origin reference EOR located substantially at the center of the orifice 44 of the lifter 12. The location of the lifter origin reference EOR may be defined by the nominal dimensions of the palletizing unit 10 (e.g., from a computer-aided drawing (CAD) file) or in any other suitable manner. The lifter origin reference EOR is common to each robot 14 and enables (in addition to or in lieu of the telescopic fork origin reference TFRP) synchronization or otherwise connection of the workspaces of the robots 14. The lifter origin reference EPR has an origin positioned at a predetermined distance in the X direction from the robot base reference RBR, and a Y origin located substantially at the midpoint between the robot base reference RBR. The lifter origin reference EOR is projected onto the plane of the empty pallet conveying mechanism 20 (see Fig.17 - so that the Z axis origin is at the top surface of the empty pallet conveying mechanism 20). The lift origin reference EOR is used to map the palletizer unit 10 working space.

[0156] See also Fig.18A and Fig.18B , a loader origin reference EOR is also defined in each of the three-dimensional cameras 310C1-310C4. The controller 10C uses the loader origin reference EOR to segment the point cloud of the corresponding three-dimensional camera 310C1-310C4. Here, the loader origin reference EOR locates the orifice 449 (and its sides 44S1-44S4 / boundaries) of the loader 12 in the workspace of the palletizer unit 10 relative to each of the three-dimensional cameras 310C1-310C4. As described above, the loader origin reference EOR is defined where the Z axis points upward from the plane defined by the top surfaces of the forks 20F1, 20F2, and the x axis is in line with the centerline of the orifice 44 (see Fig.18A and Fig.18B, the centerline of the orifice 44 may coincide with the centerline of the forks 20F1, 20F2), pointing away from the 3D cameras 310C1-310C4. The origin of the loader origin reference EOR of each respective 3D camera 310C1-310C4 is at the root of the X and Z planes to which the cameras 310C1-310C4 are projected. The Y origin is then in line with the centerline CL of the orifice 44. Exemplary images from, for example, 3D cameras 310C1 and 310C3 are illustrated in Fig.18A and Fig.18B middle, Fig.18A and Fig.18B Point clouds of the orifice 44 (and its sides 44S1-44S4) detected by the respective cameras and the loader origin reference EOR are shown. Here, in some aspects, given the predetermined positions and known characteristics of the three-dimensional cameras 310C1-310C4, selecting a discrete common reference (such as the loader origin reference EOR) for each three-dimensional camera 310C1-310C4 can be used to calibrate the three-dimensional cameras 310C1-310C4 to each other by resolving the pose of the loader origin reference EOR and the characteristics from the pose detection. In this case, the three-dimensional camera calibration is commensurate with the robot 14 / end effector 410 placement accuracy.

[0157] refer to Figures 3A-3D , Figure 5A-13 , Fig.15 as well as Figure 19A-26 A method is provided for automatically building a pallet load PAL of a pallet load unit CU onto a pallet support SPAL by means of a pallet building device, such as a palletizer unit 10. In the method, a pallet building device ( Fig.26, box 2600), the pallet building device comprises a frame 24, the frame 24 defines a pallet station, the pallet station having a pallet building base 301 for a pallet load support SPAL. The articulated robot 14 is connected to the frame 24 and has: a drive part 14DS, the drive part 14DS having a plurality of independent drive axes arranged in the robot reference system (e.g., the drive axes of the drive 14D); and an end effector (e.g., an end-of-arm tool 410) configured to hold a pallet load unit CU in transport. At least one three-dimensional camera 310C1-310C4 is connected to the frame 24, separate and distinct from the articulated robot 14. A controller 10C is operably connected to the at least one articulated robot 14 and the at least one three-dimensional camera 310C1-310C4, the controller 10C being configured to control the articulated robot movement relative to the pallet building base 301 and thereby achieve a pallet load building of the pallet load PAL. The palletizer unit 10 further comprises an automatic calibration system ACS for calibrating the articulated robot 14, wherein the controller 10C is communicatively connected to the automatic calibration system ACS. In the method, the articulated robot 14 transports and places the palletized object unit CU. Fig.26 , block 2610) onto the pallet supports SPAL in order to build the pallet load PAL on the pallet building base 301. At least one 3D camera 310C1-310C4 generates a 3D image ( Fig.26 , block 2620). The controller 10C automatically calibrates the articulated robot 14 from a substantially uncalibrated state ( Fig.26 , box 2630). The controller 10C automatically teaches the articulated robot 14 the location of the pallet station ( Fig.26 , block 2640), thereby resolving the position of the end effector (e.g., the end-of-arm tool 410) to the pallet station to build the pallet load PAL by placing the pallet load item unit CU on the pallet support SPAL. The controller 10C calibrates at least one three-dimensional camera 310C1-310C4 ( Fig.26 , box 2650).

[0158] refer to Figures 3A-3D , Figure 5A-13 , Fig.15 , Figure 19A-25 and Fig. 27 A method is provided for automatically building a pallet load PAL of a pallet load unit CU onto a pallet support SPAL by means of a pallet building device, such as a palletizer unit 10. In the method, a pallet building device ( Fig. 27, box 2700), the pallet building device comprises a frame 24 defining a pallet station having a pallet building base 301 for a pallet load support SPAL. An articulated robot 14 is connected to the frame 24 and has: a drive part 14DS having a plurality of independent drive axes (e.g. drive axes of a drive 14D) arranged in a robot reference frame; and an end effector (e.g. end-of-arm tool 410) configured to hold a pallet load unit CU in transport. A controller 10C is operably connected to at least one articulated robot 14, and an automatic calibration system ACS for calibrating the articulated robot comprises at least one three-dimensional camera 310C1-310C4. At least one three-dimensional camera 310C1-310C4 is connected to the frame 24 and is configured to perform three-dimensional imaging of a part of the articulated robot 14 and at least one of the pallet supports SPAL or the pallet load build on the pallet building base 301. The articulated robot 14 transports and places the palletized item unit CU ( Fig. 27 , block 2710) onto the pallet support SPAL in order to build the pallet load PAL on the pallet building base 301. The controller controls the articulated robot movement relative to the pallet building base and thereby achieves the pallet load building ( Fig. 27 , box 2720). The controller 10C generates a three-dimensional image of the pallet supports SPAL on the pallet building base 301 and the pallet loading building on the pallet supports SPAL ( Fig. 27 , block 2730). The controller 10C is communicatively connected to the automatic calibration system ACS and automatically calibrates the articulated robot 14 from a substantially uncalibrated state with the aid of the three-dimensional image ( Fig. 27 , box 2740), and automatically teach the articulated robot 14 the reference of the pallet station (as described herein) ( Fig. 27 , box 2750), thereby characterizing the placement position of each pallet load item unit CU placed by means of the end effector that builds the pallet load PAL on the pallet support SPAL.

[0159] According to one or more aspects of the present disclosure, there is provided a pallet building device for automatically building a pallet load of pallet load item units onto a pallet support, the pallet building device comprising:

[0160] a frame defining a pallet station having a pallet building base for a pallet support;

[0161] an articulated robot connected to the frame and having a drive portion having a plurality of independent drive axes arranged in a reference frame of the robot, the articulated robot being configured to transport and place a palletized item unit onto a pallet support so as to build the palletized item on a pallet building base, the articulated robot having an end effector configured to hold the palletized item unit during transport;

[0162] at least one three-dimensional camera connected to the frame, separate and distinct from the articulated robot, and configured to generate a three-dimensional image of a pallet support on the pallet build base and a palletized build on the pallet support;

[0163] a controller operably connected to the at least one articulated robot and the at least one three-dimensional camera, the controller being configured to control movement of the articulated robot relative to the pallet build base and thereby effectuate a pallet load build of the pallet load; and

[0164] An automatic calibration system for calibrating an articulated robot, the controller being communicatively connected to the automatic calibration system and configured to automatically calibrate the articulated robot from a substantially uncalibrated state and automatically teach the articulated robot the position of a pallet station, thereby resolving an end effector to the position of the pallet station to build a pallet load by placing pallet load item units on a pallet support, and the controller being configured to calibrate at least one three-dimensional camera relative to the end effector.

[0165] According to one or more aspects of the present disclosure, the controller and automatic calibration system are configured to enable automatic calibration and automatic teaching of the articulated robot, which occurs substantially simultaneously with the automatic calibration of each of the at least one three-dimensional cameras that images each pallet load item unit placed by the articulated robot to build a pallet load.

[0166] According to one or more aspects of the present disclosure, an automated calibration system includes one or more of structures and markings having calibration features coupled to an end effector, the calibration features having predetermined characteristics describing a predetermined or known shape or pattern.

[0167] According to one or more aspects of the present disclosure, these calibration features are dependent on the end effector, or these calibration features and the end effector are jointly dependent on a common component of the articulated robot, and these calibration features and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

[0168] According to one or more aspects of the present disclosure, the calibration features are arranged relative to the articulated robot such that predetermined characteristics determine a predetermined relationship between the calibration features and a reference frame of the robot.

[0169] According to one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor configured to sense these calibration features, the electromagnetic sensor is separate and different from the three-dimensional camera, and the controller is communicatively coupled to the electromagnetic sensor and records sensor signals embodying predetermined characteristics of corresponding postures for the articulated robot (and corresponding postures of the calibration features due to movement of the articulated robot), and the recording occurs substantially simultaneously with the electromagnetic sensor sensing the calibration features and generating the sensor signals.

[0170] According to one or more aspects of the present disclosure, the controller is configured to move the articulated robot and position the calibration feature to an initial position where the electromagnetic sensor senses the calibration feature, the controller is configured to resolve the posture of the calibration feature relative to the electromagnetic sensor reference system, and determine the transformation between the electromagnetic sensor reference system and the three-dimensional camera reference system based on the resolved posture, thereby calibrating the at least one three-dimensional camera.

[0171] According to one or more aspects of the present disclosure, there is provided a pallet building device for automatically building a pallet load of a pallet load item unit onto a pallet support. The pallet building device comprises: a frame, the frame defining a pallet station, the pallet station having a pallet building base for the pallet support; an articulated robot, the articulated robot being connected to the frame and having a drive portion having a plurality of independent drive axes arranged in a robot reference frame, the articulated robot being configured to transport and place the pallet load item unit onto the pallet support so as to build the pallet load on the pallet building base, the articulated robot having an end effector configured to hold the pallet load item unit during transport; a controller, the controller being operably connected to the at least one articulated robot, the controller being configured to control the movement of the articulated robot relative to the pallet building base, and thereby achieving pallet load building of the pallet load, and configured to generate three-dimensional imaging of pallet supports on a pallet building base and pallet load structures on the pallet supports; and an automatic calibration system for calibrating an articulated robot, the automatic calibration system comprising at least one three-dimensional camera connected to a frame and configured to perform three-dimensional imaging of a portion of the articulated robot and at least one of the pallet supports or pallet load structures on the pallet building base; wherein the controller is communicatively connected to the automatic calibration system to automatically calibrate the articulated robot from a substantially uncalibrated state with the aid of the three-dimensional images and automatically teach a reference of the articulated robot pallet station and thereby characterize the placement position of each of the pallet load item units placed with the aid of an end effector that builds the pallet load on the pallet supports.

[0172] According to one or more aspects of the present disclosure, the at least one three-dimensional camera is connected to the frame, separate and distinct from the articulated robot.

[0173] According to one or more aspects of the present disclosure, the controller is configured to resolve the position of the end effector to the pallet station to build a pallet load by placing pallet load item units on the pallet supports.

[0174] According to one or more aspects of the present disclosure, the controller is configured to calibrate the at least one three-dimensional camera relative to the end effector.

[0175] According to one or more aspects of the present disclosure, the controller and automatic calibration system are configured to enable automatic calibration and automatic teaching of the articulated robot, which occurs substantially simultaneously with the automatic calibration of each of the at least one three-dimensional cameras that images each pallet load item unit placed by the articulated robot to build a pallet load.

[0176] According to one or more aspects of the present disclosure, the automated calibration system includes one or more of structures and markings having calibration features coupled to an end effector, the calibration features having predetermined characteristics describing a predetermined or known shape or pattern.

[0177] According to one or more aspects of the present disclosure, these calibration features are dependent on the end effector, or these calibration features and the end effector are jointly dependent on a common component of the articulated robot, and these calibration features and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

[0178] According to one or more aspects of the present disclosure, the calibration features are arranged relative to the articulated robot such that predetermined characteristics determine a predetermined relationship between the calibration features and a reference frame of the robot.

[0179] According to one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor configured to sense the calibration features, the electromagnetic sensor being separate and distinct from the three-dimensional camera, the controller being communicatively coupled to the electromagnetic sensor and recording sensor signals embodying predetermined characteristics of corresponding postures for the articulated robot, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration features and generating the sensor signals.

[0180] According to one or more aspects of the present disclosure, the controller is configured to move the articulated robot and position the calibration features to initial positions where electromagnetic sensors sense these calibration features, and the controller is configured to resolve the posture of the calibration features relative to the electromagnetic sensor reference system, and determine the transformation between the electromagnetic sensor reference system and the three-dimensional camera reference system based on the resolved posture, thereby calibrating the at least one three-dimensional camera.

[0181] According to one or more aspects of the present disclosure, there is provided a method for automatically building a pallet load of a pallet load unit onto a pallet support by means of a pallet building device. The method includes providing a pallet building device, the pallet building device including: a frame, the frame defining a pallet station, the pallet station having a pallet building base for a pallet support; an articulated robot, the articulated robot connected to the frame and having: a drive portion having a plurality of independent drive axes arranged in a robot reference frame; and an end effector configured to hold the pallet load unit in transport; at least one three-dimensional camera, the three-dimensional camera connected to the frame, separate and distinct from the articulated robot; a controller operably connected to the at least one articulated robot and the at least one three-dimensional camera, the controller being configured to control the articulated robot movement relative to the pallet building base and thereby achieve pallet load building of the pallet load; and an automatic calibration system for calibrating the articulated robot, wherein the controller is communicatively connected to the automatic calibration system. The method further comprises: transporting and placing a pallet load item unit onto a pallet support with the aid of the articulated robot so as to build the pallet load on the pallet building base; generating a three-dimensional image of the pallet support on the pallet building base and the pallet load structure on the pallet support with the aid of the at least one three-dimensional camera; automatically calibrating the articulated robot from a substantially uncalibrated state with the aid of a controller; automatically teaching the position of the articulated robot pallet station with the aid of the controller so as to resolve the position of the end effector to the pallet station to build the pallet load by placing the pallet load item unit onto the pallet support; and calibrating the at least one three-dimensional camera relative to the end effector with the aid of the controller.

[0182] According to one or more aspects of the present disclosure, the controller and automatic calibration system enable automatic calibration and automatic teaching of the articulated robot, which occurs substantially simultaneously with the automatic calibration of each of the at least one three-dimensional cameras that images each pallet load item unit placed by the articulated robot to build a pallet load.

[0183] According to one or more aspects of the present disclosure, the automated calibration system includes one or more of structures and markings having calibration features coupled to an end effector, the calibration features having predetermined characteristics describing a predetermined or known shape or pattern.

[0184] According to one or more aspects of the present disclosure, these calibration features are dependent on the end effector, or these calibration features and the end effector are jointly dependent on a common component of the articulated robot, and these calibration features and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

[0185] According to one or more aspects of the present disclosure, the calibration features are arranged relative to the articulated robot such that predetermined characteristics determine a predetermined relationship between the calibration features and a reference frame of the robot.

[0186] According to one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor that is separate and different from the three-dimensional camera, and the method also includes: sensing calibration features with the help of the electromagnetic sensor; and recording sensor signals embodying predetermined characteristics of corresponding postures for the articulated robot with the help of a controller that is communicatively coupled to the electromagnetic sensor, wherein the recording occurs substantially simultaneously with the electromagnetic sensor sensing these calibration features and generating sensor signals.

[0187] According to one or more aspects of the present disclosure, the method further includes, with the aid of a controller: moving the articulated robot and positioning the calibration features to initial positions where the electromagnetic sensor senses the calibration features; and resolving the postures of the calibration features relative to an electromagnetic sensor reference system, and determining the transformation between the electromagnetic sensor reference system and a three-dimensional camera reference system based on the resolved postures, thereby calibrating the at least one three-dimensional camera.

[0188] According to one or more aspects of the present disclosure, there is provided a method for automatically building a pallet load of a pallet load unit onto a pallet support by means of a pallet building device. The method comprises providing a pallet building device, the pallet building device comprising: a frame defining a pallet station having a pallet building base for a pallet support; an articulated robot connected to the frame and having: a drive portion having a plurality of independent drive axes arranged in a robot reference frame; and an end effector configured to hold the pallet load unit in transport; a controller operably connected to the at least one articulated robot; and an automatic calibration system for calibrating the articulated robot and comprising at least one three-dimensional camera connected to the frame and configured to perform three-dimensional imaging of a portion of the articulated robot and at least one of a pallet support or a pallet load structure on the pallet building base. The method further comprises: transporting and placing pallet load item units onto pallet supports by means of an articulated robot so as to build the pallet load on the pallet building base; controlling the movement of the articulated robot relative to the pallet building base by means of a controller and thereby achieving pallet load building of the pallet load; and generating three-dimensional imaging of the pallet supports on the pallet building base and the pallet load building on the pallet supports by means of the controller; wherein the controller is communicatively connected to the automatic calibration system and automatically calibrates the articulated robot from a substantially uncalibrated state by means of the three-dimensional image and automatically teaches a reference of the articulated robot pallet station to characterize the placement position of each of the pallet load item units placed by means of the end effector that builds the pallet load on the pallet support.

[0189] According to one or more aspects of the present disclosure, the at least one three-dimensional camera is connected to the frame, separate and distinct from the articulated robot.

[0190] According to one or more aspects of the present disclosure, the method further includes resolving, by means of the controller, a position of the end effector to a pallet station to build a pallet load by placing pallet load item units on the pallet supports.

[0191] According to one or more aspects of the present disclosure, the controller calibrates the at least one three-dimensional camera relative to the end effector.

[0192] According to one or more aspects of the present disclosure, the controller and automatic calibration system enable automatic calibration and automatic teaching of the articulated robot, which occurs substantially simultaneously with the automatic calibration of each of the at least one three-dimensional cameras that images each pallet load item unit placed by the articulated robot to build a pallet load.

[0193] According to one or more aspects of the present disclosure, the automated calibration system includes one or more of structures and markings having calibration features coupled to an end effector, the calibration features having predetermined characteristics describing a predetermined or known shape or pattern.

[0194] According to one or more aspects of the present disclosure, these calibration features are dependent on the end effector, or these calibration features and the end effector are jointly dependent on a common component of the articulated robot, and these calibration features and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

[0195] According to one or more aspects of the present disclosure, the calibration features are arranged relative to the articulated robot such that predetermined characteristics determine a predetermined relationship between the calibration features and a reference frame of the robot.

[0196] According to one or more aspects of the present disclosure, the automatic calibration system includes an electromagnetic sensor that is separate and distinct from the three-dimensional camera, and the method further includes: sensing the calibration features with the aid of the electromagnetic sensor; wherein the controller is communicatively coupled to the electromagnetic sensor and records sensor signals embodying predetermined characteristics of the corresponding posture for the articulated robot, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration features and generating the sensor signals.

[0197] According to one or more aspects of the present disclosure, the method further includes, with the aid of a controller: moving the articulated robot and positioning the calibration features to initial positions where the electromagnetic sensors sense the calibration features; and resolving the postures of the calibration features relative to an electromagnetic sensor reference system, and determining the transformation between the electromagnetic sensor reference system and a three-dimensional camera reference system based on the resolved postures, thereby calibrating the at least one three-dimensional camera.

[0198] It should be understood that the foregoing description is merely an illustration of aspects of the present disclosure. Various alternatives and modifications may be envisioned by those skilled in the art without departing from aspects of the present disclosure. Therefore, aspects of the present disclosure are intended to include all such alternatives, modifications, and variations that fall within the scope of any appended claims. Furthermore, the fact that different features are recited in mutually different dependent or independent claims does not indicate that a combination of these features cannot be used to advantage, and such a combination remains within the scope of aspects of the present disclosure.

Claims

1. A pallet building device for automatically building a pallet load of pallet load items into a pallet support, the pallet building device comprising: a frame defining a pallet station having a pallet building base for the pallet supports; an articulated robot connected to the frame and having a drive section having a plurality of independent drive axes arranged in a reference frame of the robot, the articulated robot being configured to transport and place the palletized item unit onto the pallet support so as to build the palletized item on the pallet building base, the articulated robot having an end effector configured to hold the palletized item unit during transport; at least one three-dimensional camera connected to the frame, separate and distinct from the articulated robot, and configured to generate three-dimensional imaging of the pallet supports on the pallet build base and the pallet load build on the pallet supports; a controller operably connected to said at least one articulated robot and said at least one three-dimensional camera, said controller being configured to control movement of the articulated robot relative to said pallet build base and thereby effectuate said pallet load build of said pallet load; as well as an automatic calibration system for calibrating the articulated robot, the controller being communicatively connected to the automatic calibration system and configured to automatically calibrate the articulated robot from a substantially uncalibrated state and automatically teach the articulated robot the position of the pallet station, thereby resolving the position of the end effector to the pallet station to build the pallet load by placing the pallet load item units on the pallet supports, and the controller being configured to calibrate the at least one three-dimensional camera relative to the end effector.

2. A pallet building apparatus according to claim 1, wherein the controller and the automatic calibration system are configured to achieve automatic calibration and automatic teaching of the articulated robot, which occurs substantially simultaneously with the automatic calibration of each of the at least one three-dimensional camera that images each pallet load item unit placed by the articulated robot to build the pallet load.

3. The pallet building apparatus of claim 1 , wherein the automatic calibration system comprises one or more of a structure and a marking having a calibration feature connected to the end effector, the calibration feature having predetermined characteristics describing a predetermined or known shape or pattern.

4. A pallet building device according to claim 3, wherein the calibration feature is dependent on the end effector, or the calibration feature and the end effector are jointly dependent on a common component of the articulated robot, and the calibration feature and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

5. A pallet building apparatus according to claim 3, wherein the calibration feature is arranged relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

6. The pallet building device of claim 3, wherein the automatic calibration system comprises an electromagnetic sensor configured to sense the calibration feature, the electromagnetic sensor being separate and distinct from the three-dimensional camera, the controller being communicatively coupled to the electromagnetic sensor and recording a sensor signal embodying the predetermined characteristic of the corresponding posture for the articulated robot, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration feature and generating the sensor signal.

7. A pallet building device according to claim 6, wherein the controller is configured to move the articulated robot and position the calibration feature to an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to resolve the posture of the calibration feature relative to the electromagnetic sensor reference system, and determine the transformation between the electromagnetic sensor reference system and the three-dimensional camera reference system based on the resolved posture, thereby calibrating the at least one three-dimensional camera.

8. A pallet building device for automatically building a pallet load of pallet load items into a pallet support, the pallet building device comprising: a frame defining a pallet station having a pallet building base for the pallet supports; an articulated robot connected to the frame and having a drive section having a plurality of independent drive axes arranged in a reference frame of the robot, the articulated robot being configured to transport and place the palletized item unit onto the pallet support so as to build the palletized item on the pallet building base, the articulated robot having an end effector configured to hold the palletized item unit during transport; A controller, the controller: operably connected to the at least one articulated robot, the controller being configured to control movement of the articulated robot relative to the pallet building base and thereby effectuate pallet load building of the pallet load, and being arranged to generate a three-dimensional image of the pallet supports on the pallet building base and the pallet loading build on the pallet supports; as well as an automated calibration system for calibrating the articulated robot, the automated calibration system comprising at least one three-dimensional camera connected to the frame and configured to image in three dimensions a portion of the articulated robot and at least one of the pallet support or the pallet load build on the pallet build base; wherein the controller is communicatively connected to the automatic calibration system so as to automatically calibrate the articulated robot from a substantially uncalibrated state with the aid of the three-dimensional image and automatically teach the articulated robot a reference of the pallet station and thereby characterize a placement position of each of the pallet load item units placed with the end effector building the pallet load on the pallet support.

9. The pallet building apparatus of claim 8, wherein the at least one three-dimensional camera is connected to the frame, separate and distinct from the articulated robot.

10. The pallet building apparatus of claim 8, wherein the controller is configured to resolve the position of the end effector to the pallet station to build the pallet load by placing the pallet load item units on the pallet supports.

11. The pallet building apparatus of claim 8, wherein the controller is configured to calibrate the at least one three-dimensional camera relative to the end effector.

12. The pallet building apparatus of claim 8, wherein the controller and the automatic calibration system are configured to enable automatic calibration and automatic teaching of the articulated robot substantially simultaneously with automatic calibration of each of the at least one three-dimensional cameras that image each pallet load item unit placed by the articulated robot to build the pallet load.

13. The pallet building apparatus of claim 8, wherein the automatic calibration system comprises one or more of a structure and a marking having a calibration feature connected to the end effector, the calibration feature having predetermined characteristics describing a predetermined or known shape or pattern.

14. A pallet building device according to claim 13, wherein the calibration feature is dependent on the end effector, or the calibration feature and the end effector are jointly dependent on a common component of the articulated robot, and the calibration feature and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

15. A pallet building apparatus according to claim 13, wherein the calibration feature is arranged relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

16. The pallet building device of claim 13, wherein the automatic calibration system comprises an electromagnetic sensor configured to sense the calibration feature, the electromagnetic sensor being separate and distinct from the three-dimensional camera, the controller being communicatively coupled to the electromagnetic sensor and recording a sensor signal embodying the predetermined characteristic of the corresponding posture for the articulated robot, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration feature and generating the sensor signal.

17. A pallet building device according to claim 16, wherein the controller is configured to move the articulated robot and position the calibration feature to an initial position where the electromagnetic sensor senses the calibration feature, and the controller is configured to resolve the posture of the calibration feature relative to the electromagnetic sensor reference system, and determine the transformation between the electromagnetic sensor reference system and the three-dimensional camera reference system based on the resolved posture, thereby calibrating the at least one three-dimensional camera.

18. A method for automatically building a pallet load of palletized item units onto a pallet support by means of a pallet building device, the method comprising: A pallet building device is provided, the pallet building device comprising: a frame defining a pallet station having a pallet building base for the pallet supports, an articulated robot connected to the frame and having: a drive portion having a plurality of independent drive axes arranged in a robot reference frame; and an end effector configured to hold the palletized item unit in transport, at least one three-dimensional camera connected to the frame, separate and distinct from the articulated robot, a controller operably connected to said at least one articulated robot and said at least one three-dimensional camera, said controller being configured to control movement of the articulated robot relative to said pallet building base and thereby effectuate said pallet load build of said pallet load, and an automatic calibration system for calibrating the articulated robot, wherein the controller is communicatively connected to the automatic calibration system; transporting and placing the palletized load unit onto the pallet support by means of the articulated robot so as to build the palletized load on the pallet building base; generating a three-dimensional image of the pallet supports on the pallet building base and the pallet loading build on the pallet supports by means of the at least one three-dimensional camera; automatically calibrating the articulated robot from a substantially uncalibrated state with the controller; automatically teaching the articulated robot the position of the pallet station by means of the controller to resolve the position of the end effector to the pallet station to build the pallet load by placing the pallet load item units on the pallet supports; and The at least one three-dimensional camera is calibrated relative to the end effector by means of the controller.

19. The method of claim 18, wherein the controller and the auto-calibration system effectuate auto-calibration and auto-teaching of the articulated robot substantially simultaneously with auto-calibration of each of the at least one three-dimensional cameras that image each pallet load item unit placed by the articulated robot to construct the pallet load.

20. The method of claim 18, wherein the automated calibration system comprises one or more of a structure and a marking having a calibration feature coupled to the end effector, the calibration feature having predetermined characteristics describing a predetermined or known shape or pattern.

21. A method according to claim 20, wherein the calibration feature is dependent on the end effector, or the calibration feature and the end effector are jointly dependent on a common component of the articulated robot, and the calibration feature and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

22. The method of claim 20, wherein the calibration feature is arranged relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

23. The method of claim 20, wherein the automatic calibration system comprises an electromagnetic sensor that is separate and distinct from the three-dimensional camera, and the method further comprises: sensing the calibration feature with the electromagnetic sensor; as well as By means of the controller communicatively coupled to the electromagnetic sensor, a sensor signal embodying the predetermined characteristic for a corresponding posture of the articulated robot is recorded, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration feature and generating the sensor signal.

24. The method of claim 23, further comprising, by means of the controller: moving the articulated robot and positioning the calibration feature to an initial position where the electromagnetic sensor senses the calibration feature, and The at least one three-dimensional camera is calibrated by resolving a pose of the calibration feature relative to an electromagnetic sensor reference system and determining a transformation between the electromagnetic sensor reference system and a three-dimensional camera reference system based on the resolved pose.

25. A method for automatically building a pallet load of palletized item units onto a pallet support by means of a pallet building device, the method comprising: A pallet building device is provided, the pallet building device comprising: a frame defining a pallet station having a pallet building base for the pallet supports, An articulated robot connected to the frame and having: a drive portion having a plurality of independent drive axes arranged in a reference frame of the robot; and an end effector configured to hold the palletized item unit in transport, a controller operably connected to the at least one articulated robot, and an automatic calibration system for calibrating the articulated robot and comprising at least one three-dimensional camera connected to the frame and configured to perform three-dimensional imaging of a portion of the articulated robot and at least one of the pallet support or the pallet load build on the pallet build base; transporting and placing the palletized load unit onto the pallet support by means of the articulated robot so as to build the palletized load on the pallet building base; controlling, by means of the controller, articulated robotic movement relative to the pallet building base and thereby effecting a pallet load build of the pallet load; and generating, with the controller, a three-dimensional image of the pallet supports on the pallet building base and the pallet loading build on the pallet supports; wherein the controller is communicatively connected to the automatic calibration system and automatically calibrates the articulated robot from a substantially uncalibrated state using the three-dimensional image and automatically teaches the articulated robot a reference of the pallet station representing a placement position of each of the pallet load item units placed by the end effector building the pallet load on the pallet support.

26. The method of claim 25, wherein the at least one three-dimensional camera is connected to the frame, separate and distinct from the articulated robot.

27. The method of claim 25 further comprising resolving, with the controller, the position of the end effector to the pallet station to build the palletized load by placing the palletized load item units on the pallet supports.

28. The method of claim 25, wherein the controller calibrates the at least one three-dimensional camera relative to the end effector.

29. The method of claim 25, wherein the controller and the auto-calibration system effectuate auto-calibration and auto-teaching of the articulated robot substantially simultaneously with auto-calibration of each of the at least one three-dimensional cameras that image each pallet load item unit placed by the articulated robot to construct the pallet load.

30. The method of claim 25, wherein the automated calibration system comprises one or more of a structure and a marking having a calibration feature coupled to the end effector, the calibration feature having predetermined characteristics describing a predetermined or known shape or pattern.

31. A method according to claim 30, wherein the calibration feature is dependent on the end effector, or the calibration feature and the end effector are jointly dependent on a common component of the articulated robot, and the calibration feature and the end effector move as a whole relative to the frame with each articulation of the articulated robot.

32. The method of claim 30, wherein the calibration feature is arranged relative to the articulated robot such that the predetermined characteristic determines a predetermined relationship between the calibration feature and the robot reference frame.

33. The method of claim 30, wherein the automatic calibration system comprises an electromagnetic sensor that is separate and distinct from the three-dimensional camera, and the method further comprises: sensing the calibration feature with the electromagnetic sensor; The controller is communicatively coupled to the electromagnetic sensor and records a sensor signal embodying the predetermined characteristic for a corresponding posture of the articulated robot, the recording occurring substantially simultaneously with the electromagnetic sensor sensing the calibration feature and generating the sensor signal.

34. The method according to claim 33, further comprising, With the controller: moving the articulated robot and positioning the calibration feature to an initial position where the electromagnetic sensor senses the calibration feature; as well as The at least one three-dimensional camera is calibrated by resolving a pose of the calibration feature relative to an electromagnetic sensor reference system and determining a transformation between the electromagnetic sensor reference system and a three-dimensional camera reference system based on the resolved pose.

Images