Smart sensing for tray loading and unloading

Abstract

The invention relates to intelligent sensing for tray loading and unloading. The tray loading system may include a processor and a memory having instructions stored thereon that, when executed by the processor, cause the processor to receive parcel loading data that includes characteristics of a parcel to be placed on the tray and characteristics of a loading operator. A placement location of the parcel on the tray may be determined using an artificial intelligence (AI) or machine learning (ML) algorithm based on characteristics of the parcel and characteristics of the loading operator, and a visual marker, such as a visual projection, may be displayed at the placement location. The system may output instructions to the loading operator to place the parcel at the displayed placement location.

Description

Background Art

[0001] Pallets are commonly used to transport and store packages or objects in batches. Palletization is the act of packaging and securing goods onto a pallet in preparation for shipment. During palletization, the goods are stacked on the pallet and secured using straps or some type of wrapping (e.g., plastic) to stabilize the goods and, in some cases, allow the pallets to be stacked. Generally, large quantities of similar or identical items are loaded or stacked on the pallet and stored for (bulk) shipment or until the items are placed on shelves (e.g., at a retail store). Palletization can be performed by humans, robots, or a combination of humans and robots, which means that palletization can be performed manually or automatically. Brief Description of the Drawings

[0002] In the drawings (which are not necessarily to scale), the same numbers may describe similar components in different views. The same numbers with different letter suffixes may represent different instances of similar components. Generally, the drawings illustrate the various embodiments discussed in this document by way of example and not by way of limitation.

[0003] Figure 1A and Figure 1B An example of a flowchart illustrating the operation of an intelligent pallet loading / unloading system.

[0004] Figure 2A An example of a pallet loading active sensing area is illustrated.

[0005] Figures 2B - 2D Illustrated Figure 2A Views of different cameras included in the pallet loading active sensing area illustrated in.

[0006] Figure 3 An example of a pallet measurement platform is illustrated.

[0007] Figure 4 A - Figure 4 F illustrate candidate placement locations for packages generated using a placement selection algorithm.

[0008] Figure 5 An example of a computer - implemented method for pallet loading is illustrated.

[0009] Figure 6 is a block diagram of an example of a loading, device, or machine on which any one or more of the techniques (e.g., methods) discussed in this document can be executed. Detailed Description

[0010] A pallet is used to store and transport packages (e.g., boxes) or objects in groups or batches. Often, pallets are loaded with a certain quantity of the same or similar items to be shipped. For example, plastic wrapping can be used to hold items together and / or secure them to the pallet itself to load the pallet and secure the items. During palletization, objects may be misclassified or damaged, resulting in inventory mismatches that require attention and analysis and possible rearrangement. These issues can increase warehouse storage and retrieval times and may affect product time to market or the supply chain. These issues can also result in incorrect or damaged goods being shipped to customers and subsequently returned by the customers, which can lead to loss of time, effort, and / or money for the seller and the customer.

[0011] The pallet loading and unloading processes provide opportunities to perform packaging and integrity checks to mitigate, limit, or reduce the likelihood that items are stacked incorrectly on the pallet, that items on the pallet are damaged, or that damaged items are stored on the pallet. Additionally, the loading and unloading processes provide opportunities to store objects in an orderly manner to improve inventory management and reduce retrieval times.

[0012] The present disclosure relates to a system that may include a pallet loading area that provides assistance to an operator. The assistance can be visual, auditory (e.g., sound signals, voice), or tactile in nature, or a combination. Visual assistance can include projected markings (e.g., laser or similar light projections, etc.) that show the operator where to stack the next package or object awaiting palletization or show the path or direction that the package or object should take for palletization. Additionally, the loading area can include measuring members or devices, such as a platform or sensor array on which the pallet can be placed. The measuring members can include multiple sensors to check the package weight, determine package integrity, and provide data for planning the placement of packages on the pallet such that the pallet load has a centered center of mass (CoM) and controlled inertia properties. The currently disclosed system can utilize a combination of perception, planning, and active projection algorithms to create an intelligent pallet loading area that is capable of testing the integrity of packages stacked or loaded on the pallet, verifying package attributes (e.g., verifying the supplier, verifying the package weight, verifying the package inertia properties, etc.), and displaying the attributes of each package. Additionally, the above test or verification information can be directly recorded (e.g., by pasting a label) on, for example, the package, object, or pallet or in a digital repository, where the test or verification information is linked to the package, object, or pallet identifier. The currently disclosed system can also provide precise positioning information that shows where the package should be placed such that the operator can perform the pallet loading or unloading operation reliably, safely, and efficiently.

[0013] The algorithms used by the system can increase the productivity of manual packaging procedures that require human pallet loading and can accelerate automated pick-and-place tasks where an automated system (e.g., a robot) performs the loading and unloading. The algorithms can include artificial intelligence (AI), machine learning (ML), or other algorithms (e.g., non-AI or non-ML deterministic algorithms) or processes. Additionally or alternatively, a hardware-based feedback loop or feedback control can be used for some operations of the system (e.g., a hardware-based feedback loop or feedback control can also be used in combination with a deterministic algorithm).

[0014] Potential advantages of such systems include providing visual aids or markers for package placement, which can reduce the time and effort required for pallet loading and reduce loading errors that humans are prone to make. Additionally, the system can use measurement components and / or multiple sensors connected to or communicatively coupled to the measurement components to verify that the objects on the pallet have been placed, loaded in the correct order and orientation, and show no evidence of damage. The system can also allow for the digitization of the objects on the pallet for quick inventory, product indexing, and integrity verification. As used herein, the terms "loading" and "unloading" can be used interchangeably and can include stacking items on a pallet, reorganizing items on a pallet, stacking bundled items on a pallet, removing items from a pallet, etc. Similarly, the terms "(one or more) items", "(one or more) packages", "(one or more) objects", and "(one or more) bins" can be used interchangeably and refer to anything to be stacked or loaded onto a pallet.

[0015] Figure 1A and Figure 1B An example of a flowchart illustrating the operation of an intelligent pallet loading system. As Figure 1A and Figure 1B illustrated, one or more inputs 100A and 100B for packages to be placed on a pallet can be input into a system for intelligent palletization. As Figure 1BAs shown, the input 100B (collectively referred to as "input criteria") can include one or more package attributes (such as package weight or package dimensions (e.g., length, width, and height)), can include an order description (such as the contents of the package), and one or more placement criteria. In an example, the placement criteria can include whether the packages can be stacked, the fragility of the package contents, etc. The placement criteria can also include information related to the operator's capabilities. The operator can be the entity loading (or responsible for loading) the pallet. The operator can be human or can be a robot (e.g., a drone, or any similar autonomous loading device) or a combination of the two. When the operator is human, the human can wear an exoskeleton or use a similar device that can enhance the human's capabilities (such as how much weight the human operator can lift, how high the human operator can reach, etc.). In some cases, the human can use a forklift or other mechanical assistance device to load the pallet. In the case of a robotic operator, the robot has limitations on how much weight it can lift at one time, how large the packages, boxes, or items it can manipulate are, etc. The input criteria can also include an order description. The order description can indicate the contents of the packages in the order, the number of packages in the order, etc.

[0016] At 118, the package attributes, order description, and placement criteria can be sent or transmitted to a planning system. The planning system can calculate or determine the position of each package on the pallet. At least some of the input criteria information (e.g., supplier information, weight, etc.) can be obtained from a packing list, invoice, bill of lading, or any similar packing document that can be input into the planning system. In another example, the input criteria can be obtained from one or more sensors (such as imaging sensors, scanners, etc.). The imaging sensor can include one or more cameras for visualizing the loading area. The imaging sensor can transmit the image to the planning system. The planning system can determine the operator type (e.g., human, robot, or combination) and the operator's capabilities at least in part based on the image.

[0017] In some examples, one or more sensors can obtain information from a badge, a Quick Response (QR) code, etc., and this information can include electronically stored or embedded information about the capabilities of the operator. For example, when the operator is human, the badge can contain information related to how much weight a human operator can lift, any restrictions on the human operator, etc. Similarly, when a human operator wears an exoskeleton, or when the operator is a robot, the code located on the exoskeleton or the robot can include its operating capabilities. Based on the determination of the type of operator and the capabilities of the operator, as well as other input criteria, at 102, the system can display a visual aid for the item placement location. For example, as discussed below, the visual aid can be an augmented reality projection showing the item placement. The projection can be a laser projection on a tray that shows where the item should be placed. In another example, a QR code can be projected onto the tray and scanned by a robot operator to obtain instructions for the robot operator to load or place an item on the tray. Additionally, or alternatively, the visual display can be performed on a graphical user interface (GUI) (such as a monitor or screen). For example, the loading area (including the tray) can be displayed on the GUI, and visual markers or cues for placing packages can be displayed on the GUI. The display of the loading area on the GUI can be a real-time video display or an augmented or virtual reality display (e.g., a digital twin). In an example, the system can consider the capabilities of the operator to determine the preferred conditions of the loading area. For example, the system can determine the preferred techniques for the operator to pick up and load packages. When the operator is human, the system can determine that packages should be loaded in a way that reduces the number of times a human bends down, and accordingly select placement locations for each package. On the other hand, when the operator is a robot, the system can adjust the lighting conditions in the loading area to obtain optimal sensor data collection and quality.

[0018] In response to a visual display, at 104, a package can be loaded at the displayed location, and at 106, the system can use multiple sensors to detect that the package has been loaded. The multiple sensors can include the imaging sensors discussed above located in and around the loading area. The multiple sensors can also include sensors located on or under the pallet (e.g., physical measurement sensors or equipment). Physical sensors can include force sensing resistors (FSRs) placed at various locations on the pallet (e.g., in an array) to detect the presence of the package, to detect whether the package is in the correct position on the pallet, and to detect whether the package is properly placed (e.g., lying flat or horizontally) at the designated location. Physical sensors can also include optical distance sensors, such as infrared sensors or light detection and ranging (LIDAR) sensors. The physical sensors can be located in or included in one or more measurement platforms or similar measurement components, which can be located under the pallet (or the pallet can be placed on the one or more measurement platforms or similar measurement components). For example, the physical sensors can be located on a hydraulic platform or a series of hydraulic platforms located under the pallet. In such examples, the hydraulic platform can be raised so that at least a portion of the platform can extend above the slats in the pallet and make measurements when the package is loaded onto the pallet.

[0019] In response to detecting a package on the pallet and using at least a portion of the input criteria, an integrity check of the package can be performed at 108. The integrity check can include determining whether the package is located at the position displayed at 102, whether the package has the expected weight (and thus may contain the expected item(s)), whether the package is stable on the pallet, whether the package appears undamaged, etc. At 110, when the package passes the integrity check, the system can check at 114 to determine whether the entire order is ready (e.g., all packages expected on the pallet have been loaded). When the order is determined to be ready, at 116, the system can consider the pallet to be loaded, and at this time the pallet can be wrapped (e.g., with plastic) and moved from the loading area (e.g., stored, stacked on another pallet, loaded into a shipping container or trailer, etc.). When it is determined at 114 that the order is not ready, the system can return to 102 (as Figure 1A illustrated) or return to 118 (as Figure 1B illustrated) to determine the location for placing the next package.

[0020] If the package fails the integrity check at 110, an error can be displayed or otherwise indicated at 112. In an example, the error can include a visual indicator such as a colored or flashing light in the loading area. For example, when the package is incorrectly loaded, the visual loading aids discussed above can flash or change color (e.g., from blue or green to red). Additionally, or alternatively, the error can include an audio indicator such as a beep or warning sound. In another example, the indicator can include a message such as a pop-up on a GUI with details related to the error and instructions for mitigating the error. For example, when the error is a weight error indicating that an incorrect package has been placed on the tray, the message instructs the operator to remove the package from the tray and place it aside (e.g., in a designated area) for inspection. When the package is damaged such that the contents of the package may be broken or damaged, a similar message can be displayed. When the error is that the package is not loaded in a stable manner, the message can display instructions to reposition the package or otherwise stabilize it at a specified location on the tray.

[0021] When the error indicates that the package is in the incorrect position or needs to be repositioned, the system may instruct the operator to load the object at the displayed position (return to 104) and repeat the package detection and integrity check until the package passes the integrity check at 110. As Figure 1B illustrated by the dashed box, the operations of displaying visual aids at 102, loading the package to the displayed position at 104, detecting the package at 106, and performing the integrity check at 108 can be performed for each package in the order until the order is ready at 114.

[0022] Figure 2A An example of the tray loading active sensing area 200 is illustrated. The tray loading active sensing area 200 can include means for visualizing the tray loading active sensing area 200. The means for visualizing can include one or more imaging members or imaging devices. The imaging device can include one or more cameras (such as the front camera 202, the top camera 204, and the side camera 206 (collectively referred to as "cameras")) to visualize the tray 210, the incoming package 208, the first loaded package 216, the second loaded package 218, and any tray operator (not shown). The camera can be a Red, Green, Blue (RGB) camera, a Pan, Tilt, Zoom (PTZ) camera, a depth camera, or any similar camera capable of capturing the angles of the tray loading active sensing area 200. Although Figure 2AThree cameras are illustrated, but any number of cameras can be located at any number of positions within the tray loading active sensing area 200 to capture, image, view, etc. all desired angles of the tray loading active sensing area 200. The tray loading active sensing area 200 may also include one or more projection devices or components. For example, a laser projector or other projection device may be included on one or more of the cameras. Additionally, or alternatively, the projector 214 can be used to project the projected next package location 212 (represented by the "x" at the arrow point) onto the tray 210. Just as the tray loading active sensing area 200 can include any number of desired cameras or other imaging devices, any number of projectors can be included such that the projected next package location 212 can be displayed at any location on the tray 210.

[0023] Figures 2B - 2D Illustration Figure 2A Views of different cameras included in the tray loading active sensing area illustrated therein. As Figure 2B illustrated, the top camera 204 can be positioned to be able to image or view the tray 210 from directly above (or from substantially directly above the tray 210). In this view, the projected next package location 212 can be viewed relative to the first loaded package 216 and the second loaded package 218. As Figure 2C illustrated, the view from the front camera 202 can visualize the front of the tray 210 from above the tray 210. According to this view, the projected next package location 212 is blocked by the first loaded package 216 and the second loaded package 218. As Figure 2D illustrated, the view from the side camera 206 can visualize one side (e.g., the left or right side) of the tray 210. According to this view, the projected next package location 212 is visible behind the first loaded package 216 and beside the second loaded package 218.

[0024] Figure 3 Illustration of an example of a tray measurement platform. As Figure 3As shown, the measuring device or component can be located below the tray 210 (e.g., the tray 210 can be set on top of the measuring component or otherwise located above the measuring component). The measuring component can include a measuring platform or a series of measuring platforms such as, for example, a hydraulic platform 302. The hydraulic platform 302 can be arranged below the tray 210 such that they can be raised or protruded through the slats in the tray 210. The measuring platform can include a plurality of sensors, such as force-sensitive resistive sensors (FSR sensors 300). The FSR sensors 300 can be placed on the hydraulic platform 302 such that they are arranged at various positions over the entire area occupied by the tray 210. The FSR sensors 300 can detect the presence of a package at a specific location on the tray 210 and determine whether the package is properly positioned (or placed) and whether it is lying flat or level at the designated or determined position. The system can also include a computer or other machine, such as the computer or other machine to which the cameras or sensors discussed hereinbelow with respect to Figure 6 The cameras or sensors described in the present discussion can be connected or coupled. The processor of the machine can be connected to a memory having instructions that cause the processor to execute Figure 1A and Figure 1B the operations discussed in Figure 5 or the operations of the method discussed in

[0025] In addition to the FSR sensors 300, additional sensors can be included, such as distance sensors (e.g., optical or capacitive distance sensors, etc.), LIDAR sensors, etc. The distance sensors can be used in combination with the FSR sensors 300 or cameras to determine whether the packages are properly spaced on the tray 210 during the integrity check discussed above at 108. For example, when a package is placed on or removed from the tray 210, the system can use a camera during the integrity check to determine whether there is visible damage and use the FSR sensors 300 or another weight sensor to determine whether the package is of the expected weight. The expected weight of the package or other similar details can be determined or retrieved from a shipping manifest or similar document or from a database containing information about the package attributes or contents. By measuring the weight added to or removed from the tray, the system can determine whether the package is correct. Additionally, if all packages or the objects contained in the packages are expected to be the same, the system can estimate the number of objects contained in the package and flag the package (or instruct the operator to flag the package or set the package aside) for inspection in the case of a determined mismatch or error.

[0026] Cameras or other imaging sensors in the active sensing area 200 of the pallet load can be used to fit a cubic model of the package and inspect for package defects. When a defect in the appearance or shape of the package is detected, the system can evaluate to what extent the detected damage may endanger the integrity of the items contained in the package. Using the item description, the system can determine a fragility index, a fragility score, a fragility rating, etc. for the package. The fragility index can be determined based on at least one or more of the following: 1) common sense from a natural language artificial intelligence (AI) or machine learning (ML) model or other similar deterministic model, 2) shipping statistics, or 3) image analysis of the package fragility label (e.g., obtained via optical character recognition of the package label). Similarly, the system can identify, label, etc., or instruct the operator to set aside the damaged package for inspection based on the correlation or comparison of the package damage with the determined fragility index.

[0027] Depending on the packages already on the pallet and their estimated mass and inertia properties, the planning algorithm discussed above can decide where the next package should be placed. The placement can be determined so that the center of mass of the pallet is as close as possible to the geometric center. Such placement of packages or bundles of packages can make the movement of the pallet more stable and result in a more predictable and controllable inertia.

[0028] When different packages from different suppliers are to be loaded onto the pallet, different suppliers can use different communication protocols to electronically convey package details. The system can convert the different communication protocols to a standard communication protocol when communicating the information to the robot operator or storing the information in a database. Once the order is complete and the pallet is ready for packing (e.g., ready for shipment), the statistics of the loaded pallet (e.g., total weight, center of mass, etc.) can be displayed (e.g., on a GUI) or otherwise communicated to the operator.

[0029] As discussed above, the algorithm can include artificial intelligence (AI) or machine learning (ML) or other algorithms (e.g., non-AI or non-ML deterministic algorithms) or processes. In an example, a Monte Carlo algorithm can be used to determine different package configurations and suggest the most stable configuration by projecting the desired package locations onto the pallet (or the boxes already loaded on the pallet), as discussed above. The determination can be made not only based on the next package to be loaded but also based on the number (and characteristics) of one or more additional packages to be loaded onto the pallet.

[0030] The state of the pallet can be represented by the position (x ∈ R 3 ) of the packages placed on the pallet, the orientation (θ ∈ H), the mass (M ∈ R), the inertia (I ∈ R 3x3) and size (s ∈ R). Thus, the pallet can be represented by a tuple P = (x, θ, M, I, s) of these elements N The possible options for placing a new package on the pallet depend on the incoming package attributes and its estimated mass, inertia, and size. A Monte Carlo algorithm (or any similar algorithm) can be used to sample the possible positions and orientations on the pallet and the possible values of its mass, inertia, and size based on the uncertainty of the sensing system. The Monte Carlo algorithm can be particularly useful for determining the package position because it can take into account factors such as user capabilities that may prevent the cost function from being differentiated. For example, using the cost function, the derivative of the cost function can be used to reduce the distance from the centroid of the pallet. However, due to factors including operator capabilities, the cost function cannot be easily optimized because changes in operator capabilities may cause the cost function to change suddenly, making it impossible or impractical to differentiate the cost function.

[0031] Figure 4 A- Figure 4 FIG. A-F illustrates candidate placement positions for a package generated using a placement selection algorithm. As Figure 4 A- Figure 4 FIG. A-F illustrates, the algorithm can select or determine several different candidate placement positions for a candidate package 400 (illustrated by the transparent box) on the pallet 210. The algorithm can select one of these candidate placement positions for the candidate package 400 on the pallet 210 as the maximum likelihood position. The maximum likelihood position can minimize the distance from the centroid of the pallet 210 to the geometric center of the pallet 210 and the distance from the inertia matrix (e.g., a diagonal inertia matrix) to the identity. In a 3D rigid body, inertia (their resistance to changes in velocity) is represented by a 3x3 matrix that represents how difficult it is to linearly accelerate the object in any direction and angularly accelerate the object about any axis. For an object with an identity matrix, this means that the object is easier to manipulate and will not have an unexpected response to the forces applied to it. Thus, an object with a diagonal inertia matrix may be beneficial.

[0032] In some examples, for the placement of the next package to be placed on the pallet 210, the maximum likelihood position selected from the candidate placement positions may not be the best overall option. The algorithm can sequentially simulate future possible packages until the pallet 210 is fully loaded. Thus, the algorithm can randomly sample where all the packages to be loaded onto the pallet 210 can be placed on the pallet 210 and optimize their placement positions such that when the pallet 210 is fully loaded, it has optimized inertia and centroid properties. The placement positions can also be optimized based on package contents, operator capabilities, etc. Such random sampling of possible package positions can mitigate or reduce the artifacts and drawbacks of the greedy Monte Carlo algorithm. For example, the Monte Carlo algorithm may ignore some valid configurations and converge to a single configuration. When a valid configuration is found, such sampling algorithms may have difficulty escaping the attraction field of that solution to explore the remaining solution space and find other valid solutions. This is known as the mixing problem. Incorporating prior knowledge may enable the random sampling to generate more diverse solution candidates and converge to different valid configurations. Examples of artifacts are biased sampling, low mixing, and mode collapse. In an example, Monte Carlo tree search can be used in conjunction with the development of a reinforcement learning algorithm to find the next best position for a package on a pallet.

[0033] The examples discussed herein can be applicable to loading a pallet or unloading a pallet. When unloading a pallet, the Monte Carlo algorithm can randomly select the packages to be removed (e.g., in what order to remove packages from the pallet) such that the removal of the packages minimally disrupts the stability of the stack (or has a minimal impact on the instability of the stack). Thus, unordered removal of packages that could cause packages to fall and be damaged or injure the loading operator can be reduced. When the Monte Carlo algorithm terminates with a high degree of uncertainty, the system can mark the package as not suitable for placement on the pallet and suggest placing the package in a waiting area. As pallet loading continues, the packages in the waiting area can be reconsidered for placement.

[0034] Figure 5 Illustrative example of a computer-implemented method for pallet loading. Method 500 can include or comprise a plurality of operations (502 - 510). These operations are merely examples, and the method performed can omit one or more of the listed operations, can repeat operations, can include additional operations, or can perform the operations simultaneously, substantially simultaneously, or in other orders as appropriate or as needed. The operations can be automatically performed by a processor or controller of a machine or computer, such as described below for Figure 6 as described.

[0035] Operation 502 may include receiving one or more inputs that have information related to the package to be loaded onto the pallet and information related to the loading operator. Information related to the package and the pallet may be received from one or more sensors (e.g., imaging sensors such as cameras, etc.) or from a database. Information related to the package may include package attributes such as package size, package weight, etc. The input may also include an order description. The order description may include the contents of the package and one or more placement criteria for the package. The placement criteria may include stacking instructions (e.g., how much weight can be placed on the package), whether the contents of the package are fragile, etc.

[0036] The input may also include information about the loading operator, such as the type of loading operator and the capabilities of the loading operator. The loading operator may be human, may be a robot, or may be a human working with a robot. The capabilities of the loading operator may include how much weight the loading operator can lift, how high the loading operator can reach, the mobility of the loading operator, etc.

[0037] Operation 504 may include determining the placement location of the package on the pallet. The placement location may be determined by artificial intelligence (AI) or machine learning (ML) or other algorithms (e.g., non-AI or non-ML deterministic algorithms) or processes, and is determined at least in part based on information related to the loading operator and information related to the package. The algorithm may include a Monte Carlo algorithm. The algorithm may determine a number of different candidate placement locations and select one of these candidate placement locations as the maximum likelihood (e.g., best) placement location. For example, the best location may minimize the distance from the centroid of the package (or group of packages) on the pallet to the geometric center of the pallet.

[0038] At operation 506, a visual marker may be displayed at (or relative to) the placement location. The marker may be an augmented reality projection showing the placement of the item. The projection may be a laser projection (e.g., "X") on the pallet that shows where the item should be placed. In another example, a quick response (QR) code may be projected onto the pallet and scanned by the robot operator to obtain instructions for the robot operator to load or place the item on the pallet. At operation 508, the loading operator may be instructed to place the package at the displayed placement location (e.g., at the placement location based on the visual marker). For example, when the loading operator is a robot, the system may send an instruction to place the package at the displayed placement location. In another example, when multiple loading operators are used (e.g., a combination of human and robot), the system may instruct the human to wait for the robot to place its package first (e.g., so that the human does not have to bend down to place the next package).

[0039] At operation 510, once the package is placed at the determined placement location, the system can perform an integrity check on the package using multiple sensors. The multiple sensors can include imaging sensors (such as cameras) and physical measurement sensors located on the pallet or on a measurement platform below the pallet. The physical sensors can include force-sensing resistors (FSRs) placed at different locations on the pallet to detect the presence of the package and to detect whether the package is present at a specific point or location on the pallet, and to detect whether the package is properly placed or stable (e.g., lying flat) at the designated location. The physical sensors can also include optical distance sensors, such as infrared sensors or Light Detection and Ranging (LIDAR) sensors. The integrity check can include determining whether the package is placed at the appropriate selected location, whether the package has the expected weight (and thus may contain the expected items), whether the package is stable on the pallet, whether the package appears undamaged, etc. At least part of the integrity check can be performed at the package pick-up location of the loading operator (e.g., the loading area). For example, before placing the package on the pallet, a robotic arm can be used such that the imaging sensor can inspect or view all sides, edges, or corners of the package to check for damage to the package. Thus, the entire package can be viewed, whereas once the packages are stacked on the pallet, some parts of the packages may be blocked and not well visualized.

[0040] When the package passes the integrity check, the system can check to determine whether the entire order is ready (e.g., all packages expected to be loaded have been loaded). When the order is determined to be ready, the system can consider the pallet to be loaded, and at this time the pallet can be wrapped (e.g., with plastic) and moved from the loading area (e.g., stored, stacked on another pallet, loaded into a shipping container or trailer, etc.). When it is determined that the order is not ready, the system can repeat the operations of the method to determine the location for placing the next package. When the package fails the integrity check, an error can be displayed or otherwise indicated (e.g., using a flashing light or a light of a different color, an audible warning, etc.).

[0041] Figure 6FIG. 600 is a block diagram of an example of a device, apparatus, or machine 600 on which any one or more of the techniques (e.g., methods) discussed herein may be performed. In alternative embodiments, machine 600 may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 600 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, machine 600 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 600 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0042] Examples as described herein may include logic or a number of components or mechanisms, or may be operated by logic or a number of components or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity that includes hardware (e.g., simple circuits, gates, logic, etc.). The members of a circuit set may change over time and with underlying hardware changes. A circuit set includes members that, when operated, can perform specified operations either individually or in combination. In an example, the hardware of a circuit set may be immutably designed to perform a particular operation (e.g., hardwired). In an example, the hardware of a circuit set may include physically components with variable connections (e.g., execution units, transistors, simple circuits, etc.), including a computer-readable medium that is physically modified (e.g., magnetically, electrically, movably arranged, etc. with respect to immobile aggregating particles) to encode instructions for a particular operation. When connecting the physical components, the underlying electrical properties of the hardware components change, such as changing from an insulator to a conductor or vice versa from a conductor to an insulator. These instructions enable the embedded hardware (e.g., execution unit or loading mechanism) to create members of the circuit set in the hardware via variable connections to perform portions of a particular operation when operated. Accordingly, when the device is operating, the computer-readable medium may be communicatively coupled to other components of the members of the circuit set. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, in operation, an execution unit may be used in a first circuit of a first circuit set at one point in time and be used by a second circuit in the first circuit set or by a third circuit in a second circuit set at a different time.

[0043] A machine (e.g., a computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, a field programmable gate array (FPGA), or any combination thereof), a main memory 604, and a static memory 606, and some or all of these components may communicate with each other via an interlink (e.g., a bus) 630. The machine 600 may further include a display unit 610, an input device 612 (e.g., a keyboard or other alphanumeric input device), and a user interface (UI) navigation device 614 (e.g., a mouse). In an example, the display unit 610, the input device 612, and the UI navigation device 614 may be a touchscreen display. The machine 600 may additionally include a storage device 608 (e.g., a drive unit or other similar mass storage device or unit), a signal generation device 618 (e.g., a speaker), a network interface device 620 connected to a network 626, and one or more sensors 616 (such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors). The machine 600 may include an output controller 628 for communicating with or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.), such as serial (e.g., universal serial bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connections.

[0044] The storage device 608 may include a machine-readable medium 622 on which is stored one or more sets of data structures or instructions 624 (e.g., software), which embody or are used by any one or more of the techniques or functions described herein. The instructions 624 may also reside, wholly or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution thereof by the machine 600. In an example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the storage device 608 may constitute a machine-readable medium.

[0045] Although the machine-readable medium 622 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 624. The term "machine-readable medium" can include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 600 and that causes the machine 600 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding, or carrying a data structure used by or associated with such instructions. Non-limiting examples of machine-readable media can include solid state memories as well as optical and magnetic media. In an example, a mass machine-readable medium includes a machine-readable medium having a plurality of particles that have invariant (e.g., stationary) mass. Accordingly, a mass machine-readable medium is not a transient propagated signal. Specific examples of mass machine-readable media can include: non-volatile memories such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0046] The term "processor" is synonymous with terms such as "controller" and "computer" and should be understood to cover not only computers having different architectures (such as single / multiprocessor architectures and sequential (von Neumann) / parallel architectures), but also dedicated circuits such as field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), signal processing devices, and other devices. Additional Notes and Examples

[0047] Example 1 is a pallet transfer system that includes: a processor; and a memory that includes instructions that, when executed by the processor, cause the processor to: receive package loading data that includes characteristics of packages to be placed on a pallet and characteristics of a loading operator; determine a placement location of the packages on the pallet based on the characteristics of the packages and the characteristics of the loading operator; display a visual marker associated with the placement location; and output an instruction to place the packages at the placement location to the loading operator based on the visual marker.

[0048] In Example 2, the subject matter of Example 1 optionally includes the subject matter, wherein the instructions further cause the processor to: detect a package at the placement location or at a loading operator package pick-up location; and perform an integrity check on the package using a plurality of sensors located in a pallet loading area.

[0049] In Example 3, the subject matter of Example 2 optionally includes subject matter, wherein in response to a failure of the integrity check of the package, the instructions cause the processor to: display an error; and output a second instruction to the loading operator, the second instruction including mitigation techniques for correcting the error.

[0050] In Example 4, the subject matter of any one or more of Examples 2-3 optionally includes subject matter, wherein in response to a successful integrity check of the package, the instructions cause the processor to: receive one or more inputs having information related to a second package to be placed on the tray; determine a second placement location of the second package on the tray based on the information related to the second package and the information related to the loading operator; display a second visual marker for the second package at the second placement location; indicate to the loading operator to place the second package at the second placement location based on the second visual marker; detect the package at the second placement location or the loading operator package pick-up location; and perform an integrity check on the second package using a plurality of sensors located in the tray loading area.

[0051] In Example 5, the subject matter of any one or more of Examples 2-4 optionally includes subject matter, wherein the plurality of sensors includes one or more of the following: an image sensor, a force-sensitive resistor (FSR), an infrared sensor, or a light detection and ranging (LIDAR) sensor.

[0052] In Example 6, the subject matter of Example 5 optionally includes subject matter, wherein the image sensor includes an RGB camera, and wherein one or more of the infrared sensor, FSR, or LIDAR sensor are located on a measurement platform or sensor array below the tray.

[0053] In Example 7, the subject matter of Example 6 optionally includes subject matter, wherein one or more of the infrared sensor, FSR, or LIDAR sensor are located on the measurement platform, and wherein the measurement platform includes a hydraulic measurement platform configured to raise between one or more openings in the tray.

[0054] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes subject matter, wherein the characteristics of the package include one or more of the following: the attributes of the package, the order description, or the placement criteria of the package.

[0055] In Example 9, the subject matter of Example 8 optionally includes, wherein the instructions further cause the processor to: use the characteristics of the package to determine a fragility rating of the package, and wherein the placement location is determined at least in part based on the fragility rating.

[0056] In Example 10, the subject matter of any one or more of Examples 1-9 optionally includes subject matter in which the characteristics of the loading operator include the type of the loading operator and the ability of the loading operator.

[0057] In Example 11, the subject matter of any one or more of Examples 1-10 optionally includes subject matter in which the placement location is selected from a plurality of candidate placement locations and is determined at least in part based on an analysis of the characteristics of the package and the characteristics of the loading operator using an artificial intelligence (AI) or machine learning (ML) algorithm.

[0058] In Example 12, the subject matter of Example 11 optionally includes subject matter in which the algorithm includes a Monte Carlo algorithm.

[0059] Example 13 is a non-transitory machine-readable medium having instructions stored thereon that, when executed by a processor of a computing device, cause the processor to: receive package loading data including the characteristics of a package to be placed on a pallet and the characteristics of a loading operator; determine a placement location of the package on the pallet based on the characteristics of the package and the characteristics of the loading operator; display a visual marker associated with the placement location; and output an instruction to place the package at the placement location to the loading operator based on the visual marker.

[0060] In Example 14, the subject matter of Example 13 optionally includes, in which the instructions further cause the processor to: detect the package at the placement location or at the package pickup location of the loading operator; and perform an integrity check on the package using a plurality of sensors located in the pallet loading area.

[0061] In Example 15, the subject matter of Example 14 optionally includes subject matter in which, in response to a failure of the integrity check of the package, the instructions cause the processor to: display an error; and output a second instruction to the loading operator, the second instruction including mitigation techniques for correcting the error.

[0062] In Example 16, the subject matter of any one or more of Examples 14-15 optionally includes subject matter in which, in response to a successful integrity check of the package, the instructions further cause the processor to: receive additional package loading data including the characteristics of a second package to be placed on the pallet; determine a second placement location of the second package on the pallet based on the characteristics of the second package and the characteristics of the loading operator; display a second visual marker at the second placement location of the second package; output an additional instruction to place the second package at the second placement location to the loading operator; detect the package at the second placement location or at the package pickup location of the loading operator; and perform an integrity check on the second package using a plurality of sensors located in the pallet loading area.

[0063] In Example 17, the subject matter of any one or more of Examples 14 - 16 optionally includes subject matter in which the plurality of sensors includes one or more of the following: an image sensor, a force - sensitive resistor (FSR), an infrared sensor, or a light detection and ranging (LIDAR) sensor.

[0064] In Example 18, the subject matter of Example 17 optionally includes subject matter in which the image sensor includes an RGB camera, and in which one or more of the infrared sensor, FSR, or LIDAR sensor are located on a measurement member below the tray, and in which the measurement member includes a hydraulic measurement platform configured to raise between one or more openings in the tray.

[0065] In Example 19, the subject matter of any one or more of Examples 14 - 18 optionally includes subject matter in which the characteristics of the package include one or more of the following: an attribute of the package, an order description, or a placement criterion of the package.

[0066] In Example 20, the subject matter of Example 19 optionally includes, in which the instructions further cause the processor to: use the characteristics of the package to determine a fragility index of the package, and in which the placement location is determined at least in part based on the fragility index.

[0067] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings illustrate by way of example specific embodiments that may be implemented. These embodiments are also referred to herein as "examples". Such examples may include elements other than those shown or described. However, the inventors of this application also contemplate examples in which only those elements shown or described are provided. Additionally, the inventors of this application also contemplate examples using any combination or permutation of those elements (or one or more aspects thereof) with respect to a particular example (or one or more aspects thereof) shown or described herein or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0068] All publications, patents, and patent documents referred to in this document are incorporated herein by reference in their entirety, as if each were incorporated by reference individually. In the event of inconsistent usage between this document and those incorporated by reference, the usage in the incorporated (one or more) reference(s) shall be regarded as supplementary to the usage in this document; for irreconcilable inconsistencies, the usage in this document shall prevail.

[0069] In this document, as is common in patent documents, the term "a / an" is used to include one or more than one, independently of any other instances or uses of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive or, such that "A or B" includes "A but not B", "B but not A", and "A and B". In the appended claims, the terms "including" and "in which" are used as the ordinary English equivalents of the corresponding terms "comprising" and "wherein". Further, in the appended claims, the terms "including" and "comprising" are open-ended, that is, a system, apparatus, article, or process that includes elements other than those listed after such terms in a claim is still considered to fall within the scope of that claim. Additionally, in the appended claims, the terms "first", "second", "third", etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0070] The above description is intended to be illustrative and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used by those of ordinary skill in the art after a careful reading of the above description. The abstract is used to allow the reader to quickly ascertain the nature of the technical disclosure, and it is submitted with the understanding that it is not to be used to interpret or limit the scope or meaning of the claims. Also, in the above detailed description, various features may be grouped together to streamline the disclosure. This should not be interpreted as meaning that the disclosed features not claimed are essential to any claim. On the contrary, the inventive subject matter may lie in less than all of the features of a particular disclosed embodiment. Thus, the appended claims are hereby incorporated into the detailed description, where each claim stands on its own as a separate embodiment. The scope of each embodiment should be determined with reference to the appended claims along with the full scope of equivalents to which such claims are entitled.

Claims

1. A pallet loading system, comprising: processor; as well as a memory comprising instructions that, when executed by the processor, cause the processor to: receiving package loading data, the package loading data including characteristics of packages to be placed on the pallet and characteristics of a loading operator; determining a placement position of the package on the pallet based on characteristics of the package and characteristics of the loading operator; displaying a visual marker associated with the placement location; as well as Instructions are output to the loading operator to place the package at the placement location based on the visual indicia.

2. The pallet loading system of claim 1, wherein: The instructions further cause the processor to: detecting the package at the placement location or loading operator package pick-up location; and An integrity check is performed on the packages using a plurality of sensors located in the pallet loading area.

3. The pallet loading system of claim 2, wherein: In response to the package failing the integrity check, the instructions cause the processor to: displays an error; and Second instructions are output to the loading operator, the second instructions including a mitigation technique for correcting the error.

4. A pallet loading system according to any one of claims 2 to 3, wherein: In response to the integrity check of the package being successful, the instructions cause the processor to: receiving one or more inputs having information related to a second package to be placed on the pallet; determining a second placement location of the second package on the pallet based on the information related to the second package and the information related to the loading operator; displaying a second visual indicia at the second placement location of the second package; instructing the loading operator to place the second package at the second placement location based on the second visual indicia; detecting the package at the second placement location or the loading operator package pick-up location; as well as An integrity check is performed on the second package using the plurality of sensors located in the pallet loading area.

5. A pallet loading system according to any one of claims 2 to 4, wherein: The plurality of sensors include one or more of: an image sensor, a force sensitive resistor (FSR), an infrared sensor, or a light detection and ranging (LIDAR) sensor.

6. The pallet loading system of claim 5, wherein: The image sensor comprises an RGB camera, and wherein one or more of the infrared sensor, the FSR or the LIDAR sensor is located on a measurement platform or sensor array below the tray.

7. The pallet loading system of claim 6, wherein: One or more of the infrared sensor, the FSR, or the LIDAR sensor is located on the measurement platform, and wherein the measurement platform comprises a hydraulic measurement platform configured to be raised between one or more openings in the pallet.

8. A pallet loading system according to any one of claims 1 to 7, wherein: The characteristics of the package include one or more of the following: attributes of the package, an order description, or placement criteria for the package.

9. The pallet loading system of claim 8, wherein: The instructions further cause the processor to: A fragility rating of the package is determined using the characteristics of the package, and wherein the placement location is determined at least in part based on the fragility rating.

10. A pallet loading system according to any one of claims 1 to 9, wherein: The characteristics of the loading operator include the type of loading operator and the capabilities of the loading operator.

11. A pallet loading system according to any one of claims 1 to 10, wherein: The placement location is selected from a plurality of candidate placement locations and is determined using an artificial intelligence (AI) or machine learning (ML) algorithm based at least in part on the algorithm's analysis of characteristics of the package and characteristics of the loading operator.

12. The pallet loading system of claim 11, wherein: The algorithm comprises a Monte Carlo algorithm.

13. A non-transitory machine-readable medium having instructions stored thereon, which, when executed by a processor of a computing device, cause the processor to: receiving package loading data, the package loading data including characteristics of packages to be placed on the pallet and characteristics of a loading operator; determining a placement position of the package on the pallet based on characteristics of the package and characteristics of the loading operator; displaying a visual marker associated with the placement location; as well as Instructions are output to the loading operator to place the package at the placement location based on the visual indicia.

14. The non-transitory machine-readable storage medium of claim 13, wherein: The instructions further cause the processor to: detecting the package at the placement location or loading operator package pick-up location; and An integrity check is performed on the packages using a plurality of sensors located in the pallet loading area.

15. The non-transitory machine-readable storage medium of claim 14, wherein: In response to the package failing the integrity check, the instructions cause the processor to: displays an error; and Second instructions are output to the loading operator, the second instructions including a mitigation technique for correcting the error.

16. The non-transitory machine-readable storage medium of any one of claims 14 to 15, wherein: In response to the integrity check of the package being successful, the instructions further cause the processor to: receiving additional package loading data, the additional package loading data including characteristics of a second package to be placed on the pallet; determining a second placement location of the second package on the pallet based on characteristics of the second package and characteristics of the loading operator; displaying a second visual indicia associated with the second placement location of the second package; outputting instructions to the loading operator to place the second package at the second placement location based on the second visual indicia; detecting the package at the second placement location or the loading operator package pick-up location; as well as An integrity check is performed on the second package using the plurality of sensors located in the pallet loading area.

17. The non-transitory machine-readable storage medium of any one of claims 14 to 16, wherein: The plurality of sensors include one or more of: an image sensor, a force sensitive resistor (FSR), an infrared sensor, or a light detection and ranging (LIDAR) sensor.

18. The non-transitory machine-readable storage medium of claim 17, wherein: The image sensor comprises an RGB camera, and wherein one or more of the infrared sensor, the FSR, or the LIDAR sensor is located on a measurement member below the pallet, and wherein the measurement member comprises a hydraulic measurement platform configured to be raised between one or more openings in the pallet.

19. The non-transitory machine-readable storage medium of any one of claims 14 to 18, wherein: The characteristics of the package include one or more of the following: attributes of the package, an order description, or placement criteria for the package.

20. The non-transitory machine-readable storage medium of claim 19, wherein: The instructions further cause the processor to: A fragility index of the package is determined using the characteristics of the package, and wherein the placement location is determined at least in part based on the fragility index.

Images