Background: Method and system for generating landing solutions for containers on a landing surface

Abstract

A system and method are described for generating a landing solution for a container on a landing surface using multiple measurement points generated by a measurement device or multiple measurement devices. In particular, a method is described for analyzing multiple measurements and, from the analysis of the data, determining, within the frame of reference of the container handling equipment, a location and orientation for placing a transport container on a landing surface within the equipment's workspace. This location can then be used by the equipment's automation system to automatically place the transport container on the landing surface without or with little human interaction or input. The system and method are applicable to operations in landside transfer zones and outside such zones, and therefore applicable for installation on any container handling equipment where a container can be landed on the equipment from above.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on the disclosures of and claims priority to and the benefit of U.S. Provisional Application No. 62 / 590,443, filed November 24, 2017, and U.S. Provisional Application No. 62 / 626,990, filed February 6, 2018, which are incorporated herein by reference in their entireties.

[0002] The present invention relates to container crane operation, such as those used in international shipping. More particularly, the present invention relates to a method and system for generating landing solutions for containers on vehicle chassis or other landing surfaces. [Background technology]

[0003] Since the beginning of container shipping in 1955, the freight transportation industry has sought to increase efficiency in the movement of cargo. (http: / / www.worldshipping.org / about-the-industry / history-of-containerization) The advent of shipping containers, quickly followed by standardized equipment for handling them at various stages of transport (a system known as "intermodal") (http: / / www.worldshipping.org / about-the-industry / history-of-containerization), has led the industry to pursue faster movement, greater standardization, higher productivity, and less human intervention. These have been achieved through incremental developments in the automation of container handling equipment.

[0004] Technology exists to facilitate the automation of container handling equipment, such as container cranes (https: / / www.porttechnology.org / technical_papers / the_case_for_automated_rtg_container_handling). In the early days of crane automation, areas with unavoidable human access were considered too dangerous to automate. Pressure to ultimately increase productive movement and reduce wait times drove industry standards toward full automation in areas without direct human contact and "semi-automation," often involving remote operation, in areas where human contact is unavoidable. Recently, however, the industry has pursued fully automated movement in all areas of container handling. Of particular interest to this disclosure is the automation of the landside transfer zone (LSTZ), examples of which are shown in Figures 12-B and 13-B. The LSTZ area is where stacking cranes interface with vehicles transferring containers, not only on roadways but also through internal port equipment other than load chassis. There has been significant interest over the past several years in fully automating operations in the LSTZ.

[0005] The challenges associated with automating operations in an LSTZ are enormous. Designing a system capable of automating movements in an LSTZ requires overcoming variability introduced by nonstandard chassis and equipment types, the presence or absence of specific landing points, the safety of people in the area of ​​operation, and the placement of equipment within the area of ​​operation. The primary challenge to automation typically arises from human-introduced variability regarding how chassis are placed in the LSTZ and where people are located during operations. However, in the case of an LSTZ, this variability is mitigated by mechanical constraints such as the curbs surrounding the truck lanes (Figure 12, Item C and Figure 13, Item C) and the booths into which drivers enter (Figure 12, Item D and Figure 13, Item D), which limit the variability that humans can induce as long as certain rules are observed when operating in this area. In addition to the issue of human variability, there is the issue of non-standardization of the chassis and equipment required for cranes to land containers. While the shape and size of shipping containers comply with ISO (International Organization for Standardization) standards, the chassis and other equipment that carry the containers do not. The only constraint for a given load chassis is that there will typically be one or more, and preferably multiple, twistlocks somewhere on the chassis' landing surface that will mate with the corner castings of a container placed on the chassis. This means that at least the load chassis, but not necessarily other container-handling equipment, will typically have four points on the chassis that correspond to the four corners of a container placed on the chassis. Once found, these four points can be used to generate a solution for where to place the container on the chassis surface. However, it should be noted that these four points can be located anywhere on the surface of the load chassis, as long as they are appropriately spaced from one another to follow the four corners of the container that will mate with them. There is also variation in the size and shape of twistlocks, which affects the system's ability to identify them and thus the exact landing location of the container. Examples of chassis variation are shown in Figures 18 and 19.

[0006] If the equipment ready to accept the container does not have twistlocks, determining the landing location can be even more difficult because there are no constrained points on the surface to search and the equipment is not uniform enough to visually recognize with a camera or similar sensor based purely on its optical appearance. The appearance of the landing equipment may vary in shape, size, color, wear patterns, etc., making optical systems ineffective at identifying them. In Figures 18 and 19, the condition of several chassis can be observed. Note the missing twistlocks on the lower chassis in Figure 19 and the varying levels of dirt on the chassis in Figure 18.

[0007] Over the past few years, several systems have been developed that claim to fully automate operations and eliminate the need for a remote operator to interact with the load chassis and other equipment within the LSTZ. However, while the proposed systems and methods must overcome all of the aforementioned challenges to successfully perform the operations they claim to be fully automated, these systems and methods use sensors that do not fully address the challenges. Examples of efforts in this area include those described in U.S. Patent Application Publication No. 2012 / 0089320A1, International Patent Application Publication No. WO2015 / 022001A1, German Patent Application Publication Nos. DE102008014125A1 and DE102007055316A1, European Patent Application Publication No. EP2574587B1, European Patent Application Publication Nos. EP2724972A1, EP2327652A1, and EP2724972A1. Despite these efforts, however, there remains a need for technology that addresses the above-mentioned deficiencies. Summary of the Invention

[0008] Embodiments of the present invention provide methods and systems for addressing the above challenges in a more robust manner to generate more accurate solutions than existing systems. In embodiments, the present invention describes a method for generating a landing solution for an object, such as a container, on a target equipment and a landing surface on the target equipment using a measurement device or devices and a plurality of measurement points generated by a corresponding device for executing the method and corresponding computer program product. The landing surface may include any landing surface of any equipment, such as any equipment designed to accept a container when placed on the landing surface from above by container handling equipment.

[0009] In embodiments, the present invention describes a method for analyzing multiple measurement points, such as point cloud data (a point cloud is defined as a collection of points that can be represented as x, y, and z coordinate points in Cartesian coordinate space), and from the data analysis stage, determining a 6 DoF (Degrees of Freedom) location and orientation in the reference frame of the container handling equipment for where to place a shipping container on a landing surface within the equipment's workspace. This location can then be used by the equipment's automation system to automatically place the shipping container on the landing surface without human input or minimal human input or interaction. The method and system are applicable to operation in landside transfer zones and outside such zones, and therefore applicable for installation on any container handling equipment where a container can be landed on the equipment from above.

[0010] Specific aspects of embodiments of the present invention include Aspect 1, which is a method of determining a landing solution for an ISO container, including (a) scanning a target equipment or chassis to generate a point cloud in which features of the target equipment or chassis' landing surface, such as one or more twistlocks, are distinguishable; (b) analyzing the point cloud by a processor to determine locations of the features and / or twistlocks; and (c) determining a center of the target equipment or chassis' landing surface based on the locations of the features and / or twistlocks. Aspect 2 includes a method in which the processor determines raised and depressed features and / or locations of twistlocks using techniques such as local minima and maxima of the target equipment or chassis' landing surface and / or model convolution. Aspect 3 is a method of Aspect 1 or 2, in which the scanning is performed using one or more cameras, LiDAR, sonar, or optical sensors, or a combination thereof.

[0011] Specific aspects of embodiments of the present invention also include Aspect 4, which is a method for determining a landing position of a target equipment relative to an object on a landing surface, the method including: (a) providing a target equipment, such as a chassis, loaded with a first object, such as a container; (b) generating one or more coordinates of the first object or container in space; (c) removing the first object or container from the target equipment or chassis to reveal the landing surface of the target equipment or chassis; (d) scanning the target equipment or chassis to generate a model of the target equipment or chassis landing surface; and (e) determining a landing location on the target equipment or chassis of the first object or container by determining a center of the target equipment or chassis landing surface in the model based on the one or more coordinates of the first object or container. Aspect 5 is the method of Aspect 4, further including determining one or more object or container offsets to identify additional landing locations on the target equipment or chassis landing surface. Aspect 6 is the method of Aspects 4 or 5, wherein the one or more offsets are determined by (a) placing one or more second objects or containers on the target equipment or chassis landing surface; (b) generating one or more coordinates of the second objects or containers in space; and (c) determining a distance the second objects or containers are offset from a center of the target equipment or chassis landing surface based on the one or more coordinates of the second objects or containers. Aspect 7 is the method of any one of Aspects 4-6, wherein any number of landing locations are determined from the one or more offsets, e.g., the one or more container offsets include a front container offset, a rear container offset, or both a front and a rear container offset. Aspect 8 is the method of any one of Aspects 4-7, further including storing the landing location and the one or more offsets of the first object or container to create a library of models representing multiple landing locations relative to the target equipment or chassis landing surface.Aspect 9 is the method of any one of Aspects 4-8, wherein the generating is performed by one or more measurement devices including one or more cameras, LiDAR, sonar, or optical sensors, or a combination thereof. Aspect 10 is the method of any one of Aspects 4-9, wherein the target equipment or chassis is an Automatically Guided Vehicle (AGV) chassis, an AGV rack, a bomb cart chassis, a rail car, a street chassis, or a cassette. Aspect 11 is the method of any one of Aspects 4-10, wherein the container is a shipping container shaped and sized in accordance with the International Organization for Standardization (ISO). Aspect 12 is the method of any one of Aspects 4-11, wherein the model of the target equipment or chassis landing surface is represented by a 3D point cloud of data. Example 13 is the method of any one of Examples 4 to 12, wherein the model of the target device or chassis landing surface is generated from scanning the target device or chassis one or more times, e.g., 2 to 20 times, e.g., 3 to 18 times, or 4 to 15 times, or 5 to 10 times. Example 14 is the method of any one of Examples 4 to 13, wherein the model is generated from a single scan, and by scanning in a manner that achieves high resolution data.

[0012] Certain aspects of embodiments of the present invention also include Aspect 15, which is a method for determining a landing location for an object, such as a container, on a landing surface of a target equipment, such as a chassis, the method including: (a) scanning a first target equipment or chassis to provide a model of the first target equipment or chassis; (b) comparing the model of the first target equipment or chassis with one or more reference target equipment or chassis to identify which reference target equipment or chassis match the first target equipment or chassis within a selected degree of match; and (c) using the known object or container landing location of the matching reference target equipment or chassis as a proxy for the object or container landing location of the first target equipment or chassis. Aspect 16 is the method of Aspect 15, in which the container is an ISO shipping container. Aspect 17 is the method of Aspects 15 or 16, in which the scanning is performed using one or more of a camera, LiDAR, sonar, or optical sensor, or a combination thereof. Example 18 is the method of any one of Examples 15-17, wherein the model of the first target device or chassis and the reference target device or chassis are represented by a 3D point cloud of data. Example 19 is the method of any one of Examples 15-18, wherein the comparing includes identifying one or more reference target devices or chassis having a length within a specified range of the length of the first target device or chassis and / or using a voxel filter to identify similarity between the reference target device or chassis and the first target device or chassis. Example 20 is the method of any one of Examples 15-19, wherein the voxel filter compares data across one or more dimensions of the 3D data cloud. Example 21 is the method of any one of Examples 15-20, wherein the voxel filter compares any combination of x-, y-, and / or z-values.Example 22 is a method of any one of Examples 15 to 21, wherein the landing location of a known object or container on a matching reference target equipment chassis is determined by: (a) providing a reference target equipment or chassis loaded with a first reference object or container; (b) generating one or more coordinates of the first reference object or container in space; (c) removing the first reference object or container from the reference target equipment or chassis to reveal a landing surface of the reference target equipment or chassis; (d) scanning the reference target equipment or chassis to generate a model of the reference target equipment or chassis landing surface; and (e) determining the landing location for the first reference object or container on the reference target equipment or chassis by determining the center of the landing surface of the reference target equipment or chassis in the model based on the one or more coordinates of the first reference object or container.

[0013] A particular aspect of an embodiment of the present invention is a method for determining an orientation of a target equipment or chassis within a frame of reference for material handling equipment, the method comprising: (a) providing a data point cloud representing a reference target equipment or chassis; (b) providing a data point cloud representing a current target equipment or chassis; (c) positioning the reference target equipment or chassis data point cloud at an expected virtual position within the frame of reference for container material handling equipment; (d) estimating a first center of a landing surface of the reference target equipment or chassis and a second center of a landing surface of the current target equipment or chassis within their respective data point clouds; and (e) performing a first alignment by aligning the first center with the second center. (f) determining any translation(s) needed to achieve the first alignment, such as in X, Y, and Z coordinates, from the first central location to a second central location, (g) performing at least one second alignment by otherwise aligning the landing surface of the reference target equipment or chassis with the landing surface of the current target equipment or chassis, (h) determining any rotation(s) of the reference chassis data point cloud, such as about the X, Y, and Z axes, needed to achieve the second alignment, and (i) using the translation(s) and / or rotation(s) to estimate an actual orientation of the current target equipment or chassis within the motion reference frame of the container handling equipment. Embodiment 24 is the method of embodiment 23, wherein one or more of the at least one second alignment is performed using an iterative nearest neighbor technique. Example 25 is the method of example 23 or example 24, wherein noisy data of the data point cloud representing the current target equipment or chassis is reduced or removed to provide a filtered current target equipment or chassis point cloud. Example 26 is the method of any one of examples 23 to 25, wherein maximum and minimum measurement points of the filtered current target equipment or chassis point cloud are used to estimate a center of a landing surface for the current target equipment or chassis.

[0014] Certain embodiments of the present invention also include Aspect 27, a method for identifying a landing location on a target equipment or chassis of an ISO shipping container according to a position and orientation of the target equipment or chassis, the method including: (a) obtaining a point cloud model of the current target equipment or chassis landing surface aligned with a reference system of the container handling equipment; (b) aligning the point cloud model of the reference target equipment or chassis landing surface with the point cloud model of the current target equipment or chassis landing surface; and (c) generating a set of coordinates within the reference system as a result of the alignment, whereby the coordinates represent a landing location for placing the ISO shipping container on the landing surface of the current target equipment or chassis. Aspect 28 is the method of Aspect 27, wherein the alignment includes positioning the reference target equipment or chassis landing surface at an expected position and orientation in the reference system of the container handling equipment. Example 29 is the method of example 27 or 28, wherein the alignment includes (a) determining a first center of a landing surface of the current target equipment or chassis within the current target equipment or chassis point cloud model, (b) determining a second center of the landing surface of the reference target equipment or chassis within the reference target equipment or chassis point cloud model, (c) aligning the first and second centers to obtain the course alignment, and (d) iteratively minimizing distances between corresponding points of the current chassis point cloud model and points of the reference chassis point cloud model to obtain the fine alignment. Example 30 is the method of any one of examples 27-29, wherein iteratively minimizing the distances includes an iterative closest point (ICP) algorithm.

[0015] Specific aspects of embodiments of the present invention also include Aspect 31, which includes a method for identifying a type of target equipment or chassis capable of holding an ISO shipping container, the method including: (a) obtaining a point cloud model of a current target equipment or chassis landing surface; and (b) comparing the point cloud model of the current target equipment or chassis landing surface with one or more reference target equipment or chassis landing surfaces to identify which reference target equipment or chassis landing surfaces match the current target equipment or chassis landing surface within a selected degree of match; or (c) analyzing the point cloud model of the current target equipment or chassis landing surface to identify a type of target equipment or chassis with which the point cloud model of the current target equipment or chassis landing surface is associated. Aspect 32 is the method of Aspect 31, further including using a neural network and samples of the reference target equipment or chassis landing surfaces to train the neural network about matching target equipment or chassis clouds. Aspect 33 is the method of Aspects 31 or 32, wherein the comparison includes comparing metadata obtained from one or more of the point cloud model of the current target equipment or chassis landing surface and the point cloud model of the reference target equipment or chassis landing surface. Example 34 is a method of any one of Examples 31 to 33, wherein the comparing and / or analyzing is performed to identify similarities or differences in surface characteristics between the current target equipment or chassis landing surface and one or more of the reference target equipment or chassis landing surfaces, and / or between the current target equipment or chassis landing surface and known target equipment or chassis landing surfaces.

[0023] Example 35 is the method of any one of Examples 31-34, wherein the comparing and / or analyzing is performed to identify one or more of: (a) continuous or discontinuous surfaces; (b) data anomalies; (c) types of landing surface features; (d) the presence or absence of one or more landing surface features, such as girders, cross members, supports, and / or guides; (e) spacing of one or more landing surface features, such as girders, cross members, supports, and / or guides, relative to like landing surface features, relative to each other, or relative to another reference point; (d) actual or relative shape, dimension(s), and / or size of landing surface features, such as girders, cross members, supports, and / or guides; and / or (e) one or more characteristics of one or more landing surface profiles, such as bends, step downs, warping, and / or ramps, or a combination of one or more of these. Example 36 is the method of any one of Examples 31-35, wherein the metadata is obtained from a voxel filter.

[0037] Aspect 37 is the method of any one of Aspects 31-36, wherein the voxel filter operates to analyze one or more of the point cloud models of the current target equipment or chassis landing surface and / or the reference target equipment or chassis landing surface as a series of boxes, each covering 3D space. Aspect 38 is the method of any one of Aspects 31-37, wherein the metadata includes (a) a number of points within each box, (b) a maximum Z coordinate value for each of the boxes in the XY plane, or (c) whether a particular box is occupied. Aspect 39 is the method of any one of Aspects 31-38, wherein the metadata is compared by (a) a binary similarity comparison, (b) locating cross members and their locations, such as based on the number of boxes with visible points in the XY plane of the cloud point model(s), (c) measuring maximum values ​​along the length of the chassis and calling those target maximum value objects that define a particular chassis, or (d) one or more combinations thereof.Example 40 is the method of any one of Examples 31-39, wherein the binary similarity comparison includes subtracting metadata from the point cloud model of the current target device or chassis landing surface from metadata from the point cloud model of the reference target device or chassis landing surface to arrive at an error for each comparison. Example 41 is the method of any one of Examples 31-40, wherein the metadata is compared by binary string comparison. Example 42 is the method of any one of Examples 31-41, wherein the binary string comparison includes (a) summing the number of occupied boxes in each string for the previous target device or chassis landing surface and for each reference target device or chassis landing surface to arrive at a set of string sums, and (b) comparing the set of string sums between the current target device or chassis landing surface and each reference target device or chassis landing surface to arrive at an error for each comparison.

[0016] Specific aspects of embodiments of the present invention also include Aspect 43, which is a method for landing an object on a landing surface, the method including: (a) scanning a target device or chassis; (b) providing a model of the target device or chassis; (c) comparing the model to one or more reference devices or chassis; (d) identifying which of the one or more reference devices or chassis matches the target device or chassis within a selected degree of similarity; (e) determining the height of the target landing location on the target device or chassis using a known landing location height on the matching reference device or chassis as a proxy; and (f) landing the object or container on the target landing location by lowering the object or container at a speed, and then decreasing the speed as the object or container approaches the height of the target landing location as determined by the proxy. Aspect 44 is the method of Aspect 43, in which the object or container is an ISO shipping container. Aspect 45 is the method of Aspect 43 or 44, in which the scanning is performed using one or more of a camera, LiDAR, sonar, or optical sensor, or a combination thereof. Example 46 is the method of any one of Examples 43-45, wherein the models of the target device or chassis and the reference device or chassis are represented by a 3D point cloud of data. Example 47 is the method of any one of Examples 43-46, wherein the comparing includes identifying one or more reference devices or chassis having a length within a specified range of the length of the target device or chassis and / or using a voxel filter to identify similarities between the reference device or chassis and the target device or chassis. Example 48 is the method of any one of Examples 43-47, wherein the voxel filter compares data across one or more dimensions of the 3D data cloud. Example 49 is the method of any one of Examples 43-48, wherein the voxel filter compares any combination of x-, y-, and / or z-values.

[0017] Specific aspects of embodiments of the present invention also include Aspect 50, which is a method for landing an ISO transport container on a target equipment or chassis, the method including: (a) acquiring a point cloud model of a current target equipment or chassis landing surface aligned with a reference system of the container handling equipment; (b) aligning the point cloud model of the reference equipment or chassis landing surface with the point cloud model of the current target equipment or chassis landing surface; (c) generating a series of coordinates within the reference system as a result of the alignment, the coordinates representing a target landing location and an elevation for placing the ISO transport container on the landing surface of the current target equipment or chassis; and (d) lowering the ISO transport container at a speed to land the ISO transport container on the landing surface of the current target equipment or chassis, and then decreasing the speed as the ISO transport container approaches the elevation of the target landing location. Aspect 51 is the method of Aspect 50, in which the alignment includes positioning the reference equipment or chassis landing surface at an expected position and orientation in the reference system of the container handling equipment.

[0018] Certain aspects of embodiments of the present invention include aspect 52, which is a method of landing an ISO shipping container, the method including: (a) obtaining a point cloud model of a landing surface of a target equipment or chassis; (b) comparing the point cloud model of the target equipment or chassis landing surface with one or more reference equipment or chassis landing surfaces to identify which reference equipment or chassis landing surfaces match the target equipment or chassis landing surface within a selected degree of match; or (c) analyzing the point cloud model of the target equipment or chassis landing surface to identify the type of equipment or chassis with which the point cloud model of the target equipment or chassis landing surface is associated; (d) determining at least a height of the landing surface of the target equipment or chassis; and (e) landing the ISO shipping container on the landing surface of the target equipment or chassis by lowering the ISO shipping container at a speed, and then reducing the speed as the ISO shipping container approaches the height of the landing surface of the target equipment or chassis.

[0019] A particular aspect of an embodiment of the present invention is a method for landing an object or container on a landing surface, the method comprising: (a) providing a data point cloud representing a reference equipment or chassis; (b) providing a data point cloud representing a target equipment or chassis; (c) positioning the reference equipment or chassis data point cloud at an expected virtual location within a motion reference frame of a container handling equipment; (d) estimating a first center of the landing surface of the reference equipment or chassis and a second center of the landing surface of the target equipment or chassis within their respective data point clouds; (e) performing a first alignment by aligning the first center with the second center; and (f) performing any translation (multiple translations) required to achieve the first alignment, such as in X, Y, and Z coordinates, from the location of the first center to the location of the second center. (g) performing at least one second alignment by otherwise aligning the landing surface of the reference equipment or chassis with the landing surface of the target equipment or chassis; (h) determining any rotation(s) of the reference chassis data point cloud, such as about the X, Y, and Z axes, required to achieve the second alignment; (i) estimating the actual height of the target equipment or chassis using the translation(s) and / or rotation(s); and (j) lowering the object or container at a speed to land the object or container on the landing surface of the target equipment or chassis, and then reducing the speed as the object or container approaches the height of the landing surface of the target equipment or chassis. Embodiment 54 is the method of embodiment 53, wherein the object or container is an ISO shipping container.

[0020] Specific aspects of embodiments of the present invention also include aspect 55, which is a system comprising: (a) container handling equipment; (b) one or more measuring devices disposed on or near the container handling equipment; (c) one or more processors; and (d) a memory including a set of computer-executable instructions configured to instruct the one or more processors to perform the method of any one of aspects 1-54. Aspect 56 is the system of aspect 55, wherein the memory is stored on a non-transitory computer-readable storage medium. Aspect 57 is the system of aspect 55 or 56, wherein the one or more measuring devices comprise a camera, LiDAR, sonar, or optical sensor, or a combination of one or more thereof. Aspect 58 is the system of any one of aspects 55-57, wherein the container handling equipment comprises: (a) a motion control system; and (b) a container lifting device, such as a spreader. Aspect 59 is the system of any one of aspects 55-58, wherein the one or more measuring devices are disposed on the container handling equipment or a movable portion thereof.

[0021] Additional embodiments and aspects of the present invention will become apparent from the foregoing detailed description. [Brief explanation of the drawings]

[0022] The accompanying drawings illustrate certain aspects of embodiments of the present invention and should not be used to limit the invention. Together with the written description, the drawings serve to explain certain principles of the invention.

[0023] [Figure 1] A diagram of an AGV being unloaded. [Figure 2] A diagram of a bomb cart being loaded and unloaded in an ISO standard container. [Figure 3] Diagram of an ISO standard shipping container (40 feet). [Figure 4A] FIG. 1 is a diagram of a container crane. [Figure 4B] FIG. 1 is a diagram of a container crane. [Figure 5] 1 is a diagram of ISO standard corner casting. [Figure 6] FIG. 1 shows a diagram of the (unloaded) load chassis. [Figure 7] 1 is a point cloud rendering of a load chassis in a landside transfer zone according to one embodiment of the present invention. [Figure 8] A diagram of the spreader at 40 feet. [Figure 9] FIG. 1 is a diagram of a twist lock. [Figure 10] FIG. 1 is a diagram of a system topology according to one embodiment of the present invention. [Figure 11] FIG. 1 is a diagram of a measurement device configuration according to one embodiment of the present invention. [Figure 12] 1 is an image showing a perspective view of the landside transfer zone. [Figure 13] FIG. 1 is a diagram of the landside transfer zone. [Figure 14] FIG. 1 illustrates variations in chassis style. [Figure 15A] FIG. 1 is a diagram of a chassis and its model cloud, according to one embodiment of the present invention. [Figure 15B] FIG. 1 is a diagram of a chassis and its model cloud, according to one embodiment of the present invention. [Figure 15C] FIG. 1 is a diagram of a chassis and its model cloud, according to one embodiment of the present invention. [Figure 16] FIG. [Figure 17] FIG. [Figure 18] 10 is an image showing the state and fluctuations of the chassis. [Figure 19] 10 is an image showing the state and fluctuations of the chassis. [Figure 20] 4 is a flowchart of an exemplary operational procedure according to one embodiment of the present invention. [Figure 21] FIG. 1 is a diagram of a chassis with a container whose location is known. [Figure 22] FIG. 1 is a diagram of a chassis with a known load position. [Figure 23] FIG. 10 illustrates determining a front container offset according to one embodiment of the present invention. [Figure 24] FIG. 10 illustrates a chassis with a front container offset according to one embodiment of the present invention. [Figure 25] FIG. 10 illustrates determining aft container offset according to one embodiment of the present invention. [Figure 26] 1A and 1B are diagrams of a chassis having a front offset and a rear offset according to one embodiment of the present invention. [Figure 27] FIG. 10 is a diagram of a chassis with a 40-foot twistlock engaged and detected, according to one embodiment of the present invention. [Figure 28] FIG. 10 is a diagram of a chassis with detected twistlocks and target locations / orientations of containers according to one embodiment of the present invention. [Figure 29A] FIG. [Figure 29B] FIG. [Figure 29C] FIG. [Figure 30] FIG. 1 is a diagram of a model cloud for a chassis, according to one embodiment of the present invention. [Figure 31] FIG. 1 is an illustration of a scan cloud of a current chassis, according to one embodiment of the present invention. [Figure 32] FIG. 1 is an illustration of a binary similarity comparison according to one embodiment of the present invention. [Figure 33] FIG. 2 is a diagram of a binary string comparison according to one embodiment of the present invention. [Figure 34] FIG. 10 illustrates a diagram showing expected chassis positions in a lane according to one embodiment of the present invention. [Figure 35] FIG. 2 is a diagram of a model cloud in a predicted location according to one embodiment of the present invention. [Figure 36] FIG. 10 is a diagram of actual chassis position in a lane according to one embodiment of the present invention. [Figure 37A]FIG. 10 illustrates data sampled from an actual chassis parked in a lane, according to one embodiment of the present invention. [Figure 37B] FIG. 10 illustrates data sampled from an actual chassis parked in a lane, according to one embodiment of the present invention. [Figure 38] 1 is a diagram of a model cloud at an ideal location and a current data cloud at an actual location, according to one embodiment of the present invention. [Figure 39] FIG. 1 illustrates an initial rough alignment of a model cloud with a real cloud, according to one embodiment of the present invention. [Figure 40] FIG. 10 illustrates the final registration after ICP (iterative closest point), according to one embodiment of the present invention. [Figure 41] 1 is an image showing a typical landside transfer zone. [Figure 42A] 10 is a rendering showing raw data for a Twin Twenties loaded chassis, according to an embodiment of the present invention. [Figure 42B] 10 is a rendering showing filtered data for a Twin Twenties loaded chassis, according to an embodiment of the present invention. [Figure 43A] 10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. [Figure 43B] 10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. [Figure 43C] 10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. [Figure 43D] 10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. [Figure 43E] 10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. [Figure 43F]10 is a rendering showing a series of filtered scans of several chassis types, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference will now be made in detail to various exemplary embodiments of the invention. It should be understood that the following description of exemplary embodiments is not intended to be limiting on the invention. Rather, the following discussion is provided to provide the reader with a more detailed understanding of certain aspects and features of the invention.

[0025] definition Automated Guided Vehicle (AGV): AGVs are port equipment designed to transport ISO containers between areas of the port. The vehicles are designed for top-loading and unloading and move around the port using sensors without an operator, either local or remote. An example showing an AGV being loaded and unloaded is shown in Figure 1.

[0026] Block / container storage area / stacking area / stack(s): In this document, block is the term used to describe the container storage area where container handling cranes stack and transport containers.

[0027] Bomb Cart / Cassette: A truck chassis (trailer) designed and manufactured for the purpose of transporting standard shipping containers at container terminals. These do not typically have locking mechanisms for the containers. An example is shown in Figure 2, showing a bomb cart being unloaded and loaded with an ISO standard container. An example of an ISO standard shipping container (40 feet) is shown in Figure 3.

[0028] Booth / Driver's Booth: For the purposes of this document, a booth is a designated location, often an enclosure, within each lane of a landside transfer zone where the truck driver or equipment operator must be present during crane operations.

[0029] Container / Box / Reefer / Tank / Flat Rack: A shipping container defined by ISO standards. Used in international and domestic shipping. Standard lengths include, but are not limited to, 20, 40, and 45 feet. The standard also includes various types of containers, not just metal corrugated wall containers.

[0030] Container Crane / Container Handling Crane: A crane used to move shipping containers, for example, when containers are transferred from a ship to a shore port, when containers are moved within a storage area, or when containers are transferred to or from trucks at a container terminal or rail yard. An example of a container crane is shown in Figures 4A-B.

[0031] Container Handling Equipment (CHE): Any equipment designed to handle and transport containers around a terminal. In most cases, this is some kind of crane, but can also include equipment such as top pickers, straddle carriers, or similar. This class of equipment is usually capable of lifting and placing containers vertically.

[0032] Corner Casting / Casting / Corner Fitting: Corner casting as defined by the ISO standard. It is a standard mechanical fastener at each of the eight corners of a container that allows the container to be stacked, locked, and moved. An example of ISO standard corner casting is shown in Figure 5.

[0033] 6DoF / 6 degrees of freedom: The 6 degrees of freedom refer to the XYZ position of the corresponding orientation, that is, rotation about each of these axes XYZ, commonly referred to as trim, wrist and skew.

[0034] Crane Motion Reference Frame / Crane Coordinates / World Coordinates: This refers to a coordinate system that contains an origin, where the crane's automated movement and adjustment systems function. This often correlates to the number of millimeters / meters in each Cartesian direction. Some specific point on the crane's spreader is from a predetermined surveyed location in the crane yard. For the purposes of this specification, crane coordinates and world coordinates are the same.

[0035] Landside Transfer Zone (LSTZ): For the purposes of this document, the landside transfer zone is the part of the container handling crane servicing area where the lanes for the load chassis and other port equipment are serviced by the crane when either loading or unloading containers.

[0036] Lane: In this document, lane refers to the area where container handling equipment parks to be serviced by a crane.

[0037] Measuring Device: As referred to herein, a "measuring device" refers to any device capable of generating measurement points that can be expressed as Cartesian coordinates relative to an origin. Measuring devices include, but are not limited to, cameras, LiDAR, sonar, optical sensors, etc., any of which can be utilized to generate 3D (three-dimensional) measurement points.

[0038] Measured Points / Scanned Points / 3D Data: These terms all refer to a collection of points measured by a measuring device and exist as Cartesian coordinates within the crane's operating frame of reference. The terms model cloud and / or point cloud are sometimes used to describe these features.

[0039] Model / Template: A model is defined as an object that contains all of the model clouds and their respective relative landing locations, which are determined when "teaching" the system the landing locations of a given device.

[0040] Model Cloud: A model cloud is a high-resolution, densely sampled set of measurement points that, in the context of this disclosure, can represent any equipment that accepts containers. In an embodiment, the cloud can be filtered to only landing surfaces and can analyze metadata about the landing surfaces sufficient to classify them and provide a landing solution for where to place the container on the equipment.

[0041] Load chassis: A truck chassis (trailer) designed and manufactured for the purpose of transporting standard shipping containers onto roads outside container terminals. These chassis usually have twist locks in ISO standard positions for the corresponding size containers they are designed to carry. An example of a (loaded and unloaded) load chassis is shown in Figure 6.

[0042] Point Cloud: A point cloud is a term used to refer to a collection of 3D data points in a coordinate system. Figure 7 shows an example of a point cloud rendering of a load chassis in a landside transfer zone. The term cloud can also be used to refer to a point cloud.

[0043] Spreader / Hook / (Spreader) Bar: A spreader is a device that locks onto an ISO container and allows a crane to move the container. An example of a spreader at 40 feet is shown in Figure 8.

[0044] Truck Lane: A truck lane is either a lane in the landside transfer zone where trucks being serviced are parked, or a lane under the crane that runs the length of the block adjacent to the stack of containers in the block where trucks are being serviced by the crane.

[0045] Twistlocks: Twistlocks are devices designed to be placed inside a hole in a corner casting and then engaged by rotating the top of the device 90 degrees. This device is used to secure a container's corner casting to another object. An example of a twistlock is shown in Figure 9.

[0046] Voxel Grid: A voxel grid is the result of a voxel filter, which takes a point cloud and analyzes it as a series of boxes that cover the 3D space the cloud occupies. Each "box" is typically the same size and has configurable parameters. Points from the cloud are grouped into each box based on their (x, y, z) coordinate values, and the data for each box is then stored in an array, the voxel grid.

[0047] Physical Configuration of the Invention According to one embodiment, the physical system of the present invention is comprised of at least three main components: the first component is a measurement device(s) that can generate measurement points of physical objects within the sensor's field of view and report them to other components; the second component is one or more computing devices that can be used to run the software program components of the present invention that drive the measurement device(s) and perform analysis of the multiple measurement points to generate a landing position; and the final component of the system is a device, i.e., some container handling equipment, that performs the landing of the container using the landing positions provided by the computing device(s).

[0048] Figure 10 illustrates an exemplary system topology according to one embodiment of the present invention. The top left box of the diagram shows a programmable logic controller (PLC), two central processing units / processors, and a (Maxview( Registered trademark )CPU and Maxview4D( trademark )CPU) , software (CraneDirector ( trademark) The figure shows the E-house (or electrical house), a substation containing a crane management system (CMS) computer. This includes traditional computer functions such as a processor, memory, hard drive, and graphics processing unit (GPU), as well as input / output devices such as a display, keyboard, and mouse, such as those described in U.S. Patent Nos. 8,686,868 and 7,725,287. The box to the right of the E-house represents the crane leg, which contains a LiDAR sensor (LMS6) and left and right gantry motors. The boxes below the E-house and crane leg represent the trolley. The trolley contains a series of LiDAR sensors (LMS1-LMS5 and LMS7), a trolley-mounted camera, and a trolley pan-tilt-zoom (PTZ) camera. The trolley also contains one or more hoist motors and one or more trolley motors. The box on the bottom right represents the spreader, which contains a series of spreader cameras and one or more micromotion motors. Additionally, various paths of operable connections are shown below. The black lines represent the various paths that control the crane's movements and feedback. The dark gray lines represent the various monitoring paths as part of a monitoring network that ultimately flows into connections to central location devices such as the Remote Operating System (ROS), Terminal Operating System (TOS), and Remote Crane Management System (RCMS). The light gray lines represent the various control paths between components, as shown.

[0049] Not shown in FIG. 10 is a non-transitory computer storage medium, such as RAM, that stores a set of computer-executable instructions (software) that instruct a processor to perform any of the methods described in this disclosure. As used in the context of this specification, "non-transitory computer-readable medium(s)" may include any type of computer memory, including magnetic storage media, optical storage media, non-volatile memory storage media, and volatile memory. Non-limiting examples of non-transitory computer-readable storage media include floppy disks, magnetic tape, conventional hard disks, CD-ROMs, DVD-ROMs, Blu-rays, flash ROMs, memory cards, optical drives, solid-state drives, flash drives, erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), non-volatile ROMs, and RAM. Non-transitory computer-readable media may include a set of computer-executable instructions for providing an operating system, as well as a set of computer-executable instructions or software for implementing the methods of the present invention. The computer readable instructions may be programmed in any suitable programming language, including JavaScript, C, C#, C++, Java, Python, Perl, Ruby, Swift, Visual Basic, and Objective C.

[0050] According to embodiments, the system can be configured in various ways. In a first configuration, all measurement devices are mounted on the crane, and all measurements are reported in the crane's motion frame of reference. A second configuration exists in which at least one measurement device mounted on the crane itself reports measurement points in the crane's motion frame of reference, with other measurement devices detached from the crane reporting measurement points in any frame of reference. In embodiments, at some, none, any, or all points during the measurement collection phase, all, at least one, more than one, or none of the measurement devices can be positioned to obtain a nearly unobstructed view of surfaces arranged below the target equipment and container handling equipment, such as the target equipment's landing surface. For example, the collection of measurement devices can together provide a complete or nearly complete view of the landing surface. Figure 11 illustrates a configuration in which various LiDAR sensors (e.g., LMS1, LMS2) are positioned on the crane.

[0051] According to an embodiment, the third component, the primary device referred to in this disclosure as using the landing position generated by the system described herein, is a type of CHE. According to one embodiment, the specific CHE type is a rail-mounted gantry (RMG) crane or automated stacking crane (ASC) that moves 20-foot, 40-foot, and 45-foot ISO-standard shipping containers. However, the system outlined in this disclosure is not necessarily limited to operating only with this type of CHE. Other types of CHE with which the system may operate include ship-to-shore gantry cranes, rubber-tired gantry cranes, straddle carriers, front-end loaders, reach stackers, portanas, etc.

[0052] In an embodiment, the system relies on the crane to report its own location and relies on the placement of measuring devices on the crane being known with respect to the placement of the crane to report the measurement points of the object relative to one common point in the crane's frame of reference. The placement of any measuring devices not on the crane does not necessarily need to be known with respect to the crane. The system relies on the object operating (on a chassis or other equipment) so as to be visible to the measuring devices and that the object is within the operating area of ​​the crane.

[0053] Operation / Functionality of the Invention According to an embodiment, the operation of the system and method used is defined below.

[0054] The operation procedure according to one embodiment of the present invention will be described below.

[0055] 1. A crane receives a job request to land a load on some target chassis.

[0056] 2. The crane moves to the general target location: During the duration of the crane movement, the system collects measurement points and concatenates them into a point cloud to obtain a view of the scene below the crane, preferably the complete scene below the crane, although partial scenes can also be obtained. The system includes one or more, preferably multiple, sensors strategically positioned to obtain the amount of scene desired for a particular application. The measurement points are obtained from one or more measurement devices, preferably multiple measurement devices mounted around the crane, and the measurement points are reported to the system as 3D points within the crane's motion reference frame or the external coordinate system of an external device.

[0057] 3. Active Jobs: The system then analyzes the scene (point cloud) to determine if the job is valid: it looks for chassis, containers, prime movers, obstacles, etc., and determines if the items and locations onshore match the job requirements and can operate (without obstacles).

[0058] 4. Choose a model: a. If the system is equipped with a catalog of known chassis variables, it processes the current chassis point cloud data and compares it to the known variables to find the best fit. The result is a point cloud for the model with a target offset from the center of that cloud. See "Creating Relative Landing Positions: Teaching the System".

[0059] b. If the system is equipped with one or more additional sensors providing high-resolution data of the chassis, the system processes the high-resolution data to determine the target landing offset from the center of the cloud. The result is a point cloud of the model with a target offset from the center of the cloud. See "Creating Relative Landing Locations: Analyzing Surface Features at High Resolution."

[0060] 5. Run the landing solution generation: The method described in the "Method for Generating a Landing Solution for a Load on a Landing Surface" section of the crane's motion coordinate system is then used to generate a landing solution for where to place the container so that the system mates well with the chassis' landing surface.

[0061] 6.The crane's adjustment system then moves the spreader / spreader and container combo to the required location.

[0062] Sections 4 and 5 above refer to some of the procedures described in this disclosure.

[0063] A flowchart of an operational procedure according to one embodiment of the present invention is shown in Figure 20. Process 200 begins when a job start 205 is requested. The system then collects 3D data points 210 and extracts chassis surfaces from these points 215. At this point, the system determines whether a chassis exists 220. If the system determines that a chassis does not exist, then the process ends without a solution 225. If the system determines that a chassis exists, then the system queries a model library to determine whether the chassis has a corresponding chassis in the model library 230. If a model library exists, then the system first divides the surface into a voxel grid 235. The system then compares the voxel grid to existing model voxel grids 240. The system then determines whether a similar model of the chassis exists 245. If a similar model of the chassis does not exist, then the process ends without a solution 250.

[0064] If a model library does not exist, the system obtains a model cloud from the high-resolution live data 255. The system then extracts the chassis surface 260 and processes the target equipment to identify structures that may be associated with engaging an object or landing an object on the target equipment's landing surface. In this embodiment, the system processes the chassis to identify twistlocks and corresponding container locations 265. After this step, the system obtains the extracted chassis and localized target container locations as a model cloud 270, and the model can be included in the model library 230 for future reference and / or used in future analyses, such as in an iterative nearest neighbor (ICP) analysis.

[0065] Then, if there is a similar model, or once the system has the extracted chassis and localized target container locations as a model cloud, the system finds the best alignment between the current chassis field of view and the selected model 275. The system then applies any offset from the center of the model if it does not land on the center 280. The system then reports the transformation as the solution, and the process is complete 285.

[0066] How to analyze landing surfaces According to an embodiment, the method used for landing surface analysis is completed in the following stages.

[0067] 1. Create a relative landing position. Two options: A. Teach the system where to place the containers and create a point cloud of stored models for future reference.

[0068] B. Analyze landing surface features and create a point cloud of models on the fly, optionally at high resolution.

[0069] 2. Determine the point cloud of the model to use on the current landing surface and its relative landing position.

[0070] Creating Relative Landing Positions: Teaching the System In an embodiment, "teaching" is the process of using the crane and sensors (measurement devices) to determine the location of the load on the landing surface within the crane's operating frame of reference, then collecting a set of super-sampled measurement points of the unloaded chassis, and storing the two pieces of information in the system.

[0071] According to embodiments, the system provides a valid landing solution for equipment containers without needing to know the exact location of the twistlocks or any other characteristics of the landing surface. The system only needs to know where to place the container with respect to some common point on the landing surface so that it properly engages with the twistlocks (if the landing surface has them) or so that the container properly rests on the surface. The next section outlines a process for determining the landing location for a given load chassis, but the same process can be applied to other types of equipment that have a landing surface that can accept a container or any other object. The landing surface does not need to have twistlocks for the process to remain applicable.

[0072] In one embodiment, the process of "teaching" the system where to place a container on any given landing surface begins by placing the chassis in a resting position under the crane with the container correctly and accurately positioned. After the container is carefully placed in the desired location on the landing surface, the system generates the container's precise location within the crane's motion frame of reference (see, e.g., FIG. 21). This location can be obtained in any manner, provided the container's position is precisely known within the crane's motion frame of reference. However, most commonly, the system generates the container's location using measurement device(s) placed on or near the crane. To do this, the system collects measurement points from one or more measurement devices as the crane moves over the top of the container, generating multiple measurement points that are reported as Cartesian coordinates in the crane's motion frame of reference and represent the top of the container. The system then processes this set of measurement points (e.g., a point cloud) to generate the container's precise location within the crane's motion frame of reference (FIG. 21).

[0073] In an embodiment, this location can be generated accurately relatively easily because shipping containers are well defined in ISO standards (ISO 668) with respect to their width, height, and length. Typically, the ISO standard specifies the following approximate outer dimensions for a shipping container: Table 1: ISO container specifications JPEG0007784991000001.jpg82129

[0074] From a data processing perspective, the container itself is a rectangular box. Because the general location of the container is known and its properties are well-defined, calculating the container's position in six degrees of freedom (6DoF) can be done by model matching, line extraction, plane fitting, etc.

[0075] According to an embodiment, once the exact location of the container has been determined, the container is then carefully removed from the chassis landing surface so as not to disturb the location or orientation of the landing surface. Subsequently, as in the process described above, as the crane moves over the empty chassis, the system collects measurement points, reported as Cartesian coordinates in the crane's motion reference frame, from one or more measurement devices located on or near the crane to generate a plurality of measurement points (e.g., a point cloud) representing the shape of the landing surface. However, at this stage, the crane can be moved once over the object's surface, e.g., very slowly, or multiple times in a manner that creates a very dense set of measurement points. After filtering the measurement points using information about the known configuration of the area below the crane and the estimated landing surface height, the system then extracts points that represent only the landing surface, as shown in Figures 15A-C. The extracted points are then saved in the system as a "model" of the landing surface. Along with this model cloud, the previously extracted container locations are saved in the system, as shown in Figure 22. Using the model cloud, now stored as a collection of measurement points that exist in the crane's operating frame of reference, and the known location of the container, now stored as coordinates within the same frame of reference, the system can know exactly where the container should be placed on this landing surface.

[0076] In an embodiment, the system now has stored within it a 3D point cloud "model" of the landing surface with a known target location where a container should be placed on the landing surface. For the purposes of this system, the location of the largest container the landing surface can carry is considered the "center" of the landing surface. Therefore, the 3D model is shifted so that its coordinate center (0,0,0) becomes the target location of the largest eligible container the chassis can carry. This leaves the chassis' 3D point cloud model with the known container target location of the largest eligible container, i.e., the origin.

[0077] According to an embodiment, for chassis types that support the transport of multiple sizes of containers, offsets can be determined and recorded so that the landing locations of various sized containers are known for a given landing surface. To create multiple landing locations for a single landing surface, the system analyzes the containers that land at each eligible location on the landing surface, one at a time, and determines the location of each container in the same manner as described above. The system obtains the difference between the locations of each subsequent container from the first (largest) container, and the difference between the locations is recorded and stored as an offset from the "center" of the chassis. As a result of this process, all other landing locations become known relative to the landing location of the first container analyzed on the landing surface. Examples of such offset determination are shown in Figures 23-26.

[0078] In an embodiment, at the end of this process, the system will have stored at least one "model" of the landing surface and the location relative to the "center" of that surface to place the container. This process can be repeated for any number of chassis or other equipment to generate a library of models the system knows how to operate.

[0079] One added benefit of this method is that, if collected correctly, the data collected during the "teaching" phase is not specific to one crane and can be used by systems implemented in any CHE. In many cases, a model cloud collected for one chassis can be applied to another chassis in the same "class" (sharing key characteristics).

[0080] Creating relative landing positions: analyzing surface features at high resolution According to an embodiment, the system of the present invention can also use higher resolution data obtained from measurement devices located away from the crane in close proximity to the landing surface to locate the landing location of the container using key features on the chassis such as guides, targets, raised twistlocks, or recessed twistlocks, as shown in Figure 27.

[0081] This may be necessary in situations where the crane-mounted device cannot provide sufficient detail to create a functional model of the desired landing surface, or when other factors prevent "teaching" from being a reliable method of generating a landing solution. The system scans the chassis with a measurement device mounted close enough to the landing surface to generate multiple measurement points (e.g., a point cloud) where key features of the landing surface are distinguishable, collects them, and reports the points back to a computing device that analyzes the collection. As in the "teaching" phase, the system takes the collection of measurement points, filters only the landing surface, and then examines all distinguishable features of the chassis along the landing surface and catalogs them. The system searches along the chassis surface using local minima and maxima to determine the locations of raised and recessed twistlocks and catalog all possible landing point locations. Then, using geometric constraints imposed on landing point locations by ISO standards for container corner casting locations, the system finds the most suitable locations and reports the container's overall landing location as their mathematical center, as shown in Figure 28. Other techniques that can be used include model convolution. Mathematically speaking, convolution is the "integral of pointwise multiplication", and convolution filters are used to identify echo signals, such as in radar. Convolution filters use prior knowledge of the expected shape (wave, twistlock, etc.) to help better distinguish that shape from noise.

[0082] The system then obtains the discovered surface landing locations and a collection of landing surface measurements, as done in "teaching," and stores the two pieces of information as a model with its local origin as the overall landing location. This model can be stored in the system's long-term memory, or alternatively, or in addition, it can be passed directly to a method for generating landing solutions.

[0083] Find the right measurement point model to use In the "Creating Relative Landing Locations: Teaching the System" section, we defined how to create and store a collection of measurement points centered around the landing location of the largest container that each surface can accept, and the associated offsets of all other containers that the surface can accept. Once multiple models are stored in the system, a method is needed to select the correct model to apply for a given operation. This section outlines the steps performed by the system to perform a selection operation given the current set of measurement points and select the best fit match for a stored model.

[0084] According to an embodiment, after a method for "teaching" a system how to land on a particular piece of equipment has been performed, the system collects information about that piece of equipment through an analysis of the landing surface represented by a collection of measurement points reported by measurement devices, at least one of which is attached to the crane itself. The process of obtaining a representation of the landing surface and finding its associated landing location is described above. This process is completed for at least one representation of each piece of equipment belonging to a class of equipment on which the system is expected to operate, before performing the actual landing operation of the equipment.

[0085] A class is a collection of attributes that categorizes equipment into groups, with all members of a particular group sharing the same or similar landing positions. In contrast to primarily superficial attributes such as landing gear location and axle placement, classes typically consist of structural attributes such as girder location and support members. Objects within a class must have the same landing positions. An example is the skeleton chassis seen in Figures 29A-C. Of the three chassis shown in Figures 29A-C, the leftmost chassis is in a single class. They both share a landing position on the landing surface. This is sufficient for one 40-foot container to lock onto the chassis at both its front and rear ends simultaneously, and these two chassis can only carry 40-foot containers. The first and second chassis have the same overall shape, and the main members supporting the container's load are in roughly the same location, but the smaller cross members are in different locations as shown, and the wheels and landing gear are different. The third chassis belongs to the second class. This is because, unlike the first two, it can land two 20-foot containers on the chassis and can also carry a single 40-foot container load. The system requires one model of either the first or second chassis, and another model of the third chassis.

[0086] According to an embodiment, once the library is filled with representations of each class of equipment the crane is expected to service, the system can find an appropriate match for a given current chassis. In doing so, as the crane moves over the empty chassis, the system collects measurement points from one or more measurement devices located on or near the crane, which can be reported as Cartesian coordinates in the crane's operating frame of reference, generating multiple measurement points representing the shape of the landing surface, similar to the "teaching" phase. Depending on the application, other coordinate systems are equally applicable. For example, non-gantry cranes, such as luffing, tower, and jib cranes, may operate in a cylindrical / polar coordinate system. However, this time the crane moves the chassis only once. Data acquired in this way can be of low quality / noisy due to limitations in the resolution / quality of the measurement devices, the instability of the crane's mechanical features, and the possibility that the measurement device's field of view is obstructed by the load suspended by the spreader hanging from the crane during the operation. Because it is usually impossible to reliably detect small objects such as twistlocks in low-quality data samples, it is not possible to calculate the landing position using chassis features. Furthermore, with only one path, depending on the state of the operating chassis, portions of the chassis may not be visible at all due to obstructions or poor visibility from the measurement device. Due to these factors, the system may not have a complete sample of the landing surface represented by the measurement points. Regardless of the quality of the data from the measurement device, the system processes the current chassis measurement data using similar filtering and processing techniques to eliminate all but representative landing surfaces from the provided data. The filtered and processed subset of multiple measurement points (e.g., a point cloud) is then considered the cloud of the current field of view of the landing surface. An example of a densely sampled model cloud is shown in Figure 30, and the data for the operating chassis is shown in Figure 31.

[0087] Now that the system has a cloud of the current chassis, it can begin to compare this object to the entire library of stored models by cross-referencing the object against the library of model clouds and measuring its similarity to known chassis using various metrics and metadata generated for the model cloud and the current cloud. The comparison process begins by taking each cloud of measured 3D points for each item and applying a voxel filter to them.

[0088] In an embodiment, a voxel filter takes a cloud and analyzes it as a series of boxes covering the 3D space the cloud occupies. Each "box" is the same size and has configurable parameters. Points from the cloud are grouped into each box based on their (x, y, z) coordinate values, and data about each box is then stored in an array. Types of metadata include, but are not limited to, the number of points in each box, the maximum Z coordinate value of each box in the XY plane, or whether a given box is occupied.

[0089] According to an embodiment, once raw metadata is generated that knows some basic principles of the landing surface, the system applies several methods to fill holes where noisy data may be missing information. Because model clouds generated by the "teaching" method are well sampled and are expected to be noise-free or substantially noise-free, hole filling / noise reduction is only applied to clouds generated at the active landing surface. The result of this process is a collection of metadata about each stored model cloud and the model cloud of the current scan.

[0090] In embodiments, this metadata is then used in a series of calculations and metrics designed to test the similarity of two landing surface objects. In one example, the system determines the length of each landing surface in terms of the largest container the system can accept. The system then ignores all landing surfaces that do not match the length of the currently operated landing surface. The remaining stored landing surface metadata is then analyzed for specific surface characteristics that can be seen at a macro level and compared to the characteristics that apply to the current landing surface as determined by the analysis of that metadata. In embodiments, the surface characteristics considered in the analysis depend on the specific application. Surface characteristics include, but are not limited to, continuous or discontinuous surfaces that are interpreted as landing surfaces; data anomalies that are recognized as obstacles or defects; landing surface features such as the presence and spacing of girders, cross members, supports, or guides; surface profile characteristics such as bends, step-downs, warping, and / or ramps that constitute visible or measurable changes in the flatness or profile of the landing surface, or one or more combinations thereof.

[0091] In an embodiment, each collection of metadata from each eligible model cloud is looped through in turn and then compared to the metadata of the current cloud to generate a confidence score indicating the level of similarity that the given model cloud has with the current cloud. By comparing the metadata in different ways, various different scores can be generated. Each of these scores can then be weighted and summed to generate an overall score of match confidence. At the end of the process, if a sufficient score threshold has been met, the stored model with the highest score is selected as the match.

[0092] An embodiment of a simple metadata comparison is shown in Figure 32 - a binary similarity comparison. The top row represents the metadata from the current view of the chassis, followed by the stored metadata of two known chassis types. This metadata represents occupied / unoccupied boxes along the X,Y plane. One way to compare this data is to simply subtract the current view array from the model array. The result of this subtraction is displayed on the second row. For Model A, only one box is not commonly used in both arrays, while for Model B, five boxes are not commonly used. This results in an overall error score for this metric, called binary similarity, of 1 for Model A and 5 for Model B.

[0093] In another embodiment, another way to generate a score from the same metadata is to compress the data along a given dimension and compare again. By summing the number of occupied boxes in each column and comparing the results, Figure 33 - Binary Column Comparison is generated. In this comparison, A has a match error of 1, but B only has a match error of 3.

[0094] Now, the two sets of scores can be used to generate an overall match probability. Assuming each score is equally weighted, Model A will have a score of 2 and Model B will have a score of 7. Since the score in this case is the match error, the smallest number indicates the best match. Therefore, Model A is selected as the best match.

[0095] According to an embodiment, in practical applications, additional similarity comparisons can be performed to generate many scores, each of which can be weighted and summed to generate an overall similarity error, and finally the model with the best qualifying score can be selected. The type of metric and weighting applied is determined by analyzing the types of objects expected to be displayed, generating metrics, and weighting them accordingly.

[0096] Analysis steps for model selection: 1. Voxel Filter Key points of AX-Y BY-Z Points CX-Y value 2. Apply hole filling techniques to the current scan, but expect the data quality to be low. 3. Establish length classes for each model (10ft, 20ft, 40ft, 45ft, plus size, etc.) 4. For the next topic, compare each item in the length class that is most similar to the model cloud of the current scan. A. Compare the characteristics of each chassis by checking the similarity between the two sequences B. Generate a similarity confidence score C. The model with the highest qualified confidence score is selected as the best match

[0097] According to embodiments, in some cases, testing has shown that a particular landing surface may lack defining features when viewed as measurement points taken from above the surface. This makes it very difficult to select the correct landing surface from a library of models that matches the current landing surface present beneath the crane, as there may be no visible features between the two surfaces. Despite the lack of defining features in the two views, the two surfaces may result in significantly different container landing locations.

[0098] For example, take two flatbed chassis. The landing surfaces of the two flatbed chassis may be the same width, length, and height. Both chassis, when viewed from above, produce an indistinct rectangle of measurement points that are the same dimensions. For this example, we'll call them Chassis A and Chassis B. Assume both are 42 feet long and 8 feet wide (the width of an ISO container, 2 feet longer than a 40-foot container). Now, suppose Chassis A has twistlocks suitable for 40-foot containers, positioned from the front edge of the landing surface. Chassis B has twistlocks suitable for 40-foot containers, positioned 1 foot from the front edge of the chassis. Now, Chassis A and Chassis B are "taught" to the system and stored in its library for future use. Because the starting location of the two sets of twistlocks differs by 1 foot between the two chassis, if the wrong chassis is selected later, the container will not land properly. Now, later, Chassis B2 arrives and receives the box from the crane. This chassis is the same as chassis B, but because both chassis A and B appear identical in the system, there is no way to determine which is the correct model, and no way to get a solution for the container.

[0099] Ambiguities in the landing surface such that the correct one cannot be selected from the library via the companion will generate a landing solution by mounting sensors near the chassis or equipment (such as on the ground or on fixed or mobile equipment or structures) and using methods to analyze the landing surface at high resolution and identify the landing point or other distinctive features from the higher resolution data.

[0100] How to generate landing solutions for loads on landing surfaces In one embodiment, the above-defined method for creating a measurement point model is a precursor to generating a container landing solution on the landing surface of the container handling equipment, unless the system uses high-resolution surface feature analysis. Once a suitable model is selected or, in the case of "high-resolution surface feature analysis," generated, the model is passed to the next defined method, which uses the model in combination with measurement points collected by the measurement device(s) located on the crane of the current chassis to generate a solution. The following method uses a well-studied process called iterative nearest neighbor (ICP) (see Besl, Paul J., N. D. McKay (1992) "A Method for Registration of 3-D Shapes," iEEE Trans. on Pattern Analysis and Machine Intelligence. Los Alamitos, CA, USA: iEEE Computer Society. 14(2):239-256) to convert relative positions associated with the measurement point model into absolute positions in the crane's motion reference frame.

[0101] In an embodiment, by using a set of potentially low-quality measurements of the current landing surface, along with the model of the chassis created and acquired prior to this step, the system can determine exactly where on this chassis to land the container using well-studied methods of comparing the positions and orientations of two similar objects. The system starts with a rough guess of where the chassis is located under the crane. Typically, this rough guess of the current chassis location is provided by an external system (Terminal Operating System - TOS) that has a catalog of current movements and its estimated location in the crane's coordinate system; Figure 34 shows the chassis's typically expected location in the lane. To start the algorithm, a model cloud is placed at this expected location, as shown in Figure 35.

[0102] According to an embodiment, the system then attempts to generate a more accurate, yet rough, guess as to where to place the container near this location. The rough guess is obtained by filtering the noisy data corresponding to the current chassis data cloud to only the chassis landing surface and averaging the maximum and minimum measurement points of the filtered chassis point cloud to obtain the center of the landing surface. Figure 36 shows how a chassis is actually parked in a lane. Figures 37A-B show how this parked chassis can generate a current data cloud. Figure 38 shows the initial misalignment between the model cloud at the ideal location and the actual data cloud at the actual location.

[0103] In an embodiment, once the system has made a rough guess at the location of the landing surface, the system orients the model of the landing surface so that its origin, the landing location in the model of the center of the largest container supported by the chassis, is aligned with this center of the current chassis. Figure 39 shows an initial rough alignment by roughly aligning the centers of the ideal model cloud and the actual data cloud. While the model alignment at this stage is somewhat poor, it serves as a starting point for the rest of the process.

[0104] In an embodiment, after this rough alignment, the system has two collections of measurement points, the current point and the model point, that represent the chassis, which are approximately aligned within the crane's motion frame of reference. The system then attempts to create a fine alignment of the two objects using methods to obtain a more accurate alignment of the two items. The concept of taking two views of the same object, or in this case, two point clouds of the same object, and finding the transformation that best aligns them is a well-studied problem in computer vision and 3D data processing. There are many different ways to align these two clouds. One such algorithm that can be used is the iterative closest point (ICP) algorithm. (See Besl, Paul J.; N.D. McKay (1992) "A Method for Registration of 3-D Shapes." IEEE Trans. on Pattern Analysis and Machine Intelligence. Los Alamitos, CA, USA: IEEE Computer Society. 14(2):239-256.)

[0105] According to an embodiment, the algorithm takes two roughly aligned data sets and attempts to find, for each point in Set A, the corresponding point in Set B. Correspondence is determined by finding the closest points, typically determined by Cartesian distance. This generates a set of correspondences between points in Set A and points in Set B. Once correspondences are generated, a new position estimate can be generated that more accurately aligns the points in each set by attempting to minimize the distance between all correspondences. Once the distance is minimized, a new alignment is generated and another set of correspondences is found. This process is repeated until the change in position is sufficiently small. At this point, the most accurate alignment is achieved and the iterative process ends.

[0106] According to an embodiment, by using the ICP and stored chassis model, the system can generate a very accurate positioning of the model cloud on the current chassis cloud within the crane's motion frame of reference, as shown in Figure 40.

[0107] In an embodiment, the local origin of the model cloud represents the landing position of the largest container the model landing surface can support, there is an offset from that position to any other container the surface can support, and because the system positioned the model chassis the same as the current chassis using ICP, the system now has the landing position and orientation in crane coordinates of where to place the container on the current chassis. The output of the system is a translation in XYZ coordinates and a rotation about the XYZ axes that best aligns the two models. By placing the container at the target XYZ coordinates in a given orientation, the container will mate with the chassis in the desired position.

[0108] Example: The need for land-side automation in automated terminals Due to the increasing size of ships and the consolidation of shipping lines, competition among container terminals has become extremely intense. In the past decade alone, ship size has nearly doubled, from an average capacity of less than 4,000 twenty-foot equivalent container units (TEUs) in 2008 to a current average of 13,772 TEUs on the Asia-North Europe trade route. The world's largest vessels can hold 21,413 TEUs, and there are 57 ultra-large container ships with a capacity of at least 18,000 TEUs currently on order. Furthermore, environmental impact is a growing concern worldwide, with increasing regulations and incentives to reduce emissions. If container terminals want to attract larger ships and remain competitive, they must improve terminal throughput and efficiency while reducing carbon dioxide emissions. Landside automation offers a concrete solution to achieving these goals.

[0109] Automation has evolved in the container industry over the past two decades. Automated stacking cranes (ASCs) emerged in the early 2000s, revolutionizing the layout and operation of container yards. Ideally, an ASC yard is oriented perpendicular to the quay with a waterside transfer zone (WSTZ), stacking area, and landside transfer zone (LSTZ). The WSTZ serves as an exchange area between the quay and stacking area and is typically serviced by straddle carriers or automated guided vehicles (AGVs). These vehicles unload containers in the WSTZ. The ASC can automatically receive and unload containers in these areas as the vehicles depart. The stacking area is an unmanned area where the ASC operates in fully automated mode. The LSTZ is where containers are loaded onto road trucks for transport. Figure 41 shows the LSTZ of a typical ASC yard. Traditionally, these ASCs only allow fully automated operation within the unmanned areas (WSTZ and stacking area) and rely on a remote operator for all movements within the LSTZ. While this method offers many improvements over purely manual operation, it relies on a remote operator for all movements within the LSTZ, which introduces delays into the transport process.

[0110] Operators rely on various camera views to accurately place and land containers on road chassis from a remote location. These remote operators introduce human variability into the landing process. While some operators are highly efficient, others require more time or multiple attempts to successfully receive or land a container. Another delay arises from waiting for the remote operator to be able to execute the move. At one of the world's most advanced container terminals, the average wait time for a remote operator to connect to a waiting crane is approximately 33 seconds. This connection time is quite impressive when evaluated against a single container movement. However, over a 24-hour period, a 33-second delay represents a significant loss of productivity. This wait time can amount to up to 2.1 hours of wasted productivity per crane, or more than 50 hours per day depending on the size of the yard. These wait times only include the time spent waiting for an operator to become available; they do not include the time it takes for the operator to execute the move once connected. Implementing a successful automation solution at the LSTZ will significantly improve terminal productivity.

[0111] An embodiment of the present invention provides a new system that provides fully autonomous landing for top-loaded container handling equipment, including load chassis. This system utilizes the same sensors required for traditional ASCs and typically requires no additional infrastructure. Furthermore, it provides an economical solution for new or retrofitting existing ASCs. The system uses a crane-mounted laser scanner (LiDAR) to create a model of the area below the spreader. When the ASC enters the LSTZ, it begins scanning the area below the crane, generating a series of Cartesian coordinates in the crane's frame of reference. Collecting these points over time creates a point cloud representing the area of ​​interest. Filtering the raw point cloud allows targets to be clearly displayed. Figures 42A and 42B show both raw (Figure 42A) and filtered (Figure 42B) point cloud data for a loaded chassis. The filtered data provides the exact location of the container, which the crane uses to autonomously remove the container from the chassis and transfer it to the stacking area.

[0112] Automating the process of landing a container onto a load chassis poses many more challenges than receiving it. The inherent variability between different chassis types, the 25mm mating accuracy required for twistlocks, and the resolution capabilities of the crane-mounted scanner are major obstacles. Embodiments of the present invention provide a novel LSTZ automation solution that addresses these issues by analyzing the geometry and shape of the chassis, instead of attempting to "see" the twistlocks to determine the appropriate landing solution. Point cloud data for the empty chassis is constructed as described above, and the system evaluates key geometric features to determine the correct loading location for the container and place it on the empty chassis. Figures 43A-F show scan data from several different chassis types.

[0113] The system has proven highly successful where it is currently operational. It has an average success rate of 99% for automatic pick-up from the chassis and 70% for automatic landing on the chassis, resulting in a total success rate of 84.5% for shore-side automatic operations. The system significantly improves efficiency and success rates. The system generates a solution in approximately three seconds after completing a scan. The total time required to land on the load chassis is 66 seconds, from the time the crane enters the LSTZ to the time the spreader is completely off the container.

[0114] Such a landside automation system reduces the average wait time at the LSTZ from 33 seconds waiting for a remote operator to 3 seconds, enabling the system to generate the correct solution for 84.5% of all movements. This is an extra 1.66 hours of production per crane each day, without considering improvements in active landing time. Furthermore, when an operator must intervene, their involvement typically consists of a simple adjustment to the landing position, which takes significantly less time to complete than a full remote move.

[0115] Another benefit of the LSTZ's autonomous landing is that hard landings are virtually eliminated. The automatic landing system accurately detects chassis height and appropriately slows the vehicle before touchdown. This reduces spreader damage and maintenance and increases overall crane uptime.

[0116] Finally, an effective landside automation system reduces a terminal's overall emissions. The average truck uses approximately 3.1 L (0.82 gallons) of fuel for every hour it spends idling. The additional 30 seconds of waiting time is inconsequential to individual truck drivers and provides no practical incentive to turn the truck off while waiting. However, at the terminal level, these additional waiting times can add up to 50 hours of truck idling emissions per day. Using the described LSTZ automation system, idling time can be reduced by 38 hours per day, reducing CO2 and NOx emissions by up to 312 kg and 5.47 kg daily, respectively. LSTZ automation is an important step toward improving environmental performance and enabling container terminals to accommodate ever-increasing ship sizes, meet increasing throughput requirements, and remain competitive in the global marketplace.

[0117] The present invention has been described with reference to specific embodiments having various features. In light of the disclosure provided above, it will be apparent to those skilled in the art that various modifications and variations can be made in the practice of the present invention without departing from the scope or spirit of the invention. Those skilled in the art will recognize that the disclosed features may be used alone, in any combination, or omitted based on the requirements and specifications of a given application or design. The disclosed methods can be used with the disclosed systems or other known systems, and the disclosed systems can be used with the disclosed methods or other known methods. When an embodiment refers to "comprising" certain features, it is understood that the embodiment can alternatively "consist of" or "consist essentially of" any one or more features. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention.

[0118] It should be noted in particular that where a range of values ​​is provided herein, each value between the upper and lower limits of that range is also specifically disclosed. The upper and lower limits of these smaller ranges may also be independently included or excluded from the range. The singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. The specification and examples are to be considered exemplary in nature, and variations that do not depart from the essence of the invention are intended to be within the scope of the invention. Furthermore, all references cited herein are individually incorporated by reference in their entirety, and are therefore intended to provide an efficient way to supplement the effective disclosure of the invention and to provide a detailed background for those skilled in the art.

Claims

1. 1. A method for identifying a landing position of an object or container on a target device or chassis according to a position and orientation of said target device or chassis, comprising: Teaching a landing registration system which reference point cloud models stored in said landing registration system match; obtaining a point cloud model of the current target equipment or chassis landing surface aligned to the crane's frame of reference; aligning the point cloud model of the current target equipment or chassis landing surface to the stored reference point cloud, and determining which of the stored reference point cloud models is the matching reference point cloud model that matches the point cloud model of the current target equipment or chassis landing surface; generating a set of coordinates in the reference system as a result of the alignment, the coordinates representing landing positions for placing the object or container on the landing surface of the current target equipment or chassis.

2. The alignment is The method of claim 1 , further comprising positioning the stored reference point cloud model at an expected position and orientation of the frame of reference of the crane.

3. The alignment is determining a first center of the point cloud model of the current target device or chassis landing surface; determining a second center of a landing surface of the stored reference point cloud model; 2. The method of claim 1, comprising: obtaining a course alignment by aligning the first and second centers; and obtaining a fine alignment by iteratively minimizing distances between corresponding points of the point cloud model of the current target equipment or chassis landing surface and points of the stored reference point cloud model to arrive at the fine alignment.

4. The method of claim 3 , wherein iteratively minimizing the distance comprises an iterative nearest neighbor (ICP) algorithm.

5. 1. A method for identifying a type of target equipment or chassis capable of holding an object or container, comprising: Obtaining a point cloud model of a current target equipment or chassis landing surface; comparing the point cloud model of the current target equipment or chassis landing surface with one or more point cloud models of the reference equipment or chassis landing surface by comparing specific characteristics of the current target equipment or chassis landing surface with specific characteristics of one or more reference equipment or chassis landing surfaces to identify which point cloud models of the reference equipment or chassis landing surface match the point cloud model of the current target equipment or chassis landing surface within a selected degree of match, wherein the specific characteristics are selected from one or more of a surface profile or surface characteristics, an obstacle or defect, the presence or absence of girders, cross members, supports, or guides, and their spacing or size; and i) using a neural network and examples of point cloud models of reference equipment or chassis landing surfaces to train the neural network as to what the corresponding point cloud models of said reference equipment or chassis landing surfaces are; or ii) analyzing the point cloud model of the current target equipment or chassis landing surface to identify a type of target equipment or chassis with which the point cloud model of the current target equipment or chassis landing surface is associated.

6. 6. The method of claim 5, wherein the comparison includes comparing metadata obtained from one or more of the point cloud model of the current target equipment or chassis landing surface and the point cloud model of the reference equipment or chassis landing surface.

7. The method of claim 6 , wherein the metadata is obtained from a voxel filter.

Images