System and method for calibrating the attitude of sensors associated with a material handling vehicle

Abstract

A method and system for a material handling vehicle including a processor and a sensor to record warehouse features. The processor is configured to generate and extract features from sensor data, create a factor graph (FG) including a sensor extrinsic node (eO), and generate an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO), the initial sensor feature node including selected extracted features associated with cO and vO in an initial data association. A subsequent vehicle node (v1) is generated based on accumulated mileage, and a subsequent sensor frame node (c1) is generated and associated with one of fO or a subsequent sensor feature node (f1) and eO, v1 in a subsequent data association. The FG is optimized based on the data association to provide a sensor calibration output, and the vehicle is navigated based on the sensor calibration output.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims priority to U.S. Provisional Application No. 62 / 807,536, filed on February 19, 2019, entitled “SYSTEMS AND METHODS FOR CALIBRATION OF A POSE OF A SENSOR RELATIVE TO A MATERIALS HANDLING VEHICLE,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments described herein relate generally to material handling vehicle sensor calibration, and more particularly to systems and methods for calibrating sensors of a material handling vehicle relative to the material handling vehicle in a warehouse environment. Background Art

[0004] Material handling vehicles may utilize one or more sensors to capture features in a warehouse environment to facilitate positioning, attitude determination, and navigation. However, if the sensors are not properly calibrated, operational challenges may arise. Manufacturing or installation tolerances of the roll, pitch, yaw, x, y, and / or z position of the sensors relative to the material handling vehicle may include too much variation to reliably locate the material handling vehicle in a warehouse environment. Thus, in industry, regardless of location, such as but not limited to factories where material handling vehicles are built, there is a need for accurate calibration of sensors on material handling vehicles. Summary of the invention

[0005] According to the subject matter of the present disclosure, in a first aspect, a material handling vehicle includes a sensor, a vehicle position processor, one or more vehicle dead reckoning components, and a drive mechanism configured to move the material handling vehicle along an inventory transit surface. The sensor is configured to record one or more features of a warehouse. The vehicle position processor is configured to generate sensor data from a record of the one or more features recorded by the sensor, extract one or more features from the sensor data, create a factor graph including a sensor extrinsics node (eO), the sensor extrinsics node including an initial seed extrinsics associated with the sensor, and generate an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO) in the factor graph. The initial sensor feature node (fO) includes selected features of the one or more extracted features that are associated with the initial sensor frame node (cO) and the initial vehicle node (vO) in an initial data association. The vehicle position processor is further configured to navigate the material handling vehicle along the inventory transport surface using the drive mechanism, generate a subsequent vehicle node (v1) in the factor graph based on accumulated mileage from one or more vehicle dead reckoning components, and generate a subsequent sensor frame node (c1) in the factor graph that is associated with the initial sensor feature node (f0) or one of the subsequent sensor feature nodes (f1) and the sensor extrinsic node (e0) and the subsequent vehicle node (v1) in a subsequent data association. The vehicle position processor is further configured to optimize the factor graph based on the initial data association and the subsequent data association to provide a calibration output associated with the sensor, and navigate the material handling vehicle along the inventory transport surface based on the calibration output associated with the sensor.

[0006] In a second aspect, including the material handling vehicle of the first aspect, the vehicle position processor is further configured to: when a vehicle seed position and a feature map including one or more mapped features are provided, generate an initial vehicle node (vO) in a factor graph as (i) a map-based initial vehicle node at the vehicle seed position, with (ii) associated vehicle prior factors, and generate (iii) the one or more map features from a site map as corresponding sensor feature nodes, and (iv) corresponding map feature prior factors.

[0007] In a third aspect, including the material handling vehicle of the second aspect, the associated vehicle prior factor of the map-based initial vehicle node at the vehicle seed location comprises an error function of the accuracy of the vehicle seed location relative to the site map for the material handling vehicle.

[0008] In a fourth aspect, including the material handling vehicle of the first aspect, the vehicle position processor is further configured to: when no vehicle seed position and feature map are provided, generate an initial vehicle node (vO) as a map-free initial vehicle node at the origin in the factor graph.

[0009] In a fifth aspect, including a material handling vehicle of any one of the first to fourth aspects, the vehicle position processor is further configured to: iteratively generate one or more subsequent sensor feature nodes and one or more additional subsequent vehicle nodes in a factor graph based on cumulative mileage from one or more vehicle dead reckoning components between a previous vehicle node and a directly subsequent vehicle node, and iteratively generate one or more additional subsequent sensor frame nodes in the factor graph that are associated with the sensor extrinsic reference node (eO), the associated directly subsequent vehicle node, and the associated sensor feature node among the one or more subsequent feature nodes in a subsequent data association.

[0010] In a sixth aspect, including the material handling vehicle of the fifth aspect, the vehicle position processor is further configured to generate an inter-vehicle factor between a directly subsequent vehicle node and a preceding vehicle node in the factor graph based on the accumulated mileage.

[0011] In a seventh aspect, including a material handling vehicle of any one of aspects 1 to 6, the vehicle position processor is further configured to select one of the one or more features extracted, associate the one feature with an existing feature as a matched sensor feature node when the one feature matches an existing feature, and generate a factor in a factor graph linking a generated sensor frame node with a matched feature node.

[0012] In an eighth aspect, including the material handling vehicle of the seventh aspect, the generated sensor frame node comprises one of an initial sensor frame node (c0) or a subsequent sensor frame node (c1).

[0013] A ninth aspect includes the material handling vehicle of the seventh aspect or the eighth aspect, wherein the matched sensor feature node includes one of an initial sensor feature node (f0) or a subsequent sensor feature node (f1).

[0014] In the tenth aspect, comprising a material handling vehicle of any one of the first to ninth aspects, the vehicle position processor is further configured to select one of the one or more extracted features, generate a new sensor feature node associated with the one feature in a factor graph when the one feature does not match an existing feature, and generate a factor in the factor graph that links the generated sensor frame node with the new sensor feature node.

[0015] In an eleventh aspect, including the material handling vehicle of the tenth aspect, the new sensor feature node comprises one of an initial sensor feature node (f0) or a subsequent sensor feature node (f1).

[0016] In a twelfth aspect, including a material handling vehicle of any one of aspects 1 to eleven, the vehicle position processor is further configured to optimize the factor graph when no other features of the one or more extracted features remain associated with an existing feature as a corresponding matching sensor feature node or are added to the factor graph as a corresponding new sensor feature node.

[0017] In a thirteenth aspect, including the material handling vehicle of any one of the first to twelfth aspects, the vehicle position processor is further configured to optimize the factor graph by generating a constrained optimization problem based on one or more variables and one or more factors defined in the factor graph.

[0018] In the fourteenth aspect, including the material handling vehicle of the thirteenth aspect, the one or more variables represent unknown random variables in the constrained optimization problem and include one or more nodes, and the one or more nodes include one or more of a sensor extrinsic parameter node (eO), an initial vehicle node (vO), a subsequent vehicle node (v1), an initial sensor frame node (cO), a subsequent sensor frame node (c1), an initial sensor feature node (fO) and a subsequent sensor feature node (f1).

[0019] In a fifteenth aspect, including the material handling vehicle of the thirteenth aspect or the fourteenth aspect, the one or more factors represent probabilistic information about a selection factor among the one or more factors and include a priori factors, between factors, reference frame factors, projection factors, bearing-range factors or combinations thereof.

[0020] In a sixteenth aspect, the material handling vehicle including any one of aspects thirteen to fifteen uses one or more smoothing and mapping (SAM) libraries and optimizers to construct a constrained optimization problem.

[0021] In a seventeenth aspect, the material handling vehicle comprising any one of aspects thirteen to sixteen, wherein the one or more features include one or more overhead lights of the warehouse, and a constrained optimization problem is constructed using an image recognition algorithm, a data association algorithm, a modeling technique, or a combination thereof.

[0022] In an eighteenth aspect, including the material handling vehicle of any one of the first to seventeenth aspects, the vehicle position processor is further configured to terminate the optimization when a state of the intrinsic factor graph within a period of time is determined to be acceptable based on a predetermined threshold.

[0023] In a nineteenth aspect, comprising a material handling vehicle of any one of aspects one to eighteen, the sensor is a camera, a laser-based sensor, or a combination thereof, the camera is configured to capture one or more features of the warehouse, the laser-based sensor is configured to detect one or more features of the warehouse, and the vehicle position processor is further configured to generate sensor data based on a record of the one or more features captured by the camera, detected by the laser-based sensor, or a combination thereof.

[0024] According to another embodiment of the present disclosure, and in the twentieth aspect, a method of operating a material handling vehicle of any one of the first to nineteenth aspects includes: generating sensor data from a record of one or more features recorded by a sensor via a vehicle position processor; extracting one or more features from the sensor data; creating a factor graph including a sensor extrinsic parameter node (eO), the sensor extrinsic parameter node including an initial seed extrinsic parameter associated with the sensor; and generating an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO) in the factor graph. The method also includes navigating the material handling vehicle along an inventory transport surface using a drive mechanism; generating a subsequent vehicle node (v1) in the factor graph based on accumulated mileage from one or more vehicle dead reckoning components; and generating a subsequent sensor frame node (c1) in the factor graph that is associated with the initial sensor feature node (fO) or one of the subsequent sensor feature nodes (f1) and the sensor extrinsic parameter node (eO), the subsequent vehicle node (v1) in a subsequent data association. The method also includes optimizing the factor graph based on the initial data association and the subsequent data association to provide a calibrated output associated with the sensor, and navigating the material handling vehicle along the inventory transport surface based on the calibrated output associated with the sensor.

[0025] Although the concepts of the present disclosure are primarily described herein with reference to material handling vehicles in a warehouse environment, it is contemplated that these concepts will be applicable to any autonomous, partially autonomous, or manual vehicle in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The embodiments illustrated in the accompanying drawings are illustrative and exemplary in nature and are not intended to limit the present disclosure. The following detailed description of the illustrative embodiments may be understood when read in conjunction with the following drawings, in which like structures are indicated with like reference numerals and in which:

[0027] Figure 1depicts a material handling vehicle in a warehouse environment including an overhead feature according to one or more embodiments shown and described herein;

[0028] Figure 2 Depicted is a computing infrastructure that may be used in a materials handling vehicle according to one or more embodiments shown and described herein;

[0029] Figure 3 Depicted in Figure 1 Illustrative time-lapse diagrams of material handling vehicles at multiple locations during mapping and calibration of a warehouse environment;

[0030] Figure 4 depicts a graphical representation of a calibration path executed by a material handling vehicle within a calibration coordinate frame according to one or more embodiments shown and described herein;

[0031] Figure 5A schematically depicts a factor graph of a mapped sensor calibration for an image-based sensor of a material handling vehicle according to one or more embodiments shown and described herein;

[0032] Figure 5B Schematically depicts a factor graph for map sensor calibration of a laser-based sensor of a material handling vehicle according to one or more embodiments shown and described herein;

[0033] Figure 6 schematically depicts a factor graph for mapless sensor calibration of an image-based sensor of a material handling vehicle in the context of one or more embodiments shown and described herein;

[0034] Figure 7 schematically depicts a factor graph for map-less calibration of a laser-based sensor of a material handling vehicle in the context of one or more embodiments shown and described herein;

[0035] Figure 8 schematically depicts a factor graph for mapless combined sensor calibration including an image-based sensor and a laser-based sensor of a material handling vehicle in an environment according to one or more embodiments shown and described herein;

[0036] Fig. 9 depicts a flow chart illustrating another method of calibrating a sensor of a material handling vehicle according to one or more embodiments shown and described herein;

[0037] Fig.10depicts another flow chart illustrating a method of calibrating a sensor of a material handling vehicle by applying a calibration algorithm process in accordance with one or more embodiments shown and described herein;

[0038] Fig.11A depicts a first subset of flow charts illustrating methods of constructing a factor graph and utilizing the factor graph to calibrate sensors of a material handling vehicle in accordance with one or more embodiments shown and described herein;

[0039] Fig. 11B Depicted Fig.11A A second subset of the flowcharts of ; and

[0040] Fig. 11C Depicted Fig.11A The third subset of flowcharts. DETAILED DESCRIPTION

[0041] Embodiments disclosed herein include systems and methods for calibrating sensors relative to material handling vehicles. Some embodiments relate to the calibration of image-based sensors, such as image capture devices (e.g., cameras) disposed on material handling vehicles. A material handling vehicle can be located within a warehouse environment by tracking multiple static features in the environment, wherein the static features may include static features tracked by using one or more sensors such as cameras that capture image data. Such features may be overhead features and / or other static infrastructure. Overhead features may include, for example, skylights, ceiling lights, reflectors, and / or other suitable overhead components for capture or detection. Static infrastructure may include such features whether or not overhead, and may include, for example, rack legs and / or readable tags. Other position and location sensor systems, such as laser-based sensors (such as LIDAR), and / or other sensor systems (such as RADAR), ultrasonic sensor systems disposed on material handling vehicles, and the like, may also be calibrated using the systems and methods described herein. For purposes of defining and describing the concepts and scope of the present disclosure, it should be noted that "warehouse" includes any indoor or otherwise covered facility where material handling vehicles transport goods, including, but not limited to, warehouses used primarily for the storage of goods, such as those in which single or multi-story warehouse racks or storage units are arranged in aisles or otherwise, and manufacturing facilities in which goods are transported around the facility by material handling vehicles for one or more manufacturing processes.

[0042] More specifically, the embodiments described herein can be configured to reduce manufacturing tolerances of sensors coupled to material handling vehicles such that the calibrated sensors can locate material handling vehicles in an environment with improved accuracy compared to when the sensors are installed but not calibrated relative to the material handling vehicle. By means of systems and methods for calibrating sensors, sensors can be calibrated relative to sensor locations on and relative to a particular material handling vehicle. For example, the roll, pitch, and yaw of the sensor relative to the material handling vehicle can be more accurately determined compared to the initial manufacturing tolerances of the sensor to more accurately determine the vehicle attitude based on the use of the sensor in a warehouse environment. In addition, the systems and methods described herein for calibrating sensors can be easily employed in near real-time processing when the material handling vehicle is delivered and set up in the environment, for maintenance, when performing component replacement, when modifying existing trucks, when placing sensors (e.g., to account for changes in sensor extrinsic parameters over a long period of time), after a period of time (e.g., to adjust for sensor drift), etc. In some embodiments, the deployment and calibration of positioning sensors for material handling vehicles may or may not implement a site map of the environment.

[0043] The term "positioning" is used herein to refer to any of a variety of system configurations that enable location tracking of material handling vehicles in a warehouse environment. The concepts of the present disclosure are not limited to any particular positioning system configuration. A system implementing one or more positioning sensors may be applicable to any of a variety of conventional and yet to be developed positioning systems. Such positioning systems may include those described in U.S. Patent No. 9,349,181, entitled “LOST VEHICLE RECOVERY UTILIZING ASSOCIATED FEATURE PAIRS,” issued on May 24, 2016, U.S. Patent No. 9,984,467, entitled “VEHICLE POSITIONING OR NAVIGATION UTILIZING ASSOCIATED FEATURE PAIRS,” issued on May 29, 2018, and / or U.S. Published Patent Application No. 2012 / 0303255, entitled “METHOD AND APPARATUS FOR PROVIDING ACCURATE LOCALIZATION FOR AN INDUSTRIAL VEHICLE,” published on November 29, 2012. In addition, such positioning systems may utilize sensors such as image capture devices to determine the pose of a material handling vehicle in a warehouse environment by using image capture of overhead features, as described in the positioning system patents referenced above. Such sensors may be calibrated to provide more accurate positioning determinations by the positioning system. An example of such a calibration technique is described in U.S. Patent Publication No. 2016 / 0353099 assigned to Crown Equipment Corporation, which involves comparing image data to a storage site map of a warehouse environment to determine a calibration value for an image capture device. The concepts described herein provide calibration techniques for positioning sensors on material handling vehicles that may be calibrated regardless of whether a storage site map of the warehouse environment has been referenced. Thus, calibration may be simplified and universally implemented in an environment regardless of whether a site map has been created for the warehouse environment.

[0044] Positioning sensors operating with such positioning systems can be used to locate and / or navigate material handling vehicles through warehouse environments, such as warehouses, yards, etc. In some embodiments, image-based sensors (e.g., cameras) and / or laser-based sensor systems can be mounted to material handling vehicles navigating through warehouses (e.g., automated guided vehicles or manually guided vehicles as material handling vehicles) and can assist in vehicle positioning. Laser-based sensor systems can include laser scanners, laser rangefinders, 2D / 3D mapping lasers, LIDAR, or combinations thereof.

[0045] In some embodiments, calibration of an image-based sensor (e.g., a camera) can be used for a material handling vehicle that utilizes overhead feature detection to determine the location of the material handling vehicle and / or plan a route. Camera calibration parameters may include calibration of roll, pitch, yaw, x-position, y-position, z-position, focal length, other camera extrinsic parameters, intrinsic parameters, and the like. Camera calibration may include intrinsic calibration and extrinsic calibration. Intrinsic calibration includes determining the parameters of the camera model itself within a suitable error range. Extrinsic camera calibration may include determining the position of the camera on the material handling vehicle within a suitable error range. The embodiments described herein focus at least on extrinsic calibration of cameras. Systems and methods for vehicle calibration incorporating the same will be described in more detail below.

[0046] Reference now Figure 1 , illustrates a material handling vehicle 100 according to an embodiment described herein, which includes material handling hardware configured to utilize overhead features such as lighting for positioning and navigation services. Overhead features may include lamps, skylights, etc. of various styles, types, shapes, and sizes. As shown, the material handling vehicle 100 may be configured to navigate through a warehouse environment 10, such as a warehouse. The material handling vehicle 100 may be configured as a material handling vehicle for lifting and moving goods. Examples of material handling vehicles include, but are not limited to, forklifts, reach trucks, turret trucks, portable stackers, tractors, pallet trucks, high / low stacker trucks, trailer loaders, side loaders, forklifts, etc. The material handling vehicle 100 may be configured to automatically navigate and / or manually / remotely operate around the floor of the warehouse environment 10 along a desired route. Therefore, the material handling vehicle 100 may be guided forward and backward by the rotation of one or more wheels. In addition, the material handling vehicle 100 may change direction by manipulating one or more wheels. The material handling vehicle 100 may be autonomously controlled and / or include operator controls for controlling functions of the material handling vehicle 100 , such as, but not limited to, the speed of the wheels, the orientation of the wheels, etc.

[0047] The operator controls may include inputs and outputs assigned to functions of the material handling vehicle 100, such as, for example, switches, buttons, joysticks, handles, pedals, calibration indicators, etc. The operator controls may additionally include a position system, a positioning system, an accelerator, a brake, autonomous mode options, and / or other controls, outputs, hardware, and software for operating the material handling vehicle 100 manually, semi-autonomously, and / or fully autonomously. The operator controls may additionally include an odometer for determining the distance traveled by the material handling vehicle 100. The odometer may be configured to determine a determined number of rotations of one or more wheels (e.g., using one or more motion sensors 114 ( Figure 2), such as an encoder coupled to a drive shaft or one or more wheels) and calculates the distance traveled based on a predetermined circumference of the wheel. In some embodiments, the odometer can be determined based on one or more inertial measurement units (IMUs), laser scan matching, and / or visual odometers to calculate the distance traveled.

[0048] The materials handling vehicle 100 may also include a sensor 110 disposed on and relative to the materials handling vehicle 100 , whereby the sensor 110 may be an image-based sensor, such as a camera 110A. Figure 2 Depicted is a computing infrastructure that can be implemented and used with a material handling vehicle 100, configured to calibrate a sensor 110, as described in further detail further below. The sensor 110 may additionally or alternatively be other sensor types configured to assist in vehicle positioning, such as a laser-based sensor 110B as described in more detail below.

[0049] As about Figure 1-4 As a non-limiting example of the description, the sensor 110 may be an image-based sensor, such as a camera. With respect to such image-based sensor embodiments, the sensor 110 may be a digital still camera, a digital video camera, an analog still camera, an analog video camera, a red-green-blue (RGB) camera, an RGB-depth (RBGD) camera, and / or other device for capturing overhead images. The captured image may be formatted as a JPEG, JPEG 2000, Exif, TIFF, raw image format, GIF, BMP, PNG, Netpbm format, WEBP, raster format, vector format, and / or other types of formats. Thus, the sensor 110 may include an image sensor as an image capture device, such as a charge coupled device (CCD), a complementary metal oxide semiconductor sensor, or a functional equivalent thereof. In some embodiments, the material handling vehicle 100 may be located within the warehouse environment 10 and configured to capture an overhead image of the ceiling of the warehouse environment 10. To capture the overhead image, the sensor 110 may be mounted to the material handling vehicle 100 and focused toward the ceiling. The sensor 110 may be positioned within an image area 112 (e.g., such as Figure 3 The sensor data 144 (e.g., image data) of the ceiling and / or warehouse environment 10 is captured within the field of view of the camera (i.e., image area 112A-112L) of the image area 112A-112L, which can be based on the camera specifications and the mounting location on the material handling vehicle 100. For example, the image area 112 is depicted with reference to the sensor 110 focused on the ceiling of the warehouse environment 10, which includes features 150, such as Figure 3 Ceiling lights 150A-150S.

[0050] The ceiling of the warehouse environment 10 may include overhead lights, such as, but not limited to, ceiling lights 150A-S for providing lighting from the ceiling or generally from above a material handling vehicle 100 operating in the warehouse. The ceiling lights 150A-S may include skylights, fluorescent lights, industrial lighting fixtures, and / or other types of lights of various shapes and sizes; and may be mounted or suspended in a ceiling or wall structure to provide lighting. Figure 4 Depicted are embodiments of multiple positions of a material handling vehicle 100 in a coordinate reference system and positions of ceiling lights 150A-S in the coordinate reference system as determined by at least one or more mapless calibration processes described herein and in further detail below.

[0051] The systems and methods described herein may also employ one of a variety of image recognition, data association, and modeling techniques to construct the optimization problem. In some data association embodiments, tracking of static overhead features (such as ceiling lights between frames) may be accomplished using a pixel velocity nearest neighbor image space tracking algorithm or any algorithm capable of tracking features frame by frame without prior knowledge of the location of the features in the camera calibration frame. In addition, image recognition embodiments may include edge detection, feature detection, and / or object recognition routines for tracking static features or portions of static overhead features frame by frame. In addition, modeling techniques may include techniques such as Structure from Motion (SFM), Simultaneous Localization and Mapping (SLAM), Georgia Technology Smoothing and Mapping (GTSAM) libraries, or other smoothing and mapping (SAM) libraries that may be implemented using factor graphs and / or Bayesian networks as underlying paradigms. It should be understood that other modeling techniques are contemplated within the scope of the present disclosure.

[0052] Specifically, you can use libraries to construct optimization problems, such as the C++-like GTSAM library or other Smoothing and Mapping (SAM) libraries. The GTSAM library uses factor graphs (FGs) derived from Bayesian networks as the underlying computational paradigm to implement smoothing and mapping in robotics and vision.

[0053] The SAM library and other similar libraries provide a general framework for constructing optimization problems related to satisfying multiple spatial constraints with various associated uncertainties. It is expected that parts of the library for SLAM can also be utilized. The SLAM part of the SAM library specifically provides the function of optimizing the calibration of sensor extrinsics using reference frame factors.

[0054] Reference now Figure 5A-Figure 8 , depicts illustrative examples of different factor graphs for calibrating various sensors (e.g., cameras and / or laser systems) coupled to a material handling vehicle. Figure 5A A factor graph FG, 500 is involved that is constructed from a mapped configuration using an image-based sensor (such as camera 110A), with reference to a site map 142 of the warehouse environment 10 that includes one or more map features, such as stored overhead feature locations and known initial vehicle seed positions. Figure 5B A factor graph FG, 500' is involved that is constructed from a map configuration using a laser-based sensor 110B (such as a laser rangefinder, LIDAR, etc.), and in which a site map 142 of the warehouse environment 10 is referenced. The site map of the warehouse environment 10 includes one or more map features, such as stored overhead feature locations and a calibrated coordinate system, so that the initial vehicle seed position can be known relative to the site map. Figure 6 A factor graph FG, 600, is involved that is constructed from a mapless configuration using an image-based sensor (such as a camera), where the initial vehicle position is unknown and is set to the origin of a calibrated coordinate system and features are extracted by the sensor, as described in more detail below, and reference is made to the calibrated coordinate system mapping. Figure 7 Involving similar Figure 6 The factor graph FG, 700, is constructed using a map-less configuration of FIG. 1 , but using a laser-based sensor 110B configured to determine a bearing range from a feature that is a laser-reflecting surface disposed within the environment. Figure 8 It is related to the factor graph FG, 800 constructed from a map-free configuration (e.g., if the dashed position prior factor 804 and the dashed prior factors 805, 806, 807 are not included) or a map configuration (e.g., if the dashed position prior factor 804 and the dashed prior factors 805, 806, 807 are included), which is Figure 6-Figure 7 (e.g. no map) or Figure 5A-Figure 5B (e.g., mapping), where both the image-based sensor 110A and the laser-based sensor 110B are used to construct the factor graph. Figures 11A-11C Describes Figure 5A-Figure 8 How each of the factor graphs of is constructed through one or more calibration processes as described herein.

[0055] Although the initial reference Figure 5AReference is made, but the following generally applies to each factor graph depicted and described herein. Factor graph 500 may include nodes 510, 512, 514, 516, 518, 520, 522, 524, 526 (e.g., observations) and factors 502, 504, 506, 508, 519, 521, 523, 525, 527, 529, 531, 533. Factors set between nodes represent formulas indicating uncertainty of a probability function or a transformation or relationship between nodes. An initial factor (e.g., a priori factor) before a node represents an initial uncertainty of the corresponding node. For example, in an embodiment, a priori factors model the initial uncertainty of various states utilized by the optimizer as described herein. Node 510 represents a sensor external parameter. As a non-limiting example, camera extrinsics may include manufacturing tolerances and specifications of the camera model and the pose (e.g., roll, pitch and yaw, x, y and / or z position) of the camera (or sensor in general) measured and mounted with tolerances on the material handling vehicle 100. Another node may be vehicle nodes 512, 514, and 516. The vehicle nodes 512, 514, and 516 may indicate vehicle coordinates at corresponding positions in the calibration coordinate system CCF. These positions may be derived from sensors that determine the odometer of the material handling vehicle 100. Another node may be sensor frame nodes 518, 520, 522 (also referred to as camera frame nodes, CFN), which may be defined by a frame captured by the camera at the vehicle node. In addition, image-based sensor feature nodes 524 and 526 of an image-based sensor such as a camera may be determined and populated within the factor graph 500.

[0056] As used herein, a "frame" may be about Figure 3 112A, which may include at least one static overhead feature captured by an image sensor (e.g., an image-based sensor) or other type of sensor being calibrated (e.g., a laser-based sensor). For example, the frames are certain instances of data generated from a sensor to be calibrated disposed on the material handling vehicle 100 during travel of the material handling vehicle 100. Certain instances may be selected when the material handling vehicle 100 exceeds a linear or angular absolute travel distance threshold while capturing images and / or when a set of observations (e.g., not necessarily from a single sensor) are selected by a set of heuristics indicating that the observations should provide information that will help reduce optimization errors. In embodiments, a combination of such frames may be used to determine the position of one or more material handling vehicles 100 in a calibration coordinate system (CCF) and the pose of one or more sensors capturing images relative to one or more vehicles on which the one or more sensors are positioned.

[0057] In embodiments utilizing image-based sensors (such as cameras) in a map-free configuration (e.g., in Figure 6 ), the frame data may be utilized to determine the location of the material handling vehicle 100 in the CCF as a vehicle node, and a camera frame node (CFN) may be attached to the vehicle node and the camera extrinsic node (CEN) to model the pose of the camera relative to the material handling vehicle 100 in the CCF. Features from the current image may be associated with previous images to be attached to the CFN and construct a factor graph (FG). For example, such association may occur by using a pixel velocity nearest neighbor image space tracking algorithm or any algorithm that tracks features frame by frame without requiring a priori knowledge of the location of the features in the CCF. In some embodiments, features may be grouped over multiple (e.g., consecutive or non-consecutive) frames. The associated features may then be attached to the CFN, which may result in multiple CFNs referencing a single sensor feature node, and this may be accomplished by using projection factors in, for example, a map-free configuration.

[0058] In general, as each frame is selected from the data collected by the sensor 110, a factor graph FG can be populated. The factor graph models observations with uncertainty, where the observations are directly related to each other through factors. In addition, the factor graph does not include time as a variable and does not necessarily rely on forward time factors. A factor graph is a graphical model used to model complex estimation problems, such as simultaneous localization and mapping (SLAM) or structure from motion (SFM). A factor graph behavior is a bipartite graph consisting of factors connected to nodes (also called variables), as described below. Variables represent unknown random variables in the estimation problem, and factors represent probabilistic information about those variables derived from measurements or a priori known information. For example, referring to Figure 5A For reference, the nodes may include a sensor extrinsic node (eO) 510, a first vehicle node (vO) 512, a second vehicle node (v1) 514, a third vehicle node (v2) 516, a first sensor frame node (cO) 518, a second sensor frame node (c1) 520, a third sensor frame node (c2) 522, a first sensor feature node (fO) 524 and a second sensor feature node (f1) 526 and factors 513, 515, 519, 521, 523, 525, 527, 529, 531 and 533 linking the nodes to each other.

[0059] A node (e.g., 510, 512, 514, 516, 518, 520, 522, 524, 526) can be linked to one or more other nodes (e.g., 510, 512, 514, 516, 518, 520, 522, 524, 526) by a factor (e.g., 519, 521, 523, 525, 527, 529, 531, 533). The factor graph FG can be represented and used to construct a constrained optimization problem to be solved. The optimization problem can be constructed using the GTSAM library and optimizer. The optimization problem can then be solved and an error metric can be derived from the factor graph FG. The error metric can then be used to determine when to stop calibration by displaying the calibration factors to the user to select such a minimum error metric, or the algorithm can automatically stop calibration when the error is within or below a predetermined value. The following is about Figure 9-10 A non-limiting example of such a calibration algorithm is described.

[0060] In an embodiment, a toolkit such as the GTSAM library can provide solutions to SLAM and SFM problems, and can also be used to model and solve optimization problems, using a C++ library based on factor graphs. In general, the optimization of factor graphs is intended to reduce the overall error of all factors in the factor graph. Factors are value elements, such as direct or indirect observations or values ​​of optimization problems. Factors and variables / nodes in a factor graph FG can exist in multiple coordinate systems. Factor graphs can include multiple types of nodes and multiple types of factors.

[0061] As non-limiting examples, the types of nodes may include one or more vehicle nodes, one or more sensor frame nodes (image-based and / or laser-based as described herein), sensor extrinsic reference nodes for each corresponding sensor, and one or more sensor feature nodes (image-based and / or laser-based). These nodes may be applied to the map or mapless configurations described herein.

[0062] Each vehicle node may be associated with a two-dimensional (2D) pose, a 3D pose (e.g., x, y, z, roll, pitch, yaw, or any subset thereof), etc. in a global reference frame, including, for example, an x ​​position value, a y position value, and a yaw. Yaw indicates rotation about the z axis, also known as heading. In an embodiment, the x and y position values ​​may be in meters, and the yaw may be in radians.

[0063] Each sensor frame node, whether image-based or laser-based, is associated with a three-dimensional (3D) pose in a global reference frame, including, for example, an x-position value, a y-position value, a z-position value, roll, pitch, and yaw. Roll indicates rotation about the x-axis, and pitch indicates rotation about the y-axis. In an embodiment, the x, y, and z-position values ​​may be in meters, and the roll, pitch, and yaw values ​​may be in radians.

[0064] Each sensor extrinsic node of each corresponding sensor, whether image-based or laser-based, is associated with a three-dimensional position or attitude (e.g., x, y, z, roll, pitch, yaw, or any subset thereof), such as relative to the vehicle in a local reference frame to describe the attitude delta between the kinematic vehicle center and a stored origin of the sensor. The three-dimensional attitude includes, for example, an x ​​position value, a y position value, a z position value, roll, pitch, and yaw.

[0065] Each sensor feature node, whether image-based or laser-based, is associated with a three-dimensional pose of the feature in a global reference frame. In an embodiment, the feature may be an overhead feature, including, but not limited to, a ceiling light, skylight, etc., as described herein for capture by an image-based sensor and / or a laser-based sensor.

[0066] By way of example and not limitation, the types of factors may include a priori factors, between factors (BF), reference frame factors (RFF), projection factors (PF), orientation range factors, etc. The a priori factors provide an initial estimate of the value (node / variable). The a priori factors may be a 2D pose estimate (e.g., including x, y, and yaw) of the initial vehicle position in the map calibration process, which will be related to Figure 5A The location prior factor 504 in FIG. 5 is further described. In an embodiment, the location prior factor 504 may be provided by a user or automatically provided based on the positioning of the vehicle relative to the site map. Another example of a prior factor may be a 3D point estimate (e.g., including x, y, and z) of a map feature in a map calibration process, which will be related to Figure 5A The prior factors 506 and 508 and Figure 5B The prior factors 506' and 508' are further described.

[0067] Additionally, the prior factors in a mapped or mapless configuration may be a 3D pose estimate (e.g., including x, y, z, roll, pitch, and yaw) such as a seed sensor extrinsic value provided based on manufacturing tolerances, which will be relative to Figure 5A A priori factor 502 of the image-based sensor 110A, Figure 5B A priori factor 502' of the laser-based sensor 110B, Figure 6 A priori factor 602 of the image-based sensor 110A, Figure 7 A priori factor 702 of the laser-based sensor 110B, Figure 8 A priori factor 802 of the laser-based sensor 110B, and Figure 8 The prior factors 803 of the image-based sensor 110A are further described.

[0068] A priori factors (e.g. Figure 5AThe factors 502, 504, 506, and 508) include prior and / or initial values ​​that encode known values ​​and / or uncertainties when creating the factor graph, such as the location prior factor 504 that sets the location of the material handling vehicle 100. Figure 5A-Figure 8 The factor graphs shown in and described in detail below each illustrate only various calibrations of various sensors for a site map of an environment in which a material handling vehicle is deployed with and without a corresponding map and no-map configuration. Figure 5A The description of the nodes and / or factors applies to Figure 5B-Figure 8 Nodes and factors.

[0069] The Between factor (BF) describes the relationship between two nodes, such as two vehicle nodes. For example, the Between factor represents the 2D pose delta (e.g., provided by the accumulated odometer) indicating the x, y, and yaw changes between the two vehicle nodes. Figure 5A Factors 513 and 515 in illustrative examples depict and describe an embodiment of such an inter-factor.

[0070] A reference frame factor (RFF) may be used to calculate a relationship between two reference frames, which may be, for example, two different types of nodes. For example, a reference frame factor may indicate a 3D pose describing the relationship between the kinematic center of the vehicle and the reference frame of the sensor (e.g., as an incremental change). In an embodiment, Figure 5A The reference frame factors 519 , 521 , and 523 are used to calculate and / or optimize the sensor extrinsic reference node 510 based on the relationship between the vehicle nodes 512 , 514 , and 516 and the sensor frame nodes 518 , 520 , and 522 .

[0071] A projection factor (PF) can be used to describe the relationship between observed features and the reference frame sensor that observed them. Projection factors can be used for sensors that do not have range information. That is, the projection factor uses a projection model to calculate the observation result error (e.g., using a pinhole camera model). In some embodiments, feature positions are set in local sensor coordinates (e.g., in pixels (u, v) in the uv coordinate space associated with a UY map for 3D modeling involving 2D images to 3D model surfaces) and can be projected as an infinite line into global 3D coordinates (e.g., x, y, and z position values ​​in meters). Reference Figure 5A, projection factors 525, 527, 529, 531, and 533 describe how to observe image-based sensor feature nodes 524 and 526 from sensor frame nodes 518, 520, and 522 (e.g., camera frame node CFN), taking into account the inherent characteristics of image-based sensors. In embodiments, the projection factors may be used in a map image-based calibration process (e.g., Figure 5A ). In addition, the projection factors can be used for map-free image-based calibration processes that do not require initial a priori factor estimates for matching features (e.g., Figure 6 ).

[0072] As another type of factor, the bearing-range factor (BRF) describes the relationship between the observed feature and the reference frame of the sensor that observed it, such as, for example, from a sensor that provides range and bearing information. Figure 7 The azimuth range factors may include factors 725, 727, 729, 731, and 733 and factors associated with the laser-based sensor and features detected by the sensor. Figure 5B and Figure 8 The azimuth range factor may provide a range measurement (e.g., in meters) and an azimuth measurement (e.g., in radians) in local coordinates for each sensor, and the error may be calculated based on a transformation to global 3D space to include x, y, and z positions (e.g., in meters).

[0073] Still reference Figure 5A, a factor graph 500 for map camera calibration of a material handling vehicle is depicted and described in more detail. In the present embodiment, calibration of a sensor as a camera is accomplished by using a site map. A camera extrinsic node 510 includes the roll, pitch, yaw, x, y and / or z position of the camera and the expected variance (i.e., standard deviation) is set with reference to a prior factor 502 of a camera seed position having an extrinsic parameter value with uncertainty. When a site map is used, a material handling vehicle is seeded within the site map by associating the vehicle with a position within the site map, as indicated by a position prior factor 504. Because the material handling vehicle is seeded (i.e., the actual position of the vehicle in the environment is associated with a position in the site map), there is a level of uncertainty modeled by the position prior factor 504. The seed position indicates the origin of the initial vehicle pose (vO). As referred to herein, "origin" refers to the seed position (or initial position vO) of the vehicle in the calibration coordinate system. The origin vO may be set at the origin O of the calibration coordinate system CCF, or may be an arbitrary origin vO that is not the origin O of the calibration coordinate system CCF. The seed extrinsic values ​​(e.g., sensor extrinsic node eO) are based on the manufacturing tolerances of the sensors mounted on the material handling vehicle. The initial sensor frame (e.g., sensor frame node, cO) may be calculated by combining the sensor extrinsic node eO on the initial vehicle pose vO. This becomes the origin vO of the calibration coordinate system CCF (at the origin O or an arbitrary starting position set in some other way), and the initial vehicle pose vO models the position of the vehicle at the seed position. The observed V x (e.g., v1 514) and V x-1 (vO 512) to calculate the new V x The sensor frame nodes 518, 520, and 522 representing the camera frame nodes are each V x The factors model the pose of the camera in the same coordinate system, and the sensor frame nodes 518, 520, and 522 are linked to the reference frame factors 519, 521, and 523 for each camera frame at the corresponding vehicle nodes 512, 514, and 516. Observations, such as overhead features observed by the camera, are used as common projection factors for multiple observations and are associated between frames. In an embodiment, logic is provided to derive the location of the observed features from multiple observations. Projection factors 525, 527, 529, 531, and 533 are created to link to each sensor feature node 524 and 526 observed in the current frame. There can be a many-to-many, one-to-one, one-to-many, or many-to-one relationship between the sensor frame nodes 518, 520, and 522 and the sensor feature nodes 524 and 526.

[0074] In some embodiments, once there is a fully formed factor graph, the optimizer is run to minimize the error in all factors by moving all nodes. However, in some embodiments, the optimizer can be implemented at any time when the factor graph is constructed. Thus, when enough information is built into the factor graph, the construction of the factor graph can stop so that the optimizer can formulate an optimization problem for determining the calibration factors. In addition, reading eO with the error minimized can determine the optimized camera extrinsics for camera calibration, as described in more detail below.

[0075] Reference now Figure 5B , depicting and describing in more detail a factor graph 500' for mapping laser calibration of a material handling vehicle. In some embodiments, Figure 5A The depicted image-based sensor / camera 110A may alternatively be a laser-based sensor 110B calibrated using a site map. For example, the sensor extrinsic node 510' may include the roll, pitch, yaw, x, y and / or z position of the laser and the expected variance (i.e., standard deviation) which are prior factors 502' of the laser seed position referenced to the extrinsic parameter values ​​with uncertainty. Additionally, when a site map is used, material handling vehicles are seeded within the site map by associating the vehicle with a location within the site map, as indicated by the location prior factors 504'.

[0076] The laser records the sensor nodes (e.g., 518', 520', 522') as laser nodes associated with vehicle nodes 512', 514', and 516', such that laser node 518' is associated with vehicle node v0, 512', laser node 518' is associated with vehicle node v1, 514', and laser node 518' is associated with vehicle node v2, 516'. Laser-based sensor feature nodes 524' and 526' are determined by the laser, plotted relative to each laser node 518', 520', and 522', and can be associated by azimuth range factors 525', 527', 529', 531', and 533', and 533', defining the relationship between laser nodes 518', 520', and 522' and laser-based sensor feature nodes 524' and 526'.

[0077] Reference now Figure 6 , depicts a factor graph 600 for mapless camera calibration of a material handling vehicle in an environment. The calibration depicted in factor graph FG 600 does not include the use of a site map (e.g., is mapless), and therefore, the initial position information of the material handling vehicle 100 is represented as vehicle node v0, 612. For example, the initial position v0 is at Figure 4142, which in this case is not the origin O. As a non-limiting example, the initial position information can be any location within the environment and need not be positioned relative to the site map 142 or any other reference location. As the material handling vehicle 100 travels, the estimated location of the material handling vehicle 100 as vehicle positions 0, 1, 2, etc. (i.e., v0, v1, v2, etc.) is recorded as, for example, odometer-based vehicle nodes 612, 614, 616. Interval factors 613, 615 are set between such vehicle nodes 612, 614, 616. In addition, at the location of the calibration coordinate system CCF of the vehicle node, the sensor as the camera records the sensor frame nodes 618, 620, 622 associated with each vehicle node 612, 614, 616, such as the sensor frame node c0, 618 associated with the vehicle node v0, 612; the sensor frame node c1, 614 associated with the vehicle node v1, 620; and the sensor frame node c2, 616 associated with the vehicle node v2, 622. The image-based sensor feature nodes 624, 626 depicted relative to each sensor frame node 618, 620, 622 can be associated by defining the projection factors 625, 627, 629, 631, 633 between the adjacent sensor frame nodes 618, 620, 622 and the image-based sensor feature nodes 624, 626.

[0078] Reference now Figure 7 , depicts a factor graph 700 for mapless laser calibration of a material handling vehicle in an environment. Factor graph 700 is similar to factor graph 600 ( Figure 6 ), except that the sensors being calibrated here are part of a laser-based system modeled in the factor graph. The sensor extrinsic node 710, directed to the laser extrinsic node (LEN), may include the roll, pitch, and yaw, x, y, and / or z positions and expected variances (i.e., standard deviations) of the laser, which are set with reference to a priori factors 702 of the laser seed position having uncertain extrinsic values. The calibration depicted in the factor graph 700, FG does not include the use of a site map (e.g., is mapless). Therefore, the initial position information as the vehicle node 712 is set to the origin of the material handling vehicle 100 and represented as the vehicle node vO (e.g., having coordinate position vO, Figure 4). As a non-limiting example, the initial position information can be an arbitrary location within the environment and does not need to be positioned relative to the site map 142 or any other reference location. As the material handling vehicle 100 travels, the estimated position of the material handling vehicle 100 as vehicle positions 0, 1, 2, etc. (i.e., vehicle nodes v0, v1, v2, etc.) is recorded as, for example, odometer-based vehicle nodes 712, 714, 716. Interval factors 713 and 715 are set between such vehicle nodes 712, 714, 716. In addition, relative to the position of the vehicle node within the calibration coordinate system CCF, the laser records the corresponding laser nodes 718, 720, 722 associated with each vehicle node 712, 714, 716, such as laser node L0, 718 associated with vehicle node v0, 712; laser node L1, 720 associated with vehicle node v1, 714; and laser node L2, 722 associated with vehicle node v2, 716. Laser-based sensor feature nodes 724, 726 representing feature information determined by the laser are depicted relative to each sensor frame node 718, 720, 722 and can be associated by azimuth range factors 725, 727, 729, 731, 733 that define (one or more) relationships between the sensor frame nodes 718, 720, 722 and the laser-based sensor feature nodes 724, 726.

[0079] The laser positioning system can identify features similar to those identified by the camera's image data, such as the edges of an object or features within the environment, by employing a laser system such as a LIDAR system. In some embodiments, the laser-based positioning system can identify a range of azimuths of reflector objects positioned throughout the environment. The reflector objects can be similar to the camera's light in that the reflector objects are features that the laser system uses to map the environment and calibrate the laser-based sensor 110B, just as the camera 110A uses light.

[0080] Reference now Figure 8 , depicts a factor graph 800 for calibrating a mapless localization system that implements a combination camera and laser for a material handling vehicle within an environment. In an embodiment, including a dashed position prior factor 804 (indicated by a dashed line and as described herein with respect to using a position prior factor associated with an initial vehicle position and using a site map) will allow calibration of a mapped localization system that implements a combination camera and laser for a material handling vehicle within an environment.

[0081] Sensor extrinsic node 808 may include the roll, pitch, yaw, x, y and / or z position of the laser and the expected variance (i.e., standard deviation) set with reference to the prior factor 802 of the laser seed position with uncertain extrinsic values. Sensor extrinsic node 810 may include the roll, pitch, yaw, x, y and / or z position of the camera and the expected variance (i.e., standard deviation) set with reference to the prior factor 803 of the camera seed position with uncertain extrinsic values. Similar to Figure 6 and 7 The factor graph in , refer to Figure 8 The calibration described and modeled in the factor graph 800, FG does not include the use of a site map (e.g., is mapless). Therefore, the initial position information of the vehicle node 812 is set to the origin of the material handling vehicle 100 and is represented as the vehicle node v0 (e.g., at Figure 4 As the material handling vehicle 100 travels, the estimated position of the material handling vehicle 100 as vehicle positions 0, 1, 2, 3, etc. (i.e., vehicle nodes v0, v1, v2, v3, etc.) is recorded as, for example, odometer-based vehicle nodes 812, 814, 816, 818. Interval factors 813, 815, 817 are provided between such vehicle nodes 812, 814, 816, and 818.

[0082] In some positions of the vehicle nodes 812, 814, 816, 818 in the calibration coordinate system CCF, the camera records camera frame nodes 820, 824 associated with the vehicle nodes 812, 816, such as camera frame node c0, 820 associated with the vehicle node v0, 812, and camera frame node c1, 824 associated with the vehicle node v2, 816. Feature nodes 828 depicted with respect to the camera frame nodes 820, 824 can be associated by projection factors 827, 829 that define relationships between adjacent camera frame nodes 820, 824 and feature nodes 828.

[0083] Additionally, the laser records laser nodes 822, 826 associated with vehicle nodes 814, 818 relative to vehicle nodes 812, 814, 816, 818 such that laser node L0, 822 is associated with vehicle node v0, 814 and laser node L1, 826 is associated with vehicle node v3, 818. Laser-based sensor feature nodes 830, 832 determined by the laser are depicted relative to each laser node 822, 826 and may be associated by azimuth range factors 831, 833, 835 defining the relationship between laser nodes 822, 826 and laser-based sensor feature nodes 830, 832.

[0084] It should be understood that factor graphs provide a method of associating observed information collected by one or more sensors being calibrated with vehicle positions and features within the environment. Lasers and cameras can be calibrated through the relationship between vehicle positions (i.e., vehicle nodes) and observed features (e.g., as laser-based sensor feature nodes and / or image-based sensor feature nodes). As described above, the relationship between variables / nodes is defined by factors. These factors can represent probabilistic information about these variables. In some embodiments, factors can define factor functions associated with transformations between nodes. During optimization, the error of each variable can be minimized so that the probabilistic information associated with these variables is maximized.

[0085] Now turn to Figure 9-11C , depicts several flow charts illustrating methods of calibrating sensors of a material handling vehicle. Figure 9-10 Depicted is the process of applying a calibration algorithm by using an optimizer in either a map or map-free configuration. Figures 11A-11C A calibration method is described that constructs a factor graph such as the one above Figure 5A-Figure 8 A factor map is provided for calibrating the sensor relative to the position of the sensor on the material handling vehicle, as described in this article.

[0086] refer to Fig. 9 , depicts a flow chart 900 of a sensor calibration method that can be applied to a map or mapless calibration system. Sensor calibration begins at box 902. Calibration of the sensor can be initiated automatically by the system, or can be initiated manually through a GUI interface or command prompt interface of a computing device. In some embodiments, calibration can be automatically initiated when a material handling vehicle is first activated. In some embodiments, as shown in box 910, the initial step of calibration includes determining and / or confirming the vehicle model of the material handling vehicle. The vehicle model can be manually determined by entering the vehicle model or selecting the vehicle model through a computing device. Alternatively, the vehicle model can be automatically determined by a pre-programmed identifier or by a scanning and registration of a component connected thereto by a computing device. That is, the computing device may be able to automatically determine the vehicle model of the material handling vehicle by scanning the installed hardware. For example, a model matching method can be implemented to determine the model of the material handling vehicle. That is, when sensor data, such as from an encoder, an IMU, an imaging device, a laser sensor, etc., is collected, the data and the source can be compared with a set of known mathematical models to select the type of material handling vehicle.

[0087] Once the vehicle model is determined, two tasks may be initiated. One task may include initiating a kinematic model so that the sensors and systems used to determine the odometer may be calibrated, as shown at block 920. For example, the odometer may be calculated by moving the material handling vehicle back and forth between two known points several times and comparing the difference between the wheel encoders and the actual distance. Such an odometer calibration may be used as described in U.S. Patent No. 9,921,067, issued on March 20, 2018, and assigned to Crown Equipment Corporation, entitled “SYSTEMS AND METHODS FOR MATERIALS HANDLING VEHICLE ODOMETRY CALIBRATION.”

[0088] At box 930, the odometer calibration and sensor seed extrinsics determined at box 920 may be received as inputs to a calibration algorithm. In a map embodiment, the calibration algorithm at box 930 may also receive a site map and a truck seed location as inputs from box 940. However, as described above, calibration of sensors used for positioning material handling vehicles may not include a site map (e.g., may be mapless). In this case, box 940 may only provide input of the truck seed location as having a coordinate reference on a calibration coordinate system as described herein (which may be an arbitrary coordinate reference or set to (0,0,0) relative to x, y, and yaw). In some embodiments, the seed location may be any arbitrary coordinate within the calibration coordinate system. At box 930, the calibration algorithm is executed and the calibration extrinsics of the sensor are determined. The example calibration method described herein may be used for the calibration algorithm.

[0089] If it is determined at box 950 that the calibration extrinsics are acceptable, then the calibration extrinsics are saved as calibrated extrinsics at box 970. However, if it is determined at box 950 that the calibration extrinsics determined at box 930 are unacceptable, then calibration can be restarted at box 960, returning to box 910 or box 930. Before restarting calibration at box 960, analysis of the calibration extrinsics determined at box 930 and / or analysis of previously completed calibration processes can be completed at box 960. The analysis at box 960 can be configured to determine whether a specific problem with the calibration process that resulted in unacceptable calibration extrinsics can be identified. For example, the analysis can determine that an incorrect truck vehicle model was determined at box 910 or that insufficient calibration paths were performed when executing the calibration algorithm (e.g., motion in a particular direction was not included, i.e., the vehicle was only traveling in a straight line and no turns were performed during calibration). Therefore, box 960 can determine that calibration can be restarted at the intermediate step at box 930 instead of from the beginning. For example, if the vehicle has not completed a sufficiently diverse set of paths to generate acceptable calibration extrinsics, calibration may be restarted by continuing the calibration algorithm at block 930. However, if the vehicle model was incorrectly determined at block 910, calibration may be restarted at block 930 to determine the correct vehicle model.

[0090] Reference now Fig.10 , another example flow chart 1000 illustrating a method of calibrating a sensor of a material handling vehicle is depicted as a non-limiting example of a calibration algorithm. For example, Fig.10 The flowchart depicted in Fig. 9 Implementation of the calibration algorithm implemented at block 930 in FIG. Calibration is initiated at block 1002. As shown in FIG. Fig. 9As discussed, calibration can be initiated automatically or manually. Once initiated, sensor data is captured by (one or more) sensors of the material handling vehicle at box 1010. In some embodiments, sensor data can be captured by the sensor being calibrated as well as other sensors. For example, sensor data from sensors used to determine odometers (e.g., as determined at box 1070) and sensor data from cameras can be collected at box 1010. Features are then extracted from the raw sensor data collected at box 1010. For example, in box 1020, features are extracted by one or more algorithms that are configured to identify, isolate and / or extract features from sensor data. Feature extraction can employ algorithms, such as utilizing object and image detection and recognition techniques, such as scale-invariant feature transforms ("SIFT"), accelerated robust features ("SURF"), blob detection methods, and edge-detection algorithms. Such algorithms can identify features and / or detect objects within sensor data, which can be used as features that the calibration method can associate with future or previously observed features from future or previously collected sensor data. Examples of such feature extraction techniques that may be used herein are described in US Pat. Nos. 9,170,581 and 9,606,540, assigned to Crown Equipment Corporation.

[0091] At block 1030, the calibration method associates the extracted sensor data with previously observed features. Some data association methods that may be implemented include, but are not limited to, image to image, pixel velocity tracking nearest neighbor, joint compatible branch and boundary "JCBB", etc. For example, but not limited to, the extracted sensor data may include features defining a first luminaire and the previously observed features may also include the first luminaire observed from a previous iteration of sensor capture and feature extraction. Features from the current image may be associated with the previous image to be attached to the camera frame node and a factor graph FG may be constructed. At block 1030, the extracted features and previously observed features may be matched. For example, such association may occur by using a pixel velocity nearest neighbor image space tracking algorithm or any algorithm that tracks features frame by frame without prior knowledge of the location of the features in the calibration coordinate system (CCF). The associated features may then be attached to the camera frame node (CFN), which may result in multiple camera frame nodes referencing a single sensor feature node, and this may be achieved by using a projection factor. In some embodiments, such as those utilizing maps or mapless systems, data association techniques may match observations from the current frame to features already present in the model (e.g., utilizing algorithms such as nearest neighbor or JCBB). That is, data association of sensor feature nodes may be accomplished through image-to-image mapping or image-to-model mapping.

[0092] That is, at block 1040, with reference to the factor graphs discussed herein, the projection factor and / or the orientation range factor may associate the sensor data (e.g., the camera extrinsics at the current location of the material handling vehicle) with the first light fixture previously observed from a different location and at the current location of the material handling vehicle. If no previous observations were made, the extracted sensor data may be added as a new node to the factor graph in the mapless calibration. One or more frames may be created accordingly and added to the factor graph FG to create an optimization problem.

[0093] The location of the material handling vehicle can be provided to the model based on the odometry information generated by the movement of the material handling vehicle at box 1070 and the raw sensor data captured at box 1010 in the initial iteration of the method. The data association at box 1030 can be repeated for one or more extracted features and previously observed features, where each matching feature is added to the factor graph or calibration model and / or each unmatched feature is added as a new feature to the factor graph or calibration model. In addition, at box 1040, a model error value can be determined and an optimization problem can be defined based on the model, such as a factor graph.

[0094] The optimization problem is solved at block 1050 to produce a calibration output. Sensor extrinsics and / or confidence values ​​are determined based on the calibration output determined by solving the optimization problem. If the sensor extrinsics and / or confidence values ​​indicate a local minimum at block 1060 such that adding additional information to the factor graph does not result in a significant change in the overall graph error, then the calibration is complete and the sensor extrinsics based on the calibration output are stored as calibration values ​​at block 1080 and the calibration ends. The calibration results may be passed on to Fig. 9 950. However, in some embodiments, if adding additional information to the factor graph at block 1060 does result in a significant change in the overall graph error, then the material handling vehicle continues to move in the environment at block 1070, thereby generating further odometry and raw sensor data from at least the sensor being calibrated and returning to block 1010. This can continue until adding additional information to the factor graph does not result in a significant change in the overall error of the graph. In this case, calibration can be terminated. It should be understood that the foregoing method is only a non-limiting embodiment of a method of calibrating a sensor of a material handling vehicle.

[0095] Reference now Figures 11A-11C, depicts a flow chart of a subset of an example method for combining constructing a factor graph FG and calibrating a sensor 110 of a material handling vehicle 100 using the factor graph. The material handling vehicle 100 includes a sensor 110, a vehicle position processor 130, one or more vehicle dead reckoning components, and a drive mechanism configured to move the material handling vehicle 100 along an inventory transport surface. The one or more dead reckoning components may include an odometer that includes odometer logic 126. The sensor 110 is configured to record one or more features of a warehouse (e.g., warehouse environment 10), and the vehicle position processor 130 is configured to generate sensor data 144 from the record of the one or more features 150 recorded by the sensor 110. In an embodiment, the sensor 110 is a camera configured to capture the one or more features 150 of the warehouse, and the vehicle position processor is further configured to generate sensor data 144 from the record of the one or more features captured by the camera. The corresponding image-based factor graph FG generated in the embodiments described herein and described in more detail below is Figure 5A and Figure 6 500, 600. Additionally or alternatively, the sensor 110 is a laser configured to detect one or more features of the warehouse, and the vehicle position processor is further configured to generate sensor data 144 from a record of the one or more features detected by the laser. The corresponding laser-based factor graph FG generated in the embodiments described herein and described in more detail below is Figure 5B and Figure 7 The corresponding image-based and laser-based factor graphs FG generated in the embodiments described herein and described in more detail below are depicted as corresponding factor graphs 500', 700. Figure 8 800 is depicted in FIG.

[0096] about Fig.11A , calibration is initiated at block 1102. As shown in FIG. Fig. 9 As discussed, calibration can be initiated automatically or manually. Once initiated, sensor data 144 from the sensor 110 being calibrated is collected and received at box 1104. The sensor data may include image data collected by a camera and / or range distance data collected by a laser sensor, such as by laser scanning, or a combination of both. The sensor data is processed and features are extracted at box 1106. For example, the vehicle position processor 130 is configured to extract one or more features 150 from the sensor data 144.

[0097] Features can be extracted in several ways. For example, as described above, feature extraction can employ algorithms, such as utilizing object and image detection and recognition techniques, such as Scale Invariant Feature Transform ("SIFT"), Speeded Up Robust Features ("SURF"), Blob detection methods, and edge detection algorithms. Such algorithms can identify features and / or detect objects within sensor data, which can be used as features that a calibration method can correlate with future or previously observed features from future or previously collected sensor data.

[0098] At block 1108, the system determines whether one or more features have been extracted from the sensor data. If no features have been extracted, or at least successfully extracted, the system collects additional sensor data by returning processing to block 1104. If no additional features have been extracted from the sensor data, as determined at block 1108, the system proceeds to determine whether the factor graph has been initialized at block 1110. If the factor graph has not been initialized, the system proceeds to the step of initializing the factor graph at block 1112. At block 1112, the system creates a sensor extrinsic node with a prior factor containing a known seed extrinsic parameter (e.g., Figure 5A For example, the vehicle position processor 130 is configured to create a factor graph FG including a sensor extrinsic parameter node (eO) including an initial seed extrinsic parameter associated with the sensor 110.

[0099] Next, at box 1114, the system determines whether a site map 142 of the warehouse environment 10 (which may be combined with a feature map of one or more features within the warehouse environment) and a material handling vehicle seed location are provided. In other words, this step determines whether the calibration method is a map-free calibration or a map-based calibration. The vehicle position processor 130 can be configured to generate an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO) in a factor graph FG of a map or map-free configuration as described below, the initial sensor feature node (fO) including selected features of the one or more extracted features that are associated with the initial sensor frame node (cO) and the initial vehicle node (vO) in the initial data association.

[0100] The vehicle position processor 130 is also configured to generate an initial vehicle node (vO) as a map-free initial vehicle node at the origin in the factor graph FG when the vehicle seed position and feature map are not provided. Thus, if the site map 142 and material handling vehicle seed positions are not provided, the system operates in a map-free configuration and creates a first vehicle node (e.g., Figure 6In this case, at block 1116, the first vehicle node may be configured at an arbitrary origin defined by an origin coordinate value (0, 0, 0), such that the origin is defined by an origin coordinate value (0, 0, 0) in a sensor calibration frame associated with the sensor 110.

[0101] The vehicle position processor 130 is also configured to, when a vehicle seed position and a feature map including one or more map features are provided, generate an initial vehicle node (vO) in the factor graph FG as (i) a map-based initial vehicle node at the vehicle seed position, having (ii) associated vehicle prior factors, and generating (iii) one or more map features from the site map 142 as corresponding sensor feature nodes, and (iv) corresponding map feature prior factors. The associated vehicle prior factors of the map-based initial vehicle node at the vehicle seed position may include an error function regarding the accuracy of the vehicle seed position of the material handling vehicle relative to the site map 142. Therefore, if a site map 142 and a material handling vehicle seed position are provided instead, then the system operates under the map configuration at box 1118 and creates a first vehicle node (e.g., Figure 5A The vehicle node will also include a priori position factors in the map configuration (e.g., Figure 5A 142). The a priori position factors may define a probability or error function regarding the accuracy of the seeding location of the material handling vehicle with reference to the site map 142. At block 1120, each map feature based on the site map 142 is added to the factor graph as a node along with the a priori factors for each map feature. In embodiments that include a site map 142 for calibrating sensors, the map features defined in the site map 142 are the features to which the system will associate sensor data from the sensors during calibration.

[0102] The vehicle position processor 130 is also configured to navigate the material handling vehicle 100 along the inventory transport surface using the drive mechanism, and to generate a subsequent vehicle node (v1) in the factor graph FG based on the accumulated odometer from one or more vehicle dead reckoning components. Referring back to block 1110, if the factor graph is initialized at item B, then at block 1122 a second vehicle node (e.g., at each of the first and second vehicle nodes) is transformed relative to the first vehicle node based on the accumulated odometer from vehicle movement (i.e., described in more detail with respect to blocks 1146, 1148, and 1150 between items C and D). Figure 5A and Figure 6Node v1, 514 or 614 shown in FIG. 1 is created and added to the factor graph. In addition, an inter-factor that defines a probability relationship or an error function of the vehicle position transformation, linking the first vehicle node to the second vehicle node, is generated at box 1124. The vehicle position processor 130 can be configured to generate a vehicle inter-factor between a subsequent vehicle node (v1) and an initial vehicle node (vO) in the factor graph FG based on the accumulated mileage. The inter-factor can define the error in determining the second vehicle node from the first vehicle node. In other words, the inter-factor describes the transformation between the two vehicle postures and the associated uncertainty of the transformation.

[0103] The vehicle position processor 130 is also configured to generate in the factor graph FG a subsequent sensor frame node (c1) associated with one of the initial sensor feature node (f0) or the subsequent sensor feature node (f1) and the sensor extrinsic reference node (e0) and the subsequent vehicle node (v1) in subsequent data association. Once the factor graph is initialized in block 1110, such as during the first iteration of the calibration process or during the determination and creation of the second vehicle node position within the factor graph through blocks 1112-1124, the factor corresponding to item A is added to the factor graph at block 1126. By way of example and not limitation, Figure 6 618, or Figure 7 7 shows the addition of a reference frame factor 719 linking the first vehicle node v0, 712 to the sensor node L0, 718. The link reference frame factor defines the relationship between the location of the material handling vehicle 100 and the sensor node.

[0104] At block 1128, one of the extracted features determined at block 1106 is selected. The selected extracted feature is attempted to be associated with an existing feature from, for example, one or more previously extracted features and / or features already defined from the site map 142. If the selected extracted feature matches an existing feature, then at block 1132, a factor (e.g., Figure 5A) and added to the factor graph at box 1136, linking the sensor node to the matching feature defined in the factor graph. As a non-limiting example, the added factor can be a projection factor for a camera feature or an azimuth range factor for a laser feature. Therefore, the vehicle position processor 130 can be configured to select one of the one or more extracted features, associate the one feature with the existing feature as a matched sensor feature node when the one feature matches the existing feature, and generate a projection factor linking the generated sensor frame node with the matched feature node in the factor graph FG. The generated sensor frame node can include one of the initial sensor frame node (cO) or the subsequent sensor frame node (c1), the matched sensor feature node can include the initial sensor feature node (fO) or one of the subsequent sensor feature nodes (f1), and the existing sensor feature can be defined in the site map 142.

[0105] Alternatively, in an embodiment, if the selected features in the extracted features do not match the existing features, then a new feature node can be created and added to the factor graph at block 1134. However, in other embodiments, it is not necessary to add the extracted features that do not match the existing features. Subsequently, at block 1136, the factors (e.g., Figure 6 The projection factor 625 shown in FIG. 6 is added to the factor graph and the sensor node is linked to the new feature node defined in the factor graph. Therefore, the vehicle position processor 130 can be configured to select one of the one or more extracted features, generate a new sensor feature node associated with the one feature in the factor graph when the one feature does not match the existing feature, and generate a projection factor in the factor graph FG to link the generated sensor frame node to the new sensor feature node. The new sensor feature node may include one of the initial sensor feature node (fO) or the subsequent sensor feature node (f1). Although the above steps and processes are described with respect to the map-free calibration method, it should be understood that similar steps and processing flows may be used in the process of implementing the map calibration system and method.

[0106] The system then determines at block 1138 whether there are more extracted features. For example, the system determines whether there are any additional extracted features identified during the current iteration of the calibration method that are associated with the existing features or added to the factor graph. If there are additional extracted features associated with the existing features or added to the factor graph, then block 1138 causes the system to return to block 1128 to further process the next extracted feature. However, if there are no additional extracted features associated with the existing features or added to the factor graph, then block 1138 causes the system to proceed to block 1140, where the vehicle position processor 130 is configured to optimize the factor graph FG to provide a calibration output associated with the sensor 110 based on the initial data association and the subsequent data association. The vehicle position processor 130 may also be configured to navigate the material handling vehicle 100 along the inventory transport surface based on the calibration output associated with the sensor 110.

[0107] Referring to box 1140, the system optimizes the factor graph to provide a solution as a calibration output. In addition, the vehicle position processor 130 can also be configured to optimize the factor graph when no other features of the extracted one or more features remain associated with existing features as corresponding matching sensor feature nodes or are added to the factor graph as corresponding new sensor feature nodes. In some embodiments, the system can be optimized at any time rather than at each step, such as the optimization can be triggered at a specific time. For example, the system can optimize the factor graph by generating a constrained optimization problem based on the factors and variables defined in the factor graph. At box 1142, the system determines whether a local minimum is reached during the optimization of the factor graph. If a local minimum is detected and, for example, a threshold minimum is reached, an acceptable calibration error is determined, etc., then at box 1144 the optimized external parameter values ​​in the factor graph are returned and can then be used in box 950 ( Fig. 9) is passed to a second filter to determine acceptability. In some local minimum embodiments, determining at block 1142 that the optimizer has reached a local minimum may include identifying a situation where providing any more data to the optimizer will not result in a significant change in the output. In some embodiments, block 1142 determines that a local minimum has been reached when a threshold is reached and maintained for a predefined time period. In an acceptability embodiment, block 950 may determine that a calibration value determined from the optimization of the factor graph is acceptable based on a rate of change of error being below a predetermined value. In some acceptability embodiments, tolerances matched to expected calibration values, analysis of optimized path characteristics (total travel distance, total angular travel, time taken, speed, etc.), sensor convergence, feature diversity, characteristics of the factor graph (distribution weights, structure, discontinuities), spot checks using existing positioning algorithms, or other mechanisms may be implemented to determine when a calibration value is an acceptable calibration value. In an embodiment, the acceptability determination determines that the calibration value is acceptable based on the provided context through one or more heuristic checks, including but not limited to path characteristics, sensor coverage, geometric constraint checks, comparisons to predetermined acceptable calibrations, comparisons to calibration averages for a fleet population, and / or spot checks using other positioning systems. Such acceptability determinations can utilize domain-specific knowledge to detect one or more failures of one or more optimizers, taking into account the broader context of a particular truck or sensor.

[0108] The vehicle position processor 130 may also be configured to optimize the factor graph by generating a constrained optimization problem based on one or more factors and one or more variables defined in the factor graph. The one or more variables represent unknown random variables in the constrained optimization problem and include one or more nodes as described herein, such as one or more of a sensor extrinsic node (eO), an initial vehicle node (vO), a subsequent vehicle node (v1), an initial sensor frame node (cO), a subsequent sensor frame node (c1), an initial sensor feature node (fO), and a subsequent sensor feature node (f1). The one or more factors as described herein may represent probabilistic information about a selected factor in the one or more factors and may include a priori factors, inter factors, reference frame factors, projection factors, azimuth range factors, or a combination thereof.

[0109] As further described herein, the constrained optimization problem can be constructed using one or more smoothing and mapping (SAM) libraries and optimizers. In an embodiment, the one or more features 150 include one or more overhead lights of a warehouse, and the constrained optimization problem is constructed using an image recognition algorithm, a data association algorithm, a modeling technique, or a combination thereof. As described herein, the image recognition algorithm is configured to track at least a portion of a static overhead light that is one of the one or more features frame by frame and includes an edge detection routine, a feature detection routine, an object recognition routine, or a combination thereof. In addition, the data association algorithm includes using a pixel velocity nearest neighbor image space tracking algorithm to track the feature frame by frame without requiring a priori knowledge of the location of the feature in a sensor calibration frame associated with the sensor. In addition, the modeling technique includes a structure from motion (SFM) technique, a simultaneous localization and mapping (SLAM) technique, a smoothing and mapping (SAM) library technique, or a combination thereof.

[0110] Return to block 1142 ( Fig. 11B ), if no local minimum is detected at box 1142, then the material handling vehicle moves to a new location, for example, by executing a command associated with item C at box 1146. The odometer is accumulated from the last vehicle location to the new vehicle location at box 1148. In some embodiments, the system determines at box 1150 whether the material handling vehicle has moved a predetermined distance from the last location. If the material handling vehicle has not moved at least a predetermined threshold distance, then in response to box 1146, the material handling vehicle continues to move. When the vehicle moves at least the predetermined threshold distance, as determined at box 1150, then the calibration method returns to box 1104 associated with item D, and a new sensor data set is collected by the sensor being calibrated. It should also be understood and within the scope of the present disclosure that in embodiments, observations from sensors as described herein rather than odometer sensors may be utilized to provide data to the optimizer.

[0111] It should be understood that Figures 11A-11C The calibration method illustrated and described in flowchart 1100 is only in an example embodiment of calibration of a sensor of a material handling vehicle utilizing a factor graph. Those skilled in the art will appreciate that modifications to the method are contemplated within the scope of the present disclosure.

[0112] In an embodiment, a factor graph (FG) is used to construct and solve a constrained optimization problem to reduce an error metric of, for example, a sensor extrinsic node to determine the pose of the sensor relative to the material handling vehicle 100, such as by reducing a desired threshold for minimization. Figure 4 An example calibration coordinate system CCF is illustrated, which is created by the travel of a material handling vehicle 100 starting from an origin O to map overhead features 150 such as ceiling lights 150A-S to generate Figure 5A The factor graph FG and through Figure 3 A constrained optimization problem for the driving path shown in is solved for an error metric indicating the calibration and acceptable poses of the camera relative to the material handling vehicle 100.

[0113] about Figure 2-Figure 4 , the material handling vehicle 100 may include and / or be coupled to a vehicle computing device 102, reference Figure 2 The vehicle computing device 102 is shown and described in further detail. The vehicle computing device 102 may include a processor 130, which may be implemented as one or more processors communicatively coupled to the sensor 110. The processor 130 may be configured to execute logic that implements any of the methods or functionality described herein. A memory 120 may also be included and may be utilized to store logic, including machine-readable instructions, that may be communicatively coupled to the processor 130, the sensor 110, or any combination thereof.

[0114] In particular, Figure 2 A computing infrastructure is depicted that may be implemented and used for a material handling vehicle 100. The computing infrastructure may include a vehicle computing device 102 that includes and / or is coupled to sensors 110, motion sensors 114, memory 120, processor 130, accelerometer 132, gyroscope 134, input / output hardware 136, network interface hardware 138, and data storage component 140, in communication with one another via a local communication interface 105. The components of the vehicle computing device 102 may be physically coupled together and / or may be communicatively coupled via the local communication interface 105 (e.g., which may be implemented as a bus) or other interface to facilitate communication between the components of the vehicle computing device 102.

[0115] The sensor 110 may be any device capable of collecting sensor data that may be used for positioning of the material handling vehicle 100. The sensor 110 may be communicatively coupled to the processor 130 and other components of the material handling vehicle 100. The sensor 110 may include a digital still camera, a digital video camera, an analog still camera, an analog video camera, and / or other devices for capturing overhead images. Additionally or alternatively, the sensor 110 may include other position and positioning sensor systems coupled to the material handling vehicle, such as, for example, a LIDAR system, a RADAR system, an ultrasonic sensor system, and the like.

[0116] Processor 130 may include an integrated circuit, a microchip, and / or other device capable of executing machine-readable instructions or that has been configured to perform functions in a manner similar to machine-readable instructions. Memory 120 may be configured as volatile and / or non-volatile memory and may thus include random access memory (including SRAM, DRAM, and / or other types of RAM), ROM flash memory, a hard drive, secure digital (SD) memory, registers, compact disks (CDs), digital versatile disks (DVDs), and / or other types of non-transitory computer-readable media capable of storing logic (such as machine-readable instructions). Depending on the particular embodiment, these non-transitory computer-readable media may reside within vehicle computing device 102 and / or outside vehicle computing device 102, as described in more detail below. Thus, memory 120 may store operating logic 122, sensor logic 124 and logic 126, and calibration logic 128 for providing instructions and facilitating the functions described herein.

[0117] For example, the operating logic 122 may include an operating system and / or other software for managing the components of the vehicle computing device 102. The sensor logic 124 may enable the material handling vehicle 100 to determine the location of the material handling vehicle 100 relative to the warehouse environment 10 (e.g., ceiling lights 150A-S) via sensor data 144 captured by the sensors 110.

[0118] As described herein, the odometer logic 126 can be used to generate and calibrate odometer data. The odometer logic 126 can also enable the material handling vehicle 100 to navigate along the floor of the warehouse environment 10. In some embodiments, the odometer logic 126 can receive one or more signals from one or more sensors, such as one or more motion sensors 114, one or more accelerometers 132, one or more gyroscopes 134, one or more wheel angle sensors, etc., to determine the path that the material handling vehicle 100 has traveled over a period of time. Some other odometer methods may include laser scan matching and / or visual odometers. More specifically, the odometer logic 126 can be configured to determine a change in the position of the material handling vehicle 100 from a first position to a second position. The sensor logic 124 and the odometer logic 126 can each include multiple different logic blocks, each of which can be implemented as a computer program, firmware, and / or hardware, as an example.

[0119] In operation, for example, the material handling vehicle 100 may determine its current location via user input, via the vehicle computing device 102 (such as the material handling vehicle 100 passing through a radio frequency identifier, via a location system, etc.), and / or via a remote computing device 170. In some embodiments, the sensor 110 may be a camera configured to capture an image of a ceiling, which may include one or more ceiling lights 150A-S. Some embodiments are configured such that sensor data 144 (e.g., image data) captured by the sensor 110 may additionally be compared to a site map 142 to determine the current vehicle location. However, in order to accurately determine the vehicle location based on the sensor data 144 captured by the sensor 110, the sensor 110 may need to be calibrated relative to the material handling vehicle 100 in which it is operating, prior to relying on or not relying on such a site map 142. Systems and methods for calibrating the sensor 110 and, in some embodiments, aiding in generating the site map 142, as described herein.

[0120] Still reference Figure 2 As described herein, the calibration logic 128 may be configured to track multiple static features (e.g., ceiling lights 150A-S) in the warehouse environment 10 by using the sensor 110 to be calibrated, such as an image-based system (e.g., a camera) and / or a laser-based system (e.g., a LIDAR system) to construct a constrained optimization problem over multiple observations of the multiple static features from different locations in the warehouse environment 10. This data may be used to create an optimization problem that may be used to optimize the pose of the sensor 110 as it relates to the material handling vehicle 100. In embodiments described herein, the calibration logic 128 may not use or rely on a priori knowledge of the warehouse environment 10, such as a site map 142. In some embodiments, the process of calibrating the sensor 110 may also result in the formation of a site map 142, which may later be used to locate or supplement the positioning of the material handling vehicle 100 in the warehouse environment 10.

[0121] Reference again Figure 2, the motion sensor 114 may be any sensor capable of detecting or determining motion of the material handling vehicle 100. For example, the motion sensor 114 may include an optical, microwave, acoustic, or illumination sensor configured to detect motion or changes in the warehouse environment 10 as a function of detected motion. Some such sensors may include infrared sensors, ultrasonic range sensors, photo resistors, etc. In some embodiments, the motion sensor 114 may include one or more encoders (e.g., optical, mechanical, or electromechanical) positioned to detect movement of one or more wheels of the material handling vehicle 100. The motion sensor 114 may also include one or more sensors capable of and configured to detect an angular or directional heading of the material handling vehicle 100. In addition, the material handling vehicle 100 may include at least an accelerometer 132 and / or a gyroscope 134 in communication with the vehicle computing device 102.

[0122] The input / output hardware 136 may include and / or be configured to communicate with Figure 2 The network interface hardware 138 may include and / or be configured to communicate with any wired or wireless networking hardware, including antennas, modems, LAN ports, Wi-Fi cards, WiMax cards, Bluetooth cards, etc. TM Modules, mobile communication hardware, and / or other hardware for communicating with other networks and / or devices. Depending on the connection, communication between vehicle computing device 102 and other computing devices (such as remote computing device 170) can be facilitated through, for example, network 160. Network 160 can include a wide area network such as the Internet, a local area network (LAN), a mobile communication network, a public service telephone network (PSTN), and / or other networks and can be configured to electronically and / or communicatively connect vehicle computing device 102, remote computing device 170, and / or any other network-enabled device.

[0123] Still reference Figure 2, the data storage component 140 may include any storage device capable of storing logic such as machine-readable instructions and / or data. The data storage component 140 may be located locally and / or remotely from the vehicle computing device 102, and may be configured to store one or more pieces of data for access by the vehicle computing device 102 and / or other components. As shown, for example, the data storage component 140 may include data defining a site map 142, which may include information generated by the sensor 110 of the material handling vehicle 100 during calibration or may be otherwise generated and stored, sensor data 144 captured by the sensor 110 of the material handling vehicle 100 during calibration, and / or calibration data 146 for the sensor 110. The calibration data 146 may include manufacturer tolerances for the sensor to be calibrated and positioned on the material handling vehicle 100, installation calibration data 146, and / or calibration data 146 determined based on calibrating the sensor 110 relative to the material handling vehicle 100 in the warehouse environment 10.

[0124] It should be understood that although Figure 2 The components in are illustrated as residing as part of the vehicle computing device 102, but this is merely an example. In some embodiments, as described above, one or more components may reside external to the vehicle computing device 102. It should also be understood that while the vehicle computing device 102 is illustrated as a single device, this is also merely an example. In some embodiments, the operation logic 122, sensor logic 124, odometer logic 126, and calibration logic 128 may reside on different computing devices. As an example, one or more of the functions and / or components described herein may be provided by a remote computing device 170 and / or other devices, which may be communicatively coupled to the vehicle computing device 102. These computing devices may also include hardware and / or software (e.g., implementing Figure 9-1 2 illustratively shown in FIG.

[0125] Furthermore, while the vehicle computing device 102 is illustrated as having the sensor logic 124, odometer logic 126, and calibration logic 128 as separate logic components, this is also an example. In some embodiments, a single logic may enable the vehicle computing device 102 to provide the described functionality.

[0126] Reference now Figure 3 , depicts an illustrative time-lapse diagram of a material handling vehicle 100 providing calibration of sensors derived from a map of a warehouse environment 10 at multiple locations. In particular, Figure 3Depicted are material handling vehicles 100 at discrete locations in a warehouse environment 10 and associated image regions 112 that define areas of ceiling and overhead features such as ceiling lights 150A-S in sensor data 144 captured by sensors 110 (in this example, cameras 110A) coupled to the material handling vehicles 100 at each location. That is, Figure 3 Each material handling vehicle 100A depicted in the drawings is simply the same material handling vehicle 100A positioned at a different location within the warehouse environment 10. Furthermore, as shown, many of the image regions 112A-112L overlap, thereby capturing the same static overhead features (e.g., ceiling lights 150A-150S) in one or more images from different locations along the path of the material handling vehicle 100A through the warehouse environment 10.

[0127] As described in more detail below, the calibration method generally relies on calibrating the camera 110A on the material handling vehicle 100A by collecting image data from the camera 110A at multiple locations in the warehouse environment 10. The image data captured from the camera 110A includes static overhead features that overlap between one or more images captured from different locations.

[0128] Still reference Figure 3 , the calibration method in the warehouse environment 10 is generally described with respect to the time lapse diagram shown of the material handling vehicle 100 at multiple locations. For purposes of explanation and not limitation, the material handling vehicle 100A, the camera 110A, and the defined Figure 3 Each unique position of the image area 112 (i.e., represented by Figure 3 150A and 150B are associated with an additional letter (e.g., "A"-"L") indicating a unique location (e.g., material handling vehicle 100A at location A captures image data in image area 112A). More specifically, camera 110A at location A has image area 112A that includes two static features captured by the image data, ceiling lights 150A and 150B. Location A may be referred to as an initial location in the coordinate system for the remainder of the calibration.

[0129] Unlike calibration using the site map 142, the material handling vehicle 100A can be seeded anywhere within the warehouse environment 10 and does not need to be associated with a corresponding location in the site map 142. As a result, the error of associating the material handling vehicle 100A with the site map 142 can be eliminated, thereby removing the possibility of seeding errors caused by incorrect calibration and / or calibration failures due to incorrect or inaccurate initial seeding of the material handling vehicle 100A. However, it should be understood that the optimizer embodiments described herein can be used in a mapped configuration and / or a mapless configuration.

[0130] When the material handling vehicle 100A begins to move from location A to location B, a path to a reference coordinate system is determined and recorded. The path may be determined using any positioning method that does not include calibrated sensors. For example, the path may be determined based on at least the use of an odometer, motion sensor 114, wheel angle sensor, gyroscope 134, and / or accelerometer 132. Thus, a vehicle node representing the pose of the vehicle within the coordinate system may be determined. For example, if location A is represented as (x, y, z) coordinates, where the initial position is seeded at the origin (O) of a coordinate reference system (CRF) (e.g., as Figure 4 ) as (0, 0, 0) and the material handling vehicle 100A is determined to have traveled, for example, 4 meters in the x-direction, then position B may be (4, 0, 0).

[0131] In some embodiments, the path and / or distance traveled from a previous location to a next location may be determined by an algorithm configured to cause the material handling vehicle 100A to travel a linear or angular distance such that at least one static overhead feature within image region 112A captured at location A is visible within image region 112B captured at location B. As referred to herein, these unique image regions 112A that may include at least one static overhead feature are referred to as "frames." Although in Figure 3 Discrete locations are depicted in FIG. 1 , but the camera 110A may continuously record image data as the material handling vehicle 100A traverses the warehouse environment 10 , and frames including overlapping static overhead features may be selected from the image data for calibration purposes.

[0132] As mentioned above about Figure 3As described, a static overhead feature (i.e., ceiling lights 150A-S) captured in one image region 112F is also captured and visible in another image region 112G. For example, image region 112F and image region 112G share visibility of ceiling light 150J, which overlaps with image region 112F and image region 112B. Tracking of static overhead features such as ceiling lights between frames may be accomplished using a pixel-speed nearest neighbor image space tracking algorithm or any algorithm capable of tracking features frame-by-frame without requiring a priori knowledge of the location of the features in the camera calibration frames. Some image tracking algorithms may include edge detection, feature detection, and / or object recognition routines for tracking static features or portions of static overhead features frame-by-frame.

[0133] Furthermore, as the material handling vehicle 100A moves throughout the warehouse environment 10, a coordinate position relative to an initial position (eg, a seeding position) may be associated with each image data (ie, with each frame) of each image region 112. Figure 4 The tiles depicted in FIG. 1 visually represent the path of the material handling vehicle 100A and the collection of data (eg, image data from the camera 110A) as the material handling vehicle 100A travels around the warehouse environment 10. Figure 1 As discussed, the sensor 110 (eg, camera 110A) of the material handling vehicle 100 may capture views of the same static overhead features (eg, ceiling lights 150A-S) from various locations within the warehouse environment 10 . Figure 4 The graph in FIG. 1 depicts the position of the material handling vehicle 100 in a coordinate reference system and the position of the ceiling lights 150A-S in the same coordinate reference system (ie, represented by Figure 4 However, note that Figure 4 The curve graphs in the figure are only example curve graphs and do not directly depict Figure 3 Warehouse environment in 10.

[0134] It should be understood that in this calibration method, i.e. one that does not exploit a priori knowledge of the environment, calibration of the camera can be achieved by observing the same static object from more than one position within the environment. Therefore, the transformations between positions and the uncertainties between the transformations can be estimated within the factor graph FG. The factor graph FG can be used to construct an optimization problem that can be generated using the GTSAM library and optimizer, where each factor of the factor graph FG is optimized while satisfying constraints defined by the uncertainties of the inter-factors and extrinsics (such as camera extrinsics) until calibration values ​​within acceptable predefined values ​​are determined.

[0135] The sensor calibration technology disclosed herein is well suited for use at customer sites, specialized or general warehouse configurations. Using optimization and statistical techniques, sensor calibration is estimated as the material handling vehicle 100 travels through the environment. The calibration confidence in the estimate is also calculated and provided to the commissioning engineer as real-time feedback on the calibration progress to help determine when to end the calibration process. In some embodiments, the calibration process may include online and offline portions. For example, as the material handling vehicle traverses the environment, sensor data may be collected by sensors to be calibrated online. The sensor data may be stored in a computer-readable medium for offline analysis. The stored sensor data is input into an offline system, such as a server or computing device. The offline system may then generate a factor graph or similar model, an optimization model based on the sensor data, and generate calibration values ​​for the sensor, which may then be loaded into the computing device of the material handling vehicle.

[0136] In addition, the systems and methods described herein improve the ability to determine the position of a sensor (e.g., a camera coupled to a material handling vehicle) relative to a material handling vehicle by removing the requirement for prior information about the location of static features in a warehouse environment. Therefore, calibration can be performed before generating the site map 142, which eliminates the reliance on generating prior information and time associated with bringing the material handling vehicle to a functional state. In addition, potential errors associated with using map-based calibration, such as using a SLAM map in one example and manually seeding a material handling vehicle within the map, may not be associated with the calibration pose of the camera. In addition, the information of the site map 142 can be generated in parallel with the calibration of the camera 110A or similar sensor 110, because each frame and the associated position coordinates can be compiled into the site map 142 of the warehouse environment 10. For example, this information can also be used to update an existing site map 142 or help establish a new site map 142 for a new warehouse environment 10. SLAM is just one example of how to generate a map for use by the systems and methods described herein, and other map generation techniques are expected and within the scope of the present disclosure. Another example of a map source includes a surveyed map.

[0137] For purposes of describing and defining the present disclosure, it should be noted that references herein to a variable that is a "function" of a parameter or another variable are not intended to mean that the variable is a function of only the listed parameter or variable. Rather, references herein to a variable that is a "function" of the listed parameters are intended to be open-ended such that the variable may be a function of a single parameter or multiple parameters.

[0138] It should also be noted that recitation of “at least one” of a component, element, etc. herein should not be used to create an inference that alternative use of the article “a” or “an” should be limited to a single component, element, etc.

[0139] It should be noted that the description herein of a component of the present disclosure being "configured" in a particular manner to implement a particular property or function in a particular manner is a structural description, not a description of intended use. More specifically, references herein to the manner in which a component is "configured to" represent the existing physical condition of the component and are thus to be regarded as an explicit description of the structural characteristics of the component.

[0140] For purposes of describing and defining the present invention, it should be noted that the term "substantially" is utilized herein to represent the inherent degree of uncertainty that may be attributed to any quantitative comparison, value, measurement, or other representation. The term "substantially" is also utilized herein to represent the degree by which a quantitative representation may vary from a stated reference without resulting in a change in the basic function of the subject matter at issue.

[0141] Having described the subject matter of the present disclosure in detail and with reference to specific embodiments thereof, it should be noted that the various details disclosed herein should not be understood as implying that these details relate to elements that are essential parts of the various embodiments described herein, even in the case where specific elements are illustrated in each of the figures accompanying this specification. In addition, it is apparent that modifications and changes may be made without departing from the scope of the present disclosure, including but not limited to the embodiments defined in the appended claims. More specifically, although some aspects of the present disclosure are identified herein as preferred or particularly advantageous, it is contemplated that the present disclosure is not necessarily limited to these aspects.

[0142] Although specific embodiments and aspects of the present disclosure have been illustrated and described herein, various other changes and modifications may be made without departing from the spirit and scope of the present disclosure. In addition, although various aspects have been described herein, these aspects need not be used in combination. Accordingly, therefore, the appended claims are intended to cover all such changes and modifications within the scope of the embodiments shown and described herein.

[0143] It should now be understood that the embodiments disclosed herein include systems, methods, and non-transitory computer readable media describing systems for calibrating material handling vehicles. It should also be understood that these embodiments are exemplary only and are not intended to limit the scope of the present disclosure.

Claims

1. A material handling vehicle comprising a sensor, a vehicle position processor, one or more vehicle dead reckoning components, and a drive mechanism configured to move the material handling vehicle along an inventory transport surface, wherein: The sensor is configured to record one or more characteristics of the warehouse; as well as The vehicle position processor is configured as generating sensor data from a record of one or more characteristics recorded by a sensor, extracting the one or more features from the sensor data, creating a factor graph including a sensor extrinsic parameter node (eO), the sensor extrinsic parameter node including initial seed extrinsic parameters associated with the sensor, generating an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO) in a factor graph, wherein the initial sensor feature node (fO) includes selected features of the one or more extracted features that are associated with the initial sensor frame node (cO) and the initial vehicle node (vO) in the initial data association, Using a drive mechanism to navigate a material handling vehicle along a stock transport surface, generating a subsequent vehicle node (v1) in the factor graph based on accumulated mileage from the one or more vehicle dead reckoning components, In the factor graph, a subsequent sensor frame node (c1) is generated that is associated with one of the initial sensor feature node (fO) or the subsequent sensor feature node (f1) and the sensor extrinsic reference node (eO) and the subsequent vehicle node (v1) in the subsequent data association. optimizing the factor graph based on the initial data association and the subsequent data association to provide a calibrated output associated with the sensor, When the calibration output is below the threshold minimum value such that the calibration is terminated, an optimized calibration output is generated, and A material handling vehicle is navigated along the inventory transport surface based on optimized calibration outputs associated with the sensors.

2. The material handling vehicle of claim 1, wherein the vehicle position processor is further configured to: When a vehicle seed position and a feature map including one or more map features from a site map are provided, an initial vehicle node (vO) is generated in the factor graph as a map-based initial vehicle node at the vehicle seed position, with associated vehicle prior factors, and the one or more map features are generated as corresponding sensor feature nodes, with corresponding map feature prior factors.

3. The material handling vehicle of claim 2, wherein the associated vehicle prior factor of the map-based initial vehicle node at the vehicle seed location comprises an error function regarding the accuracy of the vehicle seed location of the material handling vehicle relative to the site map.

4. The material handling vehicle of claim 1, wherein the vehicle position processor is further configured to: When no vehicle seed position and feature map are provided, the initial vehicle node (vO) is generated as a map-free initial vehicle node at the origin in the factor graph.

5. The material handling vehicle of claim 1, wherein the vehicle position processor is further configured to: iteratively generating one or more subsequent sensor feature nodes and one or more additional subsequent vehicle nodes in the factor graph based on cumulative mileage from the one or more vehicle dead reckoning components between a previous vehicle node and an immediately subsequent vehicle node; and One or more additional subsequent sensor frame nodes associated with the sensor extrinsic reference node (eO), the associated directly subsequent vehicle node, and the associated sensor feature node of the one or more subsequent feature nodes in a subsequent data association are iteratively generated in the factor graph.

6. The material handling vehicle of claim 5, wherein the vehicle position processor is further configured to: An inter-vehicle factor between a directly subsequent vehicle node and a preceding vehicle node is generated in the factor graph based on the accumulated mileage.

7. The material handling vehicle of claim 1, wherein the vehicle position processor is further configured to: selecting one of the one or more extracted features; When the one feature matches an existing feature, associating the one feature with the existing feature as a matching sensor feature node; and Factors linking the generated sensor frame nodes with the matched feature nodes are generated in the factor graph.

8. The material handling vehicle of claim 7, wherein the generated sensor frame node comprises one of an initial sensor frame node (c0) or a subsequent sensor frame node (c1).

9. The material handling vehicle of claim 7, wherein the matched sensor feature node comprises one of an initial sensor feature node (f0) or a subsequent sensor feature node (f1).

10. The material handling vehicle of claim 1, wherein the vehicle position processor is further configured to: selecting one of the one or more extracted features; When the one feature does not match the existing features, generating a new sensor feature node associated with the one feature in the factor graph; Factors are generated in the factor graph linking the generated sensor frame node with the new sensor feature node.

11. The material handling vehicle of claim 10, wherein the new sensor characteristic node comprises one of an initial sensor characteristic node (f0) or a subsequent sensor characteristic node (f1).

12. The materials handling vehicle of claim 1, wherein the vehicle position processor is further configured to: The factor graph is optimized when no other features of the extracted one or more features remain associated with existing features as corresponding matching sensor feature nodes or are added to the factor graph as corresponding new sensor feature nodes.

13. The materials handling vehicle of claim 1, wherein the vehicle position processor is further configured to: The factor graph is optimized by generating a constrained optimization problem based on one or more variables and one or more factors defined in the factor graph.

14. The material handling vehicle of claim 13, wherein the one or more variables represent unknown random variables in a constrained optimization problem and include one or more nodes, the one or more nodes including one or more of a sensor extrinsic node (eO), an initial vehicle node (vO), a subsequent vehicle node (v1), an initial sensor frame node (cO), a subsequent sensor frame node (c1), an initial sensor feature node (fO), and a subsequent sensor feature node (f1).

15. The material handling vehicle of claim 13, wherein the one or more factors represent probabilistic information about a selected factor of the one or more factors and include a priori factors, inter factors, reference frame factors, projection factors, orientation range factors, or combinations thereof.

16. The material handling vehicle of claim 13, wherein the constrained optimization problem is constructed using one or more Smoothing and Mapping (SAM) libraries and optimizers.

17. The material handling vehicle of claim 13, wherein the one or more features include one or more overhead lights of a warehouse, and the constrained optimization problem is constructed using an image recognition algorithm, a data association algorithm, a modeling technique, or a combination thereof.

18. The materials handling vehicle of claim 1, wherein the vehicle position processor is further configured to: The optimization is terminated when the state of the intrinsic factor graph within a period of time is determined to be acceptable based on a predetermined threshold.

19. The materials handling vehicle of claim 1, wherein: The sensor is a camera, a laser-based sensor, or a combination thereof; The camera is configured to capture the one or more features of the warehouse; The laser-based sensor is configured to detect the one or more features of the warehouse; and The vehicle position processor is further configured to generate the sensor data based on a record of the one or more features captured by the camera, detected by the laser-based sensor, or a combination thereof.

20. A method of operating a material handling vehicle, the material handling vehicle comprising a sensor, a vehicle position processor, one or more vehicle dead reckoning components, and a drive mechanism configured to move the material handling vehicle along an inventory transport surface, wherein the sensor is configured to record one or more characteristics of a warehouse, the method comprising, via the vehicle position processor: generating sensor data from a record of one or more characteristics recorded by a sensor; extracting the one or more features from the sensor data; Creating a factor graph including a sensor extrinsic parameter node (eO), wherein the sensor extrinsic parameter node includes an initial seed extrinsic parameter associated with the sensor; generating an initial vehicle node (vO), an initial sensor frame node (cO), and an initial sensor feature node (fO) in a factor graph, wherein the initial sensor feature node (fO) includes selected features of the one or more extracted features that are associated with the initial sensor frame node (cO) and the initial vehicle node (vO) in the initial data association; navigates a material handling vehicle along a stock transport surface using a drive mechanism; generating, in the factor graph, a subsequent vehicle node (v1) based on accumulated mileage from the one or more vehicle dead reckoning components; In the factor graph, generating a subsequent sensor frame node (c1) associated with one of the initial sensor feature node (fO) or the subsequent sensor feature node (f1) and the sensor extrinsic reference node (eO) and the subsequent vehicle node (v1) in the subsequent data association; optimizing the factor graph based on the initial data association and the subsequent data association to provide a calibrated calibration output associated with the sensor; When the calibration output is below a threshold minimum value such that calibration is terminated, generating an optimized calibration output; and A material handling vehicle is navigated along the inventory transport surface based on optimized calibration outputs associated with the sensors.

Images