Precise altitude estimation using sensor fusion

By integrating distance and visual inertial sensors, flying robots can accurately estimate altitude, overcoming the limitations of barometers in indoor environments and ensuring precise navigation and collision avoidance.

JP2024542033AActive Publication Date: 2024-11-13BROOKHURST GARAGE INC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2024525710
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-01
Filing Date
2022-11-01
Publication Date
2024-11-13
Estimated Expiration
2042-11-01

AI Technical Summary

Technical Problem

Conventional barometers used in flying robots are not sensitive enough to provide accurate altitude estimates, especially in indoor environments where atmospheric pressure changes are insignificant or unmeasurable, making it difficult for flying robots to navigate and avoid collisions.

Method used

The use of a combination of distance sensors and visual inertial sensors to estimate altitude, where distance sensors provide accurate measurements over flat surfaces and visual inertial sensors are used in transition regions to correct for sudden changes in surface level, ensuring precise altitude estimation.

Benefits of technology

This method allows flying robots to maintain level flight and navigate accurately by complementing distance sensor data with visual inertial data, enhancing collision avoidance and navigation capabilities, particularly in environments with varying surface levels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542033000001_ABST
    Figure 2024542033000001_ABST
Patent Text Reader

Abstract

The flying robot may include a distance sensor and a visual inertial sensor. The flying robot may use the distance sensor data to determine a first estimated altitude of the flying robot relative to a first region having a first surface level. The flying robot may fly over at least a portion of the first region based on the first estimated altitude. The flying robot may determine that it is in a transition region between the first region and a second region having a second surface level different from the first surface level. The flying robot may use data from the visual inertial sensor to determine a second estimated altitude of the flying robot. The flying robot may use the second estimated altitude to control flight within the transition region. In the second region, the flying robot may revert to using the distance sensor in estimating altitude.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 274,448, filed November 1, 2021, which is incorporated herein by reference in its entirety.

[0002] (Technical field) The present disclosure relates generally to altitude estimation for flying robots, and more particularly to robots that use different sensors to accurately estimate altitude. [Background technology]

[0003] For flying robots, such as drones, to be autonomous, they must navigate their environments without colliding with objects. Estimating the robot's altitude at any given time instance is important for the robot's navigation and collision avoidance, especially in indoor environments. Conventionally, flying robots may be equipped with a barometer that measures air pressure changes at various altitudes to enable the flying robot to estimate its altitude. However, the measurements obtained from the barometer are often not sensitive enough to provide high-accuracy altitude estimation. Additionally, air pressure changes indoors are not significant enough or even immeasurable. Therefore, altitude estimation for flying robots is challenging. Summary of the Invention [Means for solving the problem]

[0004] Embodiments relate to a flying robot that may include a range sensor and a visual inertial sensor. Embodiments also relate to a method for a robot that performs altitude estimation using a range sensor and a visual inertial sensor. The method may include determining a first altitude estimation method for the flying robot relative to a first region having a first surface level using data from the range sensor of the flying robot. The method may also include controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude. The method may further include determining that the flying robot is in a transition region between the first region and a second region having a second surface level different from the first surface level. The method may further include determining a second altitude estimate for the flying robot using data from the visual inertial sensor of the flying robot. The method may further include controlling flight of the flying robot using the second altitude estimate in the transition region. The flying robot may include one or more processors and memory for storing instructions for executing the altitude estimation method. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating a system environment of an example storage site, according to some embodiments. [Figure 2] FIG. 1 is a block diagram illustrating components of an example robot and an example base station, according to some embodiments. [Figure 3] 1 is a flowchart illustrating an example process for managing inventory at a storage site, according to some embodiments. [Figure 4] FIG. 1 is a conceptual diagram of an example layout of a storage site with a robot, according to some embodiments. [Figure 5] 1 is a flowchart illustrating an example navigation process of a robot, according to some embodiments. [Figure 6A] FIG. 1 is a conceptual diagram showing the flight path of a flying robot. [Figure 6B]FIG. 1 is a conceptual diagram illustrating a flight path of a flying robot, according to some embodiments. [Figure 6C] 1 is a flowchart illustrating an example process for estimating a vertical altitude level of a flying robot, according to some embodiments. [Figure 7A] FIG. 2 is a block diagram illustrating an example altitude estimation algorithm, according to some embodiments. [Figure 7B] FIG. 10 is a conceptual diagram illustrating different functions of the altitude estimation algorithm and the use of sensor data when an aerial robot flies over obstacles and maintains level flight, according to some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating an example machine learning model, according to some embodiments. [Figure 9] FIG. 1 is a block diagram illustrating components of an example computing machine, according to some embodiments.

[0006] For purposes of illustration only, the figures depict and the detailed description sets forth various non-limiting embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] The figures and the following description relate only to preferred embodiments by way of example, and those skilled in the art may recognize alternative embodiments of the structures and methods disclosed herein as practical alternatives that may be employed without departing from the essence of what is disclosed.

[0008] Reference will now be made to the details of several embodiments, examples of which are shown in the accompanying figures. It should be noted that wherever feasible, similar or similar reference numerals are used in numerals, the similar or similar reference numerals may indicate similar or similar functionality. The figures represent embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the essence described herein.

[0009] An embodiment relates to an aerial robot that navigates an environment in level flight by using a range sensor and a visual inertial sensor together to accurately estimate the robot's altitude. The range sensor and the visual inertial sensor may use different methods to estimate altitude. Data generated from the two sensors may be used to complement each other to provide an accurate altitude estimate. In some embodiments, the aerial robot may use a range sensor to estimate altitude as the aerial robot moves across a flat surface. The aerial robot may monitor bias between data from the two different sensors. In a transition region between two flat surfaces, the aerial robot may switch to using the visual inertial sensor. The aerial robot may use the monitored bias to adjust the data from the visual inertial sensor.

[0010] System Overview FIG. 1 is a block diagram illustrating an exemplary robot-assisted or fully autonomous storage site system environment 100, according to some embodiments. By way of example, the system environment 100 includes a storage site 110, a robot 120, a base station 130, an inventory management system 140, a computer server 150, a data store 160, and user devices 170. The entities and components of the system environment 100 communicate with each other via a network 180. In various embodiments, the system environment 100 may include different, fewer, or additional components. Also, even when each component of the system environment 100 is referred to in the singular, the system environment 100 may include one or more of each component. For example, the storage site 110 may include one or more robots 120 and one or more base stations 130. Each robot 120 may have a corresponding base station 130, or multiple robots 120 may share a base station 130.

[0011] The storage site 110 may be any suitable facility for storing, selling, or displaying inventory, such as goods, merchandise, groceries, items, and collectibles. Exemplary storage sites 110 may include warehouses, inventory lots, bookstores, shoe stores, outlets, other retail stores, libraries, museums, and the like. The storage site 110 may include any number of regularly shaped structures. A regularly shaped structure may be a structure, fixture, equipment, furniture, frame, shell, rack, or other suitable object within the storage site that has a readily identifiable regular shape or contour, whether the object is permanent or temporary, fixed or movable, load-bearing or not. Regularly shaped structures are often used to store inventory at the storage site 110. For example, racks (including metal racks, shells, frames, or other similar structures) are often used to store goods and merchandise in warehouses. However, regularly shaped structures need not necessarily be used for inventory storage. A storage site 110 may include a particular layout in which various items are placed and systematically stored. For example, in a warehouse, racks may be organized into sections and separated by aisles. Each rack may include multiple pallet locations that may be identified using row and column numbers. A storage site may include high and low racks, and in some cases, may carry most inventory items primarily near ground level.

[0012] Storage site 110 may include one or more robots 120 used to track and manage inventory within storage site 110. For ease of reference, robot 120 may be referred to in the singular even if more than one robot 120 is used. Also, in some embodiments, more than one type of robot 120 may be present at storage site 110. For example, some robots 120 may be specialized in scanning inventory at storage site 110, while other robots 120 may be specialized in moving items. Robots 120 may also be referred to as autonomous robots, inventory cycle count robots, inventory inspection robots, inventory detection robots, or inventory control robots. Inventory robots may be used to track inventory items, move inventory items, and perform other inventory management tasks. The degree of autonomy may vary between embodiments. For example, in some embodiments, robot 120 may be fully autonomous, such that robot 120 automatically performs assigned tasks. In other embodiments, robot 120 may be semi-autonomous, such that robot 120 can move around storage site 110 with minimal human command or control. In some embodiments, regardless of the degree of autonomy robot 120 has, robot 120 may be remotely operated or switched to a manual mode. Robot 120 may take various forms, such as an aerial drone, a ground robot, a vehicle, a forklift, or a mobile picking robot.

[0013] The base station 130 may be a device to which the robot 120 returns or to which the aerial robot lands. The base station 130 may include one or more return locations. The base station 130 may be used to repower the robot 120. Various methods of repowering the robot 120 may be used in various embodiments. For example, in some embodiments, the base station 130 serves as a battery exchange station that replaces the robot 120's batteries when the robot arrives at the base station so that the robot 120 can quickly resume its mission. The replaced batteries may be charged at the base station 130 via a wired or wireless connection. In other embodiments, the base station 130 serves as a charging station that includes one or more charging terminals that mate with the robot 120's charging terminals to recharge the robot 120's batteries. In still other embodiments, the robot 120 may use fuel for power, and the base station 130 may repower the robot 120 by filling its fuel tank.

[0014] The base station 130 may also serve as a communications station for the robots 120. For example, in certain types of storage sites 110, such as warehouses, network coverage may not exist or may only exist in certain locations. The base station 130 may communicate with other components in the system environment 100 using wireless or wired communications channels, such as Wi-Fi or an Ethernet cable. When the robot 120 returns to the base station 130, the robot 120 may communicate with the base station 130. The base station 130 may send input, such as commands, to the robot 120 and download data captured by the robot 120. In embodiments where multiple robots 120 are used, the base station 130 may include a swarm controller or algorithms for coordinating movements among the robots. The base station 130 and the robots 120 may communicate using any suitable method, such as radio frequency, Bluetooth, near-field communication (NFC), or wired communication. In some embodiments, robot 120 communicates primarily with a base station, while in other embodiments, robot 120 may have the ability to communicate directly with other components within system environment 100. In some embodiments, base station 130 may serve as a wireless signal amplifier for robot 120 that communicates directly with network 180.

[0015] The inventory management system 140 may be a computer system operated by an administrator (e.g., a company that owns inventory, a warehouse management administrator, or a retailer selling inventory) using the storage site 110. The inventory management system 140 may be a system used to manage the location of inventory items. The inventory management system 140 may include a database that stores data about inventory items and item-related information, such as quantity, metadata tags, asset type tags, barcode labels, and item location coordinates within the storage site 110. The inventory management system 140 may provide both front-end and back-end software for the administrator to access the central database and inventory metrics, analyze data, generate reports, forecast future demand, and manage the location of inventory items to ensure that items are correctly located. The administrator may rely on the item coordinate data from the inventory management system 140 to ensure that items are correctly located at the storage site 110 so that they can be quickly retrieved from their storage locations. This prevents misplaced items from taking up space reserved for upcoming inventory and also reduces the time it takes to locate missing items during the outbound process.

[0016] The computer server 150 may be a server tasked with analyzing data provided by the robot 120 and providing instructions for the robot 120 to perform various inventory recognition and management tasks. The robot 120 may be controlled by the computer server 150, the user device 170, or the inventory management system 140. For example, the computer server 150 may instruct the robot 120 to scan and capture images of inventory stored in various locations at the storage site 110. Based on the data provided by the inventory management system 140 and the ground truth data captured by the robot 120, the computer server 150 checks for discrepancies between the two sets of data and determines whether an item should be flagged for various reasons, such as being misplaced, lost, or damaged. The computer server 150 may then instruct the robot 120 to resolve any potential problems, such as moving the misplaced item to its correct location. In some embodiments, the computer server 150 may generate a report of flagged items so that site personnel can manually resolve the issues.

[0017] The computer server 150 may include one or more computers operating at various locations. For example, a portion of the computer server 150 may be a local server residing at the storage site 110. Computer hardware, such as a processor, may be associated with a computer on-site or may be included in the base station 130. Another portion of the computer server 150 may be a geographically distributed cloud server. The computer server 150 may serve as a ground control station (GCS), providing data processing and maintaining end-user software used by the user devices 170. The GCS may be responsible for controlling, monitoring, and maintaining the robot 120. In some embodiments, the GCS may reside on-site as part of the base station 130. The data processing pipeline and end-user software server may reside remotely or on-site.

[0018] The computer server 150 may host software applications that allow users to manage inventory, base stations 130, and robots 120. The computer server 150 and the inventory management system 140 may or may not be operated by the same entity. In some embodiments, the computer server 150 may be operated by an entity separate from the storage site administrator. For example, the computer server 150 may be operated by a robotics service provider that supplies robots 120 and related systems to modernize and automate the storage site 110. The software applications hosted by the computer server 150 may take various forms. In some embodiments, the software applications may be integral to the inventory management system 140 or integrated as add-ons to the inventory management system 140. In other embodiments, the software applications may be separate applications that supplement or replace the inventory management system 140. In some embodiments, the software application may be provided as software as a service (SaaS) to an administrator of the storage site 110 by a robot service provider supplying the robot 120.

[0019] Data store 160 includes one or more storage devices, such as memory, in the form of a non-transitory, non-volatile computer storage medium that stores various data uploaded by robot 120 and inventory management system 140. For example, data stored in data store 160 may include images, sensor data, and other data captured by robot 120. The data may also include inventory data maintained by inventory management system 140. A computer-readable storage medium is a medium that does not involve a transitory medium, such as a propagated signal or carrier wave. Data store 160 may take various forms. In some embodiments, data store 160 communicates with other components over network 180. This type of data store 160 may be referred to as a cloud storage server. Exemplary cloud storage service providers may include AWS, AZURE STORAGE, GOOGLE CLOUD STORAGE, etc. In other embodiments, data store 160 is a storage device controlled by and connected to computer server 150, instead of a cloud storage server. For example, data store 160 may take the form of memory (e.g., hard drive, flash memory, disk, ROM, etc.) used by computer server 150, such as storage in a storage server room operated by computer server 150.

[0020] The user device 170 may be used by an administrator of the storage site 110 to provide commands to the robot 120 and manage inventory at the storage site 110. For example, the administrator can use the user device 170 to provide task commands to the robot 120 for the robot to automatically complete a task. In one case, the administrator can specify a specific target location or a range of storage locations for the robot 120 to scan. The administrator may also specify specific items for the robot 120 to locate or confirm placement. Examples of user devices 170 include personal computers (PCs), desktop computers, laptop computers, tablet computers, smartphones, wearable electronic devices such as smartwatches, or other suitable electronic devices.

[0021] User device 170 may include user interface 175 taking the form of a graphical user interface (GUI). A software application provided by computer server 150 or inventory management system 140 may be displayed as user interface 175. User interface 175 may take different forms. In some embodiments, user interface 175 is part of a front-end software application that includes a GUI displayed on user device 170. In one case, the front-end software application is a software application that may be downloaded and installed on user device 170 via, for example, an application store (e.g., App Store) on user device 170. In other cases, user interface 175 takes the form of a web interface of computer server 150 or inventory management system 140 where a client may perform actions via a web browser. In other embodiments, user interface 175 does not include graphical elements and communicates with computer server 150 or inventory control system 140 through other suitable methods, such as a command window or application program interfaces (APIs).

[0022] Communications between the robot 120, base station 130, inventory control system 140, computer server 150, data store 160, and user devices 170 may be transmitted over a network 180, such as the Internet. In some embodiments, the network 180 uses standard communication technologies and / or protocols. As such, the network 180 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, LTE, 5G, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express, etc. Similarly, network protocols used on network 180 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), user datagram protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), etc. Data exchanged on network 180 may be represented using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc.Additionally, all or part of the link may be encrypted using conventional encryption techniques such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet protocol security (IPsec), etc. Network 180 also includes links and packet switching networks such as the Internet. In some embodiments, two computer servers, such as computer server 150 and inventory management system 140, may communicate via an API. For example, computer server 150 may retrieve inventory data from inventory management system 140 via the API.

[0023] Illustrative robot and base station FIG. 2 is a block diagram illustrating components of an example robot 120 and an example base station 130, according to some embodiments. The robot 120 may include an image sensor 210, a processor 215, a memory 220, a flight control unit (FCU) 225 including an inertial measurement unit (IMU) 230, a state estimator 235, a visual reference engine 240, a planner 250, a communications engine 255, an I / O interface 260, and a power source 265. The functionality of the robot 120 may be distributed among various components in different ways than described below. In various embodiments, the robot 120 may include different, fewer, and / or additional components. Also, while each component in FIG. 2 is described in the singular, multiple components may be included. For example, the robot 120 may include one or more image sensors 210 and one or more processors 215.

[0024] The image sensor 210 captures images of the storage environment for purposes of navigation, localization, collision avoidance, object recognition and identification, and inventory tracking. The robot 120 may include one or more image sensors 210 and one or more types of such image sensors 210. For example, the robot 120 may include a digital camera that captures optical images of the environment for the state estimator 235. For example, data captured by the image sensor 210 may be provided to a VIO unit 236 included in the state estimator 235 for localization purposes, such as determining the position and orientation of the robot 120 relative to an inertial system, such as a global frame, whose position is known and fixed. The robot 120 may also include a stereo camera with two or more lenses to enable the image sensor 210 to capture three-dimensional images through stereo photography. The stereo camera may generate point cloud data for each image frame, including, for example, red, green, and blue (RGB) pixel values ​​and depth information. Images captured by the stereo camera may be provided to the visual reference engine 240 for object recognition purposes. The image sensor 210 may be other types of image sensors, such as a light detection and ranging (LIDAR) sensor, an infrared camera, and a 360-degree depth camera. The image sensor 210 may also capture images of item labels (e.g., barcodes) for inventory cycle counting purposes. In some embodiments, a single stereo camera may be used for multiple purposes. For example, the stereo camera may provide image data to the visual reference engine 240 for object recognition. The stereo camera may be used to capture images of labels (e.g., barcodes). In some embodiments, the robot 120 includes a gimbal-like rotating stage that can rotate the image sensor 210 to various angles and stabilize the images captured by the image sensor 210. In some embodiments, the image sensor 210 may capture data along a path for the purpose of mapping a storage site.

[0025] The robot 120 includes one or more processors 215 and one or more memories 220 that store one or more sets of instructions. The one or more sets of instructions, when executed by the one or more processors, cause the one or more processors to perform methods implemented as one or more software engines. Various components of the robot 120, such as the FCU 225 and state estimator 235, may be implemented as a combination of software and hardware (e.g., sensors). The robot 120 may use a single general-purpose processor to execute the various software engines or may use separate, more specialized processors for different functions. In some embodiments, the robot 120 may use a general-purpose computer (e.g., CPU) capable of executing various sets of instructions for the various components (e.g., the FCU 225, the visual reference engine 240, the state estimator 235, the planner 250). The general-purpose computer may run on a suitable operating system such as LINUX, ANDROID, etc. For example, in some embodiments, the robot 120 may carry a smartphone that includes an application used to control the robot. In other embodiments, the robot 120 includes multiple processors specialized for different functions. For example, some of the functional components, such as the FCU 225, the visual referencing engine 240, the state estimator 235, and the planner 250, may be modularized, each including its own processor, memory, and instruction set. The robot 120 may include a central processor unit (CPU) to coordinate and communicate with each modularized component. Therefore, depending on the embodiment, the robot 120 may include a single processor or multiple processors 215 to perform various tasks. The memory 220 may store images and video captured by the image sensor 210. The images may include images of the surrounding environment and images of inventory items, such as barcodes and labels.

[0026] The flight control unit (FCU) 225 may control the movement of the robot 120 through a combination of software and hardware, such as an inertial measurement unit (IMU) 230 and other sensors. For a ground robot 120, the flight control unit 225 may be referred to as a microcontroller unit (MCU). The FCU 225 controls the movement of the robot 120 based on information provided by other components. For example, a planner 250 determines the path of the robot 120 from a start point to a destination and provides commands to the FCU 225. Based on the commands, the FCU 225 generates electrical signals to various mechanical components of the robot 120 (e.g., actuators, motors, engines, wheels) to adjust the movement of the robot 120. The precise mechanical components of the robot 120 may vary depending on the embodiment and the type of robot 120.

[0027] The IMU 230 may be part of the FCU 225 or may be a separate component. The IMU 230 may include one or more accelerometers, gyroscopes, and other suitable sensors to generate measurements of forces, linear acceleration, and rotation of the robot 120. For example, an accelerometer measures forces acting on the robot 120 and senses linear acceleration. Multiple accelerometers work together to sense acceleration of the robot 120 in three-dimensional space. For example, a first accelerometer senses acceleration in the x-direction, a second accelerometer senses acceleration in the y-direction, and a third accelerometer senses acceleration in the z-direction. The gyroscope senses rotation and angular acceleration of the robot 120. Based on the measurements, the processor 215 may obtain an estimated position of the robot 120 by integrating the movement and rotation data of the IMU 230 over time. The IMU 230 may also measure the orientation of the robot 120. For example, the gyroscopes of the IMU 230 may provide readings of the pitch, roll, and yaw angles of the robot 120.

[0028] State estimator 235 may represent a set of software instructions stored in memory 220 that may be executed by processor 215. State estimator 235 may be used to generate localization information for robot 120 and may include various sub-components for estimating the state of robot 120. For example, in some embodiments, state estimator 235 may include a visual-inertial odometry (VIO) unit 236 and an altitude estimator 238. In other embodiments, other modules, sensors, and algorithms may be used by state estimator 238 to determine the location of robot 120.

[0029] The VIO unit 236 receives image data from the image sensor 210 (e.g., a stereo camera) and measurements from the IMU 230 and generates localization information, such as the position and orientation of the robot 120. Position data, obtained by double integration of acceleration measurements from the IMU 230, is often prone to drift errors. The VIO unit 236 extracts image features and tracks them in image sequences to generate optical flow vectors, which represent the movement of edges, boundaries, and surfaces of objects in the environment captured by the image sensor 210. Various signal processing techniques, such as filtering (e.g., Wiener filter, Kalman filter, bandpass filter, particle filter) and optimization, and data / image transformations may be used to reduce various errors in determining the localization information. The localization data generated by the VIO unit 236 may include an estimate of the pose of the robot 120, which may be expressed in terms of the roll, pitch, and yaw angles of the robot 120.

[0030] The altitude estimator 238 may be a combination of software and hardware used to determine the absolute and relative altitude of the robot 120 (e.g., distance from an object on the floor). The altitude estimator 238 may include a distance sensor 239 that measures the altitude of the robot 120 relative to the ground or relative to an object below the robot 120. The distance sensor 239 may be an electromagnetic, laser, optical, acoustic, ultrasonic, or other suitable signal type. For example, the distance sensor 239 may be a laser range finder, a lidar range finder, an acoustic range finder, an ultrasonic range finder, or a radar. A range finder may also include one or more emitters that emit a signal (e.g., infrared, laser, acoustic, etc.) and one or more sensors that detect the round-trip time of the signal reflecting off an object. In some embodiments, the robot 120 may be equipped with a single-emitter range finder. The altitude estimator 238 may receive data from the VIO unit 236 that estimates the altitude of the robot 120, typically in a less accurate manner than the distance sensor 239. When the robot 120 flies over various objects and inventory placed on the floor or other horizontal surface, the altitude estimator 238 may include software algorithms for combining data generated by the distance sensor 239 and data generated by the VIO unit 236. The data generated by the altitude estimator 238 may be used for collision avoidance and target location. The altitude estimator 238 may set a global maximum altitude to prevent the robot 120 from hitting the ceiling. The altitude estimator 238 also provides information about how many rows of racks are below the robot 120 so that the robot 120 can locate the target location. The altitude data may be used, along with the number of rows the robot 120 has passed, to determine the vertical level of the robot 120. Altitude estimation will be described in further detail with reference to FIGS. 6A through 7B.

[0031] The visual referencing engine 240 may represent a set of software instructions stored in memory 220 that may be executed by the processor 215. The visual referencing engine 240 may include various image processing and position algorithms to determine the current position of the robot 120, identify objects, edges, and surfaces in the environment near the robot 120, and determine an estimated distance and orientation (e.g., yaw) of the robot 120 relative to nearby surfaces. The visual referencing engine 240 may receive pixel data of successive images and point cloud data from the image sensor 210. The position information generated by the visual referencing engine 240 may include the distance and yaw from the object and the offset of the center from a target point (e.g., the center of the target object).

[0032] The visual reference engine 240 may include one or more algorithms and machine learning models for creating image segmentations from images captured by the image sensor 210. The image segmentation may include one or more segments separating frames (e.g., vertical or horizontal bars of a rack) or contours of regularly shaped structures appearing in the captured image from other objects and the environment. Algorithms used for image segmentation may include convolutional neural networks (CNNs). Other image segmentation algorithms may be used to perform the segmentation, such as edge detection algorithms (e.g., Canny operator, Laplace operator, Sobel operator, Prewitt operator), corner detection algorithms, Hough transforms, and other suitable feature detection algorithms.

[0033] The visual reference engine 240 also performs object recognition (e.g., object detection and further analysis) and keeps track of the relative movement of objects across successive images. The visual reference engine 240 may track the number of regularly shaped structures at the storage site 110 that the robot 120 has passed over. For example, the visual reference engine 240 may identify a reference point (e.g., the center of gravity) on the frame of a rack and, across successive images, determine whether the reference point passes through a particular location in the image (e.g., whether the reference point passes through the center of the image). If so, the visual reference engine 240 increments the number of regularly shaped structures that the robot 120 has passed over.

[0034] The robot 120 may use various components to generate various position information, including position information and localization information for nearby objects. For example, in some embodiments, the state estimator 235 may process data from the VIO unit 236 and the altitude estimator 238 to provide localization information to the planner 250. The visual referencing engine 240 may count the number of regularly shaped structures passed by the robot 120 to determine its current position. The visual referencing engine 240 may generate position information for nearby objects. For example, when the robot 120 reaches a target position on a rack, the visual referencing engine 240 may use point cloud data to reconstruct the surface of the rack and may use depth data from the point cloud to more accurately determine the yaw and distance between the robot 120 and the rack. The visual referencing engine 240 may determine a center offset corresponding to the distance between the robot 120 and the center of the target position (e.g., the center point of the target position on the rack). The planner 250 uses the centroid offset information to control the robot 120 to move to a target location and take a picture of the inventory at the target location. When the robot 120 changes direction (e.g., turns, transitions from horizontal to vertical motion, transitions from vertical to horizontal motion, etc.), the centroid offset information may be used to determine the exact position of the robot 120 relative to an object.

[0035] The planner 250 may correspond to a set of software instructions stored in the memory 220 that may be executed by the processor 215. The planner 250 may include various path planning algorithms for planning a path for the robot 120 as the robot moves from a first location (e.g., a start location, the current location of the robot 120 after completing a previous journey) to a second location (e.g., a target location). The robot 120 may receive input, such as a user command, to perform a specific action (e.g., scanning inventory, moving an item, etc.) at a specific location. The planner 250 may include two types of routes, corresponding to spot checks and range scans. In spot checks, the planner 250 may receive input including coordinates of one or more specific target locations. In response, the planner 250 plans a path for the robot 120 to move to the target locations and perform the action. In range scans, the input may include a range of coordinates corresponding to a range of target locations. In response, the planner 250 plans a path for the robot 120 to perform actions for a full scan or range of target locations.

[0036] Planner 250 may plan a route for robot 120 based on data provided by visual reference engine 240 and data provided by state estimator 235. For example, visual reference engine 240 may estimate the current location of robot 120 by tracking the number of regularly shaped structures in storage site 110 that robot 120 has passed. Based on the location information provided by visual reference engine 240, planner 250 may determine a route for robot 120 and adjust the movement of robot 120 as robot 120 moves along the route.

[0037] Planner 250 may include fail-safe mechanisms in case the movement of robot 120 deviates from the plan. For example, if planner 250 determines that robot 120 has passed the target path and is traveling too far from the target path, planner 250 may send a signal to FCU 225 in an attempt to correct the path. If the error is not corrected after a timeout or within a reasonable distance, or if planner 250 cannot accurately determine the current location, planner 250 may instruct the FCU to land or stop robot 120.

[0038] Relying on various position information, planner 250 may also include algorithms for collision avoidance purposes. In some embodiments, planner 250 relies on distance information, yaw angle, and center offset information relative to nearby objects to plan the movement of robot 120 to provide sufficient clearance between robot 120 and nearby objects. Alternatively or additionally, robot 120 may include one or more depth cameras, such as a set of 360-degree depth cameras, that generate distance data between robot 120 and nearby objects. Planner 250 uses position information from the depth cameras to perform collision avoidance.

[0039] The communications engine 255 and I / O interface 260 are communication components that enable the robot 120 to communicate with other components within the system environment 100. The robot 120 may communicate with external components, such as the base station 130, wirelessly or via wired connections using different communication protocols. Exemplary communication protocols may include Wi-Fi, Bluetooth, NFC, USB, etc., which connect the robot 120 to the base station 130. The robot 120 may transmit various types of data, such as image data, flight logs, position data, inventory data, and robot status information. The robot 120 may receive input from external sources to identify actions the robot 120 needs to perform. Commands may be generated automatically or manually by an administrator. The communications engine 255 may include algorithms for various communication protocols and standards, encoding, decoding, multiplexing, traffic control, data encryption, etc., for various communication processes. The I / O interface 260 may include software and hardware components, such as hardware interfaces, antennas, etc., for communication.

[0040] The robot 120 also includes a power source 265 used to power the various components and movements of the robot 120. The power source 265 may be one or more batteries or a fuel tank. Exemplary batteries may include lithium-ion batteries, lithium polymer (LiPo) batteries, fuel cells, and other suitable types of batteries. The batteries may be permanently installed or easily replaced. For example, the batteries may be removable so that they can be replaced when the robot 120 returns to the base station 130.

[0041] While Figure 2 shows various example components, robot 120 may include additional components. For example, mechanical features and components of robot 120 are not shown in Figure 2. Robot 120 may include various types of motors, actuators, robotic arms, lifts, other moving components, and other sensors, depending on the type of robot, to perform various tasks.

[0042] 2, the exemplary base station 130 includes a processor 270, a memory 275, an I / O interface 280, and a repowering unit 285. In various embodiments, the base station 130 may include different and / or fewer and / or additional components.

[0043] The base station 130 includes a processor 270 and one or more memories 275 containing one or more sets of instructions that cause the processor 270 to execute various processes implemented as one or more software modules. The base station 130 may provide input and commands to the robot 120 to perform various inventory control tasks. The base station 130 may include sets of instructions for performing group control among multiple robots 120. Group control may include task allocation, routing and planning, and coordinating movements between the robots to avoid collisions. The base station 130 may serve as a central control unit for coordinating the robots 120. The memory 275 may also include various sets of instructions for performing analysis of data and images downloaded from the robot 120. The base station 130 may provide various degrees of data processing, from formatting raw data to complete data processing to generate information useful for inventory management. Alternatively or additionally, base station 130 may directly upload data downloaded from robot 120 to a data store, such as data store 160. Base station 130 may provide operational, management, and administrative commands to robot 120. In some embodiments, base station 130 may be remotely controlled by user device 170, computer server 150, or inventory management system 140.

[0044] The base station 130 may include various types of I / O interfaces 280 for communication with the robot 120 and for communication with the Internet. The base station 130 may continuously communicate with the robot 120 using wireless protocols such as Wi-Fi or Bluetooth. In some embodiments, one or more components of the robot 120 of FIG. 2 may reside within the base station, which may provide commands to the robot 120 for movement and navigation. Alternatively or additionally, when the robot 120 lands or stops at the base station 130, the base station 130 may communicate with the robot 120 via a short-range communication protocol such as NFC or a wired connection. The base station 130 may be connected to a network 180, such as the Internet. The storage site 110's wireless network (e.g., a LAN) may not have sufficient coverage. The base station 130 may be connected to the network 180 via an Ethernet cable.

[0045] The repowering unit 285 includes components used to detect the power level of the robot 120 and repower the robot 120. Repowering can be accomplished by replacing batteries, recharging batteries, refilling fuel tanks, etc. In some embodiments, the base station 130 includes a mechanical actuator, such as a robot arm, to replace batteries in the robot 120. In other embodiments, the base station 130 may be used as a charging station for the robot 120, either via wired charging or inductive charging. For example, the base station 130 may include a landing or resting pad with an induction coil underneath to wirelessly charge the robot 120 via an induction coil within the robot. Other suitable methods of repowering the robot 120 are also possible.

[0046] Example inventory management method 3 is a flowchart illustrating an example method for managing inventory at a storage site, according to some embodiments. The method may be performed by a computer, which may be a single operating unit in the traditional sense (e.g., a single personal computer) or a set of distributed computers cooperating to execute a set of instructions (e.g., a virtual machine, a distributed computer system, cloud computing, etc.). Also, although a computer is described in the singular, the computer performing the method of FIG. 3 may include one or more computers in conjunction with computer server 150, inventory management system 140, robot 120, base station 130, or user device 170.

[0047] According to some embodiments, a computer receives 310 a configuration of the storage site 110. The storage site 110 may be a warehouse, a retail store, or other suitable location. The configuration information for the storage site 110 may be uploaded to the robot 120 for the robot to navigate the storage site 110. The configuration information may include the total number of regularly shaped structures in the storage site 110 and dimensional information for the regularly shaped structures. The provided configuration information may be in the form of a computer-aided design (CAD) drawing or other type of file format. The configuration may include a layout of the storage site 110, such as the layout of racks and the arrangement of other regularly shaped structures. The layout may be a two-dimensional layout. The computer extracts the number of sections, aisles, and racks, and the number of rows and columns for each rack, from the CAD drawing by counting their appearance in the CAD drawing. The computer may also extract the height and width of the rack cells from the CAD drawing or other source. In some embodiments, the computer does not need to extract the exact distance between a given set of racks, or the width of each aisle, or the total length of the racks. Instead, the robot 120 may measure the aisle, rack, and cell dimensions from depth sensor data, or may use a counting method implemented by the planner 250 in conjunction with the visual referencing engine 240 to traverse the storage site 110 by counting the number of rows and columns traversed by the robot 120. Therefore, in some embodiments, the exact dimensions of the racks may not be required.

[0048] Some configuration information may be manually entered by an administrator of storage site 110. For example, the administrator may provide the number of sections, the number of aisles and racks in each section, and the cell dimensions of the racks. The administrator may also enter the number of rows and columns for each rack.

[0049] Alternatively or additionally, configuration information may be obtained from a mapping process, such as pre-flight mapping or a mapping process performed as the robot 120 performs inventory control tasks. For example, for a storage site 110 newly implementing an automated management method, an administrator may provide the size of the navigable space of the storage site to one or more mapping robots, which may count sections, aisles, and rows and columns of regularly shaped structures within the storage site 110. Also, in some embodiments, mapping or configuration information need not measure precise distances between racks or other structures within the storage site 110. Instead, the robot 120 may navigate the storage site 110 with only a rough layout of the storage site 110 by counting regularly shaped structures along a path to identify target locations. As the robot 120 continues to perform various inventory control tasks, the robotic system may gradually perform mapping or estimates of the size and location of various structures.

[0050] The computer receives (320) inventory control data for inventory control operations at the storage site 110. Certain inventory control data may be manually entered by an administrator, while other data may be downloaded from the inventory management system 140. The inventory control data may include scheduling and planning for inventory control activities, including operation frequency, time windows, etc. For example, the management data may specify that each location on the racks at the storage site 110 should be scanned at predetermined intervals (e.g., daily) and that the inventory scanning process should be performed by the robot 120 overnight after the storage site is closed. The inventory management system 140 data may provide information such as the item's barcode and label, the exact coordinates of the inventory item, and information about the rack and other storage space that needs to be cleared for upcoming inventory. The inventory control data may also include items that need to be retrieved from the storage site 110 each day (e.g., purchase order items that need to be shipped), and the robot 120 may need to focus on those items.

[0051] The computer creates 330 a plan to perform the inventory control. For example, the computer may create an automated plan that includes various commands to instruct the robot 120 to perform various scans. The commands may specify the range of locations the robot 120 needs to scan or one or more specific locations the robot 120 needs to go to. The computer may estimate the time per scanning trip and design a plan for each work interval based on the time available for robotic inventory control. For example, at a particular storage site 110, robotic inventory control may not be performed during business hours.

[0052] The computer generates 340 various commands to operate one or more robots 120 to move through the storage site 110 based on the plan and information derived from the configuration of the storage site 110. The robots 120 may move through the storage site 110 by at least visually recognizing the regularly shaped structures within the storage site and counting the number of regularly shaped structures. In some embodiments, in addition to location techniques such as those used by VIO, the robot 120 counts the number of racks, rows, and columns that it has passed through to determine its current location along a path from a start location to a target location without knowing the exact distance and direction that the robot 120 has traveled.

[0053] Inventory scanning or other inventory control operations may be performed autonomously by the robot 120. In some embodiments, the scanning operation begins at a base station 130 where the robot 120 receives (342) input including coordinates of a target location or a range of target locations within the storage site 110. The robot 120 departs (344) from the base station 130. The robot 120 navigates (346) the storage site 110 by visually recognizing regularly shaped structures. For example, the robot 120 tracks the number of regularly shaped structures it passes. The robot 120 pivots and translates based on the recognized regularly shaped structures captured by the image sensor 210. Upon arriving at the target location, the robot 120 may align itself with a reference point (e.g., a center position) of the target location. At the target location, the robot 120 captures target location data (e.g., measurements, photographs, etc.) including the inventory item, the barcode, and the label on the inventory item's box (348). If the initial command before the robot 120 departs includes multiple target locations or a range of target locations, the robot 120 continues to the next target location by moving up, down, or sideways to the next location to continue the scanning operation.

[0054] After completing a scanning trip, the robot 120 returns to the base station 130 by counting the number of regularly shaped structures it has passed in reverse (350). The robot 120 may potentially recognize the structures it has passed as it moves to its target location. Alternatively, the robot 120 may return to the base station 120 by reversing its path without counting. The base station 130 repowers the robot 120. For example, the base station 130 may provide the robot 120 with a next command and replace the robot's 120's batteries (352) so that the robot 120 can quickly return to service for another scanning trip. Used batteries may be charged at the base station 130. The base station 130 may download the data and images captured by the robot 120 and upload the data and images to a data store 160 for further processing. Alternatively, the robot 120 may include wireless communication components for transmitting its data and images to the base station 130 or directly to the network 180 .

[0055] The computer performs (360) an analysis of the data and images captured by the robot 120. For example, the computer may compare barcodes (including serial numbers) in images captured by the robot 120 with data stored in the inventory management system 140 to determine if items are misplaced or missing within the storage site 110. The computer may also determine other conditions of the inventory. The computer may generate reports for display on the administrator user interface 175 to take corrective action for misplaced or missing inventory. For example, reports may be generated daily so that personnel at the storage site 110 manually locate and remove misplaced items. Alternatively or additionally, the computer may generate an automation plan for the robot 120 to remove misplaced inventory. The data and images captured by the robot 120 may be used to verify the removal or arrival of inventory items.

[0056] Example of movement method FIG. 4 is a conceptual diagram of an example layout of a storage site 110 equipped with a robot 120, according to some embodiments. FIG. 4 illustrates a two-dimensional layout of the storage site 110, with a close-up view of an example rack shown in inset 405. The storage site 110 may be divided into various regions based on regularly shaped structures. In this example, the regularly shaped structures are racks 410. The storage site 110 may be divided into sections 415, aisles 420, rows 430, and columns 440. For example, a section 415 is a collection of racks. Each aisle may have racks on both sides. Each rack 410 may include one or more columns 440 and multiple rows 430. A storage unit of a rack 410 may be referred to as a cell 450. Each cell 450 may hold one or more pallets 460. In this particular example, two pallets 460 are placed in each cell 450. Inventory for storage site 110 is transported on pallets 460. The division and designation shown in Figure 4 is used as an example only; storage site 110 in other embodiments may be divided differently.

[0057] Each inventory item at storage site 110 may reside on a pallet 460. The target location (e.g., pallet location) of an inventory item may be identified using a coordinate system. For example, an item placed on pallet 460 may have an aisle number (A), a rack number (K), a row number (R), and a column number (C). For example, pallet location coordinates of [A3, K1, R4, C5] mean that pallet 460 is located at rack 410 in the north rack of aisle 3. Pallet 460's location in rack 410 is the fourth row and fifth column (counting from the ground). In some cases, such as the particular layout shown in FIG. 4, an aisle 420 may have racks 410 on both sides. Additional coordinate information may be used to identify racks 410 on the north side of aisle 420 and racks 410 on the south side of aisle 420. Alternatively, the upper and lower racks may have different aisle numbers. For spot checks, the robot 120 may be provided with a single coordinate for a single location or multiple coordinates for more than one location. For a range scan to check for pallets 460 within a range, the robot 120 may be provided with a range of coordinates, such as an aisle number, a rack number, a starting row, a starting column, an ending row, and an ending column. In some embodiments, the pallet location coordinates may be referred to in different ways. For example, in one case, the coordinate system may adopt the format "aisle-rack-shelf-location." The shelf number may correspond to the row number, and the location number may correspond to the column number.

[0058] Referring to FIG. 5 in conjunction with FIG. 4, FIG. 5 is a flowchart illustrating an example method of movement of the robot 120, according to some embodiments. The robot 120 receives 510 a target location 474 at the storage site 110. The target location 474 may be expressed in a coordinate system, as described above with respect to FIG. 4. The target location 474 may be received as an input command from the base station 130. The input command may also include an action the robot 120 needs to take, such as taking a picture at the target location 474 to capture the barcode and label of an inventory item. The robot 120 may generate location information based on the VIO unit 236 and the altitude estimator 238. In some cases, the starting location of the route is the base station 130. In some cases, the starting location of the route may be anywhere at the storage site 110. For example, the robot 120 may have recently completed a task and may begin another task without returning to the base station 130.

[0059] A processor of the robot 120, such as executing the planner 250, controls 520 the robot 120 along a path 470 to a target location 474. The path 470 may be determined based on the coordinates of the target location 474. The robot 120 may rotate so that the image sensor 210 faces a regularly shaped structure (e.g., a rack). Movement of the robot 120 to the target location 474 may include moving to a particular aisle, rotating to enter the aisle, moving horizontally to a target column, moving vertically to a target row, and rotating to the correct angle to face the target location 474 to take a picture of the inventory items on the pallet 460.

[0060] As the robot 120 moves to the target location 474, the robot 120 captures (530) images of the storage site 110 using the image sensor 210. The captured images may be sequential images. The robot 120 receives (540) images captured by the image sensor 210 as the robot 120 moves along the path 470. The images may capture objects in the environment, including regularly shaped structures such as racks. For example, the robot 120 may use algorithms in the visual reference engine 240 to visually recognize regularly shaped structures.

[0061] The robot 120 determines (540) its current location on the path 470 by analyzing the images captured by the image sensor 210 and tracking the number of regularly shaped structures in the storage site that the robot 120 has passed. The robot 120 may use various image processing and object recognition techniques to identify regularly shaped structures and track the number of structures that the robot 120 has passed. With respect to the path 470 shown in FIG. 4 , the robot 120 may move toward the racks 410 until it reaches a turning point 476. The robot 120 determines that it has reached the target aisle because it has passed two racks 410. In response, the robot 120 rotates counterclockwise and enters the target aisle facing the target racks. The robot 120 counts the number of rows it has passed until it reaches the target row. Depending on the target row, the robot 120 may move vertically up or down to reach the target location. Upon reaching the target location, the robot 120 performs the action specified in the input command, such as taking a picture of the inventory item at the target location.

[0062] Example Level Flight Maneuvers FIG. 6A is a conceptual diagram illustrating the flight path of a flying robot 602. The flying robot 602 travels over a first region 604 having a first surface level 605, a second region 606 having a second surface level 607, and a third region 608 having a third surface level 609. For example, the first region 604 may correspond to a floor, and the second and third regions 606 and 608 may correspond to obstacles on the floor (e.g., objects on the floor, or pallets and inventory items placed on the floor in a storage site setting). FIG. 6A illustrates the challenges of moving a flying robot in level flight at a nearly constant altitude, especially in settings where accurate altitude measurement is required, such as indoor flight or low-altitude outdoor flight. Traditionally, flying robots may measure changes in air pressure with a barometer to estimate their altitude. However, in indoor or low-altitude settings, air pressure changes may not be large enough, or even immeasurable, to allow the flying robot 602 to measure altitude.

[0063] FIG. 6A illustrates a flying robot 602 that uses a distance sensor to measure altitude. The flying robot 602 is programmed to maintain a constant distance from the surface over which the flying robot 602 is traveling. While the distance sensor may produce a relatively accurate distance measurement between the flying robot 602 and the surface below, the distance sensor cannot determine changes in level between different regions because the distance sensor often measures the round-trip time of a signal (e.g., a laser) traveling from the sensor's emitter, being reflected by the surface, and returning to the sensor's receiver. Because the second region 606 is elevated from the first region 604, and the third region 608 is even elevated, a flying robot 602 maintaining a constant distance from the surface below may exhibit the flight path illustrated in FIG. 6A and be unable to perform level flight.

[0064] The inability to maintain level flight can pose various challenges to the movement of the flying robot 602. For example, an undesired change in altitude during a flight of the type shown in FIG. 6A can affect the generation of position and localization data for the flying robot 602 due to the drift that occurs with the change in altitude. In an indoor setting, an undetected increase in altitude could cause the flying robot 602 to crash into the ceiling of a building. In a storage site 110 setting, the flight path shown in FIG. 6A could prevent the flying robot 602 from performing inventory scans or moving across the same row of storage racks.

[0065] FIG. 6B is a conceptual diagram illustrating a flight path of a flying robot 610 according to some embodiments. The flying robot 610 may be an example of the robot 120 discussed in FIGS. 1-5. While the description in FIGS. 1-5 focuses on the movement of the robot 120 in a storage site, the altitude estimation discussed in FIGS. 6B-7B is not limited to indoor settings. In addition to functioning as the robot 120, the flying robot 610 may also be used in outdoor environments, such as low-altitude flight, requiring accurate altitude measurements. In some embodiments, the altitude estimation methods described in this disclosure can also be used for high-altitude flying robots, in combination with or instead of a barometer. The flying robot 610 may be a drone, an unmanned vehicle, an autonomous vehicle, or any other suitable machine capable of flight.

[0066] In some embodiments, the flying robot 610 includes a distance sensor (e.g., distance sensor 239) and a visual inertial sensor (e.g., VIO unit 236). The flying robot 610 may rely on the integration of distance sensor and visual inertial sensor analysis to navigate the flying robot 610 to maintain level flight despite changes in surface level in regions 604, 606, and 608. Again, the first region 604 may correspond to a floor, and the second and third regions 606 and 608 may correspond to obstacles on the floor (e.g., objects on the floor, or pallets and inventory placed on the floor in a storage site setting).

[0067] The flying robot 610 may use data from both sensors to compensate and adjust each other's data to determine an estimated vertical altitude, regardless of whether the flying robot 610 is traveling over the first region 604, the second region 606, or the third region 608. The distance sensor may return highly accurate distance readings (within feet, sometimes within inches, or even less) based on the round-trip time of the signal transmitted from the distance sensor's transmitter and reflected by a nearby surface toward which the transmitter is pointing. However, distance measurements from a distance sensor may be affected by changes in the nearby environment, such as the presence of an obstacle that elevates the surface toward which the distance sensor's transmitter is pointing. Additionally, depending on the orientation of the flying robot 610, the orientation of the distance sensor may not be pointing directly downward. For example, in FIG. 6B , the flying robot 610 is shown with a negative pitch angle 620 and a positive roll angle 622. As a result, the signal emitted by the distance sensor travels along a path 624 that is not a perfectly vertical path. The flying robot 610 determines a pitch angle 620 and a roll angle 622 using an IMU (such as IMU 230). The pitch angle 620 and roll angle 622 data may be part of the VIO data provided by a visual inertial sensor or may be independent data provided directly by the IMU. Using the pitch angle 620 and roll angle 622, the flying robot 610 may determine a first estimated altitude 630 based on readings from the distance sensors. Flight of the flying robot 610 over at least a portion of the first region 604 may be controlled based on the first estimated altitude. However, as the flying robot 610 moves over the second region 606, distance measurements from the distance sensors decrease rapidly due to the increasing height of the second region 606.

[0068] A visual inertial sensor (e.g., the VIO unit 236), or simply an inertial sensor, may be less susceptible to environmental changes, such as the presence of obstacles in the second region 606 and the third region 606 and 608. The inertial sensor may also be simply an inertial sensor, such as the IMU 230, or may include a visual element, such as the VIO unit 236. The inertial sensor provides localization data for the flying robot 610 based on the accelerometers and gyroscopes in the IMU. Because the IMU is built into the flying robot 610, the localization data is not measured relative to nearby objects or surfaces. Therefore, the data is also typically not affected by nearby objects or surfaces. However, the position data (including estimated vertical altitude) generated from the inertial sensor is often obtained by integrating acceleration data obtained from the IMU's accelerometer twice with respect to time. The localization data is prone to drift and may become less accurate as the flying robot 610 moves over relatively long distances.

[0069] The flying robot 610 may use data from the visual inertial sensors to compensate for data generated by the distance sensors in transition regions associated with changes in surface level. In some embodiments, in transition regions such as regions 640, 642, 644, and 646, data from the distance sensors may become unstable due to sudden changes in surface level. The flying robot 610 may temporarily switch to the visual inertial sensors to estimate its vertical altitude. After the transition region, the flying robot 610 may return to the distance sensors. Relying on both types of sensor data, the flying robot 610 may move relatively horizontally (at relatively the same horizontal level), as shown in FIG. 6B. Details of altitude estimation methods and determining transition regions are further described with reference to FIGS. 6C through 7B.

[0070] Example Altitude Estimation Method FIG. 6C is a flowchart illustrating an exemplary method for estimating the vertical altitude level of a flying robot 610 as the flying robot 610 moves over different areas with various surface levels, according to some embodiments. The flying robot 610 may include a distance sensor and a visual inertial sensor. The flying robot 610 may also include one or more processors and a memory for storing code instructions. When executed by the one or more processors, the instructions may cause the one or more processors to perform the method described in FIG. 6C. The one or more processors may correspond to the processors in the processor 215 and the FCU 225. For simplicity, the one or more processors may be referred to hereinafter as a "processor," even though each step of the method described in FIG. 6C may be performed by the same or different processors in the flying robot 610. The method illustrated in FIG. 6C will also be discussed in conjunction with the visual illustration in FIG. 6B.

[0071] In some embodiments, the flying robot 610 may use data from a range sensor to determine 650 a first estimated altitude 630 of the flying robot 610 relative to a first area 604 having a first surface level 605. For example, the data from the range sensor may take the form of a time series of distance measurements from the range sensor. For particular cases, a processor of the flying robot 610 may receive distance readings from the range sensor data. The processor may also receive an attitude of the flying robot 610. The attitude may include a pitch angle 620, a roll angle 622, and a yaw angle. In some embodiments, the flying robot 610 may use one or more angles related to the attitude to determine a first estimated altitude 630 from the distance readings adjusted by the pitch angle 620 and the roll angle 622. For example, the processor may convert the distance readings to the first estimated altitude 630 using one or more trigonometric relationships.

[0072] The processor controls (655) the flight of the flying robot 610 over at least a portion of the first region based on the first estimated altitude 630. As the flying robot 610 moves over the first region 604, measurements from the distance sensor should be relatively stable. The flying robot 610 may also monitor data from a visual inertial sensor. The visual inertial sensor data may also be a time series of localization data readings that include altitude estimation readings. The distance data readings from the distance sensor may be generated, for example, by a laser rangefinder, while the z-direction position data readings from the visual inertial sensor may be generated by double-integrating z-direction accelerometer data with respect to time. Because the two sensors estimate altitude from different sources and methods, the measurements from the two sensors may not match. Furthermore, the measurements from the visual inertial sensor may also be subject to drift. The flying robot 610 may monitor measurements from the visual inertial sensor and determine a bias between the measurements from the visual inertial sensor and the measurements from the distance sensor. The bias may be the difference between the two measurements.

[0073] The processor determines (660) that the flying robot 610 is in a transition region 640 between a first region 604 and a second region 606 having a second surface level 607 that is different from the first surface level 605. A transition region may be a region where the surface level is changing. A transition region may indicate the presence of a ground-level obstacle, such as an object that prevents a distance sensor signal from reaching the ground. For example, in a storage site setting, a transition region may be the boundary of a pallet or inventory placed on the floor.

[0074] In various embodiments, the transition region and its size may be defined differently depending on the implementation of the altitude estimation algorithm. In some embodiments, the transition region may be defined based on a predetermined horizontal length. For example, the transition region may be a fixed length after a distance sensor detects an abrupt change in distance readings. In other embodiments, the transition region may be defined based on a duration. For example, the transition region may be the duration after a distance sensor detects an abrupt change in distance readings. This time may be a predetermined period of time or a relative period determined based on the horizontal velocity of the flying robot 610.

[0075] In yet another example, a transition region may be defined as a region where the processor can no longer determine with certainty that the flying robot 610 is in a flat region. For example, the flying robot 610 may include in its memory one or more probabilistic models that determine the likelihood that the flying robot 610 is moving over a flat region. The likelihood may be determined based on distance data readings from a distance sensor that should be relatively stable when the flying robot 610 is moving over a flat region. If the likelihood that the flying robot 610 is traveling over a flat region is less than a threshold, the processor may determine that the flying robot 610 is in a transition region. For example, in some examples, the processor may determine a first likelihood that the flying robot 610 is in the first region 604. The processor may determine a second likelihood that the flying robot 610 is in the second region 606. The processor may determine that the flying robot is in the transition region 640 based on the first likelihood and the second likelihood. For example, if a first likelihood indicates that flying robot 610 is unlikely to be in first region 604 and a second likelihood indicates that flying robot 610 is unlikely to be in second region 606, the method may determine that flying robot 610 is in transition region 640.

[0076] In yet another example, the transition region may be defined based on the presence of an obstacle. For example, the processor may determine whether an obstacle is present based on distance readings from a distance sensor. The processor may determine an average value of distance readings from the distance sensor data, e.g., an average value of time-series distance data from a period prior to the most recent value. The processor may determine the difference between the average value and a particular distance reading at a particular instance, such as the most recent instance. In response to the difference being greater than a threshold, the processor may determine that there was a rather significant and sudden change in distance readings at the particular instance, indicating the likelihood that an obstacle is present. The processor may determine that the flying robot 610 has entered the transition region until the readings from the distance sensor stabilize again.

[0077] In yet another embodiment, a transition may be defined based on any suitable combination of the above criteria, or other criteria not expressly discussed.

[0078] The processor determines (665) a second estimated altitude 632 of the flying robot 610 using data from the visual inertial sensor during at least a portion of the time period during which the flying robot 610 is in the transition region 640. In the transition region 640, the abrupt change in surface level from the first surface level 605 to the second surface level 607 prevents the range sensor from accurately determining the second height estimate 632 because the range sensor signal cannot penetrate the obstacle and travel to the first surface level 605. Instead of using the range sensor data, the flying robot 610 switches to visual inertial sensor data. However, as described above, there may be a bias between the range sensor readings and the visual inertial sensor readings. The processor may determine a visual inertial bias. For example, the visual inertial bias may be determined from an average of the visual inertial sensor readings from a period prior to the transition region 640, such as the period during which the flying robot 610 is in the first region 604. In determining the second estimated altitude 632, the processor receives readings from the visual inertial sensor data. The processor determines the second estimated altitude 632 using the readings adjusted by the visual inertial bias.

[0079] The processor controls 670 the flight of the flying robot 610 using the second estimated altitude 632 in the transition region 640. The size of the transition region 640 may depend on various factors, as described in step 660. When traveling within or just past the transition region 640, the processor may determine a range sensor bias. For example, in the transition region, the visual inertial sensor may provide the second estimated altitude 632, while the range sensor may provide a range reading D because the range sensor's signal is reflected off the second surface level 607. Thus, the range sensor bias may be the difference between the second estimated altitude 632 and the range reading D that is approximately equal to the difference between the first surface level 605 and the second surface level 607.

[0080] Based on one or more factors defining the transition region as described above in step 660, the processor may determine that the flying robot 610 has exited the transition region. For example, the processor determines (675) that the flying robot 610 has been in the second region 606 for more than a threshold period of time. The threshold period may be a predetermined length or may be measured based on the stability of the distance sensor data. The processor returns to using the distance sensor data to determine a third estimated altitude 634 of the flying robot 610 while the flying robot 610 is in the second region 606 (680). In using the distance sensor data to determine the third estimated altitude 634, the processor may adjust the data using a distance sensor bias. For example, the processor may add a distance sensor bias to distance measurements from the distance sensor.

[0081] The flying robot 610 may continue to move to the third region 608 and return to the second region 606 via transition region 642 and transition region 644. The flying robot 610 may cycle through the method of switching between data from the range sensors and data from the visual inertial sensors, monitoring for various biases between the two sets of data.

[0082] Example Altitude Estimation Algorithm FIG. 7A is a block diagram illustrating an example altitude estimation algorithm 700, according to an embodiment. Altitude estimation algorithm 700 may be an example algorithm that may be used to implement the altitude estimation method illustrated in FIG. 6C. Altitude estimation algorithm 700 is merely one example for implementing the method described in FIG. 6C. In various embodiments, the method described in FIG. 6C may be implemented using other algorithms. Altitude estimation algorithm 700 may be part of an algorithm used in state estimator 235, such as altitude estimator 238. Altitude estimation algorithm 700 may be implemented by a general processor executing code instructions stored in memory or may be programmed into a special-purpose processor, depending on the design of flying robot 610.

[0083] Altitude estimation algorithm 700 may include various functions for making different decisions. For example, altitude estimation algorithm 700 may include obstacle detection function 710, downward condition detection function 720, visual inertial bias correction function 730, range sensor bias correction function 740, and sensor selection and disclosure function 750. In various embodiments, altitude estimation algorithm 700 may include different, fewer, or additional functions. Functions may also be combined or further separated. The decisions made by each function may also be distributed among the various functions in different ways as described in FIG. 7A.

[0084] The flow described in the altitude estimation algorithm 700 may correspond to a particular instance in time. A processor of the flying robot 610 may repeat the altitude estimation algorithm 700 to generate one or more time series of data. The altitude estimation algorithm 700 may receive range sensor data 760, attitude data 770, and visual inertial data 780 as inputs and generate an estimated altitude 790 as output. The range sensor data 760 may be measured using m data, such as distance readings from a range sensor as shown by line 624 in FIG. 6B. r The pose data 770 may include the following:

number

[0085] The obstacle detection function 710 uses z^, φ^, and θ^ from the attitude data 770 and m from the distance sensor data 760. r For example, the obstacle detection function 710 may determine whether a distance reading from the distance data 760 matches a distance reading calculated from the attitude data 770 (e.g., whether the absolute or squared difference between the two readings is less than or greater than a threshold). If the two data sources match, the obstacle detection function 710 may generate a first label as an output of the obstacle detection function 710. The first label indicates that no obstacle is detected. If the two data sources do not match, the obstacle detection function 710 may generate a second label as an output indicating that an obstacle is detected. The obstacle detection function 710 may be represented by the following mathematical equation: 1 G may be an output of the obstacle detection function 710.

number

number

number

[0086] The lower state detection function 720 may include one or more probability models for determining the likelihood P(H1) that the flying robot 610 is flying over a first region (e.g., a floor) and the likelihood P(H2) that the flying robot 610 is flying over a second region (e.g., over an obstacle). The lower state detection function 720 assigns a state S to the flying robot 610. The state may correspond to the first region, the second region, or a transition region. For example, if the likelihood P(H1) and the likelihood P(H2) indicate that the flying robot 610 is not in either the first region or the second region, the lower state detection function 720 assigns that the flying robot 610 is in the transition region. The lower state detection function 720 may be represented by the following mathematical equation:

number

number

number

number

[0087] The visual inertial bias correction function 730 uses the distance sensor data 760m r Visual inertial data for 780m v, and monitors the averaged bias of the visual inertial sensor. As mentioned above, data from the visual inertial sensor is prone to errors due to drift. Data from the visual inertial sensor may also have a certain bias compared to data from the range sensor. The flying robot 610 monitors the visual inertial data 780 and determines an average of the visual inertial data 780 over a certain period of time. The average may be used to determine a visual inertial bias and correct the visual inertial data 780 based on the bias. The visual inertial bias correction function 730 may be expressed by the following mathematical formula: b z (k) represents the visual inertial bias, and MA represents the moving average.

number

number

[0088] The range sensor bias correction function 740 compensates for the range sensor data 760 from the range sensor when the flying robot 610 is flying over an obstacle. The presence of an obstacle can prevent the signal from the range sensor from reaching the ground, causing the range sensor data 760 to be less than the actual altitude. The range sensor bias correction function 740 makes adjustments when the flying robot 610 returns to using the range sensor to estimate altitude after a transition region. The range sensor bias correction function 740 may be expressed as the following equation: b r (k) represents the range sensor bias,

number

number

number

[0089] The sensor selection and publishing function 750 selects which sensors to use in various situations and generates the final determination of estimated altitude z. For example, in one embodiment, when the flying robot 610 is in a first region, the flying robot 610 uses range sensor data 760 to determine estimated altitude z. When the flying robot 610 is in a transition region, the flying robot 610 uses visual inertial data 780. When the flying robot 610 is in a second region after the transition region (e.g., over an obstacle) within a threshold time, the flying robot 610 may also use visual inertial data 780. Thereafter, the flying robot 610 reverts to using range sensor data 760. The sensor selection and publishing function 750 may be represented by the following pseudocode: If S=0, (on the floor)

number

number

number

number

[0090] Altitude estimation algorithm 700 provides an example of estimating the altitude of an aerial robot that may be implemented in a location with layers of obstacles. In various embodiments, similar principles may be extended to multiple layers of obstacles.

[0091] FIG. 7B is a conceptual diagram illustrating the use of different functions of an altitude estimation algorithm 700 and sensor data as the flying robot 610 flies over obstacles and maintains level flight, according to an embodiment. An obstacle detection function 710, a downward state determination function 720, and a sensor selection and disclosure function 750 are used throughout the method. In a region 792 where the flying robot 610 is flying over a first region (e.g., a floor), range sensor data 760 is used because readings from the range sensor should be relatively stable. A visual inertial bias correction function 730 is also implemented to monitor the bias of the visual inertial data 780. In a transition region 794, visual inertial data 780 is used instead of range sensor data 760 because the range sensor data 760 may become unstable when obstacle boundaries cause abrupt changes in the range sensor data 760.

[0092] Shortly after the transition region 794, within the threshold ε 796, the flying robot 610 may determine that the range sensor data 760 may again stabilize. During this period, the flying robot 610 may continue to use the visual inertial data 780 and execute the range sensor bias correction function 740 to determine a correction value to be added to the range sensor data 760 to account for the depth of the obstacle. When the flying robot 610 is in the second region 798 (e.g., above an obstacle) and determines that it is also ready to switch to the range sensor (e.g., the range sensor data has stabilized again), the flying robot 610 again estimates its altitude using the range sensor data 760, along with adjustments due to the range sensor bias. The flying robot 610 also again executes the visual inertial bias correction function 730 and monitors the bias of the visual inertial data 780. This method may continue in a similar manner as the flying robot 610 moves across different surface levels.

[0093] Example machine learning model In various embodiments, a wide variety of machine learning techniques may be used. Examples include various forms of supervised, unsupervised, and semi-supervised learning, such as decision trees, support vector machines (SVMs), regression, Bayesian networks, and genetic algorithms. Deep learning techniques such as neural networks, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs), may also be used. For example, various object recognition, localization, and other processes performed by visual reference engine 240 may apply one or more machine learning and deep learning techniques.

[0094] In various embodiments, training techniques for machine learning models may be supervised, semi-supervised, or unsupervised. In supervised learning, the machine learning model may be trained with a set of labeled training examples. For example, for a machine learning model trained to classify objects, the training examples may be various photographs of objects labeled with the object type. The labels for each training example may be binary or multi-class. For training a machine learning model for image segmentation, the training examples may be photographs of regularly shaped objects from various repository sites with manually identified portions of the images. In some cases, unsupervised learning techniques may be used. The samples used for training are unlabeled. Various unsupervised learning techniques, such as clustering, may be used. In some cases, training may be semi-supervised, with the training set having a mixture of labeled and unlabeled examples.

[0095] Machine learning models may be associated with an objective function that generates a metric that describes the desired goal of the training method. For example, training may be intended to reduce the model's error rate in generating predictions. In this case, the objective function may monitor the error rate of the machine learning model. In object recognition (e.g., object detection and classification), the objective function of a machine learning algorithm may be the training error rate in classifying objects in a training set. Such an objective function may be called a loss function. Other forms of objective functions may be used, particularly for unsupervised learning models that are not labeled and therefore have an error rate that cannot be easily determined. In image segmentation, the objective function may correspond to the difference between the segments predicted by the model and the segments manually identified in the training set. In various embodiments, the error rate may be measured as a cross-entropy loss, an L1 loss (e.g., the sum of absolute differences between predicted and observed values), or an L2 loss (e.g., the sum of squared distances).

[0096] Referring to FIG. 8, an exemplary CNN structure is shown, according to some embodiments. CNN 800 may receive input 810 and generate output 820. CNN 800 may include various types of layers, such as convolutional layers 830, pooling layers 840, recurrent layers 850, fully connected layers 860, and custom layers 870. A convolutional layer 830 convolves the layer's input (e.g., an image) with one or more kernels to generate various types of images filtered by the kernels and generate feature maps. Each convolution result may be associated with an activation function. A convolutional layer 830 may be followed by a pooling layer 840 that selects the maximum value (max pooling) or the average value (average pooling) from the portion of the input covered by the kernel size. The pooling layer 840 reduces the spatial size of the extracted features. In some embodiments, a pair of convolutional layers 830 and pooling layers 840 may be followed by a recurrent layer 850 that includes one or more feedback loops 855. Feedback 855 may be used to evaluate spatial relationships between image features or temporal relationships between objects in the image. Layers 830, 840, and 850 may be followed by multiple fully connected layers 860 with interconnected nodes (represented by squares in FIG. 8 ). Fully connected layers 860 may be used for classification and object detection. In some embodiments, one or more custom layers 870 may be present for generating output 820 in a specific format. For example, a custom layer may be used in image segmentation for labeling pixels of an image input with various segment labels.

[0097] The layer order and number of layers in the CNN 800 in FIG. 8 are for illustrative purposes only. In various embodiments, the CNN 800 includes one or more convolutional layers 830, but may or may not include pooling layers 840 or recurrent layers 850. When a pooling layer 840 is present, not all convolutional layers 830 are necessarily followed by a pooling layer 840. Recurrent layers may be arranged differently in other positions in the CNN. The size of the kernels (e.g., 3×3, 5×5, 7×7, etc.) and the number of kernels allowed to be trained in each convolutional layer 830 may be different from the other convolutional layers 830.

[0098] A machine learning model may include specific layers and / or nodes and / or kernels and / or coefficients. Training a neural network such as the CNN800 may include forward propagation and backpropagation. Each layer of a neural network may include one or more nodes that are fully or partially connected to other nodes in adjacent layers. In forward propagation, a neural network performs a forward calculation based on the output of the previous layer. The work of a node may be defined by one or more functions. The functions defining the work of a node may include various computational tasks, such as convolution of data with one or more kernels, pooling, recurrent loops in an RNN, and various gates in an LSTM. The functions may also include activation functions that adjust the weights of the node's output. Nodes in different layers may be associated with different functions.

[0099] Each of the neural network's functions may be associated with various coefficients (e.g., weights and kernel coefficients) that can be adjusted during training. Additionally, some of the nodes in a neural network may be associated with activation functions that determine the node's output weights in forward propagation. Common activation functions may include step functions, linear functions, sigmoid functions, hyperbolic tangent (tanh) functions, and rectified linear unit (ReLU) functions. After an input is provided to the neural network and passed forward through the neural network, the results may be compared with training labels or other values ​​in the training set to determine the neural network's performance. The prediction method may be repeated for other images in the training set to calculate the value of the objective function for a particular training round. The neural network then performs backpropagation using a gradient descent method, such as stocgastic gradient descent (SGD), to adjust the coefficients of the various functions and improve the value of the objective function.

[0100] Multiple rounds of forward and backpropagation may be performed. Training may terminate when the objective function is sufficiently stable (e.g., the machine learning model has converged) or after a predetermined number of rounds on a particular set of training examples. The trained machine learning model may be used for prediction, object detection, image segmentation, or any other suitable task for which the model was trained.

[0101] Computer Architecture 9 is a block diagram illustrating components of an exemplary computing machine capable of reading instructions from a computer-readable medium and executing the instructions on a processor (or controller). The computer described herein may include a single computing machine as shown in FIG. 9, a virtual machine, a distributed computing system including multiple nodes of the computing machine as shown in FIG. 9, or any other suitable arrangement of computing devices.

[0102] 9 shows a diagrammatic representation of a computing machine in the exemplary form of a computer system 900, within which instructions 924 (e.g., software, program code, or machine language) stored on a computer-readable medium may be executed to cause the machine to perform one or more methods described herein. In some embodiments, the computing machine may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or may operate as a peer machine in a peer-to-peer (or distributed) network environment.

[0103] The computing machine configuration described in Figure 9 may correspond to any software or hardware components or combinations of components shown in Figures 1 and 2, including, but not limited to, inventory management system 140, computer server 150, data store 160, user device 170, and the various engines, modules, interfaces, terminals, and machines shown in Figure 2. Although Figure 9 shows various hardware and software elements, each of the components described in Figures 1 and 2 may include additional or fewer elements.

[0104] By way of example, the computing machine may be a personal computer (PC), or a tablet PC, or a set-top box (STB), or a personal digital assistant (PDA), or a mobile phone, or a smartphone, or a web appliance, or a network router, or an internet of things (IoT) appliance, or a switch or bridge, or any machine capable of executing instructions 924 that specify actions to be taken by the machine. Additionally, while only one machine is illustrated, the term "machine" may be taken to include a collection of machines that individually or collectively execute instructions 924 to perform one or more methodologies described herein.

[0105] The exemplary computer system 900 includes one or more processors (generally, processor 902) (e.g., a central processing unit (CPU), or a graphics processing unit (GPU), or a digital signal processor (DSP), or one or more application-specific integrated circuits (ASICs), or one or more radio-frequency integrated circuits (RFICs), or a combination thereof), a main memory 904, and a non-volatile memory 906, which are configured to communicate with each other via a bus 908. The computer system 900 may further include a graphics display device 910 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system 900 may also include an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing means), a storage device 916, a signal generating device 918 (e.g., a speaker), and a network interface device 920, which are also configured to communicate via the bus 908.

[0106] The storage device 916 includes a computer-readable medium 922 having stored thereon instructions 924 embodying one or more methodologies or functions described herein. The instructions 924 may reside, completely or at least partially, within the main memory 904 or within the processor 902 (e.g., within a processor's cache memory) during execution by the computer system 900, with the main memory 904 and the processor 902 also constituting computer-readable media. The instructions 924 may be transmitted or received over a network 926 via a network interface device 920.

[0107] Although computer-readable medium 922 is shown to be a single medium in the illustrated embodiment, the term "computer-readable medium" should be interpreted to include one medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 924). Computer-readable medium may include any medium capable of storing instructions (e.g., instructions 924) for execution by a machine, causing the machine to perform one or more of the methodologies described herein. Computer-readable medium includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media. Computer-readable medium does not include transitory media such as signals or carrier waves.

[0108] Further compositional considerations Particular embodiments are described herein as including logic circuits, or several components, engines, modules, or mechanisms. An engine may comprise a software module (e.g., code embodied on a computer-readable medium) or a hardware module. A hardware engine is a tangible device capable of performing specific tasks, and may be configured or arranged in a particular way. In example embodiments, one or more hardware engines in one or more computer systems (e.g., standalone, client, or server computer systems) or computer systems (e.g., processors or groups of processors) may be configured with software (e.g., applications or application portions) as hardware engines that operate to perform specific tasks as described herein.

[0109] In various embodiments, a hardware engine may be implemented mechanically or electronically. For example, a hardware engine may include dedicated circuitry or logic circuitry that is permanently configured to perform specific tasks (e.g., as a dedicated processor such as a field programmable gate array (FPGA) or application-specific integrated circuit (ASIC)). A hardware engine may also include programmable logic circuitry or circuitry that is temporarily configured by software to perform specific tasks (e.g., contained within a general-purpose processor or other programmable processor). The decision to implement a hardware engine mechanically, dedicated, permanently configured circuitry, or temporarily configured circuitry (e.g., configured in software) is, of course, based on cost and time considerations.

[0110] Various operations of the example methods described herein may be performed, at least in part, by one or more processors, such as processor 902, that are temporarily or permanently configured (e.g., by software) to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute a processor-implemented engine that operates to perform one or more operations or functions. An engine as referred to herein may, in some example embodiments, include a processor-implemented engine.

[0111] Performance of a particular task may be distributed across one or more processors located across several machines, rather than being located solely within one machine. In some example embodiments, one or more processors or processor-implemented modules may reside in one geographic location (e.g., in a home environment, or in an office environment, or in a server farm). In other example embodiments, one or more processors or processor-implemented modules may be distributed across several geographic locations.

[0112] Upon reading this disclosure, those skilled in the art will recognize still further alternative structural and functional designs for similar systems or methods through the principles disclosed herein. Thus, while specific embodiments and applications are shown and described, it should be understood that the disclosed embodiments are not limited to the exact structures and components disclosed herein. It will be apparent to those skilled in the art that various modifications, changes, and variations may be made in the details of the arrangements and operations and methods disclosed herein without departing from the spirit and scope, as defined in the appended claims.

[0113] (Addendum) (Appendix 1) 1. A method of operating a flying robot, comprising: determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from a range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that the flying robot is in a transition region between the first region and a second region having a second surface level different from the first surface level; determining a second estimated altitude of the flying robot using data from a visual inertial sensor of the flying robot; and controlling the flight of the flying robot using the second estimated altitude in the transition region; A method comprising:

[0114] (Appendix 2) 2. The method of claim 1, wherein the first region corresponds to ground level and the second region corresponds to an obstacle located at the ground level.

[0115] (Appendix 3) Determining the first estimated altitude of the flying robot relative to the first area having the first surface level using data from the range sensor includes: receiving a distance reading from the data of the distance sensor; receiving an attitude of the flying robot, including a roll angle and a pitch angle of the flying robot; and determining the first estimated altitude from the range readings adjusted by the roll angle and the pitch angle; 2. The method of claim 1, comprising:

[0116] (Appendix 4) Determining that the flying robot is in the transition region between the first region and the second region includes: determining a first likelihood that the flying robot is within the first region; determining a second likelihood that the flying robot is within the second region; and determining that the flying robot is within the transition region based on the first likelihood and the second likelihood; 2. The method of claim 1, comprising:

[0117] (Appendix 5) Determining that the flying robot is in the transition region based on the first likelihood and the second likelihood includes: determining that the flying robot is in the transition region in response to both the first likelihood indicating that the flying robot is unlikely to be in the first region and the second likelihood indicating that the flying robot is unlikely to be in the second region; 5. The method of claim 4, comprising:

[0118] (Appendix 6) determining that the flying robot is in the transition region between the first region and the second region includes determining the presence of an obstacle; Determining the presence of the obstacle comprises: determining an average distance reading from the data of the distance sensors; determining the difference between the average and a particular distance reading at a particular instance; and determining that the obstacle is likely to be present at the particular instance in response to the difference being greater than a threshold; 2. The method of claim 1, comprising:

[0119] (Appendix 7) Determining the second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot includes: determining a visual inertial bias, which is an estimated difference between the range sensor readings and the visual inertial sensor readings; receiving a reading from the data of the visual inertial sensor; and determining the second estimated altitude using the readings adjusted by the visual inertial bias; 2. The method of claim 1, comprising:

[0120] (Appendix 8) 8. The method of claim 7, wherein the visual inertial bias is determined from an average of the readings of the visual inertial sensor from a previous period.

[0121] (Appendix 9) determining that the flying robot is within the second region for more than a threshold period of time; and reverting to using the data from the distance sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; 2. The method of claim 1, further comprising:

[0122] (Appendix 10) reverting to using the data from the distance sensor to determine the third estimated altitude of the flying robot while the flying robot is within the second region; Determining a range sensor bias; and determining the third estimated altitude using the data from the range sensor adjusted by the range sensor bias; 10. The method of claim 9, comprising:

[0123] (Appendix 11) distance sensor, Visual inertial sensors, one or more processors coupled to the range sensor and the visual inertial sensor; and a memory configured to store instructions; A flying robot comprising: The instructions, when executed by the one or more processors, determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from the range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that the flying robot is within a transition region between the first region and a second region having a second surface level different from the first surface level; determining a second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot; and controlling the flight of the flying robot using the second estimated altitude within the transition region; causing the one or more processors to perform steps including: Flying robot.

[0124] (Appendix 12) 12. The flying robot of claim 11, wherein the first region corresponds to ground level and the second region corresponds to an obstacle placed at the ground level.

[0125] (Appendix 13) The instructions for determining the first estimated altitude of the flying robot relative to the first area having the first surface level using data from the distance sensor include: instructions for receiving a distance reading from the data of the distance sensor; An instruction to receive an attitude of the flying robot, including a roll angle and a pitch angle of the flying robot; and instructions for determining the first estimated altitude from the range readings adjusted by the roll angle and the pitch angle; 12. The flying robot of claim 11, comprising:

[0126] (Appendix 14) The instructions for determining that the flying robot is in the transition region between the first region and the second region include: instructions for determining a first likelihood that the flying robot is within the first region; instructions for determining a second likelihood that the flying robot is within the second region; and instructions for determining that the flying robot is within the transition region based on the first likelihood and the second likelihood; 12. The flying robot of claim 11, comprising:

[0127] (Appendix 15) The instructions for determining the second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot include: instructions for determining a visual inertial bias, which is an estimated difference between the range sensor readings and the visual inertial sensor readings; instructions for receiving a reading from the data of the visual inertial sensor; and instructions for determining the second estimated altitude using the readings adjusted by the visual inertial bias; 12. The flying robot of claim 11, comprising:

[0128] (Appendix 16) 16. The flying robot of claim 15, wherein the visual inertial bias is determined from an average of the visual inertial sensor readings from a previous period.

[0129] (Appendix 17) The instructions, when executed, further cause the one or more processors to: determining that the flying robot is within the second region for more than a threshold time period; and reverting to using the data from the distance sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; 12. The flying robot of claim 11,

[0130] (Appendix 18) The instructions for reverting to using the data from the distance sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region include: instructions for determining a range sensor bias; and instructions for determining a third estimated altitude using the data from the range sensor adjusted by the range sensor bias; 18. The flying robot of claim 17, comprising:

[0131] (Appendix 19) 1. A method of operating an aerial robot including a range sensor and a visual inertial sensor, comprising: determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from a range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that a first likelihood that the flying robot is within the first region is less than a first threshold; and determining a second estimated altitude of the flying robot using the data from the visual inertial sensor in response to the first likelihood being less than the first threshold; A method comprising:

[0132] (Appendix 20) determining that a second likelihood that the flying robot is within a second region exceeds a second threshold; and reverting to using the data from the distance sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; 20. The method of claim 19, further comprising:

Claims

1. 1. A method of operating an aerial robot, comprising: determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from a range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that the flying robot is in a transition region between the first region and a second region having a second surface level different from the first surface level; determining a second estimated altitude of the flying robot using data from a visual inertial sensor of the flying robot; and controlling the flight of the flying robot using the second estimated altitude in the transition region; A method comprising:

2. The method of claim 1 , wherein the first region corresponds to a ground level and the second region corresponds to an obstacle located at the ground level.

3. Determining the first estimated altitude of the flying robot relative to the first area having the first surface level using data from the distance sensor includes: receiving a distance reading from the data of the distance sensor; receiving an attitude of the flying robot, including a roll angle and a pitch angle of the flying robot; and determining the first estimated altitude from the range readings adjusted by the roll angle and the pitch angle; The method of claim 1 , comprising:

4. Determining that the flying robot is in the transition region between the first region and the second region includes: determining a first likelihood that the flying robot is within the first region; determining a second likelihood that the flying robot is within the second region; and determining that the flying robot is within the transition region based on the first likelihood and the second likelihood; The method of claim 1 , comprising:

5. Determining that the flying robot is within the transition area based on the first likelihood and the second likelihood includes: determining that the flying robot is in the transition region in response to both the first likelihood indicating that the flying robot is unlikely to be in the first region and the second likelihood indicating that the flying robot is unlikely to be in the second region; The method of claim 4 , comprising:

6. determining that the flying robot is within the transition region between the first region and the second region includes determining a presence of an obstacle; Determining the presence of an obstacle comprises: determining an average of distance readings from the data of the distance sensors; determining the difference between the average and a particular distance reading at a particular instance; and determining that the obstacle is likely to be present at the particular instance in response to the difference being greater than a threshold; The method of claim 1 , comprising:

7. Determining the second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot includes: determining a visual inertial bias, which is an estimated difference between the range sensor readings and the visual inertial sensor readings; receiving a reading from the visual inertial sensor data; and determining the second estimated altitude using the readings adjusted by the visual inertial bias; The method of claim 1 , comprising:

8. The method of claim 7 , wherein the visual inertial bias is determined from an average of the readings of the visual inertial sensor from a previous period of time.

9. determining that the flying robot is within the second region for more than a threshold time period; and reverting to using the data from the range sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; The method of claim 1 further comprising:

10. reverting to using the data from the distance sensor to determine the third estimated altitude of the flying robot while the flying robot is within the second region; Determining a range sensor bias; and determining the third estimated altitude using the data from the range sensor adjusted by the range sensor bias; 10. The method of claim 9, comprising:

11. Distance sensor, Visual inertial sensor, one or more processors coupled to the range sensor and the visual inertial sensor; and a memory configured to store instructions; A flying robot comprising: The instructions, when executed by the one or more processors, determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from the range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that the flying robot is within a transition region between the first region and a second region having a second surface level different from the first surface level; determining a second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot; and controlling the flight of the flying robot using the second estimated altitude within the transition region; causing the one or more processors to execute steps including: Flying robot.

12. The flying robot of claim 11 , wherein the first region corresponds to a ground level and the second region corresponds to an obstacle located at the ground level.

13. The instructions for determining the first estimated altitude of the flying robot with respect to the first area having the first surface level using data from the distance sensor include: instructions for receiving a distance reading from the data of the distance sensor; An instruction to receive an attitude of the flying robot, including a roll angle and a pitch angle of the flying robot; and instructions for determining the first estimated altitude from the range readings adjusted by the roll angle and the pitch angle; The flying robot of claim 11 .

14. The instructions for determining that the flying robot is in the transition region between the first region and the second region include: instructions for determining a first likelihood that the flying robot is within the first region; instructions for determining a second likelihood that the flying robot is within the second region; and instructions for determining that the flying robot is within the transition region based on the first likelihood and the second likelihood; The flying robot of claim 11 .

15. The instructions for determining the second estimated altitude of the flying robot using data from the visual inertial sensor of the flying robot include: instructions for determining a visual inertial bias, which is an estimated difference between the range sensor readings and the visual inertial sensor readings; instructions for receiving a reading from the data of the visual inertial sensor; and instructions for determining the second estimated altitude using the readings adjusted by the visual inertial bias; The flying robot of claim 11 .

16. The flying robot of claim 15 , wherein the visual inertial bias is determined from an average of the visual inertial sensor readings from a previous period of time.

17. The instructions, when executed, further cause the one or more processors to: determining that the flying robot is within the second region for more than a threshold time period; and reverting to using the data from the range sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; The flying robot of claim 11 .

18. The instructions for reverting to using the data from the distance sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region include: instructions for determining a range sensor bias; and instructions for determining a third estimated altitude using the data from the range sensor adjusted by the range sensor bias; The flying robot of claim 17 , comprising:

19. 1. A method for operating an aerial robot including a range sensor and a visual inertial sensor, comprising: determining a first estimated altitude of the flying robot relative to a first area having a first surface level using data from a range sensor of the flying robot; controlling flight of the flying robot over at least a portion of the first region based on the first estimated altitude; determining that a first likelihood that the flying robot is within the first region is less than a first threshold; and determining a second estimated altitude of the flying robot using the data from the visual inertial sensor in response to the first likelihood being less than the first threshold; A method comprising:

20. determining that a second likelihood that the flying robot is within a second region exceeds a second threshold; and reverting to using the data from the range sensor to determine a third estimated altitude of the flying robot while the flying robot is within the second region; 20. The method of claim 19, further comprising:

Citation Information

Patent Citations

  • Method, apparatus and unmanned aerial vehicle for terrain following flight of an unmanned aerial vehicle

    JP2018527669A

  • Flying device, flying method, and program

    JP2019057185A

  • Laser scanner with real-time online egomotion estimation.

    JP2020507072A

  • Electronic Control Device for Controlling a Drone, Related Drone, Controlling Method and Computer Program

    US20180164122A1

  • Detecting and following terrain height autonomously along a flight path

    US20180314268A1