Automatic docking of unmanned aerial vehicle

Abstract

To provide a technique for automatically docking an unmanned aerial vehicle (for instance, a drone) to a base station for the unmanned aerial vehicle.SOLUTION: A dock platform comprises an unmanned aerial vehicle that includes a propulsion mechanism, an image sensor and a processing device, and a dock 100 that includes a landing surface 110 constituted to hold the unmanned aerial vehicle and a fiducial 120 on the landing surface 110. The processing device is configured to: control the propulsion mechanism so that the unmanned aerial vehicle flies up to a first position in the vicinity of the dock 100; access one or more images photographed using the image sensor; detect the fiducial 120 in at least one of the one or more images; determine a posture of the fiducial 120 on the basis of the one or more images; and control the propulsion mechanism so that the unmanned aerial vehicle lands on the landing surface 110, on the basis of the posture of the fiducial 120.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application claims priority to U.S. Provisional Application No. 62 / 915,639, filed October 15, 2019, the contents of each of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates to automated docking of unmanned aerial vehicles. [Background technology]

[0003] Previous unmanned aerial vehicle (e.g., drone) base stations typically consist of a large enclosure at least three times larger on each side than the drones they serve. The drones typically rely on the availability of highly accurate Global Positioning System (GPS) location data to land somewhere within the envelope of the large base station, and then rely on complex mechanical methods to align the drone on a landing pad so that an articulated robotic arm can physically replace the drone's batteries, or some other scheme to connect the drone to a power source that can charge the batteries. These traditional systems are large, mechanically complex, expensive, and may not function properly in GPS-free environments. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings, in which: It is emphasized that, according to common practice, the various objects in the drawings are not to scale. On the contrary, dimensions of various features are arbitrarily expanded or reduced for clarity. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is an illustration of an example dock for facilitating autonomous landing of an unmanned aerial vehicle. [Diagram 2] FIG. 2 is an illustration of an example unmanned aerial vehicle configured for autonomous landing on a landing surface. [Figure 3A] FIG. 3A is an illustration of an example system for facilitating autonomous landing of an unmanned aerial vehicle. [Figure 3B] FIG. 3B is an illustration of an example system for facilitating autonomous landing of an unmanned aerial vehicle while the unmanned aerial vehicle is enclosed within a dock. [Figure 4] FIG. 4 is a block diagram of an example of a hardware configuration of an unmanned aerial vehicle. [Diagram 5] FIG. 5 is a block diagram of an example of a hardware configuration of a dock for an unmanned aerial vehicle. [Figure 6] FIG. 6 is a flowchart of an example process for autonomously landing an unmanned aerial vehicle using at least one fiducial on a landing surface. [Figure 7] FIG. 7 is a flow chart of an example process for autonomously landing an unmanned aerial vehicle. [Figure 8] FIG. 8 is a flow chart of an example process for performing a final approach to autonomously land an unmanned aerial vehicle. [Figure 9] FIG. 9 is a flowchart of an example process for autonomously landing an unmanned aerial vehicle using a sequence of fiducials at different scales. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Much of the value and challenge of autonomous unmanned aerial vehicles lies in enabling robust, fully autonomous missions. Disclosed herein is a dock platform that enables unmanned charging, takeoff, landing, and mission planning for unmanned aerial vehicles (UAVs). Some implementations enable reliable operation of such platforms and associated application programming interface designs that make the system accessible by a wide variety of consumer and commercial applications.

[0007] One of the biggest limiting factors for operating a drone is the battery. A typical drone can operate for 20-30 minutes before needing a new battery pack. This places a limit on how long an autonomous drone can operate without human intervention. When the battery pack depletes, the operator must land the drone and replace the battery pack with a fully charged one. Battery technology continues to improve, achieving higher energy densities, but improvements are incremental and it may not be possible to draw a clear roadmap for sustained autonomous operation. An approach to alleviate the need for periodic human intervention is to automate battery management operations using some type of automated base station.

[0008] Some methods disclosed herein can leverage visual tracking and control software to perform pinpoint landings on much smaller targets. By using visual fiducials to assist absolute position tracking relative to the base station, a UAV (e.g., a drone) may be able to reliably hit a 5 cm x 5 cm target in a variety of environmental conditions. This means that the UAV can be positioned very accurately with the help of a small passive funnel shape that helps direct the UAV's battery to a set of charging contacts that extend below the rest of the UAV's structure, without the need for complex actuation or large structures. This allows the basic implementation of a base station to simply consist of a funnel-shaped nest with a set of spring contacts and a visual tag inside. Also, to mitigate the effects of the turbulent ground that the UAV encounters when landing, this nest can be elevated above the ground and the shape of the nest itself can be small enough to remain in the center of the UAV's prop wash when landing. To enable reliable operation in GPS-dead environments, the fiducials in the nest (e.g., small visual tags) can be supplemented with larger fiducials (e.g., large visual tags) located somewhere outside the landing nest, such as on a flexible mat laid on the ground near the base station or attached to a nearby wall. The supplemental visual tag can be easily found by the UAV from a significant distance in order for the UAV to reacquire its absolute position relative to the landing nest in GPS-dead environments, despite visual inertial odometry (VIO) navigation drift that may accumulate over the course of the UAV's mission. Finally, a reliable communication link with the UAV must be maintained in order for the UAV to cover a large area. In most cases, the ideal landing and charging location is not a good place to place a transmitter, so the communication circuitry can be placed in a separate range-extender module that is ideally placed somewhere high and centered in the desired mission space for maximum coverage.

[0009] The simplicity and low cost of such a system, when compared to more complex and expensive battery swapping systems, compensates for the amount of time the UAV is unavailable while the batteries are recharged. Intermittent operation is sufficient for many use cases, and users requiring more UAV coverage can simply increase UAV availability by adding another UAV and base station system. Such a cheaper, more prolific approach could be cost-competitive with larger, more expensive battery swapping systems, and could also significantly improve system reliability by eliminating a system-wide outage due to a single point of failure.

[0010] For use cases where a UAV (e.g., drone) needs to be protected from the elements, but no existing UAV-accessible structure is available, a UAV nest can be incorporated into a small custom shed. The shed can consist of a covered section under which the UAV lands, attached to an unroofed foyer area that acts as a windbreak and allows the UAV to enter and make precision landings, even in high winds. One useful feature of such a shelter would be an open or vented section along the entire perimeter of the bottom of the walls, allowing the drone's downdraft to leave the structure rather than turbulently circulating inside and adversely affecting stable flight.

[0011] For use cases where a UAV (e.g., a drone) needs to be more robustly secured from dust, cold, theft, etc., a mechanized "drone in a box" enclosure may be used. For example, a drawer-like box just slightly larger than the UAV itself may be used as a dock for the UAV. In some implementations, a motorized door on the side of the box can be opened 180 degrees to keep it out of the UAV's downdraft. For example, inside the box, a charging nest may be attached to a telescoping linear slide that holds the UAV far enough away from the box as it takes off or lands. In some implementations, once the UAV lands, the slide pulls the UAV back into the box, and the UAV can slowly rotate its propellers backwards to fold into a small space and move out of the way of the door. This allows the footprint of the box to be smaller than the area swept out by the UAV's propellers. In some implementations, the two-bar link connecting the door and its motor is designed to rotate over center so that once closed, the motor cannot be backdriven by pulling on the door from the outside, effectively locking the door. For example, the UAV may be physically secured within the nest by a linkage that leverages the last few centimeters of the slide's movement to press the UAV firmly against the nest with soft rollers (see, e.g., soft rollers 310 in FIG. 3B). Once secured, the box can be safely transported or even inverted without having to remove the UAV.

[0012] This actuated enclosure design can be shelf mounted or free standing, on an elevated platform that ensures the UAV is high enough above the ground to avoid ground effect when landing. The square profile of the box makes it easy to stack multiple boxes on top of each other for a multi-drone hive configuration where each box rotates 90° relative to the box below it so that multiple drones can take off and land simultaneously without interfering with each other. Because the UAV is physically secured within the enclosure when the box is closed, the box can be mounted to a car or truck to avoid experiencing charging interruptions while the vehicle is in motion. For example, in implementations where the UAV deploys sideways from the box, the box can be flush mounted to a wall to ensure it is completely out of the way when not landing or taking off.

[0013] When closed, the box can be made to have a very high Ingress Protection (IP) rating and can be equipped with a rudimentary cooling and heating system to allow the system to function in many outdoor environments. For example, high-efficiency particulate adsorption (HEPA) filters covering the intake cooling fans can be used to protect the interior of the enclosure from dust in the environment. A heater built into the top of the box can melt snow accumulations in wintry locations.

[0014] For example, the top and sides of the box can be made of materials that do not block radio frequencies, versions of range extenders can be built into the box itself for mobile applications, UAVs (e.g., drones) can maintain GPS lock while charging and can be deployed instantly, In some implementations, windows can be built into the door, or the door and side panels of the box can be transparent, allowing the UAV to see its surroundings before deploying and allowing the UAV to act as its own security camera to deter theft or vandalism.

[0015] In some implementations, a spring-loaded microfiber wiper can be placed inside the box so that the navigation camera lens is wiped clean every time the drone slides into or out of the box. In some implementations, a small diaphragm pump in the box can charge a small pressure vessel, which can then be used to clean all of the lenses on the drone by blowing air through small nozzles in the box.

[0016] For example, the box can be attached to the vehicle by three linear actuators hidden within the mounting base that can lift or tilt the box during launch or landing to compensate for the vehicle being on a slope or uneven terrain.

[0017] In some implementations, the box can include single or double doors on the top of the box that, once slid or swung open, allow the landing nest to extend into the open air rather than out to the side. This also takes advantage of the UAV's ability to land on small targets while staying clear of obstacles or surfaces that would impede the UAV's propeller wash (which would make a stable landing more difficult), and once the UAV has landed, the UAV and nest can be stored in a secure enclosure.

[0018] The autonomous landing techniques described herein can be implemented using software running on a processing unit within the unmanned aerial vehicle and / or on a processing unit in a dock for the UAV.

[0019] For example, the robust estimation and relocalization procedure may include visual relocalization of a dock with landing surfaces at multiple scales. For example, the UAV software may support a transition from GPS to visual localization. In some implementations, the design, size, and orientation of any fiducial (e.g., visual tag) around the dock may be supported. For example, the software may enable detection and rejection of spurious detections.

[0020] For example, the UAV's takeoff and landing procedures may include wind-robust planning and control using model-based wind estimation and / or model-based wind compensation. For example, the UAV's takeoff and landing procedures may include a landing "honing procedure" that allows the vehicle to stop just above the landing surface of a dock. In windy environments, state estimation and visual detection are more accurate than control, so the vehicle waits until the position, velocity, and angle errors between the actual vehicle and a fiducial on the landing surface are small before committing to landing. For example, the UAV's takeoff and landing procedures may include dock-specific landing detection and abort procedures. For example, actual contact with the dock may be detected and the system may distinguish between a successful landing and a near miss. For example, the UAV's takeoff and landing procedures may include employing slow reverse motor rotation to enable self-retracting propellers.

[0021] In some implementations, takeoff and landing procedures for the UAV may include support for failure cases and fallback behaviors, such as setting a predetermined landing location in case of failure; going to another box; or the option to land on top of a dock if the box is congested.

[0022] For example, an application programming interface design for single drone, single dock operation may be provided. For example, skills may be performed on a schedule or as long as there is battery life or recharge rate.

[0023] For example, an application programming interface design for N drone operations with M docks may be provided. In some implementations, mission parameters may be defined such that UAVs (e.g., drones) are automatically dispatched and recalled in an overlapping manner to always meet the mission parameters.

[0024] An unmanned aerial vehicle (UAV) may be configured to automatically fold its propellers to fit into a dock. For example, the dock may be smaller than the full UAV. Sustained operation may be achieved by multiple UAVs docking, charging, conducting missions, waiting to dock, and / or charging in unison. In some implementations, automated servicing of the UAV while the UAV is in place in the dock may include, for example: charging batteries, cleaning sensors, cleaning and / or drying the UAV, and more generally, replacing propellers and / or replacing batteries.

[0025] A UAV can track its state (e.g., attitude, including position and orientation) using a combination of sensing modalities (e.g., visual inertial odometry (VIO) and Global Positioning System (GPS)-based motion) to provide robustness against drift.

[0026] In some implementations, during takeoff and landing, as the UAV approaches the dock, it always horns in to the landing site. The honing process can make the takeoff and landing procedures robust to wind, ground effects, and other disturbances. For example, intelligent honing can utilize position, orientation, and trajectory to stay within very tight tolerances. In some implementations, the rear motors may reverse to enter.

[0027] Some implementations may offer advantages over previous systems such as: a small, inexpensive, and simple dock; a retraction mechanism that may allow stacking to reduce aerodynamic turbulence issues around landing; robust visual landing that may be more accurate; automatic retraction of propellers to allow tight packing during charging, maintenance, and storage of the UAV; vehicles being able to be serviced while docked without human intervention; and sustained autonomous operation of multiple vehicles via docks, SDKs, vehicles, and services (hardware and software).

[0028] FIG. 1 is an illustration of an example dock 100 for facilitating autonomous landing of an unmanned aerial vehicle. The dock 100 includes a landing surface 110 having a fiducial 120 and charging contacts 130 for a battery charger. The dock 100 includes a rectangular parallelepiped shaped box 140 having a door 142. The dock 100 includes a retractable arm 150 that supports the landing surface 110 and allows the landing surface 110 to be positioned outside the box 140 to facilitate takeoff and landing of the unmanned aerial vehicle, or the landing surface 110 to be located inside the box 140 for storage and / or servicing of the unmanned aerial vehicle. The dock 100 includes a second, auxiliary fiducial 122 on the exterior top surface of the box 140. The root fiducial 120 and the auxiliary fiducial 122 may be detected and used for visual positioning of the unmanned aerial vehicle relative to the dock to enable precision landing on the small landing surface 110. For example, process 600 of Figure 6 may be used to land an unmanned aerial vehicle on landing surface 110 of dock 100. For example, process 900 of Figure 9 may be used to land an unmanned aerial vehicle on landing surface 110 of dock 100. Dock 100 may include electronics such as those depicted in Figure 5 to control operation of dock 100 and interaction with the unmanned aerial vehicle.

[0029] The dock 100 includes a landing surface 110 configured to hold an unmanned aerial vehicle (e.g., the unmanned aerial vehicle 200 of FIG. 2 ) and a fiducial 120 on the landing surface 110. The landing surface 110 has a funnel shape formed to fit the bottom of the unmanned aerial vehicle at the bottom of the funnel. The tapered sides of the funnel may help to mechanically guide the bottom of the unmanned aerial vehicle to a centered position on the bottom of the funnel during landing. For example, corners at the bottom of the funnel may help to prevent the air vehicle from rolling on the landing surface 110 after the bottom of the air vehicle fits into the bottom of the funnel shape of the landing surface 110. For example, the fiducial 120 may include an asymmetric pattern that can robustly detect and determine the attitude (i.e., position and orientation) of the fiducial 120 relative to the unmanned aerial vehicle based on an image of the fiducial 120 captured by an image sensor of the unmanned aerial vehicle. For example, the fiducial 120 may include a visual tag of the ApriTag family.

[0030] Dock 100 includes conductive contacts 130 for a battery charger on landing surface 110 positioned at the bottom of the funnel. Dock 100 includes a charger configured to charge the battery while the unmanned aerial vehicle is on landing surface 110.

[0031] The dock 100 includes a box 140 configured to enclose the landing surface 110 in a first configuration (shown in FIG. 4) and expose the landing surface 110 in a second configuration (shown in FIGS. 1 and 3). The dock 100 may be configured to automatically transition from the first configuration to the second configuration by performing steps including opening a door 142 of the box 140 and extending a retractable arm 150 to move the landing surface 110 from inside the box 140 to outside the box 140. The auxiliary fiducials 122 are disposed on an exterior surface of the box 140.

[0032] The dock 100 includes a retractable arm 150, with the landing surface 110 disposed at the end of the retractable arm 150. When the retractable arm 150 is extended, the landing surface 110 is positioned away from the box 140 of the dock 100, which can reduce or prevent propeller wash from the propellers of the unmanned aerial vehicle during landing and simplify landing operations. The retractable arm 150 may include an aerodynamic cowling to redirect propeller wash to further mitigate the problem of propeller wash during landing.

[0033] For example, the fiducial 120 may be a root fiducial, and the auxiliary fiducial 122 is larger than the root fiducial 120 to facilitate visual positioning from a distance as the unmanned aerial vehicle approaches the dock 100. For example, the area of ​​the auxiliary fiducial 122 may be 25 times the area of ​​the root fiducial 120. For example, the auxiliary fiducial 122 may include an asymmetric pattern that allows for robust detection and determination of the attitude (i.e., position and orientation) of the auxiliary fiducial 122 relative to the unmanned aerial vehicle based on an image of the auxiliary fiducial 122 taken by an image sensor of the unmanned aerial vehicle. For example, the auxiliary fiducial 122 may include a visual tag of the ApriTag family. For example, a processing device (e.g., processing device 510) of the unmanned aerial vehicle may be configured to detect the auxiliary fiducial 122 in at least one of one or more images captured with an image sensor of the unmanned aerial vehicle; determine an attitude of the auxiliary fiducial 122 based on the one or more images; and control a propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate landing surface 110 based on the attitude of the auxiliary fiducial. Thus, the auxiliary fiducial 122 may facilitate the unmanned aerial vehicle to get sufficiently close to landing surface 110 to enable detection of the root fiducial 120.

[0034] FIG. 2 is an illustration of an example of an unmanned aerial vehicle 200 configured to autonomously land on a landing surface 110. The unmanned aerial vehicle 200 includes a propulsion mechanism 210, an image sensor 220, and a processing unit (e.g., processing unit 410) (not shown in FIG. 2). The unmanned aerial vehicle 200 also includes a battery in a battery pack 240 attached to the bottom of the unmanned aerial vehicle 200 and has conductive contacts 230 to enable battery charging. To aid in visualization of the intended connections, a close-up view of the landing surface 110 of FIG. 1 is reproduced in FIG. 2. For example, the process 600 of FIG. 6 may be used to land the unmanned aerial vehicle 200 on the landing surface 110 of the dock 100. For example, the process 900 of FIG. 9 may be used to land the unmanned aerial vehicle 200 on the landing surface 110 of the dock 100. The unmanned aerial vehicle 200 may include electronics as depicted in FIG. 4 to control the operation of the unmanned aerial vehicle 200 and its interaction with the dock 100.

[0035] The bottom surface of battery pack 240 is the bottom surface of unmanned aerial vehicle 200. Battery pack 240 is shaped to fit landing surface 110 with the bottom of a funnel shape. As unmanned aerial vehicle 200 makes final approach to landing surface 110, the bottom of battery pack 240 will contact landing surface 110 and will be mechanically guided by the tapered sides of the funnel to a location in the center of the bottom of the funnel. Once landing is complete, the conductive contacts of battery pack 240 may contact conductive contacts 130 on landing surface 110, making an electrical connection and allowing charging of the battery of unmanned aerial vehicle 200. Dock 100 may include a charger configured to charge the battery while unmanned aerial vehicle 200 is on landing surface 110.

[0036] Unmanned aerial vehicle 200 includes a propulsion mechanism 210. In this example, propulsion mechanism 210 includes propellers. For example, unmanned aerial vehicle 200 may be a quadcopter type drone. In some implementations, a processing device configured to automatically fold the propellers upon entering dock 100 may enable the dock to have a smaller footprint than the area swept out by the propellers of propulsion mechanism 210.

[0037] FIGURE 3A is an illustration of an example system 300 for facilitating autonomous landing of unmanned aerial vehicle 200. System 300 includes unmanned aerial vehicle 200 of FIGURE 2 and dock 100 of FIGURE 1. FIGURE 3A illustrates dock 100 in an orientation with door 142 open and retractable arm 150 extended to facilitate takeoff and landing. Unmanned aerial vehicle 200 is stationary on landing surface 110 of dock 100. The scene may be immediately prior to takeoff or immediately following landing.

[0038] After landing, retractable arms 150 may be retracted to retract unmanned aerial vehicle 200 into box 140 of dock 100, transitioning to the configuration depicted in FIG. 3B. A processing device (e.g., processing device 410) of unmanned aerial vehicle 200 may be configured to rotate and automatically fold the propellers while retracting retractable arms 150 to retract unmanned aerial vehicle 200 on landing surface 110 into box 140. For example, as unmanned aerial vehicle 200 slides into box 140, the slowly rotating propellers may be pressed against the sides of box 140, causing the propeller blades to fold together.

[0039] FIG. 3B is an illustration of an example of a system 300 for facilitating autonomous landing of unmanned aerial vehicle 200 with unmanned aerial vehicle 200 enclosed within dock 100. FIG. 3B depicts dock 100 in an arrangement with door 142 closed and retractable arms 150 retracted to facilitate storage and / or maintenance of unmanned aerial vehicle 200. Unmanned aerial vehicle 200 is at rest on landing surface 110, inside box 140 of dock 100. Dock 100 includes soft rollers 310 attached to an upper inner surface of box 140, which are positioned to engage the body of unmanned aerial vehicle 200 as unmanned aerial vehicle 200 is retracted by retractable arms 150 to a resting position inside box 140. Downward pressure from soft rollers 310 can push unmanned aerial vehicle 200 down onto landing surface 110 and hold it securely in place. This prevents unmanned aerial vehicle 200 from becoming dislodged within box 140 and disrupting the connection between unmanned aerial vehicle 200's battery and the charger in dock 100. This allows system 300 to be safely transported (e.g., in a car, truck, or other large vehicle).

[0040] 3B, in some implementations, dock 100 includes components configured to automatically clean parts (e.g., lenses) of unmanned aerial vehicle 200. For example, dock 100 may include one or more pneumatic blowers positioned to blow air onto a lens of unmanned aerial vehicle 200 covering image sensor 220 while unmanned aerial vehicle 200 is on landing surface 110. For example, dock 100 may include one or more spring-loaded microfiber wipers positioned to wipe clean a lens of unmanned aerial vehicle 200 covering image sensor 220 each time unmanned aerial vehicle 200 slides into or out of box 140.

[0041] FIG. 4 is a block diagram illustrating an example of a hardware configuration of unmanned aerial vehicle 400. The hardware configuration may include a processing unit 410, a data storage device 420, a sensor interface 430, a communication interface 440, a propulsion control interface 442, a user interface 444, and an interconnect 450 through which processing unit 410 may access other components. For example, unmanned aerial vehicle 400 may be an unmanned aerial vehicle (e.g., unmanned aerial vehicle 200) or may be part of an unmanned aerial vehicle. Unmanned aerial vehicle 400 may be configured to detect one or more fiducials on a dock (e.g., dock 100) and land on a small landing surface using an estimate of the attitude of the one or more fiducials to facilitate automated maintenance of unmanned aerial vehicle 400. For example, unmanned aerial vehicle 400 may be configured to perform process 600 of FIG. 6 and / or process 900 of FIG. 9.

[0042] The processing device 410 is operable to execute instructions stored in the data storage device 420. In some implementations, the processing device 410 is a processor with a random access memory for temporarily storing instructions read from the data storage device 420 while the instructions are being executed. The processing device 410 may include a single or multiple processors, each having a single or multiple processing cores. Alternatively, the processing device 410 may include another type of device, or multiple devices, capable of manipulating or processing data. For example, the data storage device 420 may be a non-volatile information storage device, such as a solid state drive, a read-only memory device (ROM), an optical disk, a magnetic disk, or any other suitable type of storage device, such as a non-transitory computer-readable memory. The data storage device 420 may include another type of device, or multiple devices, capable of storing data for retrieval or processing by the processing device 410. The processing device 410 may access and manipulate data stored in the data storage device 420 via the interconnect 450. For example, the data storage device 420 may store instructions executable by the processing device 410 that, when executed by the processing device 410, cause the processing device 410 to perform an operation (e.g., an operation to implement process 600 of FIG. 6 and / or process 900 of FIG. 9).

[0043] The sensor interface 430 may be configured to control and / or receive data (e.g., temperature measurements, pressure measurements, Global Positioning System (GPS) data, acceleration measurements, angular velocity measurements, magnetic flux measurements, and / or visible spectrum images) from one or more sensors (e.g., including the image sensor 220). In some implementations, the sensor interface 430 may implement a serial port protocol (e.g., I2C or SPI) for communication with one or more sensor devices over wires. In some implementations, the sensor interface 430 may include a wireless interface for communicating with one or more sensor groups via low-power, short-range communications (e.g., vehicle area network protocols).

[0044] The communication interface 440 facilitates communication with other devices, such as a paired dock (e.g., dock 100), a dedicated controller, or a user computing device (e.g., a smartphone or tablet). For example, the communication interface 440 may include a wireless interface that may facilitate communication over a Wi-Fi network, a Bluetooth link, or a ZigBee link. For example, the communication interface 440 may include a wired interface that may facilitate communication over a serial port (e.g., RS-232 or USB). The communication interface 440 facilitates communication over a network.

[0045] The propulsion control interface 442 may be used by the processing unit to control a propulsion system (e.g., including one or more propellers driven by electric motors). For example, the propulsion control interface 442 may include circuitry for converting digital control signals from the processing unit 410 into analog control signals for the actuators (e.g., the electric motors driving the respective propellers). In some implementations, the propulsion control interface 442 may implement a serial port protocol (e.g., I2C or SPI) for communication with the processing unit 410. In some implementations, the propulsion control interface 442 may include a wireless interface for communicating with one or more sensor groups via low-power, short-range communications (e.g., vehicle area network protocols).

[0046] The user interface 444 allows for the input and output of information from / to a user. In some implementations, the user interface 444 can include a display, which can be a liquid crystal display (LCD), a light emitting diode (LED) display (e.g., an OLED display), or other suitable display. For example, the user interface 444 can include a touch screen. For example, the user interface 444 can include buttons. For example, the user interface 444 can include a position input device, such as a touch pad, a touch screen, or other suitable human or machine interface device.

[0047] For example, interconnect 450 may be a system bus or a wired or wireless network (e.g., a vehicle area network). In some implementations (not shown in FIG. 4), some components of unmanned aerial vehicle 400 may be omitted, such as user interface 444.

[0048] 5 is a block diagram illustrating an example of a hardware configuration of a dock 500 for an unmanned aerial vehicle. The hardware configuration may include a processing unit 510, a data storage device 520, a sensor interface 530, a communication interface 540, an actuator control interface 542, a user interface 544, and an interconnect 550 through which the processing unit 510 may access other components. For example, the dock 500 may be a dock (e.g., dock 100) or may be part of a dock. The dock 500 may be configured to accept an unmanned aerial vehicle (e.g., unmanned aerial vehicle 200) on a small landing surface to facilitate automated maintenance (e.g., charging and / or cleaning) of the unmanned aerial vehicle.

[0049] The processing device 510 is operable to execute instructions stored in the data storage device 520. In some implementations, the processing device 510 is a processor with a random access memory for temporarily storing instructions read from the data storage device 520 while the instructions are being executed. The processing device 510 may include a single or multiple processors, each having a single or multiple processing cores. Alternatively, the processing device 510 may include another type of device, or multiple devices, capable of manipulating or processing data. For example, the data storage device 520 may be a non-volatile information storage device, such as a hard drive, a solid state drive, a read-only memory device (ROM), an optical disk, a magnetic disk, or any other suitable type of storage device, such as a non-transitory computer-readable memory. The data storage device 520 may include another type of device, or multiple devices, capable of storing data for retrieval or processing by the processing device 510. The processing device 510 may access and manipulate data stored in the data storage device 520 via the interconnect 550. For example, data storage device 520 may store instructions executable by processing device 510 that, when executed by processing device 510, cause processing device 510 to perform an action (e.g., open or close a door, extend or retract a retractable arm, charge a battery, or clean a portion of the UAV).

[0050] The sensor interface 530 may be configured to control and / or receive data (e.g., temperature measurements, pressure measurements, Global Positioning System (GPS) data, acceleration measurements, angular velocity measurements, magnetic flux measurements, and / or visible spectrum images) from one or more sensors (e.g., including the image sensor 220). In some implementations, the sensor interface 530 may implement a serial port protocol (e.g., I2C or SPI) for communication with one or more sensor devices over wires. In some implementations, the sensor interface 530 may include a wireless interface for communicating with one or more sensor groups via low-power, short-range communications (e.g., vehicle area network protocols).

[0051] The communications interface 540 facilitates communication with other devices, such as a paired UAV (e.g., unmanned aerial vehicle 200), a dedicated controller, or a user computing device (e.g., a smartphone or tablet). For example, the communications interface 540 may include a wireless interface that may facilitate communication over a Wi-Fi network, a Bluetooth link, or a ZigBee link. For example, the communications interface 540 may include a wired interface that may facilitate communication over a serial port (e.g., RS-232 or USB). The communications interface 540 facilitates communication over a network.

[0052] The actuator control interface 542 may be used by the processor to control one or more actuators (e.g., an electric motor that drives the retractable arm 150). For example, the actuator control interface 542 may include circuitry for converting digital control signals from the processor 510 into analog control signals for the actuators (e.g., an electric motor that drives the opening and closing of the door 142 or a pneumatic sprayer). In some implementations, the actuator control interface 542 may implement a serial port protocol (e.g., I2C or SPI) for communication with the processor 510. In some implementations, the actuator control interface 542 may include a wireless interface for communicating with one or more sensor groups via low-power, short-range communication.

[0053] The user interface 544 allows for the input and output of information from / to a user. In some implementations, the user interface 544 can include a display, which can be a liquid crystal display (LCD), a light emitting diode (LED) display (e.g., an OLED display), or other suitable display. For example, the user interface 544 can include a touch screen. For example, the user interface 544 can include buttons. For example, the user interface 544 can include a positional input device, such as a touch pad, a touch screen, or other suitable human or machine interface device.

[0054] For example, interconnect 550 may be a system bus or a wired or wireless network (e.g., a vehicle area network). In some implementations (not shown in FIG. 5), some components of dock 500 may be omitted, such as user interface 544.

[0055] 6 is a flow chart of an example of a process 600 for autonomously landing an unmanned aerial vehicle using at least one fiducial on a landing surface. Process 600 includes controlling 610 the unmanned aerial vehicle to fly to a first location proximate a dock including a landing surface for the unmanned aerial vehicle having the fiducial; accessing 620 one or more images taken with an image sensor of the unmanned aerial vehicle; detecting 630 the fiducial in at least one of the one or more images; determining 640 an attitude of the fiducial based on the one or more images; and controlling 650 a propulsion mechanism for the unmanned aerial vehicle to land on the landing surface based on the attitude of the fiducial. For example, process 600 may be performed by unmanned aerial vehicle 200. For example, process 600 may be performed by unmanned aerial vehicle 400.

[0056] The process 600 includes controlling 610 a propulsion mechanism of the unmanned aerial vehicle (e.g., unmanned aerial vehicle 200) such that the unmanned aerial vehicle flies to a first location proximate a dock (e.g., funnel nest or dock 100) that includes a landing surface (e.g., landing surface 110) configured to hold the unmanned aerial vehicle, and a fiducial (e.g., fiducial 120) on the landing surface. For example, the unmanned aerial vehicle can store GPS coordinates of the dock and navigate to a location proximate the dock 100 using GPS data and / or visual inertial odometry. The resolution of these localization techniques may be insufficient to localize with sufficient accuracy to land on the dock's small landing surface, but may be sufficient to get within view of the dock or associated auxiliary fiducials. In some implementations, the location of the dock in the navigation frame can be used to store the corresponding GPS (lat, long) coordinate. This coordinate is global and therefore less susceptible to VT0 drift. When returning to the dock, the drone may first fly to GPS coordinates (~3-5m accuracy) and then use visual detection of one or more fiducials to further hone into the dock. For example, a processing unit (e.g., processing unit 410) may control 610 a propulsion system (e.g., one or more propellers driven by an electric motor) using a propulsion control interface (e.g., propulsion control interface 442).

[0057] The process 600 includes accessing 620 one or more images taken with an image sensor (e.g., image sensor 220) of the unmanned aerial vehicle. For example, the one or more images may include a series of images (e.g., frames of a video) taken as the unmanned aerial vehicle flies toward the landing surface of the dock. For example, the one or more images may include infrared images and / or visible spectrum images. The one or more images may undergo various amounts of processing prior to accessing 620. For example, pre-processing of the one or more images may combine image data from multiple sources (e.g., stitching images from multiple image sensors on the unmanned aerial vehicle). The one or more images may be accessed 620 in various ways. For example, the one or more images may be accessed 620 by directly reading from an image sensor via a sensor interface (e.g., sensor interface 430) or by directly reading from a memory (e.g., data storage device 420) via an interconnect (e.g., interconnect 450).

[0058] The process 600 includes detecting 630 a fiducial in at least one of the one or more images. For example, computer vision techniques may be applied in detecting 630 a fiducial in one of the one or more images. In some implementations, one or more additional auxiliary fiducials at a larger scale may be associated with the dock and used for visual positioning to navigate closer to the dock and get close enough to capture a view of the root fiducial, which is the smallest fiducial located directly on the small landing surface. A use case for auxiliary fiducials at a larger scale is described in connection with FIG. 9.

[0059] The process 600 includes determining 640 the attitude of the fiducial based on one or more images. The attitude can be relative to the unmanned aerial vehicle. In some implementations, the coordinate frame used to estimate the attitude of the fiducial is the same navigation frame estimated using a visual inertial odometry (VIO) system (e.g., base imagery and inertial measurement unit data) running for normal flight. This can provide a low-drift inertial frame to which the camera images are aligned. For example, detection 630 on dock-specific fiducials (e.g., visual markers) may occur in the images of each camera, and a calibrated camera model is used to project those points into the navigation frame. When one or many fiducials are detected in the images, their three-dimensional positions and orientations are added to a dock-specific optimization problem that simultaneously optimizes a small "root" fiducial directly beneath the unmanned aerial vehicle's intended final resting position, and the relative three-dimensional transformations (e.g., rotation and translation) between any auxiliary fiducials and the root fiducial. In some implementations, the poses of the fiducials (e.g., visual tags) and their relative translations are added as variables to a nonlinear optimization program. Upon new observations of one or more fiducials, the optimization further refines these poses and translations. The nonlinear optimization program can be implemented as a nonlinear estimation filter that uses visual tag data, visual inertial odometry (VIO) navigation system data, and GPS navigation system data to solve for the tag and vehicle poses and translations. If the drone flies for an extended period of time without seeing any of the fiducials, the navigation frame is prone to drift. In a joint optimization problem, with only one fiducial observation, the positions of all the fiducials can be recovered, most importantly the position of the root fiducial on the landing surface. The size of the fiducials (e.g., visual tags) can be designed to provide different detection distances. The "root" fiducial (e.g., fiducial 120) is the smallest.In some implementations, the landing procedure requires a certain threshold of fresh detection of the root fiducial itself to confirm landing. The end result is an updated estimate of the relative attitude of the root fiducial, which can be used to determine an attitude for the unmanned aerial vehicle, which can be used to land the unmanned aerial vehicle on a landing surface where the root fiducial was marked (e.g., landing surface 110).

[0060] Process 600 includes controlling 650 a propulsion mechanism to land the unmanned aerial vehicle on a landing surface based on the attitude of the fiducial. In some implementations, a honing procedure using a special state machine is used to land the unmanned aerial vehicle on the landing surface. For example, process 700 of FIG. 7 may be used to land the unmanned aerial vehicle. For example, a processing unit (e.g., processing unit 410) may use a propulsion control interface (e.g., propulsion control interface 442) to control 650 a propulsion system (e.g., one or more propellers driven by electric motors) to land. In some implementations, the dock includes a retractable arm (e.g., retractable arm 150), and the landing surface is positioned at the end of the retractable arm, which may help reduce propeller wash experienced during the landing operation.

[0061] FIG. 7 is a flow chart of an example of a process 700 for autonomously landing an unmanned aerial vehicle. Process 700 includes controlling 710 the propulsion mechanism to fly to a location above a landing surface; controlling 720 the propulsion mechanism to descend the unmanned aerial vehicle toward a fiducial on the landing surface; in response to reaching a predetermined height above the landing surface, controlling 730 the propulsion mechanism to hover at a predetermined height above the landing surface until an error estimate for the attitude and velocity of the unmanned aerial vehicle satisfies a stability condition; and in response to the stability condition being satisfied, controlling 740 the propulsion mechanism to make a final approach to touch down on the landing surface. For example, process 700 may be used to refine an estimate of the attitude of a route fiducial as it approaches a final landing maneuver. For example, process 700 may be implemented by unmanned aerial vehicle 200. For example, process 700 may be implemented by unmanned aerial vehicle 400.

[0062] Process 700 includes controlling 710 a propulsion mechanism to fly the unmanned aerial vehicle (e.g., unmanned aerial vehicle 200) to a location above a landing surface (e.g., landing surface 110). For example, a processing device (e.g., processing device 410) may use a propulsion control interface (e.g., propulsion control interface 442) to control 650 a propulsion system (e.g., one or more propellers driven by an electric motor) to fly to a location above the landing surface.

[0063] Process 700 includes controlling 720 a propulsion mechanism such that the unmanned aerial vehicle descends toward a fiducial (e.g., fiducial 120) on the landing surface. For example, a processing device (e.g., processing device 410) may use a propulsion control interface (e.g., propulsion control interface 442) to control 650 a propulsion mechanism (e.g., one or more propellers driven by electric motors) to descend toward the fiducial on the landing surface. For example, the fiducial may be a root fiducial, and in some implementations, detection of the auxiliary fiducial continues to be used to update a joint model of the attitude of the root fiducial along with detection of the root fiducial. In some implementations, during a descent when the root fiducial is fully visible, only detection of the root fiducial is used to update an estimate of the attitude of the root fiducial used for landing, and detection of larger auxiliary fiducials is ignored. In some implementations, during the descent phase, the unmanned aerial vehicle slowly approaches the route fiducial and monitors the health of the detection system (e.g., the estimated variance of the dock location, when the most recent visual detection was on the auxiliary tag, and / or the timing and number of detections on the route tag).

[0064] Process 700 includes, in response to reaching a predetermined height above the landing surface, controlling a propulsion mechanism such that the unmanned aerial vehicle hovers at a predetermined height (e.g., 5 centimeters) above the landing surface until error estimates for the attitude and velocity of the unmanned aerial vehicle satisfy a stability condition. Because the final approach for landing is a critical phase of the landing process, it may be beneficial to wait for state estimates to converge in favorable conditions before making the final approach. In some implementations, during the hover phase, the unmanned aerial vehicle waits near a route fiducial so that errors in the position, velocity, and orientation of the unmanned aerial vehicle are small enough to transition to final approach.

[0065] Process 700 includes controlling 740 the propulsion mechanisms to cause the unmanned aerial vehicle to perform a final approach to touch down on a landing surface in response to the stability condition being satisfied. For example, the final approach sequence may be aimed at completing a landing on a landing surface of a dock faster than a descent. For example, process 800 of FIG. 8 may be performed when performing a final approach.

[0066] FIG. 8 is a flow chart of an example of a process 800 for performing a final approach to autonomously land an unmanned aerial vehicle. There are certain cues that can be used to detect the difference between a proper dock landing and a near miss / crash landing. In general, landing detection is done by calculating the difference between the aerodynamic forces expected on the vehicle due to the drone's rotor rate and the forces estimated by the unmanned aerial vehicle's inertial measurement unit (IMU). The forces imparted by the dock to the body of the unmanned aerial vehicle can be passively sensed as the landing surface (e.g., landing surface 110) hugs the drone in a certain way. In some implementations, torques can also be actively injected into the vehicle body through a control system to look for vehicle motion to better sense the contours of the landing surface.

[0067] Process 800 includes monitoring 810 inertial measurements (e.g., acceleration and angular velocity) of the unmanned aerial vehicle while final approach is being made to ascertain whether the landing surface has been properly engaged. The impact of contact with the landing surface (e.g., landing surface 110) produces characteristic forces on the unmanned aerial vehicle, and these forces may be reflected in the inertial data. For example, the inertial measurement data may be compared to thresholds corresponding to expected ranges of these parameters associated with proper engagement of the landing surface. For example, a machine learning module (e.g., a neural network) may be trained to distinguish whether the landing surface has been properly engaged based on the inertial measurement data.

[0068] If the landing surface is not properly engaged (step 825), abort 830 the final approach. If the landing surface is properly engaged (step 825), apply torque 840 to the unmanned aerial vehicle using the propulsion system and check 855 whether the unmanned aerial vehicle has stopped rolling. If the rolling has not stopped (step 855), abort 830 the final approach; if the rolling has stopped (e.g., by a corner of the bottom of the funnel of the landing surface 110), complete 860 the successful landing procedure.

[0069] After the unmanned aerial vehicle lands on the landing surface, the dock (e.g., dock 100) and the unmanned aerial vehicle may coordinate to complete 860 a landing procedure (e.g., to store and shut down the unmanned aerial vehicle and / or to initiate automated maintenance services (e.g., charging and / or cleaning) for the unmanned aerial vehicle). For example, the dock may include a box configured to enclose the landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock. The dock may be configured to automatically transition the dock from the first configuration to the second configuration by performing steps including opening a door of the box and extending a retractable arm to move the landing surface from inside the box to outside the box. In some implementations, the propulsion mechanism includes propellers, and the post-landing procedure includes automatically folding the propellers by rotating the propellers while the retractable arm is retracted to retract the unmanned aerial vehicle on the landing surface into the box. For example, the post-landing procedure may include automatically charging a battery of the unmanned aerial vehicle with a charger included in the dock while the unmanned aerial vehicle is on the landing surface. For example, the post-landing procedure may include automatically cleaning a lens of the unmanned aerial vehicle covering an image sensor by blowing air at the lens using a pneumatic blower positioned on the dock while the unmanned aerial vehicle is on the landing surface.

[0070] Process 800 may include, in response to determining that the landing surface is not properly engaged, aborting 830 the final approach by controlling the propulsion mechanisms so that the unmanned aerial vehicle ascends. For example, aborting 830 the landing may include turning the control system back on and commanding a desired position above the dock. This may be facilitated by a variety of recovery methods, such as an abort sequence in which the unmanned aerial vehicle leaves the vicinity of the dock and executes another preprogrammed plan (e.g., landing on the ground next to the dock, going to another location, or going to another dock).

[0071] 9 is a flow chart of an example of a process 900 for autonomously landing an unmanned aerial vehicle using a sequence of fiducials at different scales. The process 900 includes controlling 910 the unmanned aerial vehicle to fly to a first location proximate a dock including a landing surface for the unmanned aerial vehicle having the fiducial; accessing 920 one or more images taken with an image sensor of the unmanned aerial vehicle; detecting an auxiliary fiducial in at least one of the one or more images 930; determining an attitude of the auxiliary fiducial based on the one or more images 940; controlling 950 a propulsion mechanism to fly to the first location proximate the landing surface based on the attitude of the auxiliary fiducial; detecting 960 a root fiducial for the landing surface in at least one of the one or more images, determining 970 an attitude of the root fiducial based on the one or more images; and controlling 980 a propulsion mechanism to land the unmanned aerial vehicle on the landing surface based on the attitude of the root fiducial. For example, the process 900 may be performed by the unmanned aerial vehicle 200. For example, process 900 may be performed by unmanned aerial vehicle 400.

[0072] The process 900 includes controlling 910 a propulsion mechanism of the unmanned aerial vehicle (e.g., unmanned aerial vehicle 200) to fly the unmanned aerial vehicle to a first location proximate a dock (e.g., funnel nest or dock 100) that includes a landing surface (e.g., landing surface 110) configured to hold the unmanned aerial vehicle, and a fiducial (e.g., fiducial 120) on the landing surface. For example, the unmanned aerial vehicle can store GPS coordinates of the dock and navigate to a location proximate the dock 100 using GPS data and / or visual inertial odometry. The resolution of these location techniques may be insufficient to locate with sufficient accuracy to land on the dock's small landing surface, but may be sufficient to get within view of the dock or associated auxiliary fiducials. In some implementations, the location of the dock in the navigation frame can be used to store the corresponding GPS (latitude, longitude) coordinates. Because this coordinate is global, it is less susceptible to VIO drift. When returning to the dock, the drone may first fly to GPS coordinates (~3-5 m accuracy) and then use visual detection of one or more fiducials to further target the dock. For example, a processing unit (e.g., processing unit 410) may control 910 a propulsion system (e.g., one or more propellers driven by an electric motor) using a propulsion control interface (e.g., propulsion control interface 442).

[0073] The process 900 includes accessing 920 one or more images taken with an image sensor (e.g., image sensor 220) of the unmanned aerial vehicle. For example, the one or more images may include a series of images (e.g., frames of a video) taken as the unmanned aerial vehicle flies toward the landing surface of the dock. For example, the one or more images may include infrared images and / or visible spectrum images. The one or more images may undergo various amounts of processing prior to accessing 920. For example, pre-processing of the one or more images may combine image data from multiple sources (e.g., stitching images from multiple image sensors on the unmanned aerial vehicle). The one or more images may be accessed 920 in various ways. For example, the one or more images may be accessed 920 by directly reading from an image sensor via a sensor interface (e.g., sensor interface 430) or by directly reading from a memory (e.g., data storage device 420) via an interconnect (e.g., interconnect 450).

[0074] The process 900 includes detecting 930 an auxiliary fiducial (e.g., auxiliary fiducial 122) in at least one of the one or more images. The auxiliary fiducial may be larger than the root fiducial, and the root fiducial is located on the landing surface of the dock. For example, computer vision techniques may be applied to detect 930 the auxiliary fiducial in one of the one or more images. For example, the auxiliary fiducial may be larger than the root fiducial, which may be used for visual positioning to approach and navigate to the dock and get close enough to capture a view of the root fiducial, which is the smallest fiducial located directly on the small landing surface. In some implementations, multiple auxiliary fiducials (e.g., visual tags) at different scales may be used to iteratively get closer and closer to the dock, eventually zooming in to a direct view of the root fiducial showing a mini landing pad on the dock and the final target for landing.

[0075] The process 900 includes determining 940 the attitude of the auxiliary fiducial based on one or more images. The attitude may be relative to the unmanned aerial vehicle. In some implementations, the coordinate frame used to estimate the attitude of the fiducial is the same navigation frame estimated using a visual inertial odometry (VIO) system (e.g., base imagery and inertial measurement unit data) running for normal flight. This may provide a low-drift inertial frame to which the camera images are aligned. For example, detection 930 on dock-specific fiducials (e.g., visual markers) may occur in the respective camera images, and a calibrated camera model is used to project those points into the navigation frame. If one or many fiducials are detected 930 in the images, their three-dimensional positions and orientations may be added to a dock-specific optimization problem that simultaneously optimizes a small "root" fiducial directly beneath the unmanned aerial vehicle's intended final resting position, and the relative three-dimensional transformations (e.g., rotations and translations) between any auxiliary fiducials and the root fiducial. In some implementations, the poses of the fiducials (e.g., visual tags) and their relative translations are added as variables to a nonlinear optimization program. Upon new observations of one or more fiducials, the optimization further refines these poses and translations. The nonlinear optimization program can be implemented as a nonlinear estimation filter that uses visual tag data, visual inertial odometry (VIO) navigation system data, and GPS navigation system data to solve for the tag and vehicle poses and translations. If the drone flies for an extended period of time without seeing any of the fiducials, the navigation frame is prone to drift. In a joint optimization problem, with only one fiducial observation, the positions of all the fiducials can be recovered, most importantly the position of the root fiducial on the landing surface. The size of the fiducials (e.g., visual tags) can be designed to provide different detection distances. The "root" fiducial (e.g., fiducial 120) is the smallest.

[0076] Process 900 includes controlling 950 a propulsion mechanism to fly the unmanned aerial vehicle to a location proximate the landing surface based on the attitude of the auxiliary fiducial. For example, a processing unit (e.g., processing unit 410) can use a propulsion control interface (e.g., propulsion control interface 442) to control 950 a propulsion mechanism (e.g., one or more propellers driven by electric motors) to fly to a location proximate the landing surface. In some implementations, the dock includes a box (e.g., box 140) configured to enclose the landing surface (e.g., landing surface 110) in a first configuration (e.g., as shown in FIG. 3B) and expose the landing surface in a second configuration (e.g., as shown in FIG. 3A), and the auxiliary fiducial (e.g., auxiliary fiducial 122) is disposed on an exterior surface of the box.

[0077] The process 900 includes detecting 960 root fiducials on the landing surface in at least one of the one or more images. For example, the root fiducials may be detected 960 as described in connection with step 630 of the process 600 of FIG.

[0078] The process 900 includes determining 970 a pose of the root fiducial based on the one or more images. For example, the pose of the root fiducial may be determined 970 as described in connection with step 640 of the process 600 of FIG.

[0079] Process 900 includes controlling 980 a propulsion mechanism to land the unmanned aerial vehicle on a landing surface based on the attitude of the root fiducial. For example, the propulsion mechanism may be controlled 980 based on the attitude of the root fiducial to land the unmanned aerial vehicle on the landing surface as described in connection with step 650 of process 600 of Figure 6. For example, controlling 980 a propulsion mechanism to land the unmanned aerial vehicle on the landing surface based on the attitude of the root fiducial may include performing process 700 of Figure 7.

[0080] In a first aspect, the subject matter described herein may be embodied in a system including an unmanned aerial vehicle including a propulsion mechanism, an image sensor, and a processing device; a landing surface configured to hold the unmanned aerial vehicle, and a dock including a fiducial on the landing surface, where the processing device is configured to: control the propulsion mechanism to fly the unmanned aerial vehicle to a first location in the vicinity of the dock; access one or more images taken using the image sensor; detect the fiducial in at least one of the one or more images; determine an attitude of the fiducial based on the one or more images; and control the propulsion mechanism to land the unmanned aerial vehicle on the landing surface based on the attitude of the fiducial.

[0081] In a first aspect, the fiducial may be a root fiducial, and the system may further include an auxiliary fiducial larger than the root fiducial, and the processing device may be configured to: detect the auxiliary fiducial in at least one of the one or more images; determine an attitude of the auxiliary fiducial based on the one or more images; and control the propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the landing surface based on the attitude of the auxiliary fiducial. The dock may include a box configured to enclose the landing surface in a first configuration and expose the landing surface in a second configuration. The auxiliary fiducial may be disposed on an exterior surface of the box. In a first embodiment, the processing device may be configured to: control the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; control the propulsion mechanism to cause the unmanned aerial vehicle to descend toward a fiducial on the landing surface; in response to reaching a predetermined height above the landing surface, control the propulsion mechanism to cause the unmanned aerial vehicle to hover at a predetermined height above the landing surface until an error estimate of the attitude and velocity of the unmanned aerial vehicle satisfies a stability condition; in response to the stability condition being satisfied, control the propulsion mechanism to cause the unmanned aerial vehicle to perform a final approach to touch down on the landing surface. In the first embodiment, the processing device may be configured to: monitor inertial measurements of the unmanned aerial vehicle while the final approach is being performed to ascertain whether the landing surface has been properly engaged; in response to a determination that the landing surface has not been properly engaged, abort the final approach by controlling the propulsion mechanism to lift the unmanned aerial vehicle. In the first embodiment, the dock may include a retractable arm, and the landing surface is positioned at an end of the retractable arm. For example, the dock includes a box configured to enclose a landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock, and the dock is configured to automatically transition from the first configuration to the second configuration by performing steps including opening a door of the box and extending a retractable arm to move the landing surface from inside the box to outside the box.In a first embodiment, the propulsion mechanism may include a propeller, and the processing device may be configured to: automatically fold the propeller by rotating the propeller while the retractable arms are retracted to retract the unmanned aerial vehicle on the landing surface into the box. In a first embodiment, the dock may include soft rollers attached to an upper inner surface of the box and positioned to engage a body of the unmanned aerial vehicle as the unmanned aerial vehicle is retracted by the retractable arms into a resting position within the box. In a first embodiment, the landing surface may have a funnel shape with a shape that fits the bottom of the unmanned aerial vehicle at a bottom of the funnel. For example, the dock may include conductive contacts of a battery charger on the landing surface positioned at the bottom of the funnel. In a first embodiment, the unmanned aerial vehicle may include a battery, and the dock may include a charger configured to charge the battery while the unmanned aerial vehicle is on the landing surface. In a first embodiment, the dock may include one or more pneumatic blowers positioned to blow air on a lens of the unmanned aerial vehicle covering an image sensor while the unmanned aerial vehicle is on the landing surface. The first aspect may include any combination of the features described in this paragraph.

[0082] In a second aspect, the subject matter described herein may be embodied in a method that includes controlling a propulsion mechanism of an unmanned aerial vehicle to fly the unmanned aerial vehicle to a first location proximate a dock including a landing surface configured to hold the unmanned aerial vehicle and a fiducial on the landing surface; accessing one or more images taken using an image sensor of the unmanned aerial vehicle; detecting the fiducial in at least one of the one or more images; determining an attitude of the fiducial based on the one or more images; and controlling the propulsion mechanism to land the unmanned aerial vehicle on the landing surface based on the attitude of the fiducial.

[0083] In a second aspect, the fiducial may be a root fiducial, and the method may include: detecting an auxiliary fiducial in at least one of the one or more images, where the auxiliary fiducial is larger than the root fiducial; determining an attitude of the auxiliary fiducial based on the one or more images, and controlling a propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the landing surface based on the attitude of the auxiliary fiducial. For example, the dock may include a box configured to enclose the landing surface in a first configuration and to expose the landing surface in a second configuration, and the auxiliary fiducial may be disposed on an exterior surface of the box. In a second aspect, the method may include controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to descend toward a fiducial on the landing surface, and in response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to hover at a predetermined height above the landing surface until an error estimate of the attitude and velocity of the unmanned aerial vehicle satisfies a stability condition, and in response to the stability condition being satisfied, controlling the propulsion mechanism to cause the unmanned aerial vehicle to perform a final approach to touch down on the landing surface. In the second aspect, the method may include monitoring inertial measurements of the unmanned aerial vehicle while the final approach is being performed to ascertain whether the landing surface has been properly engaged, and in response to determining that the landing surface has not been properly engaged, aborting the final approach by controlling the propulsion mechanism to lift the unmanned aerial vehicle. In the second aspect, the dock may include a retractable arm, and the landing surface may be positioned at an end of the retractable arm. For example, the dock may include a box configured to enclose the landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock, and the method may further include automatically transitioning the dock from the first configuration to the second configuration by performing steps including: opening a door of the box and extending a retractable arm to move the landing surface from inside the box to outside the box.For example, the propulsion mechanism may include a propeller, and the method may include automatically folding the propeller by rotating the propeller while the retractable arm is retracted to pull the unmanned aerial vehicle on the landing surface into the box. In a second aspect, the method may include automatically charging a battery of the unmanned aerial vehicle with a charger included in the dock while the unmanned aerial vehicle is on the landing surface. In a second aspect, the method may include automatically cleaning a lens of the unmanned aerial vehicle covering an image sensor by blowing air on the lens using an air pressure blower positioned in the dock while the unmanned aerial vehicle is on the landing surface. The second aspect may include any combination of the features described in this paragraph.

[0084] In a third aspect, the subject matter described herein may be embodied in a non-transitory computer-readable storage medium including instructions that, when executed by a processor, facilitate performance of operations including: controlling a propulsion mechanism of the unmanned aerial vehicle so that the unmanned aerial vehicle flies to a first location in the vicinity of a dock including a landing surface configured to hold the unmanned aerial vehicle and a fiducial on the landing surface; accessing one or more images taken using an image sensor of the unmanned aerial vehicle; detecting the fiducial in at least one of the one or more images; determining an attitude of the fiducial based on the one or more images; and controlling the propulsion mechanism so that the unmanned aerial vehicle lands on the landing surface based on the attitude of the fiducial.

[0085] In a third aspect, the fiducial may be a root fiducial, and the non-transitory computer-readable storage medium may include instructions that, when executed by the processor, facilitate performance of an operation including: detecting an auxiliary fiducial in at least one of the one or more images, where the auxiliary fiducial is larger than the root fiducial; determining an attitude of the auxiliary fiducial based on the one or more images; and controlling the propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the landing surface based on the attitude of the auxiliary fiducial. In a third aspect, the dock includes a box configured to enclose the landing surface in a first configuration and expose the landing surface in a second configuration, and the auxiliary fiducial may be disposed on an exterior surface of the box. In a third aspect, the non-transitory computer readable storage medium may include instructions that, when executed by a processor, facilitate performance of an operation including: controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; controlling the propulsion mechanism to cause the unmanned aerial vehicle to descend toward a fiducial on the landing surface; in response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to hover at a predetermined height above the landing surface until an error estimate of the attitude and velocity of the unmanned aerial vehicle satisfies a stability condition; and in response to the stability condition being satisfied, controlling the propulsion mechanism to cause the unmanned aerial vehicle to make a final approach to touch down on the landing surface. In a third aspect, the non-transitory computer readable storage medium may include instructions that, when executed by a processor, facilitate performance of an operation including: monitoring inertial measurements of the unmanned aerial vehicle while the final approach is being made to ascertain whether the landing surface has been properly engaged; and in response to a determination that the landing surface has not been properly engaged, aborting the final approach by controlling the propulsion mechanism to lift the unmanned aerial vehicle. In a third aspect, the dock may include a retractable arm and the landing surface may be positioned at an end of the retractable arm. The third aspect may include any combination of the features described in this paragraph.

[0086] In a fourth aspect, the subject matter described herein may be embodied in a method that includes accessing one or more images taken using an image sensor of the unmanned aerial vehicle; detecting an auxiliary fiducial in at least one of the one or more images; determining an attitude of the auxiliary fiducial based on the one or more images; controlling a propulsion mechanism of the unmanned aerial vehicle to fly the unmanned aerial vehicle to a first position near the root fiducial based on the attitude of the auxiliary fiducial; detecting a root fiducial in at least one of the one or more images taken from the first position, where the auxiliary fiducial is larger than the root fiducial; determining an attitude of the root fiducial based on the one or more images; and controlling a propulsion mechanism of the unmanned aerial vehicle to land on the root fiducial based on the attitude of the root fiducial.

[0087] In a fourth aspect, the route fiducial may be on the landing surface, and the method may include: controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; controlling the propulsion mechanism to cause the unmanned aerial vehicle to perform a descent toward the route fiducial on the landing surface; in response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to hover at a predetermined height above the landing surface until attitude and velocity error estimates of the unmanned aerial vehicle satisfy a stability condition; and in response to the stability condition being satisfied, controlling the propulsion mechanism to cause the unmanned aerial vehicle to perform a final approach to touch down on the landing surface. In the fourth aspect, the method may include: monitoring inertial measurements of the unmanned aerial vehicle while the final approach is being performed to verify whether the landing surface has been properly engaged; in response to determining that the landing surface has not been properly engaged, aborting the final approach by controlling the propulsion mechanism to lift the unmanned aerial vehicle. For example, the dock may include a box configured to enclose the landing surface in a first configuration and expose the landing surface in a second configuration, and the auxiliary fiducial may be disposed on an exterior surface of the box. In a fourth aspect, the dock may include a retractable arm, and the landing surface may be positioned at an end of the retractable arm. For example, the dock may include a box configured to enclose the landing surface in a first configuration of the dock and expose the landing surface in a second configuration of the dock, and the method may further include automatically transitioning the dock from the first configuration to the second configuration by performing steps including: opening a door of the box, and extending the retractable arm to move the landing surface from inside the box to outside the box. For example, the propulsion mechanism may include a propeller, and the method may include automatically folding the propeller by rotating the propeller while the retractable arm is retracted to retract the unmanned aerial vehicle on the landing surface into the box. In a fourth aspect, the method may include automatically charging a battery of the unmanned aerial vehicle with a charger included in the dock while the unmanned aerial vehicle is on the landing surface.In a fourth aspect, the method may include automatically cleaning a lens of the unmanned aerial vehicle covering the image sensor by blowing air on the lens while the unmanned aerial vehicle is on the landing surface using an air pressure blower positioned in the dock. The fourth aspect may include any combination of the features described in this paragraph.

[0088] Although the present disclosure has been described in connection with particular embodiments, the present disclosure is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope should be understood to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures.

Claims

1. an unmanned aerial vehicle including a propulsion mechanism, an image sensor, and a processing unit; a landing surface configured to hold the unmanned aerial vehicle, and a dock including a fiducial on the landing surface; The system includes a processor, controlling a propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the dock; Accessing one or more images captured using the image sensor; Detecting a fiducial in at least one of the one or more images; determining a pose of the fiducial based on the one or more images; Based on the attitude of the fiducial, the propulsion mechanism is controlled so that the unmanned aerial vehicle lands on the landing surface. The system is configured as follows.

2. The fiducial is a root fiducial and further includes an auxiliary fiducial that is larger than the root fiducial, and the processing device: Detecting an auxiliary fiducial in at least one of the one or more images; determining a pose of the auxiliary fiducial based on the one or more images; Controlling the propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the landing surface based on the attitude of the auxiliary fiducial. The system of claim 1 configured to:

3. 3. The system of claim 2, wherein the dock includes a box configured to enclose the landing surface in a first configuration and expose the landing surface in a second configuration, and the auxiliary fiducials are disposed on an exterior surface of the box.

4. The processing device comprises: controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; controlling a propulsion mechanism so that the unmanned aerial vehicle descends toward a fiducial on a landing surface; In response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to hover at the predetermined height above the landing surface until the attitude and velocity error estimates of the unmanned aerial vehicle satisfy a stability condition; Responsive to the stability condition being satisfied, control the propulsion mechanism to cause the unmanned aerial vehicle to make a final approach to touch down on the landing surface. The system according to claim 1 , configured to:

5. The processing device comprises: While final approach is underway, inertial measurements of the unmanned aerial vehicle are monitored to determine whether the landing surface has been properly engaged; In response to determining that the landing surface is not properly engaged, aborting the final approach by controlling the propulsion mechanism to lift the unmanned aerial vehicle. The system according to claim 4 , configured to:

6. 6. The system of claim 1, wherein the dock includes a retractable arm, the landing surface being positioned at the end of the retractable arm.

7. the dock includes a box configured to enclose the landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock; 7. The system of claim 6, wherein the dock is configured to automatically transition from the first configuration to the second configuration by performing steps including opening a door of the box and extending the retractable arm to move the landing surface from inside the box to outside the box.

8. The propulsion mechanism includes a propeller, and the processing device includes: Automatically fold the propellers by rotating the propellers while the retractable arms are retracted to pull the drone on the landing surface into the box The system of claim 7 configured to:

9. 9. The system of claim 7 or 8, wherein the dock includes soft rollers attached to an upper inside surface of the box and positioned to engage the body of the unmanned aerial vehicle as the unmanned aerial vehicle is pulled into a resting position within the box by the retractable arm.

10. 10. The system of claim 1, wherein the landing surface has a funnel shape that is shaped to fit the bottom of the unmanned aerial vehicle at the bottom of the funnel.

11. The system of claim 10 , wherein the dock includes conductive contacts for a battery charger on a landing surface positioned at the bottom of the funnel.

12. 12. The system of claim 1, wherein the unmanned aerial vehicle includes a battery, and the dock includes a charger configured to charge the battery while the unmanned aerial vehicle is on the landing surface.

13. 13. The system of claim 1, wherein the dock includes one or more pneumatic blowers positioned to blow air onto a lens of the unmanned aerial vehicle covering the image sensor while the unmanned aerial vehicle is on the landing surface.

14. controlling a propulsion mechanism of the unmanned aerial vehicle to fly the unmanned aerial vehicle to a first location proximate a dock including a landing surface configured to hold the unmanned aerial vehicle and a fiducial on the landing surface; Accessing one or more images captured using an image sensor of the unmanned aerial vehicle; Detecting a fiducial in at least one of the one or more images; determining a pose of the fiducial based on the one or more images; Controlling a propulsion mechanism so that the unmanned aerial vehicle lands on a landing surface based on the attitude of the fiducial; A method comprising:

15. The fiducial is a root fiducial, detecting an auxiliary fiducial in at least one of the one or more images, the auxiliary fiducial being larger than the root fiducial; determining a pose of the auxiliary fiducial based on the one or more images; controlling the propulsion mechanism to fly the unmanned aerial vehicle to a first location proximate the landing surface based on the attitude of the auxiliary fiducial; 15. The method of claim 14, comprising:

16. 16. The method of claim 15, wherein the dock includes a box configured to enclose the landing surface in a first configuration and to expose the landing surface in a second configuration, and the auxiliary fiducials are disposed on an exterior surface of the box.

17. controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; controlling a propulsion mechanism such that the unmanned aerial vehicle descends toward a fiducial on a landing surface; In response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism such that the unmanned aerial vehicle hover at the predetermined height above the landing surface until error estimates of attitude and velocity of the unmanned aerial vehicle satisfy a stability condition; In response to the stability condition being satisfied, controlling the propulsion mechanism to cause the unmanned aerial vehicle to perform a final approach for touchdown on the landing surface.

17. The method of any one of claims 14 to 16, comprising:

18. monitoring inertial measurements of the unmanned aerial vehicle while final approach is being performed to determine whether the landing surface has been properly engaged; In response to determining that the landing surface is not properly engaged, aborting the final approach by controlling a propulsion mechanism to lift the unmanned aerial vehicle.

20. The method of claim 17, comprising:

19. 19. The method of any one of claims 14 to 18, comprising automatically charging a battery of the unmanned aerial vehicle with a charger included in the dock while the unmanned aerial vehicle is on the landing surface.

20. the dock includes a retractable arm, the landing surface being disposed at an end of the retractable arm; 20. The method according to any one of claims 14 to 19.

21. the dock includes a box configured to enclose the landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock; automatically transitioning the dock from the first configuration to the second configuration by performing steps including opening a door of the box and extending the retractable arm to move the landing surface from inside the box to outside the box.

21. The method of claim 20, further comprising:

22. the propulsion mechanism includes a propeller; Automatically folding the propellers by rotating the propellers while the retractable arms are retracted to retract the unmanned aerial vehicle on the landing surface into the box.

22. The method of claim 21 , comprising:

23. 23. The method of any one of claims 14 to 22, comprising automatically cleaning a lens of the unmanned aerial vehicle covering the image sensor by using an air pressure blower positioned in the dock to blow air against the lens while the unmanned aerial vehicle is on the landing surface.

24. Accessing one or more images captured using an image sensor of the unmanned aerial vehicle; Detecting an auxiliary fiducial in at least one of the one or more images; determining a pose of the auxiliary fiducial based on the one or more images; controlling a propulsion mechanism of the unmanned aerial vehicle to fly the unmanned aerial vehicle to a first position proximate the root fiducial based on the attitude of the auxiliary fiducial; detecting a root fiducial in at least one of the one or more images taken from a first position, the auxiliary fiducial being larger than the root fiducial; determining a pose of the root fiducial based on the one or more images; Controlling a propulsion mechanism so that the unmanned aerial vehicle lands on the route fiducial based on the attitude of the route fiducial; A method comprising:

25. The root fiducial is on the landing surface. controlling the propulsion mechanism to fly the unmanned aerial vehicle to a second location above the landing surface; controlling a propulsion mechanism such that the unmanned aerial vehicle descends toward a root fiducial on a landing surface; In response to reaching a predetermined height above the landing surface, controlling the propulsion mechanism to cause the unmanned aerial vehicle to hover at the predetermined height above the landing surface until error estimates of attitude and velocity of the unmanned aerial vehicle satisfy a stability condition; In response to the stability condition being satisfied, controlling the propulsion mechanism to cause the unmanned aerial vehicle to perform a final approach for touchdown on the landing surface.

25. The method of claim 24, comprising:

26. monitoring inertial measurements of the unmanned aerial vehicle while final approach is being performed to determine whether the landing surface has been properly engaged; In response to determining that the landing surface is not properly engaged, aborting the final approach by controlling a propulsion mechanism to lift the unmanned aerial vehicle.

26. The method of claim 25, comprising:

27. 27. The method of any one of claims 24 to 26, wherein the dock includes a box configured to enclose the landing surface in a first configuration and to expose the landing surface in a second configuration, and the auxiliary fiducials are disposed on an exterior surface of the box.

28. Automatically charging the battery of the unmanned aerial vehicle using a charger included in the dock while the unmanned aerial vehicle is on the landing surface.

28. The method of any one of claims 24 to 27, comprising:

29. the dock includes a retractable arm, the landing surface being positioned at an end of the retractable arm; 29. The method of any one of claims 24 to 28.

30. the dock includes a box configured to enclose the landing surface in a first configuration of the dock and to expose the landing surface in a second configuration of the dock; automatically transitioning the dock from the first configuration to the second configuration by performing steps including opening a door of the box and extending the retractable arm to move the landing surface from inside the box to outside the box.

30. The method of claim 29, further comprising:

31. the propulsion mechanism includes a propeller; Automatically folding the propellers by rotating the propellers while the retractable arms are retracted to retract the unmanned aerial vehicle on the landing surface into the box.

31. The method of claim 30, comprising:

32. Automatically cleaning the lens of the unmanned aerial vehicle covering the image sensor by blowing air at the lens while the unmanned aerial vehicle is on the landing surface using an air pressure blower positioned within the dock.

32. The method of any one of claims 24 to 31, comprising:

Images