Method and computing device for generating aircraft splines

Abstract

Disclosed herein are techniques for operating an unmanned aerial vehicle, UAV (101), that fly the UAV along a calculated spline while also allowing for in-flight corrections. In various implementations, the UAV includes a flight control subsystem (124) and an electromechanical subsystem (126). The flight control subsystem (124) records keyframes during flight and calculates a spline based on the keyframes. The flight control subsystem (124) then stores the calculated spline for playback, at which point the UAV automatically flies according to the calculated spline.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. patent application Ser. No. 17 / 689,414, filed March 8, 2022, entitled "ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," U.S. patent application Ser. No. 17 / 689,459, filed March 8, 2022, entitled "GRAPHICAL USER INTERFACE FOR ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," U.S. provisional patent application Ser. No. 63 / 282,725, filed November 24, 2021, entitled "ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," and U.S. provisional patent application Ser. No. 63 / 282,725, filed January 4, 2022, entitled "INTERFACES AND CONTROL FOR ENHANCED UNMANNED AERIAL VEHICLE This application claims priority to U.S. Provisional Patent Application No. 63 / 296,285, entitled "Flight," which is incorporated herein by reference in its entirety.

[0002] Various implementations of the present technology relate to unmanned aerial vehicles (UAVs), and in particular to improved UAV flight along computed splines. [Background technology]

[0003] Unmanned aerial vehicles (or drones) are commonly used to capture video, images, or other data from viewpoints or locations that are otherwise difficult or cumbersome to reach. Drones are used for a variety of purposes, including recreation, scientific exploration, military operations, intelligence gathering, and commercial use. UAVs for commercial and recreational use typically have multiple rotors to allow the UAV to be agile and respond quickly to flight commands. For example, a common configuration known as a "quadcopter" has four rotors for flight.

[0004] The ability of drones to fly programmatically, i.e., according to a set of programmed flight instructions, makes them useful for repetitive operations such as monitoring a safe perimeter. Additionally, once an appropriate flight program is determined, no specific pilot skill level is required. However, while the drone is operating under a programmed instruction set, this denies the pilot any ability to make slight adjustments to the flight path or sensor operation during flight. For example, if the drone captures something unexpected or the pilot wants to spontaneously correct a video recording, the pilot must terminate the automated flight operation and rely on manual operation of the drone. Alternatively, the pilot can reprogram the flight plan, but risk missing an opportunity to capture potentially important data in the time it takes to reprogram and return to flight.

[0005] Beyond their more prosaic uses, an occupation for which drones are particularly well suited and actively employed is dynamic aerial filming. Highly dynamic, smooth, one-shot video was previously only possible for big-budget Hollywood production companies using high-end equipment and large teams of trained professionals. In recent years, teams of world-class drone pilots with thousands of hours of "stick time" under their belts have been able to create similarly dynamic shots. However, these teams of drone pilots can be very expensive to hire and time-consuming to film. Summary of the Invention

[0006] Disclosed herein are techniques for operating an unmanned aerial vehicle (UAV) that fly the drone along a calculated spline while allowing for in-flight corrections. In various implementations, the UAV includes a flight control subsystem and an electromechanical subsystem. The flight control subsystem records keyframes during flight and calculates a spline based on the keyframes. The flight control subsystem then stores the calculated spline for playback when the UAV automatically flies according to the calculated spline.

[0007] In various implementations, the flight control subsystem can receive user input and responsively modify the calculated spline based at least on the user input, resulting in a modified version of the calculated spline. The flight control subsystem can store the modified version of the calculated spline for later playback. In some scenarios, the UAV may be able to upload (or download) the modified version of the calculated spline to a remote storage location.

[0008] Examples of user input may include one or more of changes to one or more components of the calculated spline, such as the position, direction, velocity, and orientation of the unmanned aerial vehicle along the calculated spline. Components may also include a camera focal length and camera orientation relative to the unmanned aerial vehicle.

[0009] Other examples of user input include snapping the unmanned aerial vehicle directly to a new position on the calculated spline in any order relative to the next position on the calculated spline, reversing direction along the calculated spline relative to the current direction along the calculated spline, and hovering at a point on the calculated spline.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the technical disclosure. It will be understood that this Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0011] Many aspects of the present disclosure can be better understood with reference to the following drawings. The components in the figures are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Also in the drawings, like reference numerals indicate corresponding parts throughout the several views. While several embodiments will be described in connection with these drawings, the present disclosure is not limited to the embodiments disclosed herein. Rather, the intention is to cover all alternatives, modifications, and equivalents. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an operational architecture of an unmanned aerial vehicle in one implementation. [Figure 2] FIG. 1 illustrates a method of operation of a UAV to create splines in one implementation. [Figure 3] FIG. 1 is a detailed diagram of a UAV system in one implementation. [Figure 4A] FIG. 1 illustrates an operating environment and an exemplary scenario. [Figure 4B] FIG. 1 illustrates an operating environment and an exemplary scenario. [Figure 5A] FIG. 1 illustrates an exemplary UAV flight path. [Figure 5B] FIG. 1 illustrates an exemplary UAV computed flight path around a building. [Figure 6A] 1 is an exemplary overhead view of flying a UAV around a house. [Figure 6B] FIG. 10 is an exemplary overhead view of a calculated spline flight path. [Figure 6C]FIG. 10 is an example overhead view of a calculated spline flight path with corrections to the flight path and drone orientation. [Figure 6D] 10A-10C are exemplary close-up overhead views of the flight path and corrections to the drone orientation during flight along the calculated flight path. [Figure 6E] FIG. 10 illustrates an exemplary display on a remote control device from a first-person perspective as a pilot issues flight commands during flight along a calculated flight path. [Figure 7] FIG. 1 illustrates an exemplary workflow when a pilot issues flight commands as the drone flies under autopilot control. [Figure 8] 10A-10C are exemplary overhead views of the effect of pilot commands on an autopilot flying a drone along a calculated flight path. [Figure 9] 10A-10C are exemplary overhead views of the effect of pilot commands on an autopilot flying a drone along a calculated flight path. [Figure 10] 10 is an exemplary overhead view of the calculation of new position points for corrections to a calculated flight path during flight. [Figure 11] FIG. 1 illustrates a workflow for controlling an aircraft in one implementation. [Figure 12A] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12B] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12C] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12D] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12E]FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12F] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12G] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12H] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12I] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12J] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 13A] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13B] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13C] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13D] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 14A] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14B]FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14C] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14D] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 15] 1 is a series of images showing the user interface of an autonomous flight control application in one implementation during playback. DETAILED DESCRIPTION OF THE INVENTION

[0013] The drawings are not necessarily drawn to scale. Similarly, some components and / or operations may be divided into different blocks or combined into a single block for purposes of discussion of some of the embodiments of the present technology. Also, while the present technology is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are described in detail below. However, there is no intention to limit the present technology to the particular embodiments described. Rather, the present technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.

[0014] The technology discussed herein enables anyone to capture continuous choreographed shots with complex, cinematic, and smooth camera movements. The user defines a flight path by setting points called "keyframes," and the software creates smooth splines between the points that can be flown repeatedly with varying degrees of control, speed, and complexity. In fact, the technology discussed herein enables complex, Hollywood-style camera moves that would otherwise be impossible for any pilot, regardless of their skill level. The technology does not replace human creativity or production, but rather allows pilots to unlock amazing shots with just a few taps.

[0015] Various implementations disclosed herein include an unmanned aerial vehicle that includes a flight control subsystem and an electromechanical subsystem. As discussed above, the flight control subsystem can record keyframes during flight and calculate splines based on the keyframes. The flight control subsystem saves the calculated splines for playback, at which point the flight control subsystem instructs the electromechanical subsystem to fly the UAV according to the calculated splines.

[0016] In one or more of various implementations, the flight control subsystem can also receive user input and modify the calculated spline based on at least the user input. The modified version of the calculated spline may itself be saved for later playback. The user input may cause one or more changes to the calculated spline, such as changing the position, direction, speed, and / or orientation of the unmanned aerial vehicle along the calculated spline. Other changes include modifications to the focal length and orientation of a camera on the UAV, or changes to any other peripherals on the UAV. Further examples of user input include directly snapping the unmanned aerial vehicle to a new position on the calculated spline out of order relative to the next position on the calculated spline, reversing direction along the calculated spline relative to the current direction along the calculated spline, and hovering at a point on the calculated spline.

[0017] In one operational example, a drone pilot operating a controller device identifies a set of discrete spatial locations called keyframes. Resources onboard the drone and / or the controller (or distinct from either system) calculate splines based on the keyframes. The calculated splines are then "played back" by the drone, meaning that the drone's onboard flight control subsystem commands its flight based on the calculated splines.

[0018] Continuing with the operational example, a pilot can modify the calculated spline on the fly while the drone flies the spline, allowing the pilot to focus and control one or more aspects of the drone's operation without having to actively pilot the drone. The pilot may, for example, change the drone's position, direction, speed, and / or orientation as it flies the calculated spline. These changes may cause the drone to depart from the calculated spline or change direction as it moves along the spline. The pilot may accelerate or decelerate the drone along a segment of the spline or at a point along the spline. The pilot may also modify camera movement, such as zooming in or out or adjusting exposure, as the drone flies along the spline. In another example, the pilot may cause the drone to reverse course along the calculated spline or "snap" to a new position on the spline without moving along the spline. The pilot may also stop and hover the drone at a point on the calculated spline, for example, to add a keyframe at that location. Any or all of these modifications may be saved for later use with the spline or as a new version of the spline. In some implementations, the pilot may control the orientation of the drone along the spine and / or gimbal position as the drone moves along the calculated spine, i.e., in "free look mode." In this way, the pilot can focus on the camera angle and position without having to actively pilot the drone.

[0019] Various technical effects of the disclosed technologies can be appreciated from this disclosure. For example, when a drone is used to record video, a common use of drones, pilot input incorporated into the programmed operation allows for more creative buy-in to the video product by allowing the pilot to focus on the artistic aspects and nuances of the recording without having to actively navigate the drone's flight. The ability to incorporate pilot or user input into the drone's operation while flying under the authority of its autopilot can be exercised in several ways.

[0020] In one example, the UAV flies a programmed or predetermined flight plan. The programmed or predetermined flight plan may be a calculated spline flight path whose trajectory is defined by a set of discrete positions through which the drone will pass, or it may be a flight path recorded during a previous flight. It may also be a flight path manually programmed and uploaded to the drone. The programmed flight behavior of the drone includes parameters such as the drone's position, orientation, and / or speed as it flies along the predetermined flight path.

[0021] Drone operation may include object tracking during programmed operation. For example, a drone may be deployed to record a bicycle race by flying a programmed route along the racetrack from an overhead perspective, tracking the progress of a particular cyclist. Drone operation may also include object detection and avoidance, so that the drone does not crash into trees while flying information and recording the bicycle race. In this way, the pilot can focus on a particular aspect of the drone operation, such as the view captured by an onboard video camera, without having to actively fly the drone at the same time. When the pilot stops controlling that aspect of the drone's operation, the drone smoothly returns to and resumes its programmed operation.

[0022] In another example, a drone or UAV is docked prior to flight. The dock provides the drone with positioning information that allows the drone to ascertain its position and orientation relative to the dock for navigation during flight. Position information for navigation may be specified in three-dimensional coordinates, such as a set of coordinates relative to the dock or coordinates detected by an onboard GPS sensor. Position information for navigation may also incorporate or rely on visual tracking using one or more onboard navigation cameras. Drone orientation information may refer to the drone's pitch, roll, and yaw angles. The pilot wirelessly communicates with the drone's flight control system using a remote control. The remote control may be a dedicated device or app on a computing device such as a laptop computer, tablet, or smartphone. The user interface or UI on the UAV remote control may include a display screen and one or more input mechanisms for controlling the drone, such as buttons, rocker switches, sliders, and toggles. One implementation of a user interface on a drone remote control includes a touch-enabled display screen, i.e., a touchscreen, that displays functional graphical object representations of the control input mechanisms. Wireless transmissions between the UAV and the remote control device may be carried via a WiFi connection, a Bluetooth connection, or any suitable wireless connection. The UAV may also communicate wirelessly with other devices, such as computers, that can receive, view, record, and store information transmitted from the drone.

[0023] Once the UAV pilot issues a command to launch the drone from the dock and the drone begins its flight, the pilot may actively fly the drone, or the drone may fly according to a predetermined flight plan. In one implementation, the predetermined flight includes a set of keyframes, each of which may be defined according to the visual input of one or more navigation cameras in a dead-reckoning or visual tracking process. Keyframes defined according to visual tracking are particularly useful in indoor or outdoor environments where Global Positioning System (GPS) data or other inertial navigation data is partially or completely unavailable, or when using drones that lack inertial or satellite navigation capabilities. Keyframe positions may also be defined in three-dimensional coordinates relative to the drone dock position or from satellite or inertial navigation data.

[0024] A keyframe may contain additional information about drone orientation, drone speed, or onboard sensor activity. For example, a drone may be programmed to pause at a keyframe and pivot horizontally to capture a panoramic shot as a camera's zoom lens moves from a telephoto to a wide-angle focal length. The drone's path from one keyframe position to the next is a calculated function, such as a cubic spline interpolation formula or "spline."

[0025] In another implementation, the predetermined flight plan may be a previously recorded drone flight stored in the drone's persistent storage (e.g., non-volatile memory) or on a data storage device wirelessly communicating with the drone. Pre-recorded flight plans may be uploaded to other drones for reuse. For example, a flight plan for monitoring the perimeter of a secure facility can be recorded and saved for periodic reuse or when the drone is replaced by a backup drone. The pre-recorded flight path may include the entire flight from launch from a dock to return to the dock, or it may include a subset of the drone's flight, such as a spline between two keyframes, or even a drone operation in one location (such as a sweeping video shot). In yet another implementation, the predetermined flight path may be a computer program uploaded to the drone. For example, the flight plan may be programmed based on map or terrain data.

[0026] In another implementation, the pilot can record aspects of the drone's operation, such as recording the drone's trajectory, recording the drone's camera view, pausing the recording, deleting the recording, and saving the recording. The recorded operations or recorded components of the flight may be stored in on-board non-volatile memory, or they may be transmitted to a device in communication with the drone, such as a remote control, a laptop computer that receives transmissions from the drone, or cloud data storage.

[0027] In yet another implementation of the present technology, a display screen or touchscreen of a user interface displays a view transmitted from a forward-facing camera on a drone in real time, known as a first-person perspective. An augmented reality (AR) graphic representing a predetermined flight path is superimposed on the forward-facing camera view in the form of a semi-transparent curve overlaying the camera view, optionally in a color significantly different from the background. The AR representation may indicate distance from the drone along the flight path by varying the width of the curve, e.g., the curve becomes thinner with distance from the drone. The AR representation is continuously updated as the drone flies the predetermined flight path. Additional AR graphics may indicate keyframes or waypoints identified by distinctive shapes and colors, such as diamonds on the curved representation of the predetermined flight path.

[0028] In various implementations, a pilot can add, edit, and delete keyframes on a predetermined flight path through a user interface on a remote control. For example, when a pilot wants to identify and select keyframes for a desired camera view, the user interface displays AR graphics in the form of translucent frames over the camera views that make up the camera shot. In addition to virtual framing, the UI touchscreen can display virtual buttons for adding, editing, and deleting keyframes. Keyframes can record and store information such as drone position or location, drone orientation, and information about the operation of onboard sensors such as cameras. The AR representation of a predetermined flight plan defined by keyframes can be updated according to the latest set of keyframes.

[0029] In one implementation, a user interface on a remote control device displays a linear playback track or timeline representation of a predetermined flight plan. The linear timeline representation may include keyframes identified on the timeline by distinctive shapes, such as diamonds. Distance along the timeline may be proportional to actual flight distance. As the drone flies a predetermined flight path, the timeline may show the drone's progress along the flight path, using one color to indicate completed portions and a second color to indicate remaining portions. Optionally, an arrow or other symbol may move along the timeline as the drone flies to show the drone's movement along the flight path in real time.

[0030] In one implementation, a pilot can issue commands to a drone using a flight path timeline. In a touchscreen timeline graphic implementation, the pilot can command the drone to reverse direction on the flight path or to pause at a point on the flight path by touching a specific point or location on the displayed timeline. The pilot can also command the drone to snap (i.e., fly immediately) to a point on the spline by moving along the spline or by flying directly to the indicated point. For example, the pilot may command the drone to jump to a point between two keyframes and add a keyframe at that location.

[0031] In one implementation of the present technology, the UI screen also shows the drone's speed at points along the timeline. The UI may include virtual or physical controls, such as rocker switches or sliders, to adjust the drone's speed along segments of the flight path or at points on the flight path. For example, the pilot may command the drone to pause (i.e., hover) at various keyframes to capture long still views or panoramic views from their vantage point.

[0032] In another implementation of the present technology, a virtual slider controlling drone speed is displayed in multiple colors to indicate multiple zones of dynamic feasibility. Starting at the slow end of the slider range, green can indicate a range of speeds that are dynamically feasible for the drone to fly, yellow can indicate a range of speeds that push the drone's operating envelope, and red can indicate a range of speeds that are dynamically infeasible for the drone to fly. For example, if a given flight path indicates a turn, the red portion of the slider corresponds to a speed at which the drone cannot navigate the turn without flying off the flight path.

[0033] The pilot may maintain aspects of operational control of the drone as it flies a predetermined flight plan. For example, once a completed drone flight path is recorded, the pilot may command the drone to re-fly the recorded flight path while manually controlling the camera orientation. The camera orientation can be controlled by changing the drone's pitch, roll, and / or yaw angles. Additionally, as the drone flies the predetermined flight plan, the pilot may issue commands or motion commands via the remote control device that cause the drone to slightly deviate from the flight path or change the drone's orientation as it flies. For example, the pilot may gently nudge the drone's orientation to turn westward for a few seconds as the drone flies a predetermined path heading north.

[0034] In another implementation of the present technology, a drone receives and incorporates real-time inputs into its flight behavior corresponding to a predetermined flight path. When the pilot's real-time inputs cease, the drone performs an auto-return to the predetermined flight path. In yet another implementation of the present technology, the real-time inputs are smoothed or damped, resulting in less adjustment to the drone's flight. When the real-time inputs cease, the return to the predetermined behavior is similarly smoothed or damped.

[0035] For example, as a drone flies a spline, the pilot may activate an object tracking capability that allows the drone to keep its orientation pointed toward the subject so that the subject is always within the drone camera's field of view. Alternatively, an object avoidance function may cause the drone to deviate from its programmed flight path if the flight path intersects with an obstacle. In cinematography, the ability to manually control one or more aspects of drone operation (e.g., drone flight dynamics, drone orientation in flight, and onboard camera operation) as the drone navigates a predetermined flight path can give the pilot or videographer operating the drone a greater sense of creative ownership of the video footage, as opposed to strictly programmed or mechanical operation.

[0036] In another implementation of the present technology, deviations from or adjustments to a predetermined flight plan made as the drone flies the flight path can be saved for later use. The adjustments may be saved by themselves (to be added to the predetermined flight path), or the flight path and adjustments may be saved together as an entirely new flight path. In this manner, multiple adjustments to a particular predetermined flight path may be superimposed on the flight path, enabling the ability to create flight plans of increasing complexity or variability. For example, a flight path may be re-flown multiple times, each time with a different camera orientation, to compare and avoid various viewpoints.

[0037] In one implementation of the present technology, a drone or UAV may be docked at a location remote from the pilot. The pilot typically flies the drone by maintaining line of sight to the drone in accordance with FAA regulations governing drone flight. However, in certain situations, the pilot may navigate the drone without line of sight to the drone, relying on the drone's first-person perspective, i.e., by seeing what the drone camera sees. This method of flying a drone is generally only possible in certain limited situations, such as indoor flight in a large warehouse or stadium.

[0038] A drone's autopilot receives and incorporates several internal and external operational inputs that govern the drone's flight to issue commands to the drone's microprocessor. These commands are received by the microprocessor as if issued by a human pilot and are therefore issued by the autopilot as apparent joystick commands. The autopilot integrates the drone's calculated or programmed flight path with sensor data related to drone operation, such as wind speed and direction data. The autopilot may also receive joystick inputs when the pilot issues commands via a joystick on a remote control. The joystick inputs are interpreted according to the specific function assigned to the joystick during autopilot operation. For example, a joystick may be used to change the drone's pitch or yaw during programmed operation to change the view of an onboard camera. Management functions may be applied to joystick inputs that can dampen or limit the input so that the drone does not exceed the operating envelope for its flight operation. The drone autopilot may also receive inputs from a collision avoidance system or subject tracking system. Based on inputs from these various sources, the autopilot calculates and issues apparent joystick commands to the drone's microprocessor. In response to receiving apparent joystick commands from the autopilot, the microprocessor transmits flight commands to the drone electromechanical propulsion system to cause the drone to fly according to the commands of the autopilot.

[0039] Returning to the bicycle race example, during programmed operation, the drone pilot may command the drone to change its orientation in flight to obtain views of crowds of spectators along the race route or of particularly striking vistas in the distance. Alternatively, the pilot may modify and adjust the operation of the onboard camera, such as by zooming out for a wide-angle shot of a distant mountain range or zooming in for a close-up of a cyclist. When the pilot stops correcting the drone's flight or operation, the autopilot receives input from the joystick corresponding to a return to its neutral position, thereby effecting a smooth return to its programmed or calculated flight plan.

[0040] Referring now to the drawings, Figure 1 illustrates an unmanned aerial vehicle (UAV) 101 and its components represented by operational architecture 128. Operational architecture 128 broadly includes a flight controller subsystem 124, an electromechanical subsystem 126, external operation inputs 120, and internal operation inputs 122. Flight controller subsystem 124 may include a circuit board housing one or more microprocessors, also known as flight controllers, that control various aspects of drone operation. Electromechanical subsystem 126 may include an electronic speed controller unit, as well as various rotors, power supplies, etc.

[0041] External operational inputs can include inputs received from a remote control device, typically operated by a human pilot, and sensor data measuring environmental conditions affecting UAV operation. Internal operational inputs can include a program or calculated flight or motion plan that directs drone position, drone movement, or sensor operation during flight, and other information related to the particular use or capabilities of the UAV, such as map or terrain data.

[0042] FIG. 2 illustrates a process 200 performed by one or more components of the flight control subsystem 124 of the UAV 101. The process 200 is embodied in program instructions that, when executed by one or more hardware and / or firmware elements of the flight control subsystem 124, direct the process to operate as follows: As the UAV 101 is flying, the flight control subsystem 124 of the UAV 101 records one or more keyframes (step 210). The pilot of the UAV 101 may instruct the UAV 101 to record the keyframes based on a first-person perspective from which the pilot views the remote control device 130. The recorded keyframe data may include parameters such as the drone's physical position based on visual tracking or based on the drone's three-dimensional coordinates and orientation at that location. The keyframe data may also include data regarding onboard camera or sensor movement at the keyframe locations. The flight control subsystem 124 of the UAV 101 calculates a flight path or calculated spline connecting the keyframes (step 220). The flight path may be calculated by an on-board microprocessor using a set of discrete points or waypoints identified by the pilot during the current or previous flight. The flight control subsystem of the UAV 101 saves the calculated spline for later use (step 230).

[0043] The calculated spline may be re-flown by the UAV 101 or by a backup drone while the UAV 101 is recharging. The calculated spline may be stored locally, such as in on-board persistent memory or data storage. It may also be stored remotely, such as in data storage on a device in communication with the drone, such as a remote control or laptop computer that receives transmissions from the drone. This may also include, for example, remote cloud data storage. In future flights of the stored spline, the pilot may make modifications to the flight, for example, to make incremental improvements to the drone's operation, to make temporary adjustments based on unforeseen conditions, or to explore different ways of operating the drone. These modifications may be stored as well.

[0044] FIG. 3 illustrates an exemplary system architecture 300 for the quadcopter 401 of FIG. 4 . The system architecture 300 includes a flight control subsystem 391 and an electromechanical subsystem 392. The flight control subsystem 391 includes autopilot functionality (represented by an autopilot 328, a flight controller 326, an inertial measurement unit 302, sensors 304, a transmitter 306, a receiver 308, and a memory card port 310). The electromechanical subsystem 392 includes an electronic speed controller 312 and a rotor 314. It will be understood that both the flight control subsystem 391 and the electromechanical subsystem 392 may include other elements in addition to (or instead of) those disclosed herein, which are shown for illustrative purposes. The system architecture 300 also includes operational inputs 393. The operational inputs 393 include joystick data 322 provided by a remote control device 318, as well as management factors 330 and a calculated flight path 332.

[0045] The inertial measurement unit 302 includes one or more sensors, such as gyroscopes and accelerometers, that provide motion and orientation data to the flight controller subsystem. In some implementations, the flight controller subsystem may also be connected to or include other sensors 304, such as a video camera, a global positioning system (GPS) sensor, a magnetometer, or a barometer. The UAV also carries onboard equipment for wireless communication, such as an antenna, a video transmitter 306, and a radio receiver 308, which enable communication with a remote control device 340 to which a human pilot can send commands, such as flight commands or commands related to onboard sensor operation. The remote control device may be a dedicated device or an application on a mobile computing device, such as a smartphone, tablet, or laptop computer, capable of wireless communication with the UAV 401. Wireless communication between the remote control device and the UAV may be carried over a WiFi network or a Bluetooth link. The flight controller subsystem may also be connected to onboard persistent or non-volatile memory or a memory card port 310 for recording flight and sensor operation and data. As part of the electromechanical subsystem of the UAV 401, the electronic speed controller 312 is connected to the flight controller subsystem and controls the operation of the rotor 314 according to flight commands 316 received from a microprocessor on the flight controller subsystem.

[0046] The remote control device 340 for the drone 401 includes wireless communication hardware for communicating with the drone 401, as well as a throttle device (e.g., a physical or virtual joystick) 320 for manually controlling the flight (i.e., speed and direction) of the drone 401. For example, when the pilot moves the joystick 320, the remote control device 340 transmits joystick data 322 to the UAV 401. While the pilot can control the drone 401 based on their line of sight to the drone, remote control devices for drones typically have a display screen 324 for displaying the point of view of an onboard camera, referred to as a first-person view. The first-person view capability allows the pilot to find and capture views from remote or difficult-to-access vantage points.

[0047] The exemplary operating environment 400 of FIG. 4A illustrates the process of creating a calculated spline flight path. A pilot 404 initiates flight of a drone 401 by launching the drone from a drone docking device 402. The drone dock 402 provides orientation information for the drone 401 at the start of flight so that the drone 401 can determine its position relative to the dock 402 during flight. Drone flight may be manually controlled by the pilot 404 using a remote control device 403 in wireless communication with the UAV 401. The pilot may use a joystick 320 to turn or accelerate the drone 401. The remote control device 403 transmits the pilot's 404 inputs received from the joystick 320 to the drone 401 via an onboard receiver 308 coupled to a flight controller 326. The flight controller 326 translates the pilot's inputs into flight commands 316 issued to an electronic speed controller 312, which slows the rotor 314 accordingly.

[0048] In the sequence of events shown in FIG. 4A , the drone 401 is piloted by the pilot 404 along an arbitrary route 405 (Event 1). In Event 2, the pilot 404 identifies a location to be saved for the spline at point A. Using the remote control device 403, the pilot 404 adds a keyframe to that location as the drone flies. In one aspect of the present technology, this step may be performed using a virtual button object displayed on the touchscreen of the remote control device 403. Keyframe A is saved in Event 3 along with flight and / or operational data associated with location A. The keyframe data saved in Event 3 includes location coordinates (e.g., GPS coordinates or coordinates relative to the dock 402) and may also include data regarding the speed of the drone 401 at location A, its orientation at location A, and the operation of its onboard camera. The keyframe data may be saved in data storage on the remote control device 403 or onboard data storage of the drone 401. The flight continues. Upon reaching location B, the pilot 404 adds another keyframe (Event 4). Keyframe B is saved in event 5 in a manner similar to event 3. In event 6, a spline 410 connecting keyframes A and B is calculated. The spline calculation may be calculated within an on-board processor of drone 401, or may be calculated within a processor of remote control device 403 and then transmitted to drone 401. For clarity, in this example, the spline calculation is performed by a processor in an on-board flight control subsystem of drone 401.

[0049] FIG. 4B shows drone 401 in operating environment 400 in flight after recording keyframes “A” and “B” and calculating spline 410. Keyframes “A” and “B” and spline 410 may be stored in drone 401's onboard data storage, or the keyframes and spline may be uploaded to drone 401 before or during the current flight from a remote device that stores the information, such as remote control device 403. In event 1, pilot 404 commands drone 401 to play spline 410. In event 2, drone 401 flies spline 410. As drone 401 flies spline 410, it operates under the control of its onboard autopilot, which is part of drone 401's flight control subsystem. In events 2 and 3, drone 401 passes locations A and B, respectively, which are points on spline 410 that were stored as keyframes. As drone 401 plays back spline 410, pilot 404 may issue flight or movement commands to modify its behavior under the autopilot's command. For example, pilot 404 may command drone 401 to disengage and return to spline 410; to accelerate, decelerate, or stop and hover at a location on spline 410; or to reverse direction along 410. Pilot 404 may command drone 401 to face north as it flies rather than facing forward along the spline. Pilot 404 may add additional keyframes to spline 410. Pilot 404 may operate the drone camera as the autopilot navigates drone 401 along spline 410.

[0050] A UAV can be commanded to fly a predetermined flight path. The flight path can be defined by a discrete set of position data (and optionally speed data) called waypoints. Waypoints can be specified in three-dimensional Cartesian coordinates. Waypoints can be selected for different purposes. Some waypoints may be locations where the drone is intended to stop and view points of interest, while other waypoints may specify precise locations, such as a door or window, that the drone must reach in order to pass through.

[0051] When a drone is used for videography, a pilot may use keyframes to identify desired shots. View 500 in FIG. 5A shows eight exemplary keyframes 504 selected to view the exterior of a building 506 from an aerial position. Each keyframe 504 is a static record of the waypoint, along with the drone's 101 orientation (i.e., the orientation of the drone's onboard camera) at that waypoint so that the desired viewpoint can be recaptured at different times. Drone orientation refers to the drone's 101 angular position relative to forward horizontal flight, i.e., pitch angle 112, roll angle 110, and yaw angle 114 as shown in FIG. 1. While viewing a scene captured by the drone camera at a waypoint, the pilot can capture a panorama by varying the drone's yaw angle from left to right of its forward-facing position. Similarly, varying the drone's pitch angle can change the camera's tilt. For example, a videographer piloting a drone may want to capture a construction site from an elevated vantage point looking down on different stages of construction to document the work's progress. The technique is also useful for before-and-after comparisons of large area developments or reconstruction of communities after natural disasters.

[0052] The drone 101 can record a particular location in three-dimensional space as it flies by recording sensor or telemetry data about the location, such as the drone's distance and orientation from the drone dock 402, GPS data collected from the onboard GPS sensor 304, visual data collected from the onboard camera 304, or a combination thereof. Similarly, the orientation of the drone 101 at a particular location can be recorded and stored using data from the inertial measurement unit 302.

[0053] One technique for programming a UAV's flight is to record a continuous set of keyframes 504 that define the flight path and video movement of the drone 101. For example, when checking a security perimeter, the pilot may define a set of keyframes that capture every point of a building's entrance. The drone's 101 operating system then calculates the drone's 101 flight path and its orientation during flight from one keyframe to the next. On subsequent flights, the pilot deploys the drone 101 to fly the same route and capture the same view each time, making it easier to identify when something has changed.

[0054] An alternative to recording a continuous set of static keyframes 504 to define the flight path and video movement is to record successive keyframes, where the flight path and video movement of the initial flight are recorded continuously over a period of time. A flight recorded as successive keyframes can be re-flown later if desired. Because successive keyframes record all movement, including any jerky, erratic movements or other idiosyncrasies associated with manual flight control, this mode of operation may be more appropriate for experienced drone pilots or those with more competent flying skills.

[0055] When calculating a flight path based on a set of static keyframes, calculating the route from one keyframe to the next is a mathematical operation known as spline interpolation. Spline interpolation is a method of fitting a curve to a set of points by computing a piecewise set of low-order polynomials, or splines, over successive pairs of points so that the resulting function is continuous and smooth over its domain. This is a simpler and more stable method than polynomial interpolation, which involves fitting a single high-order polynomial to all points in the set.

[0056] In a basic implementation of spline interpolation, the path between any two points in three-dimensional space is a straight line. Connecting consecutive pairs of points with straight lines generates a continuous path with a shape similar to a polygon. When using this method to create a flight plan, the resulting set of connected straight lines will have points where the direction changes abruptly. FIG. 5A shows an example flight path 508 including straight line segments defined by a set of eight keyframes 504 at various locations around a building 506. Such abrupt changes in direction are not only aesthetically undesirable for cinematography, but may also be dynamically infeasible, i.e., exceeding the flight capabilities of the drone 101.

[0057] However, higher-order splines can connect consecutive sets of points with curves more suitable for drone flight. That is, a set of cubic polynomials is generated across consecutive pairs of points with the requirement that the polynomials be twice continuously differentiable, i.e., continuous and smooth when combined. Smoothness is mathematically defined by requiring that lower-order derivatives be equal at knots (i.e., points where the functions are combined). In cubic splines, smoothness requires that both the tangent and curvature of the function be equal at the knots, which eliminates abrupt changes in velocity and acceleration. Thus, curves generated in this manner produce flight paths that are aesthetically appealing for cinematography and dynamically feasible for drone operation. Furthermore, cubic splines satisfy an additional constraint for drone flight and cinematography: that the third and fourth derivatives of the interpolation formula with respect to time, known as jerk and snap, be zero. View 520 in FIG. 5B shows flight path 510 through the same keyframes 504 of FIG. 5A but connected to a cubic spline interpolation formula. The spline interpolation calculations can be performed on the fly, as the pilot adds or removes keyframes in the sequence. Furthermore, by varying the waypoint or endpoint constraints, the characteristics of the calculated curve can be altered, thereby affecting the dynamic characteristics of the flight path.

[0058] As the drone records video while flying along the calculated flight path, the drone's 101 orientation can be programmed to provide smooth, stable video recording, eliminating any uneven camera movement or directional changes that may occur during manual camera movement, and allowing camera movement to be subsequently recaptured once a preferred orientation program is found. In a process similar to interpolating a flight path over a set of waypoints, a drone orientation function can be interpolated to provide smooth drone movement along the flight path using the drone orientation data (e.g., pitch angle 112, roll angle 110, and yaw angle 114) specified in the waypoints. And just as successive keyframes record the drone's 101 position over a period of time during its initial flight, the orientation of the drone 101, and therefore its camera or other sensors, can also be recorded during its initial flight for later use.

[0059] In one aspect of the present technology, during programmed drone operation, the pilot may desire to modify video recording or other sensor data collection while the drone 101 is flying by making small, temporary changes to the UAV's position or orientation without terminating the UAV's programmed operation. The pilot may issue flight or motion commands via the joystick 320 on the remote control device 130, which are sent to the drone's flight controller subsystem. Upon receiving new joystick data, the flight controller subsystem modifies the flight commands issued to the electronic speed controller 312 based on the calculation of the calculated spline 510, successive keyframe paths, or corrections to the programmed flight path. The corrections incorporate one or more factors governing drone flight, such as environmental conditions (e.g., wind speed and direction) or obstacle detection and avoidance. For example, if a set of keyframes is identified to interpolate a flight path, and the interpolation produces a path that passes through a tree, a proximity sensor on the drone will detect the tree and the drone will complete a modification to the spline to bypass the tree and return to the spline.

[0060] In an exemplary use shown in the overhead view 600 of FIG. 6A , a real estate agent desires to provide video recording of a residential property 602 from an exterior aerial perspective for marketing purposes. A drone system with videography capabilities, including a drone 101 and a videographer 606 flying the drone 101 using a remote control device 130, is dispatched to perform the recording. In the first flight 601 shown in FIG. 6A , the videographer 606 flies the drone 101 around the house 602, selectively choosing a set of particularly desirable viewpoints and saving that information as a set of five keyframes 608, including drone position and drone orientation information for each keyframe. Exemplary display screens 610 and 612 for identifying and adding keyframes to the set are shown in FIG. 6A , and an augmented reality image 614 can be overlaid on the first-person perspective to accurately identify the camera shot associated with the keyframe.

[0061] For subsequent flights, flight controller subsystem 124 calculates flight path 622 as shown in overhead view 620 of Figure 6B via spline interpolation. Drone 101 flies flight path 622 determined by the interpolation formula and records video of the entire perimeter of house 602, including the particularly desired viewpoint of keyframe 608.

[0062] Continuing with this exemplary use, assume that in a later flight, the real estate agent wishes to include in the video recording a shot of pond 616 behind house 602. Rather than recording a new set of keyframes around house 602, while flying drone 101 around the property on the calculated spline, videographer 604 modifies drone movement while flying on calculated spline 622. As shown in overhead view 640 of FIG. 6C , videographer 604 issues a command using joystick 320 to move drone 101 slightly leftward 624 from calculated spline 622 and steer the camera away from house 602 and toward pond 616. After achieving the desired recording, videographer 604 returns joystick 320 to its neutral position, and drone 101 calculates a return path 626 to calculated spline 622 and a reorientation to the orientation spline to continue its flight. A close-up view of the calculated corrections to flight path 622, including departure 624 and return 626 and drone transitions indicated by arrows, is shown in Figure 6D. Figure 6E shows an exemplary first-person perspective of drone 101 as seen on display screen 324 of remote control device 130 at a series of points V1 to V10 before or during a correction of its flight along flight path 622 in accordance with flight commands issued by videographer 604.

[0063] 7 illustrates a process 700 performed by one or more components of the flight control subsystem 124 of the UAV 401. The process 700 is embodied in program instructions that, when executed by one or more hardware elements of the flight control subsystem onboard the drone 401, direct the drone 401 to operate as follows: The drone pilot uses the joystick 320 to correct flight of the drone 401 along the calculated spline (step 710); the corrections may affect the drone's position or orientation, or some combination thereof. Joystick data is wirelessly transmitted from the remote control device 403 to the autopilot 328 of the drone 401 (step 712). The autopilot 328 also receives external operating inputs (i.e., wind speed and direction data) that affect drone flight (step 714). The autopilot 328 calculates corrections to the calculated spline (step 716). At event 718, this correction is sent to the flight controller 326, which at event 720 instructs the drone electromechanical subsystem of the drone 401 to fly according to the calculated correction to the spline (step 718). The electromechanical subsystem issues flight commands that simulate actual joystick commands, in other words, composite or modified joystick commands 330 (step 720). The modified joystick commands 330 include the pilot's input along with factors governing drone operation. In another implementation of the present technology, a pilot can command multiple simultaneous corrections to flight during programmed operation using a remote control device with multiple joysticks, each joystick assigned to a specific aspect of drone operation, or by assigning multiple functions to a single joystick. For example, considering the two joysticks shown in FIG. 1, input from one joystick may control the drone's yaw angle 114, while input from the other joystick controls the drone's pitch angle 112, giving the pilot the ability to focus on fine-tuning the camera while the drone flies autonomously along a predetermined flight path.

[0064] In another implementation of the present technology, the management factor may convert actual joystick commands into modified joystick commands by applying a damping function to the actual joystick input or data. In one aspect of the present technology, the damping function may be a mathematical model that simulates the response of a spring-rigged object subjected to forces corresponding to actual joystick inputs. More specifically, the spring-rigged model converts real-time inputs by a pilot using a joystick into the response of a simulated object rigged with three linear or three torsional overdamped screws subjected to displacements in one or more directions. One result of applying a damping function to the pilot's joystick is a damped input to the drone that prevents abrupt dynamic responses in the drone's velocity or orientation, resulting in a correction to the calculated spline that maintains cinematically desirable characteristics. Another result of a damping function, such as a spring model, is that when the pilot releases the joystick and allows it to return to its neutral position, the drone receives that joystick input and converts it into a modified joystick input that gradually reduces the correction to zero, resulting in a smooth return to the spline. Similarly, if the pilot uses the joystick to reorient the drone to point in a direction different from that determined by the orientation spline (rather than moving the drone off the calculated spline), the joystick input is damped to avoid abrupt mechanical responses upon reorientation, resulting in a more desirable response for cinematic purposes.

[0065] In yet another implementation of the present technology, the application of a damping function can impose limits on modifications to the calculated spline or predetermined flight behavior of the drone. This limit on the drone's dynamic response to pilot inputs creates a motion envelope around the drone's calculated or predetermined flight path. For example, in the exemplary overhead view shown in FIG. 8 , using joystick data filtered through a damping function, the pilot can cause the drone 804 to make small deflections 806 from flight path 802 as it flies along flight path 802. The damping function applied to the pilot's flight commands creates a motion envelope 810 around flight path 802. The drone 804's autopilot continues to provide motion commands to the electromechanical subsystem, taking into account the joystick data received from the remote control. Thus, the pilot can focus their attention on specific aspects of the drone's operation without having to assume complete control of the drone's operation.

[0066] FIG. 9 shows an overhead view 900 of a predetermined drone flight path for a drone flying under autopilot control. As the drone flies the predetermined flight path 904, if the drone pilot pushes the joystick to the left (event 920), such an input would normally disengage autopilot operation, resulting in the drone turning left; if the joystick is held in that position and then released shortly thereafter, the drone will fly backward (926) and eventually stop and hover. However, in one implementation of the present technology, the same joystick input is filtered through a modified joystick command attenuation function. The modified joystick command causes the drone to deviate 902 slightly to the left from the flight path 902, but still follow the flight path 902. Thus, when the joystick returns to the neutral position (event 922), i.e., when the pilot stops pushing the joystick, the drone will return 906 or resume its calculated flight path 904 or programmed operation. The net effect of such an implementation of the technology is to create an operating envelope 910 around flight path 904 in which the drone preferentially adheres to flight path 904 but can deviate from that path in terms of the drone's position or orientation based on joystick data provided by the drone pilot.

[0067] As the drone flies a predetermined flight path, the autopilot transmits modified joystick commands to the electromechanical subsystem, which then slows the rotor accordingly. These modified joystick commands are typically issued several times per second, such as every 50 milliseconds. The commands incorporate the drone's current position, the drone's desired next position according to the flight path, and external or environmental factors such as wind speed and direction. In one aspect of the present technology, the autopilot incorporates factors governing drone flight behavior into the modified joystick commands. These factors can affect drone behavior, such as by damping the drone's dynamic response to joystick data or by incorporating collision avoidance responses into object detection. In FIG. 10 , point 1002 represents position data progressively calculated by the autopilot of the flight control subsystem based on the calculated spline. Point 1002 may also include flight parameters governing the drone's transitions, but for clarity, this example is limited to a discussion of position corrections. The overhead view 1000 of a drone flight path shown in FIG. 10 further illustrates the effect of joystick data received from a remote control device causing a transient flight path deviation. At position 1020, the autopilot creates a new path including a new set of flight parameters, including incremental position data 1006, and issues modified joystick commands, which may also include orientation data. Point 1006 represents a decaying response to the joystick input received from the remote control device. When the pilot stops pushing the joystick at position 1022, the drone autopilot calculates incremental position data 1008 that returns the drone to flight path 1002. The decaying of the pilot's joystick commands results in a motion envelope 1012 around flight path 1002 that limits the drone's ability to deviate from the flight path, even when the pilot pushes the joystick continuously and perfectly.

[0068] 11 illustrates a process 1100 implemented by one or more components of the flight control subsystem 124 and remote controller 130 of the UAV 101. The process 1100 is embodied in program instructions that, when executed by one or more hardware and / or firmware elements of the flight control subsystem 124 and remote controller 130, instruct the flight control subsystem 124 and remote controller 130 to operate as follows: When the UAV 101 is flight-controlled by the remote controller 130 operated by a pilot, a computer system instructs a graphical user interface (GUI) on the remote controller 130 to display a perspective view from a sensor operably coupled to the flight control subsystem 124 (step 1110). The computer system detects input from the pilot interfacing with the GUI, including instructions to add keyframes, the keyframes including the spatial position of the UAV 101 and the sensor orientations (step 1120). The computer system continuously generates and updates a spline including a projected flight path or trajectory between each of a plurality of keyframes, including the sensor orientations (step 1130). The computer system continuously displays (step 1140) a graphical representation of the spline overlaid on a perspective view from a sensor mounted on the UAV 101. In one implementation, the sensor is a forward-facing camera that provides a first-person perspective view of the UAV 101. The orientation of the camera corresponds to the gimbal angle of the camera relative to horizontal flight.

[0069] 12A-12J show one implementation of a user interface presented to a pilot on a remote control device's display screen. In this example, the remote control device displays a user interface of a virtual reality-based autonomous flight control application that receives input from the pilot and commands the drone to fly accordingly. The remote control device displays the user interface on a touchscreen along with various input devices, such as virtual buttons, sliders, etc. In another implementation, the input devices may be physical buttons, sliders, toggles, joysticks, etc., on the remote control device. The remote control device may be a dedicated drone control device, a smartphone, tablet, or mobile device, or a computing device such as a laptop or other computer that wirelessly communicates with the drone.

[0070] In this example, the autonomous flight control application receives input from a user or pilot through a virtual input device in its user interface. When a virtual input device is described below as being "selected," this indicates that the autonomous flight control application has received an instruction from the pilot to cause the virtual input device to change its state (such as by touching, tapping, or "clicking" the virtual input device). The application responds to that change in state according to its program instructions.

[0071] In this example, a drone operating in keyframe mode generates a calculated spline flight path, or "spline," that circles a small grove of trees and then flies that spline during playback. The UI software generates an AR representation of the calculated spline and associated keyframes and overlays a live video feed from the onboard camera on the touchscreen. The UI software continuously updates the AR representation of the spline and keyframes as keyframes are added, edited, or deleted, the spline is generated or recalculated, and during playback as the drone flies the spline. The calculated spline can be recorded and saved for later use by the same drone or other drones with similar capabilities. The calculated spline may also be later edited by the user, and any changes may be saved as a new spline or as a revision that can be selectively added to the spline during later use.

[0072] To begin this example, FIG. 12A shows an implementation of the UI of an AR-based autonomous flight control application on the touchscreen of a drone remote control device as the spline is defined. The touchscreen displays a camera view 1210 captured by an onboard camera. In the center of the touchscreen, the UI displays a launch button 1201, which causes the application to launch the drone from its dock. On the left side of the screen are virtual indicators through which the UI presents various states related to the drone's operation, or virtual buttons through which the pilot can access aspects of the drone's operation: a battery charge indicator 1211, a Wi-Fi signal strength indicator 1212, a camera resolution indicator button 1213, and a settings button 1214. On the right side of the screen are a map button 1221 to access a graphical map, a home button 1222, an auto-record indicator 1223 that indicates whether video is being recorded, a settings button 1224, and an operation mode graphic 1225 that indicates the drone's operating mode (in this illustration, the drone is being operated manually).

[0073] FIG. 12B shows the UI of an autonomous flight control application when the home button 1222 is selected in one implementation. When the UI receives input indicating that the home button 1222 has been selected, the UI displays a tabbed window 1230 including a cinematic tab 1240 for selecting among several modes of automated drone flight. The motion track button 1241 causes the drone to track a moving object, such as a person or vehicle, in flight while autonomously avoiding obstacles along the way. The fixed track button 1242 initiates a fixed track mode used to track a moving object on a fixed track. The fixed track mode causes the drone to follow the object while maintaining a set distance from the object and maintaining the original camera orientation. The orbit object button 1243 initiates the drone's object-tracking flight operation, whereby the application commands the drone to fly an orbit around the object. The cable button 1244 engages a method of operating the drone, whereby the application defines two keyframes marking the endpoints of a flight path and then flies the drone between them as if tethered to a cable stretched between the two points. The hover button 1245 causes the drone to hover at a single spatial position or keyframe. The keyframe button 1246 activates a keyframe mode of drone operation, in which the application records and stores multiple keyframes, automatically and dynamically generates a spline flight path between each of the keyframes, and commands the drone to fly that spline. In keyframe mode, the application may also combine input received by the UI from the pilot's interaction with the interface's input devices. In keyframe mode, existing splines can also be edited and saved for reuse by the drone or other drones with keyframe mode capabilities. Saved splines may be stored and retrieved from non-volatile storage onboard the drone, the controller, a computer in communication with the controller or drone, or connected cloud storage.

[0074] FIG. 12C shows the UI of the autonomous flight control application when keyframe button 1246 is selected, causing the application to enter keyframe mode, which operates the drone as instructed by operation mode graphic 1225. In this mode, the touchscreen displays a camera view 1210 from the onboard camera. The UI displays text display 1203 to indicate the current mode, "Keyframe Mode." Flight parameter set 1202 is a graphic in the upper left corner of the touchscreen that displays the drone flight speed, drone distance from the dock, drone altitude, and camera gimbal angle relative to horizontal flight. In the bottom center of the touchscreen, the UI displays virtual buttons that allow the pilot to define keyframes used in generating splines. Add button 1251 causes the application to add a keyframe at the drone's current position, Undo button 1250 reverses the action triggered by Add button 1251 (i.e., undoes the addition of the last added keyframe), and Done button 1252 finishes adding a keyframe. In the upper right corner of the touchscreen is a graphic 1204 for pausing keyframe mode, allowing the pilot to stop autonomous flight and take manual control. Note that the camera view 1210 has been darkened to increase visibility of the text display 1203.

[0075] FIG. 12D shows the UI at the start of keyframe mode, with the autonomous flight control application UI prompting the pilot to add the first keyframe in text display 1203. FIG. 12E shows the UI after the first keyframe has been added. As the keyframe is added, the autonomous flight control application records the drone's spatial position. The application may also record the drone orientation, onboard camera gimbal angle, onboard camera focal length and / or exposure setting, drone speed, and / or other flight or operational parameters at the location of the newly added keyframe. In FIG. 12F, the drone is piloted closer to a wooded area for the second keyframe, and the second keyframe is added. FIG. 12F shows the UI's text display 1203 confirming that the keyframe has been added.

[0076] The example continues with the process of adding keyframes. Figures 12G and 12H show the touchscreen display as the drone is maneuvered around a wooded area and keyframes are added. In keyframe mode, the autonomous flight control application dynamically recalculates the spline as keyframes are added, and the AR representation of the spline is continuously updated by the on-screen UI. As shown in Figure 12H, a keyframe is added and the calculated spline 1260 is displayed on the touchscreen extended over the camera view 1210. Keyframe markers 1262 appear as diamonds on the calculated spline 1260.

[0077] After several more keyframes are added, FIG. 12I shows the camera view 1210 overlooking the wooded area as the 17th keyframe is added. In this illustration, the computed spline 1260 and keyframe markers 1262 are more clearly visible. The computed spline 1260 is a two-dimensional projection of a three-dimensional spline containing the flight path or trajectory between each of the keyframes. The diamond-shaped keyframe markers 1262 marking the locations of the keyframes vary in size with the order of their addition, with the most recently added keyframes being indicated by the largest diamonds. The computed spline 1260 and keyframe markers 1262 may also be scaled proportionally to the distance from the drone, dynamically increasing in size on the display as the drone approaches the keyframe location. Additionally, different geometric shapes may be used to indicate keyframes according to specific purposes, such as the beginning or end of the computed spline.

[0078] 12J shows the UI during keyframe addition and spline generation once the pilot has finished adding the keyframes. When the pilot selects the Done button 1252, the autonomous flight control application receives an indication that the spline is complete and switches to keyframe playback mode as shown in FIG. 13A.

[0079] 13A-13D show a UI implementation in which the spline definition is complete and the spline is played back. At the bottom of the touchscreen display shown in FIG. 13A, the autonomous flight control application UI displays a playback track 1320, a linear graphical representation of the calculated spline. On the playback track 1320, keyframes are represented as diamonds 1322, and the drone's current position along the spline is also shown as an arrow 1324. The relative distance between keyframes is indicated by the proportional spacing of the diamonds 1322 on the playback track 1320. FIG. 13B shows the addition of a virtual speed control slider 1330 to the touchscreen display, which causes the UI to receive manual input that causes the application to accelerate, decelerate, or hover the drone as it traverses the calculated spline 1260. Using the speed control slider 1330, the pilot can control the drone's speed between keyframes or across the entire spline. Note that in this illustration, the arrow 1324 changes color to indicate when the drone is moving. In one implementation, the relative time to move between keyframes is indicated by proportional spacing of diamonds 1322 on the playback track 1320.

[0080] An additional feature of the UI playback track 1320 is the ability to "snap" the drone to any location on the calculated spline 1260. Tapping anywhere on the playback track 1320 instructs the application to fly the drone directly to that location without crossing the spline.

[0081] 13C shows an implementation of a UI in which an autonomous flight control application displays the drone's progress as it flies a spline 1260 during playback in keyframe mode. The application may receive commands from the pilot using virtual play / stop button 1314 and forward / reverse buttons 1316 to fly the drone forward or backward along the spline, or to pause and hover. The UI continuously updates a playback track 1320 to indicate both the direction and current position of the drone's playback along the spline. Diamonds 1322 change color to indicate the drone's progress through the keyframes.

[0082] FIG. 13D shows a controller touchscreen display during playback in one implementation as the drone begins to move through the calculated spline 1260 starting from the first keyframe recorded in FIG. 12D . In this view, the autonomous flight control application UI displays several flight and movement commands that allow the application to adjust the speed of the drone's movement along the spline, stop or reverse the drone along the spline, jump or "snap" to out-of-sequence keyframes on the spline, add new keyframes, delete keyframes, or edit the speed or orientation at any keyframe. Note that on the AR representation of the calculated spline 1260, each of the keyframe markers 1262 increases in size as the drone approaches the keyframe location. The UI displays the calculated spline 1260 in a color that is highly visible from the background (first-person) view on the touchscreen. This color can be selected programmatically using an algorithm that detects the color range of the camera view 1210, or the color may be set manually by the pilot.

[0083] 14A-14D show an implementation of a user interface for an autonomous flight control system displayed on a touchscreen of a smartphone. In this example implementation, an autonomous flight control application operating in keyframe mode generates a spline between each of a plurality of keyframes. The application can edit the spline during playback by adding additional keyframes or by editing existing keyframes.

[0084] 14A shows the UI before the start of the drone's movement on the spline. The UI displays a playback track 1410 with diamonds 1412 marking the locations of keyframes proportional to the distance along the spline or proportional to the travel time between keyframes. Visually, the spline 1404 is an AR graphic displayed on the camera view 1402 along with a keyframe marker 1406 indicating the location of the next keyframe as the drone traverses the spline 1404.

[0085] FIG. 14B shows the UI as the drone moves along the calculated spline 1404 between the second and third keyframes. An arrow 1414 traverses the playback track 1410, indicating the drone's position on the spline 1404 in real time. Next to the playback track 1410 are virtual input devices: an edit button 1416 and an add button 1418. As the drone traverses the spline 1404, when it reaches a keyframe, the edit button 1416 becomes active, allowing the pilot to select the keyframe for editing. As the drone moves anywhere along the spline 1404, the add button 1418 becomes active and, when selected, prompts the application to define and add a new keyframe at the drone's position, recording and storing the drone's position as determined by visual tracking or navigation coordinates. The application may also record and store other flight or operational parameters of the new keyframe, such as the onboard camera's gimbal angle, exposure setting, or focal length.

[0086] Figure 14C shows the UI when the Add button 1418 is selected. The UI prompts the pilot to set the location of the new keyframe by tapping the Set button 1420. When the Set button 1420 is tapped, the application recalculates the spline 1404, instructs the UI to display an updated graphical representation of the spline 1404, and marks the location of the new keyframe by adding a diamond to the playback track 1410.

[0087] 14D shows the UI when the pilot taps the add button 1418 while the drone is in an existing keyframe, and the application prompts the pilot to indicate whether the new keyframe should be placed before or after the existing keyframe (or to cancel the addition). If the pilot selects before or after, the application recalculates the spline 1404 and the UI updates the display accordingly.

[0088] FIG. 15 includes a series of images illustrating yet another implementation of a UI for a drone's autonomous flight control application operating in keyframe mode during playback. Images 1510-1530 show displays on a drone remote control device. In image 1510, the display shows that the first-person camera view is a live feed from the onboard camera. At the bottom of image 1510, a drone arrow indicator moves from right to left across the playback track, indicating the drone approaching keyframe 4. As the drone approaches keyframe 4, a semi-transparent diamond marking the location of keyframe 4 on the AR overlay of the computed spline increases in size as the drone approaches and then disappears to simulate the drone passing through the AR keyframe diamond (image 1520). Next, image 1530 shows the drone's first-person perspective, continuing on the computed spline but as the drone turns right to track the paddleboard. Because it has been pivoted from its forward-facing orientation, it is ostensibly out of the camera view to the left of the screen, so the AR representation of the computed spline is no longer visible.

[0089] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied therein.

[0090] In various implementations, the systems, methods, processes, and operational scenarios may be implemented in computer software executed by a processing system in the context of an unmanned aerial vehicle, a remote control device, or any other type of device capable of executing software, such as a computer or a mobile phone. The processing system may load and execute the software from a storage system or may be pre-configured with the software. The software includes and implements processes for creating calculated splines, which represent the spline creation processes discussed in connection with previous figures, such as process 200 and process 700. The software also includes and implements programs related to the user interface of the autonomous flight control program, which represent the user interface of the autonomous flight control program discussed in connection with previous figures, such as process 1100. When executed by the processing system, the software instructs the processing system to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations.

[0091] An exemplary processing system may include a microprocessor and other circuitry that retrieves and executes software from storage. A processing system may be implemented in a single processing device, or may be distributed across multiple processing devices or subsystems that cooperate to execute program instructions. Examples of processing systems include general-purpose central processing units, graphical processing units, application-specific processors, and logic devices, as well as any other type of processing device, combination, or variation thereof.

[0092] An exemplary storage system may include any computer-readable storage medium readable by a processing system and capable of storing software. The storage system may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage medium. In no case is the computer-readable storage medium a propagating signal.

[0093] Software may be implemented with program instructions that, when executed by a processing system, may, among other functions, instruct the processing system to operate as described with respect to the various operational scenarios, sequences, and processes presented herein. In particular, the program instructions may include various components or modules that cooperate or otherwise interact to perform the various processes and operational scenarios described herein. The various components or modules may be embodied with compiled or interpreted instructions, or some other variation or combination of instructions. The various components or modules may be executed synchronously or asynchronously, serially or in parallel, in a single-threaded environment or multi-threaded, or according to any other suitable execution program, variation, or combination thereof. Software may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software may also include firmware or any other form of machine-readable processing instructions executable by a suitable processing system.

[0094] In general, software, when loaded and executed on a processing system, may transform a generally suitable apparatus, system, or device from a general-purpose computing system to a special-purpose computing system as described herein. Encoding software on a storage system may transform the physical structure of the storage system. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors include, but are not limited to, the technology used to implement the storage medium of the storage system and whether the computer storage medium is characterized as primary or secondary storage, as well as other factors.

[0095] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, software may transform the physical state of the semiconductor memory as the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of this description, and the above examples are provided solely to facilitate this discussion.

[0096] It will be further understood that an unmanned aerial vehicle, remote control device, or other device in which aspects of the present invention may be embodied may include a communications interface system. The communications interface system may include communications connections and devices that enable communication with other computing systems and devices (not shown) over a communications network (not shown). Examples of connections and devices that together enable system-to-system communications may include network interface cards, antennas, power amplifiers, RF circuits, transceivers, and other communications circuitry. The connections and devices may communicate via a communications medium such as metal, glass, air, or any other suitable communications medium for exchanging communications with other computing systems or networks of the system. The foregoing media, connections, and devices are well known and need not be discussed in detail here.

[0097] Communications between such systems and devices may occur over one or more communications networks according to various communications protocols, combinations of protocols, or variations thereof. Examples include intranets, network-to-network connections, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The foregoing communications networks and protocols are well known and need not be discussed at length here.

[0098] An unmanned aerial vehicle, remote control device, or other device in which aspects of the present technology may be embodied may include a user interface system. The user interface system may include one or more of a joystick, keyboard, mouse, voice input device, touch input device for receiving touch gestures from a user, motion input device for detecting non-touch gestures and other actions by the user, and other equivalent input devices and associated processing elements (e.g., joystick toggles) capable of receiving user input from a user. Output devices, such as displays, speakers, haptic devices, and other types of output devices, may also be included in the user interface system. In some cases, input and output devices may be combined into a single device, such as a display capable of displaying images and receiving touch gestures. The aforementioned user input and output devices are well known in the art and need not be discussed in detail here. The user interface system may also include associated user interface software executable by an appropriate processing system that supports the various user input and output devices discussed above. Separately or in conjunction with each other or other hardware and software elements, the user interface software and user interface devices may support a graphical user interface, a natural user interface, or any other type of user interface.

[0099] It will be further appreciated that aspects of the present technology describe various operational scenarios for drone operation along a calculated spline flight path or along a programmed flight path obtained from a series of keyframes or other sources. As the drone's autopilot issues modified joystick commands to the electromechanical subsystem based on the flight path or other external operating factors, the drone pilot may issue operational commands via a joystick on a remote control device. The autopilot receives the joystick data and incorporates this data into apparent joystick commands issued to the UAV microprocessor. The autopilot maintains control over the drone's operation along the flight path, and its ability to incorporate joystick data into the drone's operation results in an operational envelope along the flight path that allows the drone pilot to control one or more aspects of flight to achieve optimal drone operation.

[0100] Unless the context clearly requires otherwise, throughout the description and claims, words such as "comprises," "comprising," "such as," "the like," and the like, should be construed in an inclusive sense, i.e., "including but not limited to," rather than an exclusive or exhaustive sense. As used herein, the terms "connected," "coupled," or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, where the coupling or connection between elements may be physical, logical, or a combination thereof. Additionally, the terms "herein," "above," "below," and words of similar import, when used in this application, refer to this application as a whole and not to any particular portion of this application. Where the context allows, words in the above detailed description using singular or plural numbers may also include the plural and singular number, respectively. The word "or" in connection with a list of two or more items covers all of the following interpretations of that word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0101] The above detailed description of examples of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. While specific examples of the present technology have been described for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the present technology. For example, while processes and blocks are presented in a given order, alternative embodiments may perform routines having different orders of operations or employ systems having blocks in different orders, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are sometimes shown as being performed sequentially, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Furthermore, any specific numerical values ​​described herein are merely examples. Alternative implementations may employ different values ​​or ranges.

[0102] The teachings of the technology provided herein may be applied to other systems, not necessarily the system described above. The elements and operations of the various examples described above may be combined to provide further implementations of the technology. Some alternative implementations of the technology may include additional elements to the above examples, as well as fewer elements.

[0103] These and other changes can be made to the present technology in light of the above detailed description. While the above description describes particular examples of the present technology and sets forth the best mode contemplated, no matter how detailed the above appears in text, the present technology can be practiced in many ways. The details of the system may vary considerably in its specific implementation, but still be included in the technology disclosed herein. As noted above, specific terminology used when describing particular features or aspects of the present technology should not be construed as implying that the terminology is redefined herein so that it is not limited to any specific characteristic, feature, or aspect of the associated technology. In general, the terms used in the following claims should not be construed to limit the present technology to the specific examples disclosed herein unless the above detailed description section explicitly defines such terms. Thus, the actual scope of the present technology encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the technology under the claims.

[0104] To reduce the number of claims, certain aspects of the present technology are presented below in specific claim formats, but Applicant contemplates various aspects of the present technology in any number of claim formats. For example, while only one aspect of the present technology is recited as a computer-readable medium claim, other aspects may equally be implemented in other forms, such as as a computer-readable medium or in means-plus-function claims. Any claim intended to be treated under 35 U.S.C. 112(f) begins with the words "means for," but the use of the term "for" in any other context is not intended to invoke treatment under 35 U.S.C. 112(f). Accordingly, Applicant reserves the right to pursue additional claim formats in either this application or any continuing application, or to pursue such additional claims after filing this application.

Claims

1. 1. A method for generating splines for an aircraft, the method comprising: continuously presenting, by a user computing device, a graphical user interface (GUI) on the user computing device, the GUI including a display of a current view of a physical environment from a perspective of the aircraft, the view being generated based on sensor data from sensor devices onboard the aircraft; detecting, by the user computing device, user interaction with the GUI, the user interaction including instructions to add a plurality of key frames, each key frame including a location in three-dimensional space of the aircraft; continuously displaying, by the user computing device, a graphical representation of a spline overlaid on a current view of the physical environment from a perspective of the aircraft, the spline including a flight path between each of the plurality of key frames; the GUI further includes a linear representation of the spline, the linear representation of the spline further including graphical marker representations of the keyframes spaced proportionally to the spatial positions of the keyframes on the spline. method.

2. The method for generating splines for an aircraft of claim 1 , wherein each of the plurality of keyframes further includes an orientation of the sensor device.

3. The method for generating splines for an aircraft of claim 1 , wherein the graphical representation of the spline further includes graphical markers indicating locations of the plurality of keyframes on the spline.

4. The method for generating splines for an aircraft of claim 1 , wherein the linear representation of the spline further includes a graphical marker indicating a location of the aircraft on the spline.

5. The method for generating splines for an aircraft of claim 1 , wherein the user interaction further includes an instruction to set aircraft speed.

6. The method for generating splines for an aircraft of claim 1 , wherein the user interaction further includes instructions to edit and delete one or more keyframes.

7. The method for generating splines for an aircraft as recited in claim 1 , further comprising the step of storing said splines in a non-transitory computer readable storage medium.

8. The method for generating splines for an aircraft of claim 1 , wherein each of the plurality of keyframes further includes a sensor device focal length.

9. The method for generating splines for an aircraft of claim 1 , wherein each of the plurality of key frames further includes aircraft velocity and orientation relative to a direction of forward motion.

10. The method for generating splines for an aircraft of claim 1 , wherein the splines are generated and updated by a computing device onboard the aircraft.

11. A computing device for generating splines for an aircraft, comprising: one or more non-transitory computer-readable storage media; program instructions stored on the one or more computer-readable storage media that, when executed by one or more processors, cause the computing device to continuously presenting a graphical user interface (GUI) on the computing device, the GUI including a display of a current view of a physical environment from a perspective of the aircraft, the view being generated based on sensor data from sensor devices onboard the aircraft; detecting a user interaction with the GUI, the user interaction including an instruction to add a plurality of key frames, each key frame including a location of the aircraft in three-dimensional space; continuously displaying a graphical representation of a spline overlaid on a current view of the physical environment from a perspective of the aircraft, the spline including a flight path between each of the plurality of key frames; and program instructions to Equipped with the GUI further includes a linear representation of the spline, the linear representation of the spline further including graphical marker representations of the keyframes spaced proportionally to the spatial positions of the keyframes on the spline. computing device.

12. Each of the plurality of key frames comprises: an orientation of the sensor device; The computing device of claim 11 , further comprising one or more of a sensor device focal length, and an aircraft velocity and orientation relative to the direction of forward motion.

13. The computing device of claim 11 , wherein the graphical representation of the spline further comprises graphical markers indicating locations of the plurality of keyframes on the spline.

14. The computing device of claim 11 , wherein the linear representation of the spline further includes a graphical marker indicating a location of the aircraft on the spline.

15. The computing device of claim 11 , wherein the user interaction further comprises one or more instructions to set a speed of the aircraft or to edit or delete one or more keyframes.

Images