Improved drone flight along calculated splines
The integration of a flight control subsystem in UAVs that records keyframes and allows for real-time path modifications addresses the limitations of programmed UAV flights, enabling complex camera movements and improved data capture.
Patent Information
- Application Number
- JP2024531341
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-03-08
- Filing Date
- 2022-11-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-11-23
AI Technical Summary
Existing unmanned aerial vehicles (UAVs) struggle with the inability to make in-flight adjustments during programmed flights, requiring pilots to terminate automatic operations for manual control, which can result in lost data capture opportunities and limited creative control over camera movements.
UAVs are equipped with a flight control subsystem that records keyframes and calculates splines for autonomous flight paths, allowing for real-time user input to modify these paths, enabling in-flight corrections and enhancements.
Enables pilots of varying skill levels to perform complex, cinematic camera movements and adjustments without interrupting programmed flights, enhancing creative control and data capture capabilities.
Smart Images

Figure 2024542577000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 689,414, filed March 8, 2022, entitled "ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," U.S. Patent Application No. 17 / 689,459, filed March 8, 2022, entitled "GRAPHICAL USER INTERFACE FOR ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," U.S. Provisional Patent Application No. 63 / 282,725, filed November 24, 2021, entitled "ENHANCED UNMANNED AERIAL VEHICLE FLIGHT ALONG COMPUTED SPLINES," and U.S. Provisional Patent Application No. 63 / 282,725, filed January 4, 2022, entitled "INTERFACES AND CONTROL FOR ENHANCED UNMANNED AERIAL VEHICLE This application claims priority to U.S. Provisional Patent Application No. 63 / 296,285, entitled "FLIGHT," which is incorporated herein by reference in its entirety.
[0002] Various implementations of the present technology relate to unmanned aerial vehicles (UAVs), and more particularly, to improved UAV flight along computed splines. [Background technology]
[0003] Unmanned aerial vehicles (or drones) are commonly used to capture video, images, or other data from viewpoints or locations that are otherwise difficult or cumbersome to reach. Drones are used for a variety of purposes, such as recreation, scientific exploration, military operations, intelligence gathering, and commercial use. UAVs for commercial and recreational use typically have multiple rotors so that the UAV is agile and responds quickly to flight commands. For example, a common configuration known as a "quadcopter" includes four rotors for flight.
[0004] The ability of a drone to fly according to a program, i.e., according to a programmed flight instruction set, makes it useful for repetitive operations such as monitoring a safe perimeter. In addition, once a suitable flight program is determined, no particular skill level of the pilot is required. However, while the drone is operating under a programmed instruction set, this denies the pilot any ability to make slight adjustments to the flight path or sensor operation during flight. For example, if the drone captures something unexpected or if the pilot wants to spontaneously correct the video recording, the pilot must terminate the automated flight operation and resort to manual operation of the drone. Alternatively, the pilot can reprogram the flight plan, but at the risk of missing the opportunity to capture potentially important data in the time it takes to reprogram and return to flight.
[0005] Beyond their more prosaic uses, an occupation for which drones are particularly well suited and being actively adopted is dynamic aerial filming. Highly dynamic, smooth, one-shot video was previously only possible for big-budget Hollywood production companies using high-end equipment and large teams of trained professionals. In recent years, teams of world-class drone pilots with thousands of hours of "stick time" under their belts have been able to create similarly dynamic shots. However, these teams of drone pilots can be very expensive to hire and time-consuming to film. Summary of the Invention
[0006] Disclosed herein are techniques for operating an unmanned aerial vehicle (UAV) that fly the drone along a calculated spline while also allowing for in-flight modifications. In various implementations, the UAV includes a flight control subsystem and an electromechanical subsystem. The flight control subsystem records keyframes during flight and calculates a spline based on the keyframes. The flight control subsystem then stores the calculated spline for playback when the UAV automatically flies according to the calculated spline.
[0007] In various implementations, the flight control subsystem can receive user input and responsively modify the calculated spline based at least on the user input, resulting in a modified version of the calculated spline. The flight control subsystem may store the modified version of the calculated spline for later playback. In some scenarios, the UAV may be able to upload (or download) the modified version of the calculated spline to a remote storage location.
[0008] Examples of user input may include one or more of changes to one or more components of the calculated spline, such as the position, direction, speed, and orientation of the unmanned aerial vehicle along the calculated spline. Components may also include a camera focal length and a camera orientation relative to the unmanned aerial vehicle.
[0009] Further examples of user input include directly snapping the unmanned aerial vehicle to a new position on the calculated spline in any order relative to the next position on the calculated spline, reversing the direction along the calculated spline relative to the current direction along the calculated spline, and hovering at a point on the calculated spline.
[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the technical disclosure. It will be understood that this Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
[0011] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the figures are not necessarily to scale, emphasis instead being placed on clearly illustrating the principles of the present disclosure. Also in the drawings, like reference numbers indicate corresponding parts throughout the several views. Although several embodiments will be described in connection with these drawings, the present disclosure is not limited to the embodiments disclosed herein. Rather, all alternatives, modifications, and equivalents are intended to be covered. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 illustrates an operational architecture of an unmanned aerial vehicle in one implementation. [Diagram 2] A diagram illustrating a method of operation of a UAV to create splines in one implementation. [Diagram 3] FIG. 1 is a detailed diagram of a UAV system in one implementation. [Figure 4A] FIG. 1 illustrates an operating environment and an example scenario. [Figure 4B] FIG. 1 illustrates an operating environment and an example scenario. [Figure 5A] FIG. 1 illustrates an exemplary UAV flight path. [Figure 5B] FIG. 1 illustrates an example UAV calculated flight path around a building. [Figure 6A] 1 is an exemplary overhead view of flying a UAV around a house. [Figure 6B] 1 is an exemplary overhead view of a calculated spline flight path. [Figure 6C]FIG. 13 is an example overhead view of a calculated spline flight path with corrections to the flight path and drone orientation. [Figure 6D] FIG. 13 is an exemplary close-up overhead view of the flight path and corrections to the drone orientation during flight along the calculated flight path. [Figure 6E] FIG. 1 illustrates an exemplary display on a remote control device from a first person perspective as a pilot issues flight commands during flight along a calculated flight path. [Figure 7] FIG. 1 illustrates an example workflow when a pilot issues flight commands as the drone flies under autopilot control. [Figure 8] 13A-13C are exemplary overhead views of the effects of pilot commands on an autopilot flying a drone along a calculated flight path. [Figure 9] 13A-13C are exemplary overhead views of the effects of pilot commands on an autopilot flying a drone along a calculated flight path. [Figure 10] FIG. 13 is an exemplary overhead view of the calculation of new position points for corrections to a calculated flight path during flight. [Figure 11] FIG. 1 illustrates a workflow for controlling an aircraft in one implementation. [Figure 12A] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12B] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12C] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12D] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12E]FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12F] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12G] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12H] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12I] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 12J] FIG. 1 illustrates a graphical user interface of an autonomous flight control application on a drone remote control device in one implementation. [Figure 13A] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13B] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13C] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 13D] FIG. 1 illustrates a user interface of an autonomous flight control application of a drone remote control device in keyframe mode in one implementation. [Figure 14A] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14B]FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14C] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 14D] FIG. 1 illustrates a graphical user interface of an autonomous flight control application in one implementation. [Figure 15] 1 is a series of images showing a user interface of an autonomous flight control application in one implementation during playback. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] The drawings are not necessarily drawn to scale. Similarly, some components and / or operations may be divided into different blocks or combined into a single block for purposes of discussion of some of the embodiments of the present technology. Also, while the present technology is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are described in detail below. However, there is no intention to limit the present technology to the particular embodiments described. Rather, the present technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.
[0014] The technology discussed herein allows anyone to capture continuous choreographed shots with complex, cinematic, smooth camera movements. The user defines a flight path by setting points, called "keyframes," and the software creates smooth splines between the points that can be flown repeatedly with varying degrees of control, speed, and complexity. In fact, the technology discussed herein enables complex Hollywood-style camera movements that would otherwise be impossible for any pilot, regardless of their skill level. The technology does not replace human creativity or work, but rather allows pilots to unlock amazing shots with just a few taps.
[0015] Various implementations disclosed herein include an unmanned aerial vehicle that includes a flight control subsystem and an electromechanical subsystem. As discussed above, the flight control subsystem can record key frames during flight and calculate splines based on the key frames. The flight control subsystem stores the calculated splines for playback, at which point the flight control subsystem instructs the electromechanical subsystem to fly the UAV according to the calculated splines.
[0016] In one or more of various implementations, the flight control subsystem may also receive user input and modify the calculated spline based on at least the user input. The modified version of the calculated spline may itself be saved for later playback. The user input may cause one or more of the changes to the calculated spline, such as a change in the position, direction, speed, and / or orientation of the unmanned aerial vehicle along the calculated spline. Other changes include modifications to the focal length of the camera on the UAV and the orientation of the camera, or changes to any other peripherals on the UAV. Further examples of user input include directly snapping the unmanned aerial vehicle to a new position on the calculated spline out of order relative to the next position on the calculated spline, reversing the direction along the calculated spline relative to the current direction along the calculated spline, and hovering at a point on the calculated spline.
[0017] In one operational example, a drone pilot operating a controller device identifies a set of discrete spatial locations called keyframes. Resources onboard the drone and / or the controller (or distinct from either system) calculate splines based on the keyframes. The calculated splines are then "played back" by the drone, meaning that a flight control subsystem onboard the drone commands its flight based on the calculated splines.
[0018] Continuing with the operational example, the pilot may modify the calculated spline on the fly while the drone is flying the spline, allowing the pilot to focus and control one or more aspects of the drone operation without having to actively pilot the drone. The pilot may, for example, change the position, direction, speed, and / or orientation of the drone as it flies the calculated spline. These changes may cause the drone to move away from the calculated spline or change direction as it moves along the spline. The pilot may accelerate or decelerate the drone along a segment of the spline or at a point along the spline. The pilot may also modify the camera's movement, such as zooming in or out or adjusting exposure, as the drone flies along the spline. In another example, the pilot may cause the drone to reverse course along the calculated spline or "snap" to a new position on the spline without moving along the spline. The pilot may also cause the drone to stop and hover at a point on the calculated spline, for example to add a keyframe at that location. Any or all of these modifications may be saved for later use with the spline or as a new version of the spline. In some implementations, the pilot may control the orientation of the drone along the spine and / or gimbal position as the drone moves along the calculated spine, i.e., in "free look mode." In this way, the pilot can focus on the camera angle and position without having to actively pilot the drone.
[0019] Various technical effects of the disclosed technology may be appreciated from this disclosure. For example, when a drone is used to record video, a common use of drones, pilot input incorporated into the programmed operation allows for more creative buy-in of the video product by allowing the pilot to focus on the artistic aspects and nuances of the recording without having to actively navigate the drone's flight. The ability to incorporate pilot or user input into the drone's operation while flying under the authority of its autopilot may be exercised in several ways.
[0020] In one example, the UAV flies a programmed or predefined flight plan. The programmed or predefined flight plan may be a calculated spline flight path whose trajectory is defined by a set of discrete positions through which the drone will pass, or may be a flight path recorded during a previous flight. It may also be a flight path that is manually programmed and uploaded to the drone. The programmed flight behavior of the drone includes parameters such as the position, orientation, and / or speed of the drone as it flies along the predefined flight path.
[0021] Drone operations may include subject tracking during programmed operations. For example, a drone may be deployed to record a bicycle race by flying a programmed route along the race track from an overhead perspective while tracking the progress of a particular cyclist. Drone operations may also include object detection and avoidance so that the drone does not crash into a tree while flying information and recording the bicycle race. In this way, the pilot can focus on a particular aspect of the drone operation, such as a view captured by an on-board video camera, without having to actively fly the drone at the same time. When the pilot stops controlling that aspect of the drone operation, the drone resumes smoothly back to its programmed operation.
[0022] In another example, a drone or UAV docks prior to flight. The dock provides the drone with orientation information that allows the drone to ascertain its position and orientation relative to the dock in order to navigate during flight. The position information for navigation may be specified in three-dimensional coordinates, such as a set of coordinates relative to the dock, or coordinates detected by an on-board GPS sensor. The position information for navigation may also incorporate or rely on visual tracking using one or more on-board navigation cameras. The drone orientation information may refer to the drone's pitch, roll, and yaw angles. The pilot wirelessly communicates with the drone's flight control system using a remote control. The remote control may be a dedicated device or an app on a computing device such as a laptop computer, tablet, or smartphone. The user interface or UI on the UAV remote control may include a display screen and one or more input mechanisms for controlling the drone, such as buttons, rocker switches, sliders, toggles, etc. One implementation of a user interface on a drone remote control includes a touch-enabled display screen, i.e., a touch screen, that displays functional graphical object representations of the control input mechanisms. Wireless transmissions between the UAV and the remote control device may be carried over a WiFi connection, a Bluetooth connection, or any suitable wireless connection. The UAV may also wirelessly communicate with other devices, such as computers, that can receive, view, record, and store information transmitted from the drone.
[0023] Once the UAV pilot issues a command to launch the drone from the dock and the drone begins its flight, the pilot may actively fly the drone or the drone may fly according to a predefined flight plan. In one implementation, the predefined flight includes a set of key frames, each of which may be defined according to the visual input of one or more navigation cameras in a process of dead reckoning or visual tracking. Key frames defined according to visual tracking are particularly useful in indoor or outdoor environments where Global Positioning System (GPS) data or other inertial navigation data is partially or completely unavailable, or when using drones that lack inertial or satellite navigation capabilities. Key frame positions may also be defined in three-dimensional coordinates relative to the drone dock position, or from satellite or inertial navigation data.
[0024] A keyframe may contain additional information about drone orientation, drone speed, or onboard sensor activity. For example, a drone may be programmed to pause at a keyframe and pivot horizontally to capture a panoramic shot as a camera's zoom lens moves from a telephoto to a wide-angle focal length. The drone's path from one keyframe position to the next is a calculated function, such as a cubic spline interpolation formula or "spline."
[0025] In another implementation, the predetermined flight plan may be a previously recorded drone flight stored in the drone's persistent storage (e.g., non-volatile memory) or on a data storage device in wireless communication with the drone. The pre-recorded flight plan may be uploaded to other drones for reuse. For example, a flight plan for monitoring the perimeter of a secure facility may be recorded and stored for periodic reuse or when a drone is replaced by a backup drone. The pre-recorded flight path may include the entire flight from launch from a dock to a return to the dock, or may include a subset of the drone's flight, such as a spline between two key frames, or even a drone operation in one location (such as a sweeping video shot). In yet another implementation, the predetermined flight path may be a computer program uploaded to the drone. For example, the flight plan may be programmed based on a map or terrain data.
[0026] In another implementation, the pilot can record aspects of the drone's operation, such as recording the drone's trajectory, recording the drone's camera view, pausing the recording, deleting the recording, and saving the recording. The recorded operations or recorded components of the flight may be stored in on-board non-volatile memory, or they may be transmitted to a device in communication with the drone, such as a remote control, a laptop computer that receives transmissions from the drone, or cloud data storage.
[0027] In yet another implementation of the technology, a display screen or touch screen of the user interface displays in real time a view transmitted from a forward-facing camera on the drone, known as a first-person perspective. An augmented reality (AR) graphic representing the predetermined flight path is superimposed on the forward-facing camera view in the form of a semi-transparent curve that overlays the camera view, optionally in a color significantly different from the background. The AR representation may indicate the distance from the drone along the flight path by varying the width of the curve, e.g., the curve becomes thinner with distance from the drone. The AR representation is continuously updated as the drone flies the predetermined flight path. Additional AR graphics may indicate key frames or waypoints identified by unique shapes and colors, such as diamonds on the curve representation of the predetermined flight path.
[0028] In various implementations, a pilot may add, edit, and delete keyframes on a given flight path via a user interface on a remote control device. For example, when a pilot wants to identify and select keyframes for a desired camera view, the user interface displays AR graphics in the form of semi-transparent frames over the camera views that make up the camera shot. In addition to virtual framing, the UI touch screen may display virtual buttons for adding, editing, and deleting keyframes. Keyframes may record and store information such as drone position or location, drone orientation, and information about the operation of on-board sensors such as cameras. The AR representation of a given flight plan defined by keyframes may be updated according to the latest set of keyframes.
[0029] In one implementation, a user interface on the remote control displays a linear playback track or timeline representation of a given flight plan. The linear timeline representation may include keyframes identified on the timeline by distinctive shapes, such as diamonds. Distance along the timeline may be proportional to actual flight distance. As the drone flies the given flight path, the timeline may show the drone's progress along the flight path, using one color to indicate the completed portion and a second color to indicate the remaining portion. Optionally, an arrow or other symbol may move along the timeline as the drone flies to show the drone's movement along the flight path in real time.
[0030] In one implementation, the pilot may issue commands to the drone using the flight path timeline. In an implementation of a timeline graphic on a touch screen, the pilot may command the drone to reverse direction on the flight path or pause at a point on the flight path by touching a particular point or location on the displayed timeline. The pilot may also command the drone to snap (i.e., fly immediately) to a point on the spline by moving along the spline or by flying directly to the indicated point. For example, the pilot may command the drone to jump to a point between two keyframes and add a keyframe at that location.
[0031] In one implementation of the technology, the UI screen also shows the drone speed at points along the timeline. The UI may include virtual or physical controls, such as rocker switches or sliders, to adjust drone speed along segments of the flight path or at points on the flight path. For example, pilots may command the drone to pause (i.e., hover) at various keyframes to capture long static views or panoramas from their vantage point.
[0032] In another implementation of the technology, a virtual slider controlling drone speed is displayed in multiple colors to indicate multiple zones of dynamic feasibility. Starting at the slow end of the slider range, green can indicate a range of speeds that are dynamically feasible for the drone to fly, yellow can indicate a range of speeds that push the drone's operating envelope, and red can indicate a range of speeds that are dynamically not feasible for the drone to fly. For example, if a given flight path indicates a turn, the red portion of the slider corresponds to a speed at which the drone cannot navigate the turn without flying off the flight path.
[0033] The pilot may maintain aspects of the drone's motion control as it flies a predefined flight plan. For example, once a completed drone flight path is recorded, the pilot may command the drone to re-fly the recorded flight path while manually controlling the camera orientation. The camera orientation may be controlled by changing the drone's pitch, roll, and / or yaw angles. Additionally, as the drone flies the predefined flight plan, the pilot may issue commands or motion commands via the remote control device that cause the drone to deviate slightly from the flight path or change the drone's orientation as it flies. For example, the pilot may nudge the drone's orientation to turn westward for a few seconds as the drone flies a predefined path heading north.
[0034] In another implementation of the present technology, the drone receives and incorporates real-time input into its flight behavior corresponding to a predetermined flight path. When the pilot's real-time input stops, the drone performs an auto-return to the predetermined flight path. In yet another implementation of the present technology, the real-time input is smoothed or damped, resulting in less adjustment to the drone's flight. When the real-time input stops, the return to the predetermined behavior is similarly smoothed or damped.
[0035] For example, as the drone flies the spline, the pilot may activate a subject tracking capability that allows the drone to keep its orientation pointed toward the subject so that the subject is always within the drone camera's field of view. Alternatively, an object avoidance function may cause the drone to deviate from its programmed flight path if the flight path intersects with an obstacle. In cinematography, the ability to manually control one or more aspects of the drone's operation (e.g., drone flight dynamics, drone orientation during flight, and onboard camera operation) as the drone navigates a predetermined flight path can give the pilot or videographer operating the drone a greater sense of creative ownership of the video recording, since the operation is not strictly programmed or mechanical.
[0036] In another implementation of the technology, deviations from or adjustments to a predefined flight plan made as the drone flies the flight path may be saved for later use. The adjustments may be saved by themselves (to be added to the predefined flight path), or the flight path and adjustments may be saved together as an entirely new flight path. In this manner, multiple adjustments to a particular predefined flight path may be superimposed on the flight path, enabling the ability to create flight plans of increasing complexity or variability. For example, a flight path may be re-flown multiple times with different camera orientation movements each time to compare and avoid various viewpoints.
[0037] In one implementation of the technology, a drone or UAV may be docked at a location away from the pilot. The pilot typically flies the drone by maintaining line of sight to the drone in accordance with FAA regulations governing drone flight. However, in certain situations, the pilot may navigate the drone without having line of sight to the drone, relying on the drone's first-person perspective, i.e., by seeing what the drone camera sees. This method of flying a drone is generally only possible in certain limited situations, such as indoor flight in a large warehouse or stadium.
[0038] A drone autopilot receives and incorporates several internal and external operational inputs that govern the flight of the drone to issue commands to the drone microprocessor. These commands are received by the microprocessor as if issued by a (human) pilot and are therefore issued by the autopilot as apparent joystick commands. The autopilot integrates the drone's calculated or programmed flight path with sensor data related to the drone's operation, such as wind speed and direction data. The autopilot may also receive joystick inputs when the pilot issues commands via a joystick on a remote control device. The joystick inputs are interpreted according to a specific function assigned to the joystick during autopilot operation. For example, the joystick may be used to change the drone's pitch or yaw during programmed operation to change the view of the onboard camera. Management functions may be applied to the joystick inputs that can dampen or limit the inputs so that the drone does not exceed the operating envelope for its flight operations. The drone autopilot may also receive inputs from a collision avoidance system or a subject tracking system. Based on inputs from these various sources, the autopilot calculates and issues apparent joystick commands to the drone microprocessor. In response to receiving the apparent joystick commands from the autopilot, the microprocessor transmits flight commands to the drone electro-mechanical propulsion system to cause the drone to fly according to the commands of the autopilot.
[0039] Returning to the bicycle race example, during programmed operation, the drone pilot may command the drone to change its orientation in flight to get a view of a crowd of spectators along the race route, or of a particularly striking vista in the distance. Alternatively, the pilot may change and adjust the operation of the onboard camera, such as by zooming out for a wide-angle shot of a distant mountain range or zooming in for a close-up of a cyclist. When the pilot stops to correct the drone's flight or operation, the autopilot receives input from the joystick corresponding to a return to its neutral position, thereby making a smooth return to its programmed or calculated flight plan.
[0040] Referring now to the drawings, Figure 1 illustrates an unmanned aerial vehicle (UAV 101) and its components represented by operational architecture 128. Operational architecture 128 broadly includes flight controller subsystem 124, electromechanical subsystem 126, external operational inputs 120, and internal operational inputs 122. Flight controller subsystem 124 may include a circuit board housing one or more microprocessors, also known as flight controllers, that control various aspects of the drone operation. Electromechanical subsystem 126 may include an electronic speed controller unit, and various rotors, power supplies, etc.
[0041] External operational inputs can include inputs received from a remote control device, typically operated by a human pilot, and sensor data that measures environmental conditions that affect UAV operation. Internal operational inputs can include programs or calculated flight or motion plans that direct drone position, drone movement, or sensor operation during flight, and other information related to a particular use or capability of the UAV, such as map or terrain data.
[0042] FIG. 2 illustrates a process 200 implemented by one or more components of the flight control subsystem 124 of the UAV 101. The process 200 is implemented in program instructions that, when executed by one or more hardware and / or firmware elements of the flight control subsystem 124, direct the UAV 101 to operate as follows: As the UAV 101 is flying, the flight control subsystem 124 of the UAV 101 records one or more key frames (step 210). The pilot of the UAV 101 may instruct the UAV 101 to record the key frames based on a first-person perspective from which the pilot views the remote control device 130. The recorded key frame data may include parameters such as the drone's physical location based on visual tracking or based on the drone's three-dimensional coordinates and orientation at that location. The key frame data may also include data regarding on-board camera or sensor movements at the key frame locations. The flight control subsystem 124 of the UAV 101 calculates a flight path or calculated spline that connects the key frames (step 220). The flight path may be calculated by an on-board microprocessor using a set of discrete points or waypoints identified by the pilot during the current or previous flight. The flight control subsystem of the UAV 101 stores the calculated spline for later use (step 230).
[0043] The calculated spline may be re-flown by the UAV 101 or by a backup drone while the UAV 101 is recharging. The calculated spline may be stored locally, such as in on-board persistent memory or data storage. It may also be stored remotely, such as in data storage on a device in communication with the drone, such as a remote control device or laptop computer that receives transmissions from the drone. This may also include, for example, remote cloud data storage. In future flights of the stored spline, the pilot may make modifications to the flight, for example, to make incremental improvements to the drone's operation, to make temporary adjustments based on unexpected conditions, or to explore different ways to operate the drone. These modifications may be stored as well.
[0044] FIG. 3 illustrates an exemplary system architecture 300 of the quadcopter 401 of FIG. 4. The system architecture 300 includes a flight control subsystem 391 and an electromechanical subsystem 392. The flight control subsystem 391 includes an autopilot function (represented by an autopilot 328, a flight controller 326, an inertial measurement unit 302, sensors 304, a transmitter 306, a receiver 308, and a memory card port 310). The electromechanical subsystem 392 includes an electronic speed controller 312 and a rotor 314. It will be understood that both the flight control subsystem 391 and the electromechanical subsystem 392 may include other elements in addition to (or instead of) those disclosed herein, which are shown for illustrative purposes. The system architecture 300 also includes operational inputs 393. The operational inputs 393 include joystick data 322 provided by a remote control device 318, as well as management factors 330 and a calculated flight path 332.
[0045] The inertial measurement unit 302 includes one or more sensors, such as gyroscopes and accelerometers, that provide motion and orientation data to the flight controller subsystem. In some implementations, the flight controller subsystem may also be connected to or include other sensors 304, such as a video camera, a global positioning system (GPS) sensor, a magnetometer, or a barometer. The UAV also carries equipment for wireless communication, such as an antenna, a video transmitter 306, and a radio receiver 308, which allow communication with a remote control device 340 to which a human pilot can send commands, such as flight commands or commands related to on-board sensor operations. The remote control device may be a dedicated device or may be an application on a mobile computing device, such as a smartphone, tablet, or laptop computer, capable of wireless communication with the UAV 401. Wireless communication between the remote control device and the UAV may be carried over a WiFi network or a Bluetooth link. The flight controller subsystem may also be connected to an on-board permanent or non-volatile memory or memory card port 310 for recording flight and sensor operations and data. As part of the electromechanical subsystem of the UAV 401, the electronic speed controller 312 is connected to the flight controller subsystem and controls the operation of the rotor 314 according to flight commands 316 received from a microprocessor on the flight controller subsystem.
[0046] The remote control device 340 of the drone 401 includes wireless communication hardware for communication with the drone 401, as well as a throttle device (e.g., a physical or virtual joystick) 320 for manually controlling the flight (i.e., speed and direction) of the drone 401. For example, when the pilot moves the joystick 320, the remote control device 340 transmits joystick data 322 to the UAV 401. Although the pilot can control the drone 401 based on his or her line of sight to the drone, remote control devices for drones typically have a display screen 324 for displaying the perspective of an onboard camera, referred to as a first-person view. The first-person view capability allows the pilot to find and capture views from remote or difficult-to-access viewpoints.
[0047] The exemplary operating environment 400 of FIG. 4A illustrates the process of creating a calculated spline flight path. A pilot 404 initiates flight of a drone 401 by launching the drone from a drone docking device 402. The drone dock 402 provides orientation information of the drone 401 at the beginning of the flight so that the drone 401 can ascertain its position relative to the dock 402 during flight. The drone flight may be manually controlled by the pilot 404 using a remote control device 403 in wireless communication with the UAV 401. The pilot may use a joystick 320 to turn or accelerate the drone 401. The remote control device 403 transmits the pilot's 404 inputs received from the joystick 320 to the drone 401 via an on-board receiver 308 coupled to a flight controller 326. The flight controller 326 translates the pilot's inputs into flight commands 316 issued to the electronic speed controller 312, which slows the rotors 314 accordingly.
[0048] In the sequence of events shown in FIG. 4A, the drone 401 is piloted by the pilot 404 along an arbitrary route 405 (event 1). In event 2, the pilot 404 identifies a location to be saved for the spline at point A. The pilot 404 uses the remote control device 403 to add a key frame to that location as the drone flies. This step may be implemented using a virtual button object displayed on the touch screen of the remote control device 403 in one aspect of the present technology. Key frame A is saved in event 3 along with flight and / or operation data associated with location A. The key frame data saved in event 3 includes location coordinates (e.g., GPS coordinates or coordinates relative to the dock 402) and may also include data regarding the speed of the drone 401 at location A, its orientation at location A, and the operation of the onboard camera. The key frame data may be saved in data storage on the remote control device 403 or may be saved in data storage onboard the drone 401. The flight continues. Upon reaching location B, the pilot 404 adds another key frame (event 4). Keyframe B is saved in event 5 in a manner similar to event 3. In event 6, a spline 410 is calculated that connects keyframes A and B. The spline calculation may be calculated within an on-board processor of the drone 401, or may be calculated within a processor of the remote control device 403 and then transmitted to the drone 401. For clarity, in this example, the spline calculation is performed by a processor in the on-board flight control subsystem of the drone 401.
[0049] FIG. 4B shows drone 401 in operating environment 400 in flight after recording keyframes "A" and "B" and calculating spline 410. Keyframes "A" and "B" and spline 410 may be stored in on-board data storage of drone 401, or the keyframes and spline may be uploaded to drone 401 before or during the current flight from a remote device that stores the information, such as remote control device 403. In event 1, pilot 404 commands drone 401 to play spline 410. In event 2, drone 401 flies spline 410. As drone 401 flies spline 410, drone 401 operates under the control of its on-board autopilot, which is part of drone 401's flight control subsystem. In events 2 and 3, drone 401 passes locations A and B, respectively, which are points on spline 410 that were stored as keyframes. As drone 401 plays spline 410, pilot 404 may issue flight or movement commands to modify its behavior under the command of the autopilot. For example, pilot 404 may command drone 401 to disengage and return to spline 410; to accelerate, decelerate, or stop and hover at a location on spline 410; or to reverse direction along 410. Pilot 404 may command drone 401 to face north as it flies, rather than facing forward along the spline. Pilot 404 may add additional keyframes to spline 410. Pilot 404 may operate the drone camera as the autopilot navigates drone 401 along spline 410.
[0050] A UAV may be commanded to fly a predetermined flight path. The flight path may be defined by a discrete set of position data (and optionally speed data) called waypoints. The waypoints may be specified in three-dimensional Cartesian coordinates. Waypoints may be selected for different purposes. Some waypoints may be locations where the drone is intended to stop and view a point of interest, while other waypoints may specify a precise location, e.g., a door or window, that the drone must reach in order to pass through.
[0051] When a drone is used for videography, a pilot may use key frames to identify desired shots. View 500 in FIG. 5A shows eight exemplary key frames 504 selected to view the exterior of a building 506 from an aerial position. Each key frame 504 is a static record of a waypoint in addition to the orientation of the drone 101 (i.e., the orientation of the drone's onboard camera) at that waypoint so that the desired viewpoint can be recaptured at different times. Drone orientation refers to the angular position of the drone 101 relative to forward horizontal flight, i.e., pitch angle 112, roll angle 110, and yaw angle 114 as shown in FIG. 1. While viewing a scene captured by the drone camera at a waypoint, the pilot can capture a panorama by varying the drone's yaw angle from left to right of its forward-facing position. Similarly, the tilt of the camera can be changed by varying the drone's pitch angle. For example, a videographer operating a drone may want to capture a construction site from an elevated vantage point looking down at different stages of construction to document the progress of the work. The technique is also useful for before-and-after comparisons of developments over large areas, or for rebuilding communities after natural disasters.
[0052] The drone 101 can record a particular location in three-dimensional space as it flies by recording sensor or telemetry data about the location, such as the drone's distance and orientation from the drone dock 402, GPS data collected from the on-board GPS sensor 304, visual data collected from the on-board camera 304, or a combination of these. Similarly, the orientation of the drone 101 at a particular location can be recorded and stored using data from the inertial measurement unit 302.
[0053] One technique for programming the flight of a UAV is to record a continuous set of keyframes 504 that define the flight path and video movement of the drone 101. For example, when checking a security perimeter, the pilot may define a set of keyframes that capture every point of a building's entrance. The drone's 101 operating system then calculates the drone's 101 flight path and its orientation during flight from one keyframe to the next. On subsequent flights, the pilot deploys the drone 101 to fly the same route and capture the same views each time, making it easier to identify when something has changed.
[0054] An alternative to recording a successive set of static keyframes 504 to define the flight path and video motion is to record successive keyframes, where the flight path and video motion of an initial flight are recorded continuously over a period of time. A flight recorded as successive keyframes can be re-flown later if desired. Because successive keyframes record all motion, including any jerky or erratic movements or other idiosyncrasies associated with manual flight control, this mode of operation may be more appropriate for experienced drone pilots or those with more competent flying skills.
[0055] When calculating a flight path based on a set of static keyframes, the calculation of the route from one keyframe to the next is a mathematical operation known as spline interpolation. Spline interpolation is a method of fitting a curve to a set of points by computing a piecewise set of low-order polynomials, or splines, over successive pairs of points such that the resulting function is continuous and smooth over its domain. This is a simpler and more stable method than polynomial interpolation, which involves fitting a single higher-order polynomial to all the points in the set.
[0056] In a basic implementation of spline interpolation, the path between any two points in three-dimensional space is a straight line. Connecting consecutive pairs of points with straight lines generates a continuous path of a shape similar to a polygon. When using this method to create a flight plan, the resulting set of connected straight lines will have points where the direction changes abruptly. FIG. 5A shows an example flight path 508 that includes straight line segments defined by a set of eight key frames 504 at various locations around a building 506. Such abrupt changes in direction are not only aesthetically undesirable for cinematography, but may also be dynamically infeasible, i.e., beyond the flying capabilities of the drone 101.
[0057] However, higher order splines can connect consecutive sets of points with curves that are more suitable for drone flight. That is, a set of cubic polynomials is generated over consecutive pairs of points with the requirement that the polynomials are twice continuously differentiable, i.e., continuous and smooth when brought together. Smoothness is defined mathematically by requiring that the lower order derivatives be equal at the knots (i.e., the points where the functions are brought together). In cubic splines, smoothness requires that both the tangency and curvature of the functions be equal at the knots, which would eliminate abrupt changes in velocity and acceleration. Thus, the curves generated in this way generate flight paths that are aesthetically appealing for cinematography and dynamically feasible for drone operation. Furthermore, cubic splines satisfy an additional constraint for drone flight and cinematography that the third and fourth derivatives of the interpolation equation with respect to time, known as jerks and snaps, are zero. View 520 in FIG. 5B shows flight path 510 through the same keyframes 504 of FIG. 5A but connected to a cubic spline interpolation equation. The spline interpolation calculations can be done on the fly, so to speak, as the pilot adds or removes keyframes in the sequence. Furthermore, by varying the waypoints or end point constraints, the characteristics of the calculated curve can be altered, thus affecting the dynamic characteristics of the flight path.
[0058] When the drone is recording video while flying along the calculated flight path, the orientation of the drone 101 can be programmed to provide smooth, stable video recording, eliminating any uneven camera movement or direction changes that may occur during manual camera movement, and allowing camera movement to be subsequently recaptured once a preferred orientation program is found. In a process similar to that of interpolating a flight path over a set of waypoints, a drone orientation function can be interpolated to provide smooth drone movement along the flight path using the drone orientation data (e.g., pitch angle 112, roll angle 110, and yaw angle 114) specified in the waypoints. And just as successive keyframes record the position of the drone 101 over a period of time during the initial flight, the orientation of the drone 101, and therefore its camera or other sensors, can also be recorded during the initial flight for later use.
[0059] In one aspect of the present technology, during programmed drone operation, the pilot may wish to modify video recording or other sensor data collection while the drone 101 is flying by making small, temporary changes to the position or orientation of the UAV without terminating the UAV's programmed operation. The pilot may issue flight or motion commands via the joystick 320 on the remote control device 130 that are sent to the drone's flight controller subsystem. Upon receiving new joystick data, the flight controller subsystem modifies flight commands issued to the electronic speed controller 312 based on the calculation of the calculated spline 510, successive keyframe paths, or modifications to the programmed flight path. The modifications incorporate one or more factors governing drone flight, such as environmental conditions (e.g., wind speed and direction) or obstacle detection and avoidance. For example, a set of keyframes is identified to interpolate a flight path, and if the interpolation produces a path that passes through a tree, a proximity sensor on the drone will detect the tree and the drone will complete a correction to the spline to bypass the tree and return to the spline.
[0060] In an exemplary use shown in the overhead view 600 of FIG. 6A, a real estate agent wishes to provide a video recording of a residential property 602 from an exterior aerial viewpoint for marketing purposes. A drone system with videography capabilities, comprising a drone 101 and a videographer 606 flying the drone 101 using a remote control device 130, is dispatched to perform the recording. In a first flight 601 shown in FIG. 6A, the videographer 606 flies the drone 101 around the house 602, selectively picking a set of particularly desirable viewpoints and saving that information as a set of five key frames 608 including drone position and drone orientation information at each key frame. Exemplary display screens 610 and 612 for identifying and adding key frames to the set are shown in FIG. 6A, and an augmented reality image 614 can be overlaid on the first person viewpoint to accurately identify the camera shot associated with the key frame.
[0061] For the subsequent flight, the flight controller subsystem 124 calculates a flight path 622 as shown in the overhead view 620 of Figure 6B via spline interpolation. The drone 101 flies the flight path 622 determined by the interpolation equation and records a video of the entire surroundings of the house 602, including the particularly desired viewpoint of the key frame 608.
[0062] Continuing with this exemplary use, assume that on a later flight, the real estate agent wishes to include in the video recording a shot of the pond 616 behind the house 602. Rather than recording a new set of keyframes around the house 602, while flying the drone 101 around the property on the calculated spline, the videographer 604 modifies the drone movement while flying on the calculated spline 622. As shown in the overhead view 640 of FIG. 6C, the videographer 604 issues a command using the joystick 320 to move the drone 101 slightly leftward 624 from the calculated spline 622 and redirect the camera away from the house 602 and towards the pond 616. After achieving the desired recording, the videographer 604 returns the joystick 320 to its neutral position, and the drone 101 calculates a return path 626 to the calculated spline 622 and a reorientation to the orientation spline to continue its flight. A close-up view of the calculated corrections to flight path 622, including departure 624 and return 626 and drone transitions indicated by arrows, is shown in Figure 6D. Figure 6E shows an example first-person perspective of drone 101 as seen on display screen 324 of remote control device 130 at a series of points V1 to V10 before and during a correction of its flight along flight path 622 in accordance with flight commands issued by videographer 604.
[0063] 7 illustrates a process 700 implemented by one or more components of the flight control subsystem 124 of the UAV 401. The process 700 is implemented in program instructions that, when executed by one or more hardware elements of the flight control subsystem on board the drone 401, direct them to operate as follows: The drone pilot uses the joystick 320 to correct flight of the drone 401 along the calculated spline (step 710), where the corrections may affect the drone's position or orientation, or some combination of these. Joystick data is wirelessly transmitted from the remote control device 403 to the autopilot 328 of the drone 401 (step 712). The autopilot 328 also receives external operating inputs (i.e., wind speed and direction data) that affect the drone flight (step 714). The autopilot 328 calculates corrections to the calculated spline (step 716). At event 718, this correction is sent to the flight controller 326, which at event 720 instructs the drone electromechanical subsystem of the drone 401 to fly according to the correction to the calculated spline (step 718). The electromechanical subsystem issues flight commands that simulate the actual joystick commands, in other words, the composite or modified joystick commands 330 (step 720). The modified joystick commands 330 include the pilot's input along with factors that govern the drone operation. In another implementation of the present technology, the pilot can command multiple simultaneous corrections to the flight during a programmed operation using a remote control device with multiple joysticks, each joystick assigned to a specific aspect of the drone operation, or by assigning multiple functions to a single joystick. For example, considering the two joysticks shown in FIG. 1, input from one joystick may control the drone's yaw angle 114, while input from the other joystick controls the drone's pitch angle 112, giving the pilot the ability to focus on fine-tuning the camera while the drone flies autonomously along a predetermined flight path.
[0064] In another implementation of the present technology, the management factor may convert the actual joystick command into a modified joystick command by applying a damping function to the actual joystick input or data. In one aspect of the present technology, the damping function may be a mathematical model that simulates the response of a spring-rigged object subjected to a force corresponding to the actual joystick input. More specifically, the spring-rigged model converts real-time input by a pilot using a joystick into the response of a simulated object rigged with three linear or three torsional overdamped screws subjected to displacement in one or more directions. One result of applying a damping function to the pilot's joystick is a dampened input to the drone that prevents abrupt dynamic responses in the drone's velocity or orientation, resulting in a correction to the calculated spline that maintains cinematically desirable characteristics. Another result of a damping function such as a spring model is that when the pilot releases the joystick and allows it to return to its neutral position, the drone receives that joystick input and converts it into a modified joystick input that gradually reduces the correction to zero for a smooth return to the spline. Similarly, if the pilot uses the joystick to reorient the drone to point in a different direction than that determined by the orientation spline (rather than moving the drone off the calculated spline), the joystick input is damped to avoid abrupt mechanical responses upon reorientation, resulting in a more desirable response for cinematic purposes.
[0065] In yet another implementation of the present technology, the application of the damping function can provide a limit to the calculated spline or modifications to the drone's predetermined flight behavior. This limit to the drone's dynamic response to the pilot's input creates an operating envelope around the drone's calculated or predetermined flight path. For example, in the exemplary overhead view shown in FIG. 8, using joystick data filtered through the damping function, the pilot can cause the drone 804 to make a small deflection 806 from the flight path 802 as it flies the flight path 802. The damping function applied to the pilot's flight commands creates an operating envelope 810 around the flight path 802. The autopilot of the drone 804 continues to provide operating instructions to the electromechanical subsystem while taking into account the joystick data received from the remote control device. Thus, the pilot can focus his or her attention on certain aspects of the drone's operation without having to assume complete control of the drone's operation.
[0066] FIG. 9 shows an overhead view 900 of a predetermined drone flight path for a drone flying under autopilot control. As the drone flies the predetermined flight path 904, if the drone pilot pushes the joystick to the left (event 920), such an input would normally disengage the autopilot operation, resulting in the drone turning left, and if the joystick is held in that position and released shortly thereafter, the drone will fly backwards (926) and eventually stop and hover. However, in one implementation of the present technology, the same joystick input is filtered through a modified joystick command damping function. The modified joystick command causes the drone to depart 902 slightly to the left from the flight path 902, but still follow the flight path 902. Thus, when the joystick returns to the neutral position (event 922), i.e., when the pilot stops pushing the joystick, the drone executes a return 906 or resumption to its calculated flight path 904 or programmed operation. The net effect of such an implementation of the technology is to create an operating envelope 910 around the flight path 904 within which the drone will preferentially adhere to the flight path 904, but can deviate from that path in terms of the drone's position or orientation based on joystick data provided by the drone pilot.
[0067] As the drone flies a predetermined flight path, the autopilot communicates modified joystick commands to the electromechanical subsystem, which then slows down the rotor accordingly. These modified joystick commands are typically issued several times per second, such as every 50 milliseconds. The commands incorporate the drone's current position, the drone's desired next position according to the flight path, and external or environmental factors, such as wind speed and direction. In one aspect of the present technology, the autopilot incorporates factors that govern the drone flight operation into the modified joystick commands. These factors can affect the drone operation, such as by dampening the drone's dynamic response to joystick data or by incorporating collision avoidance responses into object detection. In FIG. 10, points 1002 represent position data calculated incrementally by the autopilot of the flight control subsystem based on the calculated spline. Points 1002 may also include flight parameters that govern the drone's transitions, but for clarity, this example is limited to a discussion of position corrections. The overhead view 1000 of a drone flight path shown in FIG. 10 further illustrates the effect of joystick data received from a remote control device causing a transient flight path deflection. The autopilot creates a new path including a new set of flight parameters, including incremental position data 1006, at location 1020 and issues modified joystick commands, which may also include orientation data. Points 1006 represent a damping response to the joystick input received from the remote control device. When the pilot stops pushing the joystick at location 1022, the drone autopilot calculates incremental position data 1008 that returns the drone to the flight path 1002. The damping of the pilot's joystick commands results in an operating envelope 1012 around the flight path 1002 that limits the drone's ability to deviate from the flight path even when the pilot is pushing the joystick fully and continuously.
[0068] FIG. 11 illustrates a process 1100 implemented by one or more components of the flight control subsystem 124 and the remote control device 130 of the UAV 101. The process 1600 is implemented in program instructions that, when executed by one or more hardware and / or firmware elements of the flight control subsystem 124 and the remote control device 130, instruct the flight control subsystem 124 and the remote control device 130 to operate as follows. When the UAV 101 is flight controlled by the remote control device 130 operated by a pilot, the computer system instructs a graphical user interface (GUI) on the remote control device 130 to display a perspective view from a sensor operably coupled to the flight control subsystem 124 (step 1110). The computer system detects input from the pilot interfacing with the GUI, including instructions to add key frames, the key frames including the spatial position of the UAV 101 and the orientation of the sensor (step 1120). The computer system continuously generates and updates a spline including a projected flight path or trajectory between each of a plurality of key frames, including the orientation of the sensor (step 1130). The computer system continuously displays (step 1140) a graphical representation of the splines overlaid on a perspective view from a sensor mounted on the UAV 101. In one implementation, the sensor is a forward-facing camera that provides a first-person perspective view of the UAV 101. The orientation of the camera corresponds to the gimbal angle of the camera relative to horizontal flight.
[0069] 12A-12J show one implementation of a user interface presented to a pilot on a display screen of a remote control device. In this example, the remote control device displays a user interface of a virtual reality-based autonomous flight control application that receives input from the pilot and commands the drone to fly accordingly. The remote control device displays the user interface on a touch screen with various input devices such as virtual form buttons, sliders, etc. In another implementation, the input devices may be physical buttons, sliders, toggles, joysticks, etc. on the remote control device. The remote control device may be a dedicated drone control device, a smartphone, tablet or mobile device, or a computing device such as a laptop or other computer that wirelessly communicates with the drone.
[0070] In this example, the autonomous flight control application receives input from a user or pilot through a virtual input device of its user interface. When a virtual input device is described below as being "selected," this indicates that the autonomous flight control application has received an instruction from the pilot to cause the virtual input device to change its state (such as by touching, tapping, or "clicking" the virtual input device). The application responds to that change in state according to its program instructions.
[0071] In this example, a drone operating in keyframe mode generates a calculated spline flight path or "spline" that circles a small copse of trees, and then flies that spline during playback. The UI software generates an AR representation of the calculated spline and associated keyframes and overlays a live video feed from the onboard camera on the touchscreen. The UI software continuously updates the AR representation of the spline and keyframes as keyframes are added, edited, or deleted and splines are generated or recalculated, and during playback as the drone flies the spline. The calculated spline can be recorded and saved for later use by the same drone or other drones with similar capabilities. The calculated spline may also be later edited by the user, and any changes may be saved as a new spline or as a revision that can be selectively added to the spline during later use.
[0072] To begin this example, FIG. 12A shows an implementation of the UI of an AR-based autonomous flight control application on a touchscreen of a drone remote control device as the splines are defined. The touchscreen displays a camera view 1210 captured by an on-board camera. In the center of the touchscreen, the UI displays a launch button 1201 that causes the application to launch the drone from its dock. On the left side of the screen are virtual indicators where the UI presents various states regarding the drone operation or virtual buttons where the pilot can access aspects of the drone operation: a battery charge indicator 1211, a Wifi signal strength indicator 1212, a camera resolution indicator button 1213, and a settings button 1214. On the right side of the screen are a map button 1221 to access a graphic map, a home button 1222, an auto record indicator 1223 that indicates whether video is being recorded, a settings button 1224, and an operation mode graphic 1225 that indicates the operation mode of the drone (in this illustration, the drone is being operated manually).
[0073] FIG. 12B shows the UI of an autonomous flight control application when the home button 1222 is selected in one implementation. When the UI receives an input indicating that the home button 1222 has been selected, the UI displays a tabbed window 1230 including a cinematic tab 1240 for selecting among several modes of automated drone flight. The motion track button 1241 causes the drone to track an object, such as a moving individual or vehicle, in flight while autonomously avoiding obstacles along the way. The fixed track button 1242 initiates a fixed track mode used to track a subject moving on a fixed track. The fixed track mode causes the drone to follow a subject while keeping a set distance from the subject as well as maintaining the original camera orientation. The orbit subject button 1243 initiates the drone's subject-tracking flight operation, whereby the application commands the drone to fly an orbit around the subject. The cable button 1244 engages a method of operating the drone whereby the application defines two keyframes that mark the end points of a flight path, and then flies the drone between them as if tethered to a cable stretched between the two points. The hover button 1245 causes the drone to hover at a single spatial location or keyframe. The keyframe button 1246 activates a keyframe mode of drone operation where the application records and stores multiple keyframes, automatically and dynamically generates a spline flight path between each of the keyframes, and commands the drone to fly that spline. In keyframe mode, the application may also combine input received by the UI from the pilot's interaction with the input devices of the interface. In keyframe mode, existing splines may also be edited and saved for reuse by the drone or other drones with keyframe mode capabilities. Saved splines may be saved to and retrieved from non-volatile storage onboard the drone, the controller, a computer in communication with the controller or drone, or connected cloud storage.
[0074] FIG. 12C illustrates the UI of the autonomous flight control application when the keyframe button 1246 is selected, causing the application to enter a keyframe mode that operates the drone as indicated by the operating mode graphic 1225. In this mode, the touchscreen displays a camera view 1210 from the onboard camera. The UI displays a text display 1203 to indicate the current mode, "Keyframe Mode." A flight parameter set 1202 is a graphic in the upper left corner of the touchscreen that displays the drone flight speed, drone distance from the dock, drone altitude, and the camera gimbal angle relative to horizontal flight. At the bottom center of the touchscreen, the UI displays virtual buttons that allow the pilot to define keyframes that will be used in generating the spline. An add button 1251 causes the application to add a keyframe at the drone's current position, an undo button 1250 reverses the action triggered by the add button 1251 (i.e., undoes the addition of the last added keyframe), and a done button 1252 finishes adding the keyframe. In the upper right corner of the touch screen is a graphic 1204 for pausing the keyframe mode, allowing the pilot to stop autonomous flight and take manual control. Note that the camera view 1210 has been darkened to increase visibility of the text display 1203.
[0075] FIG. 12D shows the UI at the start of the keyframe mode, with the autonomous flight control application UI prompting the pilot to add the first keyframe in a text display 1203. FIG. 12E shows the UI after the first keyframe has been added. As the keyframe is added, the autonomous flight control application records the spatial position of the drone. The application may also record the drone orientation, onboard camera gimbal angle, onboard camera focal length and / or exposure settings, drone speed, and / or other flight or operating parameters at the location of the newly added keyframe. In FIG. 12F, the drone is piloted closer to a wooded area for the second keyframe, and the second keyframe is added. FIG. 12F shows the UI text display 1203 confirming that the keyframe has been added.
[0076] The example continues with the process of adding keyframes. Figures 12G and 12H show the touchscreen display as the drone is maneuvered around the wooded area and keyframes are added. In keyframe mode, the autonomous flight control application dynamically recalculates the spline as keyframes are added, and the AR representation of the spline is continuously updated by the UI on the display. As shown in Figure 12H, a keyframe is added and the calculated spline 1260 is displayed on the touchscreen extended above the camera view 1210. Keyframe markers 1262 are displayed as diamonds on the calculated spline 1260.
[0077] After several more keyframes are added, FIG. 12I shows the camera view 1210 overlooking the woods as the 17th keyframe is added. In this view, the computed spline 1260 and keyframe markers 1262 are more clearly visible. The computed spline 1260 is a two-dimensional projection of a three-dimensional spline that includes the flight path or trajectory between each of the multiple keyframes. The diamond-shaped keyframe markers 1262 that mark the location of the keyframes vary in size with the order of addition, with the most recently added keyframes being shown as the largest diamonds. The computed spline 1260 and keyframe markers 1262 may also be scaled in proportion to the distance from the drone, with the keyframes dynamically increasing in size on display as the drone approaches the keyframe location. In addition, different geometric shapes may be used to indicate keyframes according to specific purposes, such as the beginning or end of the computed spline.
[0078] 12J shows the UI during keyframe addition and spline generation once the pilot has finished adding keyframes. Once the pilot selects the Done button 1252, the autonomous flight control application receives an indication that the spline is complete and switches to keyframe playback mode as shown in FIG.
[0079] 13A-13D show an implementation of a UI where the spline definition is complete and the spline is played back. At the bottom of the touchscreen display shown in FIG. 13A, the autonomous flight control application UI displays a playback track 1320, which is a linear graphical representation of the calculated spline. On the playback track 1320, keyframes are represented as diamonds 1322, and the drone's current position along the spline is also shown as an arrow 1324. The relative distance between keyframes is indicated by the proportional spacing of the diamonds 1322 on the playback track 1320. FIG. 13B shows the addition of a virtual speed control slider 1330 to the touchscreen display, whereby the UI receives manual input that causes the application to accelerate, decelerate, or hover the drone as it traverses the calculated spline 1260. Using the speed control slider 1330, the pilot can control the drone speed between keyframes or across the spline. Note that in this illustration, the arrow 1324 changes color to indicate when the drone is moving. In one implementation, the relative time to move between keyframes is indicated by proportional spacing of diamonds 1322 on the playback track 1320.
[0080] An added feature of the UI's playback track 1320 is the ability to "snap" the drone anywhere on the calculated spline 1260. Tapping anywhere on the playback track 1320 instructs the application to fly the drone directly to that location without crossing the spline.
[0081] 13C shows an implementation of a UI in which an autonomous flight control application displays the drone's progress as it flies the spline 1260 during playback in keyframe mode. The application may receive commands from the pilot using virtual play / stop buttons 1314 and forward / reverse buttons 1316 to fly the drone forward or backward along the spline, or to pause and hover. The UI continuously updates the playback track 1320 to show both the direction and current position of the drone's playback along the spline. Diamonds 1322 change color to indicate the drone's progression through the keyframes.
[0082] FIG. 13D shows a touchscreen display of the controller during playback in one implementation as the drone begins to move the calculated spline 1260 starting from the first keyframe recorded in FIG. 12D. In this view, the UI of the autonomous flight control application displays several flight and motion commands that allow the application to adjust the speed of the drone's movement along the spline, stop or reverse the drone along the spline, jump or "snap" to out-of-order keyframes on the spline, add new keyframes, delete keyframes, or edit the speed or orientation at any keyframe. Note that on the AR representation of the calculated spline 1260, each of the keyframe markers 1262 increases in size as the drone approaches the keyframe location. The UI displays the calculated spline 1260 in a color that is highly visible from the background (first person) view on the touchscreen. This color can be programmatically selected using an algorithm that detects the color range of the camera view 1210, or the color may be set manually by the pilot.
[0083] 14A-14D show an implementation of a user interface of an autonomous flight control system displayed on a touch screen of a smartphone. In this example implementation, an autonomous flight control application operating in keyframe mode generates a spline between each of a number of keyframes. The application can edit the spline during playback by adding additional keyframes or by editing existing keyframes.
[0084] 14A shows the UI before the start of the drone's movement on the spline. The UI displays a playback track 1410 with diamonds 1412 marking the locations of keyframes in proportion to the distance along the spline or in proportion to the travel time between keyframes. Visually, the spline 1404 is an AR graphic displayed on the camera view 1402 along with a keyframe marker 1406 indicating the location of the next keyframe as the drone traverses the spline 1404.
[0085] FIG. 14B shows the UI as the drone moves along the calculated spline 1404 between the second and third keyframes. An arrow 1414 traverses the playback track 1410, indicating the drone's position on the spline 1404 in real time. Next to the playback track 1410 are an edit button 1416 and an add button 1418, which are virtual input devices. As the drone traverses the spline 1404, when it reaches a keyframe, the edit button 1416 becomes active, allowing the pilot to select the keyframe for editing. As the drone moves anywhere along the spline 1404, the add button 1418 is active, and when selected, prompts the application to define and add a new keyframe at the drone's position, recording and storing the drone's position as determined by visual tracking or navigation coordinates. The application may also record and store other flight or operating parameters of the new keyframe, such as the gimbal angle, exposure setting, or focal length of the onboard camera.
[0086] 14C shows the UI when the Add button 1418 is selected. The UI prompts the pilot to set the location of the new keyframe by tapping a Set button 1420. When the Set button 1420 is tapped, the application recalculates the spline 1404, instructs the UI to display an updated graphical representation of the spline 1404, and marks the location of the new keyframe by adding a diamond to the playback track 1410.
[0087] 14D shows the UI when the pilot taps the add button 1418 while the drone is on an existing keyframe, and the application prompts the pilot to indicate whether the new keyframe should be placed before or after the existing keyframe (or to cancel the add). If the pilot selects before or after, the application recalculates the spline 1404 and the UI updates the display accordingly.
[0088] FIG. 15 includes a series of images illustrating yet another implementation of a UI for a drone autonomous flight control application operating in keyframe mode during playback. Images 1510-1530 show displays on a drone remote control device. In image 1510, the display shows that the first-person camera view is a live feed from an on-board camera. At the bottom of image 1510, a drone arrow indicator moves from right to left across the playback track, showing the drone approaching keyframe 4. As the drone approaches keyframe 4, on the AR overlay of the computed spline, a semi-transparent diamond marking the location of keyframe 4 grows in size as the drone approaches and disappears to simulate the drone passing the AR keyframe diamond (image 1520). Image 1530 then shows a first-person perspective of the drone continuing on the computed spline, but as the drone turns right to track the paddle border. Because we have pivoted from our forward facing orientation, ostensibly to the left of the screen and out of the camera view, the AR representation of the computed spline is no longer visible.
[0089] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Additionally, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied therein.
[0090] In various implementations, the systems, methods, processes, and operational scenarios may be implemented in computer software executed by a processing system in the context of an unmanned aerial vehicle, a remote control device, or any other type of device capable of executing software, such as a computer and a mobile phone. The processing system may load and execute the software from a storage system, or may be pre-configured with the software. The software includes and implements a process for creating a calculated spline, which represents the spline creation process discussed in connection with the preceding figures, such as process 200 and process 700. The software also includes and implements a program related to a user interface of the autonomous flight control program, which represents the user interface of the autonomous flight control program discussed in connection with the preceding figures, such as process 1100. When executed by the processing system, the software instructs the processing system to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the preceding implementations.
[0091] An exemplary processing system may include a microprocessor and other circuitry that retrieves and executes software from storage. A processing system may be implemented in a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of processing systems include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combination, or variation thereof.
[0092] An exemplary storage system may include any computer-readable storage medium that is readable by a processing system and capable of storing software. The storage system may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer-readable storage medium a propagating signal.
[0093] The software may be implemented with program instructions that, among other functions, when executed by a processing system, may instruct the processing system to operate as described with respect to the various operational scenarios, sequences, and processes set forth herein. In particular, the program instructions may include various components or modules that cooperate or otherwise interact to perform the various processes and operational scenarios described herein. The various components or modules may be embodied with compiled or interpreted instructions, or some other variation or combination of instructions. The various components or modules may be executed synchronously or asynchronously, serially or in parallel, in a single-threaded environment or multi-threaded, or according to any other suitable execution program, variation, or combination thereof. The software may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. The software may also include firmware, or any other form of machine-readable processing instructions executable by a suitable processing system.
[0094] In general, the software, when loaded and executed on a processing system, may transform a generally suitable apparatus, system, or device from a general-purpose computing system to a special-purpose computing system as described herein. Encoding the software on the storage system may transform the physical structure of the storage system. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors include, but are not limited to, the technology used to implement the storage medium of the storage system and whether the computer storage medium is characterized as a primary or secondary storage device, as well as other factors.
[0095] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, the software may transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of this description, and the above examples are provided only to facilitate this discussion.
[0096] It will be further appreciated that an unmanned aerial vehicle, remote control device, or other device in which aspects of the present invention may be embodied, may include a communication interface system. The communication interface system may include communication connections and devices that enable communication with other computing systems and devices (not shown) over a communication network (not shown). Examples of connections and devices that together enable inter-system communication may include network interface cards, antennas, power amplifiers, RF circuits, transceivers, and other communication circuitry. The connections and devices may communicate via a communication medium for exchanging communications with other computing systems or networks of the system, such as metal, glass, air, or any other suitable communication medium. The aforementioned media, connections, and devices are well known and need not be discussed in detail here.
[0097] Communications between such systems and devices may occur over one or more communications networks according to various communications protocols, combinations of protocols, or variations thereof. Examples include intranets, inter-network connections, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The foregoing communications networks and protocols are well known and need not be discussed at length here.
[0098] An unmanned aerial vehicle, remote control device, or other device in which aspects of the present technology may be embodied may include a user interface system. The user interface system may include one or more of a joystick, a keyboard, a mouse, a voice input device, a touch input device for receiving touch gestures from a user, a motion input device for detecting non-touch gestures and other motions by the user, and other equivalent input devices and associated processing elements (e.g., joystick toggles) capable of receiving user input from a user. Output devices, such as displays, speakers, haptic devices, and other types of output devices, may also be included in the user interface system. In some cases, the input devices and output devices may be combined into a single device, such as a display capable of displaying images and receiving touch gestures. The aforementioned user input devices and output devices are well known in the art and need not be discussed in detail here. The user interface system may also include associated user interface software executable by an appropriate processing system that supports the various user input devices and output devices discussed above. Separately or in conjunction with each other or other hardware and software elements, the user interface software and user interface devices may support a graphical user interface, a natural user interface, or any other type of user interface.
[0099] It will be further appreciated that aspects of the present technology describe various operational scenarios for drone operation along a calculated spline flight path or along a programmed flight path obtained from a series of key frames or other sources. As the drone's autopilot issues modified joystick commands to the electromechanical subsystem based on the flight path or other external operating factors, the drone pilot may issue operation commands via a joystick on a remote control device. The autopilot receives the joystick data and incorporates this data into apparent joystick commands issued to the UAV microprocessor. The autopilot maintains control over the drone's operation along the flight path, and the ability to incorporate joystick data into the drone's operation results in an operating envelope along the flight path that allows the drone pilot to control one or more aspects of the flight to achieve optimal drone operation.
[0100] Unless the context clearly requires otherwise, throughout the description and claims, words such as "comprises," "comprising," "such as," "the like," and the like, should be construed in an inclusive sense, i.e., "including but not limited to," rather than in an exclusive or exhaustive sense. As used herein, the terms "connected," "coupled," or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and the coupling or coupling between the elements may be physical, logical, or a combination thereof. In addition, the terms "herein," "above," "below," and words of similar import, when used in this application, refer to this application as a whole and not to any particular portion of this application. Where the context permits, words in the above detailed description using singular or plural numbers may also include the plural and singular numbers, respectively. The word "or" in connection with a list of two or more items covers all of the following interpretations of that word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
[0101] The above detailed description of examples of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. Although specific examples of the present technology are described for illustrative purposes, various equivalent modifications are possible within the scope of the present technology, as those skilled in the art will recognize. For example, while processes and blocks are presented in a given order, alternative embodiments may perform routines having different orders of operations or employ systems having blocks in different orders, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are sometimes shown as being performed in series, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Furthermore, any specific numerical values described herein are merely examples. Alternative implementations may employ different values or ranges.
[0102] The teachings of the technology provided herein may be applied to other systems, not necessarily the systems described above. The elements and operations of the various examples described above may be combined to provide further implementations of the technology. Some alternative implementations of the technology may include additional elements to the above examples, as well as fewer elements.
[0103] These and other changes can be made to the technology in light of the detailed description above. Although the above description describes certain examples of the technology and describes the best possible mode, no matter how detailed the above may appear in text, the technology can be practiced in many ways. The details of the system may vary considerably in its specific implementation, but still be included in the technology disclosed herein. As mentioned above, a particular term used when describing a particular feature or aspect of the technology should not be construed as implying that the term is redefined herein such that the term is not limited to any particular characteristic, feature, or aspect of the relevant technology. In general, the terms used in the following claims should not be construed as limiting the technology to the specific examples disclosed herein, unless the detailed description section above clearly defines such terms. Thus, the actual scope of the technology encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the technology under the claims.
[0104] In order to reduce the number of claims, certain aspects of the technology are presented below in certain claim formats, but the applicant contemplates various aspects of the technology in any number of claim formats. For example, only one aspect of the technology is recited as a computer-readable medium claim, but other aspects may be similarly implemented in other forms, such as as a computer-readable medium or implemented in a means-plus-function claim. Any claim intended to be treated under 35 U.S.C. 112(f) begins with the words "means for," but the use of the term "for" in any other context is not intended to conjure treatment under 35 U.S.C. 112(f). Accordingly, the applicant reserves the right to pursue such additional claims after filing this application to pursue additional claim formats in either this application or any continuing application.
Claims
1. An unmanned aerial vehicle, a flight control subsystem; an electromechanical subsystem coupled to the flight control subsystem and configured to fly the unmanned aerial vehicle as directed by the flight control subsystem; Equipped with The flight control subsystem includes: recording key frames during one or more flights of the unmanned aerial vehicle; Calculating a spline based on at least one of the keyframes; Save the spline for playback when flying the unmanned aerial vehicle 1. An unmanned aerial vehicle configured to:
2. The unmanned aerial vehicle of claim 1 , wherein the flight control subsystem is further configured to direct the electromechanical subsystem to fly the unmanned aerial vehicle based on a calculated spline during one or more subsequent flights.
3. During the one or more subsequent flights, the flight control subsystem: Receive user input; modifying the calculated spline based on at least the user input, resulting in a modified version of the calculated spline; Saving the modified version of the calculated spline for later playback The unmanned aerial vehicle of claim 2 , further configured as follows:
4. 4. The unmanned aerial vehicle of claim 3, wherein the user input includes one or more of changes to one or more components of the calculated spline, the one or more components of the calculated spline including one or more of a position, a direction, a velocity, and an orientation of the unmanned aerial vehicle along the calculated spline.
5. The unmanned aerial vehicle of claim 4 , wherein the one or more components of the calculated spline further include one or more of a camera focal length and a camera orientation for the unmanned aerial vehicle.
6. The unmanned aerial vehicle of claim 4 , wherein the user input includes at least one of: directly snapping the unmanned aerial vehicle to a new position on the calculated spline out of order relative to the calculated spline.
7. The unmanned aerial vehicle of claim 4 , wherein the user input comprises reversing a direction along the calculated spline relative to a current direction along the calculated spline.
8. The unmanned aerial vehicle of claim 4 , wherein the user input comprises hovering at a point on the calculated spline.
9. The unmanned aerial vehicle of claim 4 , wherein the flight control system is further configured to upload a modified version of the computational spline to a remote storage location.
10. The unmanned aerial vehicle of claim 4 , wherein the one or more subsequent flights include leaving a dock and arriving at a dock.
11. 1. A computing device, comprising: one or more computer readable storage media; program instructions stored on the one or more computer readable storage media, When executed by the one or more processors, the flight control subsystem of the unmanned aerial vehicle includes at least recording key frames during one or more flights of the unmanned aerial vehicle; Calculating a spline based on at least one of the keyframes; Save the spline for playback when flying the unmanned aerial vehicle and A computing device comprising:
12. 12. The computing device of claim 11, wherein the program instructions further direct the flight control subsystem to direct an electromechanical subsystem of the unmanned aerial vehicle to fly the unmanned aerial vehicle based on a calculated spine during one or more subsequent flights.
13. During the one or more subsequent flights, the program instructions cause the flight control subsystem to: Receive user input; modifying the calculated spline based on at least the user input, resulting in a modified version of the calculated spline; Saving the modified version of the calculated spline for later playback The computing device of claim 12 , further comprising:
14. 14. The computing device of claim 13, wherein the user input comprises one or more of changes to one or more components of the calculated spline, the one or more components of the calculated spline comprising one or more of a position, a direction, a velocity, and an orientation of the unmanned aerial vehicle along the calculated spline.
15. The computing device of claim 14 , wherein the one or more components of the calculated spline further include one or more of a camera focal length and a camera orientation for the unmanned aerial vehicle.
16. The computing device of claim 14 , wherein the user input comprises at least one of: directly snapping the unmanned aerial vehicle to a new position on the calculated spline out of order relative to the calculated spline.
17. The computing device of claim 14 , wherein the user input comprises reversing a direction along the calculated spline relative to a current direction along the calculated spline.
18. The computing device of claim 14 , wherein the user input comprises hovering over a point on the calculated spline.
19. 15. The computing device of claim 14, wherein the flight control system is further configured to upload a modified version of the computational spline to a remote storage location.
20. 1. A method of operating an unmanned aerial vehicle, the method comprising: a flight control subsystem for the unmanned aerial vehicle, recording key frames during one or more flights of the unmanned aerial vehicle; calculating a spline based on at least one of the keyframes; storing the spline for playback when flying the unmanned aerial vehicle; The method includes:
21. An unmanned aerial vehicle, a flight control subsystem; an electromechanical subsystem coupled to the flight control subsystem and configured to fly the unmanned aerial vehicle as directed by the flight control subsystem; Equipped with The flight control subsystem includes: calculating a spline along which the unmanned aerial vehicle will fly; receiving input from a control device remote from the unmanned aerial vehicle while flying the unmanned aerial vehicle along the calculated spline, the input including joystick data; calculating modifications to the calculated spline based at least on the joystick data and one or more governing factors related to modifications to the calculated spline; Instructing the electromechanical subsystem to fly the drone based on at least the modifications to the calculated spline.
1. An unmanned aerial vehicle configured to:
22. The unmanned aerial vehicle of claim 21 , wherein the joystick data indicates a departure from the calculated spline, and the correction includes a path that deviates from the calculated spline.
23. The unmanned aerial vehicle of claim 22 , wherein the joystick data indicates a return to the calculated spline, and the modifications to the calculated spline include a path that returns to the calculated spline.
24. The unmanned aerial vehicle of claim 21 , wherein the one or more management factors include a damping function applied to the joystick data.
25. The unmanned aerial vehicle of claim 24 , wherein the damping function prevents modifications to the calculated spline from exceeding a flight envelope around the calculated spline.
26. The unmanned aerial vehicle of claim 21 , wherein the one or more management factors include an object avoidance function applied to the joystick data.
27. 22. The unmanned aerial vehicle of claim 21, wherein the flight control subsystem includes an autopilot system, and wherein to calculate corrections to the calculated spline, the autopilot system is configured to calculate modified joystick data based on the joystick data and the one or more control factors.
28. 28. The unmanned aerial vehicle of claim 27, wherein the flight control subsystem further includes a flight controller coupled to the autopilot system, the flight controller configured to instruct the electromechanical subsystem based on at least modified joystick data calculated by the autopilot system to instruct the electromechanical subsystem to fly the drone based at least on the modified spline.
29. 22. The unmanned aerial vehicle of claim 21, wherein the calculated spline includes a set of points along a path and flight parameters at each point in the set of points, and modifications to the calculated spline include a new set of points off the path and new flight parameters at each point in the new set of points.
30. 30. The unmanned aerial vehicle of claim 29, wherein the flight parameters include a position and an orientation of the unmanned aerial vehicle, and the new flight parameters include one or more of a new position and a new orientation of the unmanned aerial vehicle.
31. 1. A method for controlling an aircraft, the method comprising: continuously presenting, by a user computing device, a graphical user interface (GUI) on the user computing device, the GUI including a display of a current view of a physical environment from a perspective of the aircraft, the view being generated based on sensor data from sensor devices onboard the aircraft; detecting, by the user computing device, a user interaction with the GUI, the user interaction including instructions to add a plurality of key frames, each key frame including a location in three-dimensional space of the aircraft; continuously displaying, by the user computing device, a graphical representation of a spline overlaid on a current view of the physical environment from the perspective of the aircraft, the spline including a flight path between each of the plurality of key frames; The method includes:
32. 32. The method for controlling an aircraft of claim 31, wherein each of the plurality of key frames further includes an orientation of the sensor device.
33. 32. The method for controlling an aircraft of claim 31, wherein the graphical representation of the spline further includes graphical markers indicating locations of the plurality of keyframes on the spline.
34. 32. The method for controlling an aircraft as described in claim 31 , wherein the GUI further includes a linear representation of the spline, the linear representation of the spline further including graphical marker representations of the plurality of keyframes spaced proportionally to spatial locations of the plurality of keyframes on the spline.
35. 32. The method for controlling an aircraft of claim 31, wherein the user interaction further includes an instruction to set aircraft speed.
36. 32. The method for controlling an aircraft as described in claim 31, wherein the user interaction further includes instructions to edit and delete one or more keyframes.
37. 32. The method for controlling an aircraft as described in claim 31, further comprising storing the splines in a non-transitory computer readable storage medium.
38. 32. The method for controlling an aircraft of claim 31, wherein each of the plurality of key frames further includes a sensor device focal length.
39. 32. The method for controlling an aircraft of claim 31, wherein each of the plurality of key frames further includes aircraft velocity and orientation relative to the direction of forward motion.
40. 32. The method for controlling an aircraft of claim 31 , wherein the splines are generated and updated by a computing device onboard the aircraft.
41. 1. A computing device, comprising: one or more non-transitory computer readable storage media; program instructions stored on the one or more computer readable storage media that, when executed by one or more processors, cause the computing device to continuously presenting a graphical user interface (GUI) on the computing device, the GUI including a display of a current view of a physical environment from a perspective of the aircraft, the view being generated based on sensor data from sensor devices onboard the aircraft; detecting a user interaction with the GUI, the user interaction including an instruction to add a plurality of key frames, each key frame including a location in three-dimensional space of the aircraft; continuously displaying a graphical representation of a spline overlaid on a current view of the physical environment from a perspective of the aircraft, the spline including a flight path between each of the plurality of key frames; and A computing device comprising:
42. 42. The computing device of claim 41, wherein each of the plurality of key frames further comprises an orientation of the sensor device.
43. 42. The computing device of claim 41, wherein the graphical representation of the spline further comprises graphical markers indicating locations of the plurality of keyframes on the spline.
44. 42. The computing device of claim 41, wherein the GUI further comprises a linear representation of the spline, the linear representation of the spline further comprising graphical marker representations of the keyframes spaced proportionally to spatial locations of the keyframes on the spline.
45. 42. The computing device of claim 41, wherein the user interaction further comprises one or more instructions for setting the aircraft speed or editing or deleting one or more key frames.
46. 42. The computing device of claim 41, further comprising program instructions directing the computing device to store the spline in the computer readable storage medium.
47. 42. The computing device of claim 41, wherein each of the plurality of keyframes further includes one or more of a sensor device focal length, and an aircraft speed and orientation relative to the direction of forward motion.
48. 42. The computing apparatus of claim 41, wherein the splines are generated and updated by a computing device onboard the aircraft.
49. 1. A computing device, comprising: one or more non-transitory computer readable storage media; Program instructions stored on the one or more computer readable storage media, which when executed by one or more processors, cause the computing device to sequentially: receiving sensor data from a sensor device installed on the aircraft; generating and updating a graphical user interface (GUI) on the computing device, the GUI including a display of a current view of a physical environment from a perspective of the aircraft, the view being generated based on the sensor data; presenting the GUI at the computing device, the GUI including a display of a current view of the physical environment from a perspective of the aircraft; detecting a user interaction with the GUI, the user interaction including an instruction to add a plurality of key frames, each key frame including a location in three-dimensional space of the aircraft; generating and displaying a graphical representation of a spline overlaid on a current view of the physical environment from a perspective of the aircraft, the spline including a calculated flight path between each of the plurality of key frames; and A computing device comprising:
50. The program instructions, when executed by one or more processors, cause the computing device to sequentially: transmitting information indicative of the user interaction to the aircraft; receiving information describing the spline for generating and updating a graphical representation of the spline; 50. The computing device of claim 49,
Citation Information
Patent Citations
Method and System for Creating the Travel Direction of a Drone
JP2017504875A
Method and system for generating aerial imaging flight path
US20190295423A1
Methods for Generating Aerial Photographing Path for Unmanned Aerial Vehicle, Computer Devices, and Storage Mediums
US20200027225A1
Cited By
Information processing device, information processing method, and computer program
JP2026155391A
Information processing device, information processing method, and computer program
JP7832391B1
