Flight path determination

Abstract

Systems and methods for determining a flight path of an aerial vehicle are provided. The systems and methods are particularly useful for autonomous flight planning or navigation of unmanned aerial vehicles.

Description

BACKGROUND

[0001] Unmanned vehicles, such as unmanned aerial vehicles (UAVs), can be used to perform surveillance, reconnaissance, and exploration tasks in a variety of environments for military and civilian applications. Unmanned vehicles can be manually controlled by a remote user, or can operate in a semi-autonomous or fully autonomous manner. Such unmanned vehicles can include a processor to automatically determine a flight path thereof.

[0002] In some cases, existing methods for determining a flight path with respect to a particular target can not be optimal. For example, the determination can not be adequate to accommodate changes in the environment or to suit characteristics of the flight area. SUMMARY

[0003] The present application provides a system and related method for determining a flight path within an airspace of a vehicle. The determination can be made based on various conditions, such as conditions of the vehicle, conditions of the flight environment, conditions of signal transmissions between the vehicle and a remote device, and the like. These conditions can generally determine which areas of the airspace the flight path can be located and how the flight path can be established.

[0004] In some aspects, a method of determining a flight path of a vehicle is provided. The method provides identifying, during a flight and by way of one or more processors, a change in a signal transmission status occurring at a first location; in response to the identifying, selecting a first destination within a range proximate a second location, wherein the signal transmission status at the second location during a previous flight is different than the signal transmission status at the first location; determining a first flight path to the first destination, wherein the first flight path includes reachable locations detected by one or more sensors on one or more vehicles; upon the vehicle reaching the first destination, assessing the signal transmission status at the first destination in real-time; and determining a second flight path to a second destination based on the assessment.

[0005] In some embodiments, the change in the signal transmission status includes a loss or weakening of a signal transmitted between the vehicle and the remote device. In some embodiments, the transmitted signal includes data related to control signals generated by the remote device for operating the vehicle, or image data acquired by the vehicle.

[0006] In some embodiments, the first destination is proximate to the second location such that it is within approximately 10 meters of the second location. In some embodiments, the first destination is proximate to the second location such that the signal transmission rate at the first location and the second location are both above a predetermined threshold. In some embodiments, the first destination is at the second location. In some embodiments, the change in signal transmission status comprises a change in signal transmission rate between the aerial vehicle and the remote device. In some embodiments, the remote device is a remote controller or a satellite. In some embodiments, the change results in a signal transmission rate below a predetermined threshold.

[0007] In some embodiments, the identifying comprises receiving a notification from a sensor on the aerial vehicle or the remote device. In some embodiments, the first destination is selected based on user input. In some embodiments, the change in signal transmission status indicates an abnormal state in signal transmission between the aerial vehicle and the remote device. In some embodiments, the first destination is selected based on an operational state of one or more sensors on the aerial vehicle. In some embodiments, the one or more sensors comprise one or more GPS receivers. In some embodiments, the first destination is selected according to a current condition of a flight environment. In some embodiments, the current condition of the flight environment comprises a weather condition or a wireless signal strength.

[0008] In some embodiments, the second location is a location detected by one or more sensors on one or more aerial vehicles during a previous flight. In some embodiments, the second location is a last point of successful signal transmission between the aerial vehicle and the remote device. In some embodiments, the first destination is within a predetermined distance from the first location. In some embodiments, the predetermined distance is less than 10 meters.

[0009] In some embodiments, the first flight path is a reverse of a last flight path of the aerial vehicle. In some embodiments, the first flight path does not include a location that was not detected by the one or more sensors during a previous flight. In some embodiments, the first location, the first destination, the second location, or the second destination is characterized by GPS coordinates. In some embodiments, the one or more sensors on one or more aerial vehicles comprise one or more of a camera, a radar, a lidar, an ultrasonic sensor, and a GPS receiver. In some embodiments, the second destination is selected based on user input from the remote device.

[0010] In some embodiments, the method further comprises, prior to or concurrently with selecting the second destination, hovering the aerial vehicle for a predetermined period of time to collect data in real time. In some embodiments, the second flight path is determined while the aerial vehicle is hovering. In some embodiments, the second destination is a starting point of the current flight when no user input is received within the predetermined period of time. In some embodiments, selecting the second destination comprises further hovering for a second predetermined period of time when no user input is received within the predetermined period of time. In some embodiments, the second destination is a location that was not detected by the one or more sensors during a previous flight. In some embodiments, the second destination is a predetermined location that the aerial vehicle reaches prior to identifying a change in signal transmission status occurring at the first location.

[0011] In some embodiments, the method further comprises selecting the second destination based on real-time information. In some embodiments, the real-time information relates to one or more of: user input, operational status of the aerial vehicle, and current conditions of the flight environment. In some embodiments, the current conditions of the flight environment include weather conditions or wireless signal strength. In some embodiments, the real-time information indicates an abnormal operational status of the aerial vehicle.

[0012] In some embodiments, the second destination is a starting point of the current flight. In some embodiments, the second destination is a service point that provides a service to restore the normal operational status, and wherein the second flight path is determined based on a flight distance to the service point. In some embodiments, the abnormal operational status indicates a low fuel level or a low battery level. In some embodiments, the change in signal transmission status includes a loss or weakening of signals transmitted between the aerial vehicle and the remote device. In some embodiments, when the signal transmission remains lost or weakened at the first destination, determining the second flight path comprises: including a location that was not detected by any of the one or more sensors of the one or more aerial vehicles during a previous flight. In some embodiments, when the signal transmission remains lost or weakened at the first destination, the second flight path includes a location that was detected by any of the one or more sensors of the one or more aerial vehicles during a previous flight. In some embodiments, when the signal transmission status at the first destination is restored to a normal status at the first destination, the second flight path includes a location that was not detected by any of the one or more sensors of the one or more aerial vehicles during a previous flight.

[0013] In some embodiments, the method further includes selecting the second flight path further depending on user input from a remote device. In some embodiments, wherein the signal transmission includes transmission of image data acquired by an imaging device on the aerial vehicle. In some embodiments, the operational status of the aerial vehicle indicates that a power level of a battery configured to power at least one or more propulsion units of the aerial vehicle has fallen below a threshold value. In some embodiments, the power level is insufficient to power the aerial vehicle from the first destination to the second destination.

[0014] In another aspect, a system for determining a flight path of an aerial vehicle is provided. The system includes one or more processors; and one or more memories having instructions stored thereon that, when executed by the one or more processors, cause the processors to perform the steps of: identifying a change in a signal transmission status occurring at a first location during a flight; in response to the identification, selecting a first destination within a range proximate to a second location, wherein the signal transmission status at the second location during a previous flight is different from the signal transmission status at the first location; determining a first flight path to the first destination, wherein the first flight path includes reachable locations detected by one or more sensors on one or more aerial vehicles; upon the aerial vehicle reaching the first destination, assessing the signal transmission status at the first destination in real-time; and determining a second flight path to a second destination based on the assessment.

[0015] In another aspect, a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a computing system, cause the computing system to perform a method of determining a flight path of an aerial vehicle. The method includes: identifying a change in a signal transmission status occurring at a first location during a flight; in response to the identification, selecting a first destination within a range proximate to a second location, wherein the signal transmission status at the second location during a previous flight is different from the signal transmission status at the first location; determining a first flight path to the first destination, wherein the first flight path includes reachable locations detected by one or more sensors on one or more aerial vehicles; upon the aerial vehicle reaching the first destination, assessing the signal transmission status at the first destination in real-time; and determining a second flight path to a second destination based on the assessment.

[0016] In another aspect, a method of planning a flight path for an aerial vehicle is provided. The method includes: obtaining, with the aid of one or more processors: (a) one or more costs each associated with a path segment connecting a first point and a second point, wherein the first point and the second point are located in a search space comprising a plurality of points including a start point and an end point, and (b) one or more costs each associated with an auxiliary segment connecting the second point and a target point, the target point comprising a two-dimensional (2D) coordinate; applying a target function to the second point, wherein the target function yields an estimated cost of a route from the start point through the second point to the end point, and wherein the target function is based on a combination of two or more components, the components including at least one of the obtained one or more costs associated with the auxiliary segment; and including the path segment into the flight path starting from the start point and sequentially connected to one or more path segments, when the target function applied to the second point yields a desirable value, thereby planning the flight path.

[0017] In some embodiments, the method further includes repeating the obtaining, applying, and including steps until the flight path includes the end point. In some embodiments, the desirable value is a minimum of values yielded by applying the target function to a plurality of candidate second points in the search space. In some embodiments, the two or more components further include one or more costs associated with a straight line connecting the second point and the end point.

[0018] In some embodiments, the method further includes directing the aerial vehicle to follow the flight path. In some embodiments, the 2D coordinate of the target point is associated with a surface height relative to a reference level for which the aerial vehicle is unaware of an actual value. In some embodiments, the target point is a nearest point to the second point.

[0019] In some embodiments, the cost associated with the auxiliary segment is related to a kind of object associated with the target point. In some embodiments, the cost associated with the auxiliary segment is related to a distance between the second point and the target point. In some embodiments, the cost associated with the auxiliary segment is related to an estimated value of the surface height at the target point or a confidence index associated with the estimated value. In some embodiments, the method further includes assigning a weight to the cost associated with the auxiliary segment in computing the value of the target function according to the confidence index associated with the estimate.

[0020] In some embodiments, the cost associated with the path segment from the first point to the second point is related to a distance between the two points. In some embodiments, the cost associated with the path segment is further related to a difference in height between the two points when the height of the second point is greater than the height of the first point. In some embodiments, the method further comprises assigning a first weight to a distance between 2D coordinates corresponding to the two points and a second weight to a difference in the third dimension between the two points when calculating the value of the cost associated with the path segment.

[0021] In some embodiments, the cost associated with the path segment is related to one or more of the following state indicators associated with the aerial vehicle: overall battery level, overall GPS signal strength, overall ground control signal strength, and overall image transmission rate at a point on the path segment.

[0022] In some embodiments, the method further comprises determining a cost associated with a path segment connecting a previous point of the first point and the second point along a line of sight. In some embodiments, the path segment connecting the previous point of the first point and the second point is added to the flight path when the cost of the path segment connecting the previous point of the first point and the second point is less than an overall cost associated with the path segment connecting the first point and the second point and the path segment connecting the previous point of the first point and the first point. In some embodiments, the determination of the cost associated with the path segment connecting the previous point of the first point and the second point is only performed when a cost of a secondary segment from the second point to a target point is greater than a predetermined threshold. In some embodiments, the cost of the path segment connecting the previous point of the first point and the second point is related to a distance from the previous point of the first point to the second point. In some embodiments, the method further comprises determining whether there is an obstacle in the path segment connecting the previous point of the first point and the second point.

[0023] In some embodiments, the method further comprises directing the aerial vehicle to follow a new flight path upon detecting in real-time that a predetermined condition exists. In some embodiments, the predetermined condition is the presence of an unforeseen obstacle or loss of signal.

[0024] In some embodiments, the end point is a previously visited point, a known safe point, a ground control center, or a user location.

[0025] In another aspect, a system for planning a flight path of a flying vehicle is provided. The system includes one or more processors; and one or more memories having instructions stored thereon that, when executed by the processors, cause the processors to perform the steps of: obtaining (a) one or more costs each associated with a path segment connecting a first point and a second point, wherein the first point and the second point are located in a space of a plurality of points including a start point and an end point, and (b) one or more costs each associated with an auxiliary segment connecting the second point and a target point, the target point including two-dimensional (2D) coordinates; applying a target function to the second point, wherein the target function yields an estimated cost of a route from the start point through the second point to the end point, and wherein the target function is based on a combination of two or more components, the components including at least one of the obtained one or more costs associated with the auxiliary segment; and including the path segment into a flight path that starts from the start point and is sequentially connected to one or more path segments when the target function applied to the second point yields a desirable value, thereby planning the flight path.

[0026] In another aspect, a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a computing system, cause the computing system to perform a method of planning a flight path of a flying vehicle. The method includes: obtaining (a) one or more costs each associated with a path segment connecting a first point and a second point, wherein the first point and the second point are located in a space of a plurality of points including a start point and an end point, and (b) one or more costs each associated with an auxiliary segment connecting the second point and a target point, the target point including two-dimensional (2D) coordinates; applying a target function to the second point, wherein the target function yields an estimated cost of a route from the start point through the second point to the end point, and wherein the target function is based on a combination of two or more components, the components including at least one of the obtained one or more costs associated with the auxiliary segment; and including the path segment into a flight path that starts from the start point and is sequentially connected to one or more path segments when the target function applied to the second point yields a desirable value, thereby planning the flight path.

[0027] In yet another aspect, a method of planning a flight path of a flying vehicle is provided. The method includes: identifying, with aid of one or more processors, a plurality of candidate points, wherein the plurality of candidate points are in a predetermined relationship with a first point; determining a cost associated with each candidate point of the plurality of candidate points, wherein the determining the cost is based at least on a distance between each candidate point and a target point, wherein the target point is associated with a surface height relative to a reference level; selecting a candidate point from the plurality of candidate points when a cost associated with the selected candidate point satisfies a predetermined condition; and including a segment connecting the first point and the selected candidate point into the flight path, thereby planning the flight path.

[0028] In some embodiments, the method further includes selecting the selected candidate point as the first point and repeating the steps of identifying, determining, selecting, and including until the flight path reaches an end point. In some embodiments, the method further includes directing the aerial vehicle to follow the flight path.

[0029] In some embodiments, the plurality of candidate points are within a predetermined distance relative to the first point. In some embodiments, the predetermined condition indicates a minimum cost among the costs associated with the plurality of candidate points. In some embodiments, determining the cost is further based on a distance between each candidate point and the first point. In some embodiments, when a height of one of the candidate points is greater than a height of the first point, determining the cost associated with the one of the candidate points is further based on a height difference between the one of the candidate points and the first point.

[0030] In some embodiments, determining the cost is further based on a distance between each candidate point and an end point. In some embodiments, determining the cost is further based on one or more state indicators associated with the aerial vehicle during a previous flight at each candidate point. In some embodiments, a height of a surface point corresponding to the target point is unknown to the aerial vehicle. In some embodiments, the target point of a particular candidate point is a nearest point to the particular candidate point where a surface height is unknown to the aerial vehicle.

[0031] In some embodiments, the method further includes determining a first cost associated with a segment connecting a previous point of the first point and the selected candidate point; determining a second cost associated with a segment connecting the previous point of the first point and the first point; determining a third cost associated with a segment connecting the first point and the selected candidate point; and including the segment connecting the previous point of the first point and the selected candidate point into the flight path when the first cost is lower than a sum of the second cost and the third cost. In some embodiments, the method further includes determining whether the segment connecting the previous point of the first point and the selected candidate point is obstacle-free.

[0032] In another aspect, a system for planning a flight path of an aerial vehicle is provided. The system includes one or more processors; and one or more memories having instructions stored thereon, which, when executed by the one or more processors, cause the processors to perform the following steps: identifying a plurality of candidate points, wherein the plurality of candidate points are in a predetermined relationship with a first point; determining a cost associated with each of the plurality of candidate points, wherein the determining the cost is based at least on a distance between each candidate point and a target point, wherein the target point is associated with a surface height relative to a reference level; selecting one of the plurality of candidate points when a cost associated with the selected candidate point satisfies a predetermined condition; and including a segment connecting the first point and the selected candidate point into the flight path, thereby planning the flight path.

[0033] In another aspect, a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a computing system, cause the computing system to perform a method of planning a flight path for a vehicle. The method includes identifying a plurality of candidate points, wherein the plurality of candidate points are in a predetermined relationship with a first point; determining a cost associated with each candidate point of the plurality of candidate points, wherein the determining the cost is based at least on a distance between each candidate point and a target point, wherein the target point is associated with a surface height relative to a reference level; selecting a candidate point from the plurality of candidate points when a cost associated with the selected candidate point satisfies a predetermined condition; and including a segment connecting the first point and the selected candidate point into the flight path, thereby planning the flight path.

[0034] It should be appreciated that different aspects of the application can be recognized alone, collectively or in combination with each other. The various aspects of the application described herein can be applied to any of the specific applications set forth below or to any other type of movable object. Any description herein with respect to a vehicle can be adapted and used for any movable object, such as any carrier vehicle. Additionally, the systems, devices, and methods disclosed herein in the context of aerial motion (e.g., flight) can also be applied in the context of other types of motion, such as motion on the ground or on water, underwater motion, or space motion. Furthermore, any description herein with respect to a rotor or rotor assembly can be adapted and used for any propulsion system, device, or mechanism configured to generate propulsion force through rotation (e.g., propellers, wheels, axles).

[0035] Other objects and features of the present application will become apparent by a review of the specification, claims, and appended figures.

[0036] INCORPORATION BY REFERENCE

[0037] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. BRIEF DESCRIPTION OF DRAWINGS

[0038] The novel features of the application are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present application will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the application are utilized, and the accompanying drawings of which:

[0039] Figure 1 A drone operating in an outdoor environment according to an embodiment is shown;

[0040] Figure 2 A drone operating in an indoor environment according to an embodiment is shown;

[0041] Figure 3 Data showing how the drone acquires the 2D elevation map.

[0042] Figure 4 A 2D elevation map based on data collected by the forward-looking sensor is shown.

[0043] Figure 5 A scenario where multiple surface heights exist for a 2D coordinate is shown.

[0044] Figure 6 An example process performed by the drone in response to an abnormal situation is shown.

[0045] Figure 7 An example process for determining a flight path in an area without obstacles is shown.

[0046] Figure 8 An example process for determining a flight path in an area with obstacles is shown.

[0047] Figure 9 Another example process for determining a flight path in an area with obstacles is shown.

[0048] Figure 10 An example method for using different path search algorithms for different areas is shown.

[0049] Figure 11 An example process performed by the drone to determine a flight path based on a cost derived from a 2D elevation map is shown.

[0050] Figure 12 An example process performed by the drone to determine a flight path by determining sub-flight paths that connect through different areas using different path search algorithms is shown.

[0051] Figure 13 A drone according to an embodiment is shown.

[0052] Figure 14 A movable object including a carrier and a payload according to an embodiment is shown. Also shown is a system for controlling a movable object according to an embodiment.

[0053] Figure 15 A system for controlling a movable object according to an embodiment is shown. DETAILED DESCRIPTION

[0054] Systems and methods for controlling movable objects, such as unmanned aerial vehicles (UAVs), are provided. In some embodiments, the unmanned aerial vehicle can be adapted to carry a plurality of sensors configured to collect environmental data. Some sensors can be of different types (e.g., a vision sensor used in combination with a proximity sensor). Data acquired by the plurality of sensors can be combined to generate an environmental map representing the surrounding environment. In some embodiments, the environmental map can include information about the locations of objects, including obstacles, in the environment. The unmanned aerial vehicle can use the generated map to perform various operations, some of which can be semi-automated or fully automated. For example, the environmental map can be used to automatically determine a flight path for the unmanned aerial vehicle to navigate from its current location to a target location. The environmental map can be useful when the unmanned aerial vehicle performs an automatic return function from its current location to a home location or any other specified location. As another example, the environmental map can be used to determine the spatial layout of one or more obstacles, thereby enabling the unmanned aerial vehicle to perform an obstacle avoidance strategy. Advantageously, the use of multiple sensor types to collect environmental data as disclosed herein can improve the accuracy of environmental mapping, even under different environmental and operating conditions, thereby enhancing the robustness and flexibility of unmanned aerial vehicle functions such as navigation and obstacle avoidance.

[0055] Embodiments presented herein can be applied to various types of unmanned aerial vehicles. For example, the unmanned aerial vehicle can be a small unmanned aerial vehicle having a weight of no more than 10 kilograms and / or a maximum dimension of no more than 1.5 meters. In some embodiments, the unmanned aerial vehicle can be a rotorcraft, such as a multicopter (e.g., quadcopter) propelled in the air by a plurality of propulsion units. Further examples of unmanned aerial vehicles and other movable objects suitable for use with embodiments presented herein are described in further detail below. The unmanned aerial vehicles described herein can be operated fully autonomously (e.g., by a suitable computing system, such as an on-board controller), semi-autonomously, or manually (e.g., by a human user). The unmanned aerial vehicle can receive commands from a suitable entity (e.g., a human user or an autonomous control system) and respond to these commands by performing one or more actions. For example, the unmanned aerial vehicle can be controlled to take off from the ground, move in the air (e.g., with up to three translational and up to three rotational degrees of freedom), move to a target location or sequence of target locations, hover in the air, land on the ground, and so on. As another example, the unmanned aerial vehicle can be controlled to move at a particular speed and / or acceleration (e.g., with up to three translational and up to three rotational degrees of freedom) or along a particular path of movement. Further, the commands can be used to control one or more unmanned aerial vehicle components, such as components described herein (e.g., sensors, actuators, propulsion units, payloads, and so on). For example, some commands can be used to control the position, orientation, and / or operation of an unmanned aerial vehicle payload, such as a camera. Optionally, the unmanned aerial vehicle can be configured to operate in accordance with one or more predetermined operational rules. The operational rules can be used to control any suitable aspect of the unmanned aerial vehicle, such as the unmanned aerial vehicle’s position (e.g., latitude, longitude, altitude), orientation (e.g., roll, pitch, yaw), speed (e.g., translational and / or angular), and / or acceleration (e.g., translational and / or angular). For example, the operational rules can be designed such that the unmanned aerial vehicle is not permitted to fly above a threshold altitude, e.g., the unmanned aerial vehicle can be configured to fly at an altitude of no more than 400 m from the ground. In some embodiments, the operational rules can be adapted to provide an automated mechanism for improving the safety of the unmanned aerial vehicle and preventing safety incidents. For example, the unmanned aerial vehicle can be configured to detect restricted flight areas (e.g., airports) and not fly within a predetermined distance of the restricted flight areas, thereby avoiding potential collisions with aircraft and other obstacles.

[0056] Reference is now made to the drawings, Figure 1A drone 102 operating in an outdoor environment 100 is shown, in accordance with an embodiment. The outdoor environment 100 can be an urban, suburban, or rural environment, or any other environment that is at least partially not within a building. The drone 102 can operate relatively close to the ground 104 (e.g., low altitude) or relatively far from the ground 104 (e.g., high altitude). For example, a drone 102 operating less than or equal to approximately 10 meters from the ground can be considered to be at low altitude, while a drone 102 operating more than or equal to approximately 10 meters from the ground can be considered to be at high altitude.

[0057] In some embodiments, the outdoor environment 100 includes one or more obstacles 108a-d. Obstacles can include any object or entity that can impede movement of the drone 102. Some obstacles can be located on the ground 104 (e.g., obstacles 108a, 108d), such as buildings, ground vehicles (e.g., cars, motorcycles, trucks, bicycles), people, animals, plants (e.g., trees, bushes), and other man-made or natural structures. Some obstacles can be in contact with and / or supported by the ground 104, water, man-made structures, or natural structures. Alternatively, some obstacles can be located entirely in the air 106 (e.g., obstacles 108b, 108c), including aircraft (e.g., airplanes, helicopters, hot air balloons, other drones) or birds. Airborne obstacles can not be supported by the ground 104, or water, or any natural or man-made structure. Obstacles located on the ground 104 can include portions that extend substantially into the air 106 (e.g., tall structures such as towers, skyscrapers, light posts, radio towers, power lines, trees, etc.).

[0058] Figure 2 A drone 152 operating in an indoor environment 250 is shown, in accordance with an embodiment. The indoor environment 250 is located inside a building 254 having a floor 256, one or more walls 258, and / or a ceiling or roof 260. Exemplary buildings include residential, commercial, or industrial buildings, such as houses, apartments, offices, production facilities, storage facilities, etc. The interior of the building 254 can be completely enclosed by the floor 256, walls 258, and ceiling 260, such that the drone 252 is confined to the interior space. Conversely, at least one of the floor 256, walls 258, or ceiling 260 can be absent, thereby enabling the drone 252 to fly from the interior to the exterior, or vice versa. Alternatively or in combination, one or more holes 264 can be formed in the floor 256, walls 258, or ceiling 260 (e.g., doors, windows, skylights).

[0059] Similar to the outdoor environment 100, the indoor environment 250 can include one or more obstacles 262a-d. Some obstacles can be located on the floor 256 (e.g., obstacle 262a), such as furniture, appliances, humans, animals, plants, and other man-made or natural objects. Conversely, some obstacles can be located in the air (e.g., obstacle 262b), such as birds or other drones. Some obstacles in the indoor environment 250 can be supported by other structures or objects. Obstacles can also be attached to the ceiling 260 (e.g., obstacle 262c), such as light fixtures, ceiling fans, beams, or other ceiling-mounted appliances or structures. In some embodiments, obstacles can be attached to the walls 258 (e.g., obstacle 262d), such as light fixtures, shelves, cabinets, and other wall-mounted appliances or structures. Notably, structural components of the building 254 can also be considered obstacles, including the floor 256, walls 258, and ceiling 260.

[0060] The obstacles described herein can be substantially stationary (e.g., buildings, plants, structures) or substantially movable (e.g., humans, animals, vehicles, or other objects capable of movement). Some obstacles can include a combination of fixed and moving components (e.g., windmills). Moving obstacles or obstacle components can move according to a predetermined or predictable path or pattern. For example, the motion of a car can be relatively predictable (e.g., according to the shape of the road). Alternatively, some moving obstacles or obstacle components can move along a random or unpredictable trajectory. For example, living creatures such as animals can move in a relatively unpredictable manner.

[0061] To ensure safe and effective operation, it can be beneficial to provide a drone with mechanisms for detecting and identifying environmental objects such as obstacles. Additionally, identifying environmental objects such as landmarks and features can facilitate navigation, particularly when the drone is operating in a semi-autonomous or fully autonomous manner. Furthermore, knowing the precise location of a drone in an environment and the spatial relationship with surrounding environmental objects can be valuable for various drone functions.

[0062] Accordingly, the drones described herein can include one or more sensors configured to collect relevant data, such as information related to the drone’s state, the surrounding environment, or objects within the environment. Exemplary sensors suitable for use with the embodiments disclosed herein include a position sensor (e.g., a global positioning system (GPS) sensor, a mobile device transmitter that enables position triangulation), a vision sensor (e.g., an imaging device capable of detecting visible, infrared, or ultraviolet light, such as a camera), a proximity or range sensor (e.g., an ultrasonic sensor, a lidar, a time-of-flight or depth camera), an inertial sensor (e.g., an accelerometer, a gyroscope, an inertial measurement unit (IMU)), an altitude sensor, an attitude sensor (e.g., a compass), a pressure sensor (e.g., a barometer), an audio sensor (e.g., a microphone), or a field sensor (e.g., a magnetometer, an electromagnetic sensor). Any suitable number and combination of sensors can be used, such as one, two, three, four, five, or more sensors. Optionally, data can be received from different types of sensors (e.g., two, three, four, five, or more types). Different types of sensors can measure different types of signals or information (e.g., position, orientation, velocity, acceleration, proximity, pressure, etc.) and / or obtain data using different types of measurement techniques. For example, sensors can include any suitable combination of active sensors (e.g., sensors that generate and measure energy from their own energy source) and passive sensors (e.g., sensors that detect available energy). As another example, some sensors can generate absolute measurement data provided in a global coordinate frame (e.g., position data provided by a GPS sensor, attitude data provided by a compass or magnetometer), while other sensors can generate relative measurement data provided in a local coordinate frame (e.g., relative angular velocity provided by a gyroscope; relative translational acceleration provided by an accelerometer; relative attitude information provided by a vision sensor; relative distance information provided by an ultrasonic sensor, lidar, or time-of-flight camera). In some cases, the local coordinate frame can be a body coordinate frame defined relative to the drone.

[0063] The sensors described herein can be carried by a drone. The sensors can be located on any suitable portion of the drone, such as an upper portion, a lower portion, a side portion, or an interior portion of the drone body. Some sensors can be mechanically coupled to the drone such that the spatial layout and / or motion of the drone corresponds to the spatial layout and / or motion of the sensor. The sensors can be coupled to the drone by a rigid coupling such that the sensor does not move relative to the portion of the drone to which it is coupled. Alternatively, the coupling between the sensor and the drone can allow movement of the sensor relative to the drone. The coupling can be a permanent coupling or a non-permanent (e.g., removable) coupling. Suitable coupling methods can include adhesives, bonding, welding, and / or fasteners (e.g., screws, nails, pins, etc.). Alternatively, the sensors can be integrally formed with a portion of the drone. Further, the sensors can be electrically coupled with a portion of the drone (e.g., a processing unit, a control system, a data store) to enable data collected by the sensors to be used for various functions of the drone (e.g., navigation, control, propulsion, communication with a user or other device, etc.), such as the embodiments discussed herein.

[0064] The sensors can be configured to collect various types of data, such as data related to the drone, the surrounding environment, or objects in the environment. For example, at least some of the sensors can be configured to provide data regarding the state of the drone. The state information provided by the sensors can include information regarding the spatial layout of the drone (e.g., position or orientation information such as longitude, latitude, and / or altitude; orientation or attitude information such as roll, pitch, and / or yaw). The state information can also include information regarding the motion of the drone (e.g., translational velocity, translational acceleration, angular velocity, angular acceleration, etc.). The sensors can be configured to determine, for example, the spatial layout and / or motion of the drone relative to up to six degrees of freedom (e.g., three degrees of freedom of position and / or translation, three degrees of freedom of orientation and / or rotation). The state information can be provided relative to a global coordinate system or relative to a local coordinate system (e.g., relative to the drone or another entity). For example, the sensors can be configured to determine a distance between the drone and a user controlling the drone, or a distance between the drone and a flight origination point of the drone.

[0065] The data obtained by the sensors can provide various types of environmental information. For example, the sensor data can indicate an environmental type, such as an indoor environment, an outdoor environment, a low-altitude environment, or a high-altitude environment. The sensor data can also provide information regarding current environmental conditions, including weather (e.g., sunny, raining, snowing), visibility conditions, wind speed, time of day, etc. Further, the environmental information collected by the sensors can include information regarding objects in the environment, such as obstacles described herein. The obstacle information can include information regarding the number, density, geometry, and / or spatial layout of obstacles in the environment.

[0066] In some embodiments, the sensing results are generated by combining sensor data obtained by multiple sensors (also referred to as "sensor fusion"). For example, sensor fusion can be used to combine sensing data obtained by different sensor types, including GPS sensors, inertial sensors, vision sensors, lidar, ultrasonic sensors, etc. As another example, sensor fusion can be used to combine different types of sensing data, such as absolute measurement data (e.g., data provided relative to a global coordinate system, such as GPS data) and relative measurement data (e.g., data provided relative to a local coordinate system, such as vision sensing data, lidar data, or ultrasonic sensing data). Sensor fusion can be used to compensate for limitations or inaccuracies associated with individual sensor types, thereby improving the accuracy and reliability of the final sensing results.

[0067] In some embodiments, the drone can maintain communication with a control device that is typically operated by a user on the ground. Indirect or direct communication between the drone and the control device can be provided. For example, the drone can maintain indirect communication with the control device through a communication network, such as a cellular network. In another example, the drone can maintain direct communication with the control device through a direct, short-range wireless communication channel.

[0068] One-way or two-way communication can be provided between the drone and the control device. Typically, the control device transmits control signals to the drone to direct the flight of the drone. Control data from the control device can control the operation of the drone and / or a payload carried by the drone. The operation of the drone propulsion units, flight controllers, navigation modules, one or more sensors, a carrier of the drone, a landing pad of the drone, and / or a communication module of the drone can be controlled.

[0069] The drone can transmit data or other information generated by its sensors, motors, or other to the control device, which can be useful for further directing the flight. Any description of sensor data can include any data from the drone to the control device, including but not limited to data collected by the sensors about the external environment and / or data about the operational state of the drone. Data about the external environment of the drone can include but is not limited to image data, distance or depth profile, satellite signals, etc. Data about the operational state of the drone can include but is not limited to information about the power provided to the propulsion units of the drone, the power consumed by the propulsion units, the navigation of the drone, the attitude of the drone, the layout of the carrier, the layout of the payload, the communication of the drone, or the energy storage and / or consumption of the drone. As one example, the operational state data indicates the internal condition of the drone, such as sufficient energy level or low battery level. As another example, the sensor data indicates the current condition of the flight environment, which can include a forward obstacle not present on the existing map as described above, or a severe weather condition. The transmission is typically performed periodically, but can be adjusted as needed.

[0070] In some embodiments, the drone is configured to collect data during its flight and construct a 2D or 3D map based on the collected data. As one example, the drone can track various types of wireless transmissions. The drone can track wireless signals to monitor the wireless communication environment. For example, the drone can track incoming GPS signals, incoming control signals, outgoing data signals, etc. The drone can track signals used for navigation, flight control, sensors, image transmission, or any other type of communication. The drone can track whether a wireless communication channel is available, bandwidth, signal strength, signal-to-noise ratio, and / or any other aspect of the wireless transmission. The drone can track wireless transmissions at different locations. The final signal map can provide the drone with reference signal strength in a particular space. As another example, the drone can track the presence or absence of any objects in the air. In addition to the data collected by its sensors, the drone can incorporate data collected by other drones, or data recorded in other data sources at any time before or during the flight or at any time. For example, various topographical maps and weather maps are available to the public. The drone can also update its specific coordinate map according to new data generated by its sensors at any time or new data obtained from other sources.

[0071] In some embodiments, to make the drone more efficient in flight, the drone can include only one sensor to reduce weight or design complexity. The sensor can be located at the front, with a detection range in front of the drone. For example, within the detection range, a camera can capture a view with suitable resolution, or a laser can detect the distance to an object with sufficient accuracy, the detection range can be characterized by a certain height, width, and depth relative to the sensor location. In addition, the drone can include a single sensor located at the back or side of the drone, which can have a suitable detection range at the back or side of the drone. The sensor data can be processed to construct a 2D elevation map, which represents the height information of each point in a 2D coordinate system relative to a reference level (e.g., ground level).

[0072] Figure 3 Data is shown on how the drone obtains a 2D elevation map. An object 320 extending from a reference level 330 intersects a detection range 340 at a point 350. However, only the middle portion of the object 320 is within the detection range, while the top portion of the object 320 is not. By analyzing the data produced by the sensor 315 corresponding to the detection range 340, the drone can conclude that the top portion of the object 320 is higher than the top of the detection range 360, or in other words, the distance from the top of the detection range 360 to the reference level is the minimum value of the surface height at the point 350. Therefore, the drone can store this distance in the elevation map as an estimate of the surface height at the point 350 with the 2D coordinates relative to the reference level, and a category indicating that the estimated value is a lower threshold or minimum value of the surface height. When the top portion of the object is at or below the detection range, the drone can similarly create data for the 2D elevation map. Details regarding the construction and maintenance of the 2D elevation map can be found in the co-pending application entitled “Constructing and Updating Elevation Maps” filed on November 14, 2016, which is incorporated by reference in its entirety. Optionally, multiple sensors can be provided on the drone. Different sensors can have different fields of view.

[0073] In some embodiments, the resulting 2D elevation map includes information other than the estimated height for each 2D coordinate, such as a confidence indicator of the risk of collision to an estimated obstacle. For example, the 2D elevation map can indicate for each 2D coordinate the height estimated from the particular sensor data and a category indicating whether the estimated height is an actual value of the actual surface height at the 2D coordinate (hereinafter referred to as the "green category"), a minimum value (hereinafter referred to as the "red category"), or a maximum value (hereinafter referred to as the "blue category"). For coordinates with no available information, the 2D elevation map can also indicate a default value or a maximum value as the estimated height and a category indicating that the estimated height is not useful (hereinafter referred to as the "blank category"). The 2D elevation map can also include a confidence indicator or other indicator of the quality of the estimate, such as the capabilities of the sensor, the visibility at the time the sensor data was generated, etc. The confidence indicator can be implemented by a number, a string, or other ordinal value. In some embodiments, the confidence indicator can be represented as a category. Different values of the confidence indicator can represent different degrees of risk of an obstacle or hazard. For example, the blank, red, blue, and green categories indicate increasing levels of risk of an obstacle or hazard. In another example where the confidence indicator is represented by a numerical value, a higher confidence indicator can represent a higher risk, while a lower confidence indicator can represent a lower risk. Alternatively, a lower confidence indicator can represent a higher risk, and vice versa. Such a 2D elevation map is relatively easy to construct and maintain, and can be used to avoid obstacles or other undesirable locations when determining a flight path.

[0074] Figure 4 A 2D elevation map based on data collected by a forward-looking sensor is shown. The landscape displays the estimated height of the 2D coordinates on the ground, while the shading of the landscape displays the confidence indicator of the estimated surface height for the corresponding 2D coordinates. The shading can also indicate the category of the estimated surface height. For example, black can correspond to the red category, dark gray can correspond to the green category, light gray can correspond to the blue category, and no color or a blank color can correspond to the blank category. Thus, the 2D coordinate of point 402 has the red category, which means that the surface height of point 402 is a minimum value of the actual surface height at the 2D coordinate. Similarly, the 2D coordinate of dark gray point 404 has the green category, the 2D coordinate of light gray point 406 has the blue category, and the 2D coordinate of white point 408 has the blank category. Such a visual representation can be useful for a user controlling the drone.

[0075] In some embodiments, the 2D elevation map includes, in addition to the estimated height for each 2D coordinate and the confidence indicator associated with the estimated surface height, a height range to which the estimated surface height belongs. The height range information is particularly useful for 2D coordinates with multiple surface heights, as there are objects that do not extend all the way from the ground.

[0076] Figure 5 The scenario is shown where multiple surface heights exist for a 2D coordinate. The estimated surface height and confidence score determined for the point 502 under the bridge 504 can differ depending on the height range. For example, the drone can determine that the surface height of the point 502 in a height range between 0 m and 10 m is no more than 10 m, while the drone can determine that the surface of the point 502 in a height range between 10 m and 20 m is at least 20 m. Thus, given a 2D elevation map that includes height ranges associated with estimated heights for 2D coordinates, the drone can compare the height ranges to its current flight height to better determine its flight path, potentially taking advantage of additional flight space, such as the area under the bridge. For example, the height range can be selected based on the current height of the drone. For example, if the height of the drone falls between 0 meters and 10 meters above the ground, a first height range of 0 meters to 10 meters can be selected; if the height of the drone falls between 10 meters and 20 meters above the ground, a second height range of 10 meters to 20 meters can be selected. Depending on the selected height range, different estimated surface heights and confidence indicators can be determined for the same 2D point. The flight path relative to a given 2D point can be determined based on the estimated surface heights and confidence indicators for the 2D point (2D elevation data), taking into account the height ranges relevant to the 2D elevation data and / or the drone height.

[0077] In some embodiments, the drone can classify each point in the region based on which types of information are available for that location. For example, the drone can distinguish between those points detected by its sensors during the current flight, during a previous flight (possibly any flight prior to the current time point, any flight prior to the drone’s last stop, etc.), or during any flight, by being one of a group of predetermined drones belonging to the same organization or having access to the same elevation map, or a combination thereof. The drone can also distinguish between those points that have ever collected any information and those points that have never collected any information. While the degree of safety can vary depending on the generator of the information, the manner in which the information was generated, the time at which the information was generated, etc., the availability of information makes certain distinguished points “safe” to some degree. Generally, a point is unsafe for the drone unless it can be accessed or reached by the drone. The point can be unreachable for the drone due to the nature of the point (e.g., the presence of an obstacle or inclement weather), the operational state of the drone (e.g., lack of fuel), etc. In addition, certain points can be designated for special purposes. For example, a list of start points for the current flight or commonly used takeoff and landing points can be determined and marked on the map. A list of service points where the drone can be serviced—receive energy or other supplies, exchange components, inspection, adjustment, and repair, etc.—can be identified.

[0078] In some embodiments, aerial drones can need to find new flight paths in response to abnormal and often emergency situations. For example, data transmission from or to the drone can be intermittent or otherwise interrupted, resulting in weak or disappearing signals. When a user is controlling the flight of the drone from a remote controller, loss of control signals can mean that the drone needs to be piloted automatically or risk ending in a crash. When navigation signals (e.g., GPS) are lost, the drone can be unable to accurately indicate the location of the drone. Alternatively, the drone can have to rely on other navigation techniques to determine a possibly less accurate drone location (e.g., that can "drift" over time) or that can use more energy. In some cases, data transmission from the drone (e.g., image transmission) can be lost, which can present more challenges to a user trying to remotely control the drone. Weather conditions can be stormy or otherwise affect the operation of the vehicle (drone), including sensor performance. Batteries on the drone can run out during flight, and the drone can reach a low energy level beyond which the drone can not have enough energy to return. Drone components can fail due to mechanical, electrical, or other issues, such as a part breaking off, a circuit error, a battery draining, etc.

[0079] In any of these scenarios, it would be highly desirable to first restore normal conditions by finding a new flight path to a possibly new destination. For example, upon determining that wireless transmission is intermittent or non-existent, the drone would want to reach an area where wireless transmission is successful and uninterrupted. The determination can be made from an inability to receive normal control signals from a control device or GPS signals from a satellite, a failure to transmit relevant data to the control device, repeated notifications from the control device that no or little data was received at a particular time period, etc. Similarly, the drone would want to reach an area where air conditions are calm and clear or where the drone is to provide service.

[0080] In some embodiments, when the communication channel between the drone and the control device is not interrupted, especially when transmission of control signals continues to be acceptable, the control device can instruct the drone to follow a particular flight path to reach a particular destination, such as one of the starting point (flight takeoff point) or a designated point where the abnormal situation can be resolved. The designated point can be the last known location of the user (e.g., based on the location of the remote controller), a preset "homing" point designated by the user, an earlier location on the current or a previous flight, etc. However, when the communication channel is interrupted, the drone can automatically determine a flight path to reach a suitable destination. The drone can initiate and / or execute this process autonomously or semi-autonomously to reach a suitable destination. The drone can initiate and / or execute this process to reach a suitable destination without human assistance or intervention and / or without any instructions from any device external to the drone.

[0081] As previously mentioned, the drone can initiate a process to reach an appropriate destination in response to one or more detected conditions. The one or more detected conditions can fall into one or more different cause categories. In some embodiments, a plurality of cause categories can be predetermined. Examples of cause categories can include, but are not limited to, user instruction, GPS signal loss, remote controller signal loss, image transmission loss, other wireless signal loss, low battery power, and / or component failure. The drone can identify which cause category the detected condition belongs to. Depending on the cause category, the drone can select a flight mode to reach the appropriate destination. Further descriptions of flight modes are provided in more detail elsewhere herein. In some embodiments, a single flight mode can correspond to a single cause category. Alternatively, multiple flight modes can correspond to a single cause category, or a single flight mode can correspond to multiple cause categories.

[0082] In some embodiments, the drone is configured to automatically determine a flight path in response to an abnormal condition without human intervention based on various factors, including automatically returning to one of the locations detected by on-board sensors since take-off. The drone can fly in any of different navigation modes. For example, based on sensor data previously generated by the drone or other drones, the drone can fly in a "safe" mode whose flight path is limited to "safe" points for which there is sufficient information to fly safely. Such information can relate to the presence or absence of obstacles, weather conditions, crowding, etc. The drone can also fly in an "exploring" mode whose flight path is less restricted and can flexibly achieve specific goals, such as flying the shortest distance. There can be a hierarchy of different navigation modes, starting from the safe mode, each less restricted than the next, based on the number of spatial points that can be considered in determining the flight path, the amount of time required to reach the next destination, etc. There can also be different navigation modes to achieve different goals.

[0083] In some embodiments, to recover from the abnormal condition as quickly or as simply as possible, the drone can choose to limit the next flight to "safe" points for which sufficient information is available, as described above, and thus fly in a safe mode. Specifically, the drone can find an ideal flight path between safe points for the drone to follow along the flight path. This approach tends to reduce the search space as well as the likelihood of exacerbating the abnormal condition. In the safe mode, a simple flight path is the reverse route. For example, after failing to transmit data to the control device for a period of time, the drone can backtrack to a point where wireless transmission is stronger, an area where data transmission rate is above a threshold, or any other safe point according to a map. The flight path can be determined systematically in other ways, as described below. The drone can dynamically determine when to enter or exit a mode according to real-time information related to user input, environmental conditions, drone operating status, etc.

[0084] In some embodiments, once the abnormal situation is resolved, the drone is configured to determine another flight path in real time. At this point, it is not important to limit flight to a safe point, so the drone can flexibly determine its flight route, thus entering an exploration mode. The drone can hover to collect and evaluate data, or can determine the flight path incrementally while traveling. Hovering can continue until a user instruction from a control device is received, until a new flight route is determined, until a particular time period ends, etc. The user instruction can determine a subsequent flight path. Otherwise, the drone can select a next destination and determine a corresponding flight path based on current environmental or vehicle conditions. As one example, after struggling through a poor signal area, energy or battery power can be low, and then the drone can select a service station for replenishment or recharging. As another example, it can be important to reach the original destination before the abnormal situation occurred, and the drone can return to the flight path it was following before the abnormal situation arose. As yet another example, there can be no time to go anywhere, and then the drone can go directly back to the starting point.

[0085] In some embodiments, the abnormal situation can persist upon reaching a safe point selected for restoring normal conditions. Then, for example when a number of repeated attempts or an amount of time spent on repeated attempts exceeds a predetermined threshold, the drone can repeatedly select additional safe points as subsequent destinations until the abnormal situation is resolved or some other criteria are met. The number of repeated attempts can be determined based on a distance to known safe points, a location of the drone, a time remaining before the drone needs to return to the takeoff point, etc. Alternatively, the drone can enter an exploration mode since the safe points are no longer proving to be advantageous, until a particular criteria is met or remain in the exploration mode until it reaches a final destination.

[0086] Figure 6An example procedure performed by the drone in response to an abnormal situation is shown. In step 602, the drone identifies an abnormal state of signal transmission (at a first location). The identification can be based on data generated by its sensors or motors. The abnormal state can be a loss or weakening of the signal, which can be measured by the signal transmission rate over a period of time that can be relative to a minimum threshold. In step 604, the drone selects a first destination such that signal transmission by the drone's or other drones' sensors at an earlier time is considered normal at the first destination or at a second location proximate to the first destination. Proximity can be determined relative to a predetermined distance. In step 606, the drone determines a first path to the first destination. Preferably, the drone travels to the first destination in a safe mode, which means that the drone only searches between "safe" points, which are those locations previously detected by the drone's or other drones' sensors for building the first flight path. In step 608, upon following the first flight path to reach the first destination, the drone assesses the signal transmission state at the first destination.

[0087] In steps 610 and 612, the drone determines a second destination and a second flight path to reach the second destination in real time based on the current state of signal transmission at the first destination. The determination can also be based on other current conditions, such as the control state of the remote device, the weather, the operational state of the aircraft, etc. If the current state of signal transmission is normal as expected, the drone can be less restricted in selecting the second destination and the second flight path. Specifically, the drone can no longer be in a safe mode. Without any command from the user, the drone can determine the best next destination. For example, if the current battery level is low, the drone can select a service station or a local base station as the second destination and determine the fastest way to reach the second destination as the second flight path. On the other hand, if the current state of signal transmission is still unsatisfactory, the drone can repeat selecting another safe point as the second destination until the current signal condition recovers to normal. In this process, the drone can maintain the safe mode. Alternatively, the drone can immediately enter an exploration mode and select the second destination according to real-time information.

[0088] In some embodiments, the drone can generally follow a planned flight path. However, the drone can also establish a flight path in real time by repeatedly selecting a nearby intermediate destination or simply flying in a direction each time, without preselecting all points on the flight path at once. The scope of "real time" is generally considered to mean immediate feedback without any intentional delay, which can be measured in minutes, seconds, tenths of a second, etc. By constructing the flight path in this way, the drone can have a high degree of adaptability and can better respond to exceptional situations. For example, on-board sensors can generally detect objects that constitute obstacles that the drone needs to avoid. The stationary obstacles can already be known, but moving obstacles can arise. In addition, routine motor checks can have been performed, but unexpected motor errors can not have been detected or can arise. Thus, when an adverse situation such as a motor failure or an obstacle is detected, the drone can deviate from the planned flight path and establish a new flight path in real time. When the adverse situation is overcome, the drone can appropriately return to the planned flight path. On the other hand, the continued need to determine the next step can result in fewer computing resources being available for other tasks during flight. Thus, the drone can need to select more destinations at once, or make real-time decisions less frequently, especially when the demand for computing resources is high.

[0089] In some embodiments, the control device can continue to direct the drone to follow a particular flight path as long as control signals from the control device continue to be received by the drone. In the case where the drone has already automatically determined a flight path, the drone can choose to override the control signals until it reaches the end point of the flight path or some intermediate destination. This approach can be advantageous when the drone needs to act quickly and avoid any delay that can occur during communication with the control device. Alternatively, the drone can abandon the automatically determined flight path upon receiving the control signals to follow the instructions issued by the control device.

[0090] In some embodiments, the drone can need to automatically determine a flight path through the airspace. The drone can determine a new flight path upon entering a new navigation mode, entering a different geographical region, encountering an anomaly, receiving instructions to reach a new destination, etc. The drone can focus on predetermined points within the airspace, or treat the airspace as a grid of points. The drone can then select a flight path that passes through one or more of these points according to certain objectives. Example objectives include reducing overall flight time or distance, avoiding areas that are more likely to contain obstacles, and avoiding changes in flight altitude. To be able to systematically search for an ideal flight path, the drone can construct a cost function c(s, s') by assigning a numerical cost to a segment between any two of the points s and s'. Such a segment can represent a path segment that the drone can follow while moving between points s and s'. For example, the cost can be related to the length of the segment (distance between the endpoints of the segment), average energy consumption rate of the segment, landscape of the segment, or other characteristics of the segment. The drone can also assign a cost to each of the points. For example, the cost can be related to the likely presence of obstacles, typical temperature at the point, and distance from the point to any "safe" point, etc. In general, any factor that can affect the "quality" of the flight can be represented in the cost function, such as wireless signal (control, GPS, data, etc.) transmission status (e.g. transmission rate, signal strength, signal-to-noise ratio, etc.), weather conditions (rain, snow, fog, hurricane, etc.), climate conditions (wind, pressure, temperature, brightness, etc.), gravitational effects, etc. The search can then typically be performed by building a route with a reduced or (locally or globally) minimum cost, which is a total cost that aggregates the costs of the segments and points that are aggregated in the growing route. Typically, the cost function is pre-established and the costs are assigned (obtained) and can be looked up (obtained) during the search for the desired flight path. For example, the cost function can be built for a region once, for subsequent searches within that region. However, the cost of a segment can be dynamically computed (obtained) each time that segment is considered in the search.

[0091] In some embodiments, the drone is configured to assign costs based on the 2D elevation map discussed above. The costs can depend in various ways on the estimated altitude and on the risk of encountering an obstacle or one or more confidence indicators for each 2D coordinate. For example, a 2D coordinate with a larger estimated altitude, a larger altitude range known for the estimated altitude, a lower confidence indicator associated with the estimate, or any combination thereof, can be considered less favorable for a flight path, and thus result in a higher cost. Points with such 2D coordinates (sometimes referred to as "target points" below) should typically be avoided, or be assigned a lower priority. For example, any point in the space that can be mapped to a 2D coordinate in the elevation map that is associated with a confidence indicator that indicates a red category, e.g. Figure 4Points on the object 402 in the 2D elevation map (especially those below the estimated surface height) can be considered to be part of an obstacle to avoid in the flight path. Thus, greater cost can be assigned to points or segments that, when projected to the reference level, are closer to more of these unfavorable 2D coordinates. The cost can further depend on the class of the object associated with the target point. For example, higher cost can be assigned to those with a corresponding target.

[0092] In some embodiments, the drone can incorporate the cost determined from the 2D elevation map into the cost function in different ways. It is generally better to stay away from points that are more likely to contain obstacles. Thus, the drone can associate more cost with approaching points whose estimated height is above a particular threshold, or whose corresponding 2D coordinate class is red or blank in the 2D elevation map. In a more refined plan, greater cost can be added to approaching points whose estimated height is above the current flight height, as well as to regions of points whose estimated height is not above the current flight height and whose class is red or blank. Other schemes are possible. The proximity can be related to physical distance, travel time, etc. Varying the cost with proximity to a particular 2D coordinate means that such cost tends to peak near the particular 2D coordinate and gradually decrease as other coordinates get farther and farther away from the particular 2D coordinate. It is able to choose a flight path that is less proximate to those 2D coordinates.

[0093] To incorporate the cost of proximity to the 2D coordinates of the red category, for example, the cost assigned to each segment from the first point to the second point can be adjusted to be related to a combination (e.g., sum) of the physical distance between the two points and the physical distance from the segment to the 2D coordinates of the red category. In general, the search space can include a set of points within an area that includes the departure point and the arrival point of the drone. For example, the relationship between a point in the search space and the 2D coordinates of the red category or any point of interest that is not physically within the search space can be represented as a helper segment, and the cost attached to the vicinity of the 2D coordinates in the red category can then be associated with such helper segments. Such helper segments serve to represent the relationship to the point of interest (e.g., proximity) with which the cost can be associated; it can but need not correspond to a path segment. Separate helper segments can be used to represent proximity to each point of interest, or a single helper segment can be used to represent overall proximity to all points of interest. For example, the overall proximity can correspond to the distance to the nearest point of interest, or the average distance to a set of points that are within a certain radius relative to the red category and the particular point. The distance between a segment and a particular point can be computed in various ways, such as the shortest distance or the average distance between the particular point and any point on the segment. Mathematically, the cost of proximity to the 2D coordinates can be incorporated by updating an existing cost function c(s, s') or adding another cost function red_cost(s, s'). Alternatively, the cost assigned to each point (one of the endpoints of a segment) can be adjusted to be related to the physical distance from that point to the 2D coordinates of the red category. Additional costs of proximity to the 2D coordinates of the red and white categories, weighting proximity to the 2D coordinates of different categories, or otherwise can be incorporated. For example, the weights can decrease for the white, red, blue, and green categories, respectively. Additional costs of proximity to other points that are generally to be avoided (e.g., points on a human body) can also be incorporated to avoid collisions with humans, or points that are not detected by known sensors on the drone.

[0094] In some embodiments, the drone preferably maintains a constant altitude to reduce the effects of gravity and possible complexity of altitude changes. Specifically, searching for a flight path that passes through points with different altitudes can require more computational resources and result in a more complex flight path than searching for points with the same or similar altitudes. To avoid changes in flight altitude, upon detecting an obstacle, the drone can decide to fly around it while maintaining the flight altitude, rather than going over the obstacle by increasing the flight altitude. However, in some cases, such as when the width of the obstacle at the current flight altitude is much greater than the height of the obstacle relative to the current flight altitude, it can be necessary to increase the flight altitude.

[0095] In some embodiments, the drone is configured to incorporate costs based on height information of points in the search space differently. For example, higher costs can be related to flying to higher or different altitudes. Similar to incorporating costs for 2D coordinates close to a particular class on a 2D elevation map, the cost assigned to each segment or point can be adjusted to cover costs related to flying altitude. For example, the original cost assigned to a segment from a first point to a second point can be the physical distance between the corresponding 2D coordinates on the ground, and an additional cost can be added during execution of the path search algorithm when the altitude increase from the first point to the second point is greater than a certain threshold. Furthermore, the drone can weigh the vertical and horizontal components of the cost differently. For example, the drone can assign a smaller weight to the cost associated with the distance between the corresponding 2D coordinates, but assign a larger weight to the cost associated with the altitude difference.

[0096] In some embodiments, the drone can utilize the flying altitude included in the 2D elevation map to determine the flight path. For example, returning to Figure 5 , the 2D elevation map can include (0m~10m, green), (10m~20m, red), and (20m~30m, blue) in the form of (altitude range, class). When the drone is flying at 8 meters, the 2D coordinates with (10m~20m, red) (e.g., 502) no longer correspond to obstacles for the drone. Similarly, when the current flying altitude is 15m, (10m~20m, red) has a greater impact on the drone than (0m~10m, green) and (20m~30m, blue). One approach is to prioritize elevation data with an altitude range closer to the current or desired flying altitude, for example, by selecting the class associated with the altitude range closest to the current or desired flying altitude among all elevation data for the 2D coordinates. For example, when the drone is flying at 8 meters, green will be selected as the class for point 502, and therefore all points above 0m~10m range from point 502, and therefore flying near any of these points will not be disadvantageous. Obviously, this approach is safe only when the drone maintains more or less the same flying altitude. Another approach is to incorporate all elevation data. For example, regardless of the current flying altitude, the drone will treat points above 502 as being associated with different classes and can assign costs accordingly. The drone can then choose its flying altitude, but in effect change its flying altitude to 20 meters will be impeded. Yet another approach is to dynamically prioritize elevation data during execution of the path search algorithm based on the altitude of the last point on the growing route so that the next point is added to the growing route.

[0097] In some embodiments, the drone can employ any path search algorithm known to those of ordinary skill in the art, including A*, Theta*, D*, and variants thereof, to systematically determine a prioritized flight path from a source (start point) to a destination (end point) using an objective function that represents the cost of a path from the start point to the end point. Such path search algorithms can operate by iteratively minimizing the value of the objective function over different routes in the search space from the start point, where the value of the objective function increasingly approximates the actual cost of an actual path from the start point to the end point. For example, the A* algorithm works by minimizing the following objective function f(n) = g(n) + h(n), where n is the last point on a growing route, g(n) is the cost of the route from the start point to n, and h(n) is a heuristic that estimates the cost of the cheapest route from n to the end point. g(n) and h(n) can be combined by summation or other aggregation method. The cost of a route can be the total cost or other aggregated cost assigned to all segments and / or all points on the route. The A* algorithm builds one or more growing routes until one of them reaches the end point, and the growing route that reaches the end point has the minimum possible cost when h(n) is acceptable. Specifically, g(n) can be decomposed as g(parent of n) + c(parent of n, n), where the parent refers to the previous point on the growing route; h(n) is typically just the (straight-line) distance from n to the end point. At each iteration, one n is identified and added to a growing route. Specifically, a set of candidate points for n is identified. These candidate points are typically adjacent points to the last point on an existing growing route (and thus connected to the last point by a segment), but other criteria can be employed in selecting the candidate points. For example, the set of candidate points can be restricted to those whose corresponding segment (which can correspond to the length of the segment or the distance between the two points at the ends of the segment) has a cost below a predetermined threshold. The candidate point among all candidate points that results in the minimum value of the objective function is then added to the growing route.

[0098] The Theta* algorithm works in a similar manner, except that the presence of a line of sight is considered and an existing segment can be replaced with a line-of-sight segment in the growing route. In one version, the Theta* algorithm considers, at each iteration, the path segment between the parent of the parent of n and n (referred to as a line-of-sight segment), and replaces the two segments from the parent of the parent of n to n with the corresponding line-of-sight segment when the cost of the two segments is greater than the cost of the line-of-sight segment. Typically, a line of sight is considered for each candidate point, and then the candidate point among all candidate points that results in the minimum value of the objective function is added to the growing route as in A*. However, instead, it can first identify n in the set of candidate points in A*, and consider whether to incorporate a line-of-sight segment for the identified n only.

[0099] In some embodiments, the drone can use different methods to determine a flight path in different areas. For example, the drone can switch between A* and Theta*. The main difference between A* and Theta* is that in A*, the flight path is built from predetermined segments between points, while in Theta*, the flight path can include line-of-sight segments. Since line-of-sight segments essentially skip points, Theta* generally runs faster than A*. Figure 7 An example process of determining a flight path in an area without obstacles is shown. As shown by the dashed line in Figure 7 A* can be used to find the shortest flight path from the source 702 to the destination 708. Figure 8 An example process of determining a flight path in an area with obstacles is shown. The obstacles can be near points 804 and 806, and there can no longer be a segment from point 804 to point 806. Then, depending on the proximity to points 804 and 806, the drone can assign an additional cost to the points or segments. A* can then be used to find a flight path that is short and far away from the obstacles, as shown by the dashed line from point 802 through points 810 and 812 to point 808 in Figure 8 An example process of determining a flight path in an area with obstacles is shown. The obstacles can be near points 804 and 806, and there can no longer be a segment from point 804 to point 806. Then, depending on the proximity to points 804 and 806, the drone can assign an additional cost to the points or segments. A* can then be used to find a flight path that is short and far away from the obstacles, as shown by the dashed line from point 802 through points 810 and 812 to point 808 in Figure 9 Another example process of determining a flight path in an area with obstacles is shown, as shown by the dashed line from point 902 through point 910 to point 908. Theta* can be used instead to find a flight path that is short and far away from the obstacles. Instead of using A* or Theta* to find a route in an area, line-of-sight segments can also be considered dynamically in the path search process based on how close the current point is to an obstacle. For example, when applying the A* algorithm, the system can temporarily consider the existence of a line of sight between the parent of the parent of n and n when determining that n is not close to an obstacle, and vice versa.

[0100] In some embodiments, the drone is configured to select which path search algorithm to use depending on whether the area to be searched includes a known obstacle or is close to a known obstacle. As can be seen, although the flight path found by Theta* can be shorter than the flight path found by A*, it can be closer to the obstacle, especially when the additional cost is related to points rather than segments. Since the original route is generally shorter than the line-of-sight route, the selection of the flight route can be more easily controlled better. Furthermore, adding a cost to points in the search space can be easier than adding a cost to additional line-of-sight segments used only in Theta*, which considers points in the search space regardless of which path search algorithm is used. Thus, the drone can choose to use A* in areas around known obstacles and Theta* in other areas. Figure 10An example approach to using different path search algorithms for different regions is shown. The boxes represent the airspace, and the shading (black, dark gray, light gray, and white) of the points in the airspace represents the categories of the corresponding 2D coordinates in the 2D elevation map. To go from the source 1028 to the destination 1030 through the airspace, the drone can first identify each region 1022, e.g., the point 1026, that can be located within a certain distance from where an obstacle can exist, e.g., the point 1026 at which the corresponding 2D coordinate is associated with the red category in the 2D elevation map. The drone can then use A* in the region 1022 and Theta* in the other regions 1020 and 1024 to find a flight path. In this case, for each region, the drone can select a set of possible regional sources and a set of possible regional destinations such that any flight path from a regional source to a regional destination runs in a general direction toward the (final) destination. For example, the drone can take a simple flight route between the last regional destination and the current regional source. In a simpler approach, the drone can first select a set of intermediate points 1032 and 1034, e.g., a set of points detected by the sensors during a previous flight. The drone can then generally divide the airspace into the regions 1020, 1022, and 1024 along these intermediate points. Depending on the likelihood of an obstacle existing in each region, which can be indicated by the number of points in the region for which the corresponding 2D coordinate is associated with the red category, the drone can then select an appropriate path search algorithm. More generally, depending on the properties of the different path search algorithms and the properties of the different regions within a given airspace, the drone can select other combinations of algorithms for different regions, different modes of navigation, different reasons for changing a flight path, etc.

[0101] Figure 11An example process performed by a drone to determine a flight path based on costs derived from a 2D elevation map is shown. The drone starts with a set of points in the airspace and a set of directional segments connecting some of the points. A directional segment generally exists between two points when the two points are considered to be adjacent to each other. The segments are generally directional because the cost of flying from point A to point B can be different than the cost of flying from point B to point A. The drone then builds a cost function, which can contain different types of information, such as the information available in the 2D elevation map discussed above. A particular cost can be assigned to a directional segment between adjacent points, to an individual point, to an additional line-of-sight segment, etc. In steps 1102 and 1104, the drone builds the cost function. In step 1102, the drone assigns a cost to each existing segment (each directional segment) between each ordered pair of adjacent points in the airspace. The assignment can be based on an arbitrary combination of relevant factors, including physical distance, average battery power, average GPS signal strength, average signal transmission rate between the drone and the remote device, etc. In step 1104, the drone further assigns a cost to each point based on its relationship to, for example, a particular "dangerous" point (e.g., a point whose 2D coordinates are associated with the red category in the 2D elevation map). The relationship between a point and a dangerous point can be represented by an auxiliary segment between the two points. Optionally, in searching for a flight path, the cost of each segment can be calculated at each time the segment is considered. In step 1106, the drone then performs a path search algorithm, such as A*, Theta*, or any variant, using the cost function it built. To perform Theta*, the drone needs to consider additional line-of-sight segments and the costs associated with these segments. The drone can either assign these additional costs to the line-of-sight segments in advance or calculate them on the fly in the course of performing the Theta* algorithm.

[0102] Figure 12 An example process performed by a drone to determine a flight path by determining connected sub-flight paths through different regions using different path search algorithms is shown. Essentially, Figure 11Step 1106 in FIG. 11 can be performed by this process. In steps 1202 and 1204, the drone identifies intermediate points or intermediate destinations and uses these intermediate points to break the search space into regions. Each region will then be defined at least in part by two intermediate points or one intermediate point and the source or destination. In an alternative approach, the drone first identifies regions and uses these regions to identify intermediate points. In step 1206, the drone classifies each region based on one or more factors (e.g., the likelihood of obstacles being present in the region). For example, the drone can classify a region containing a point whose 2D coordinates are associated with a red category as a "dangerous" region and classify any other region as a "safe" region. The drone will also associate each category with a particular path search algorithm. For example, dangerous regions can be associated with A* while safe regions can be associated with Theta*. In step 1208, the drone determines a flight path for each region between its two defining points using the path search algorithm associated with the region's category to determine a flight path for the drone through the airspace.

[0103] Figure 13 A drone (UAV) 800 according to an embodiment of the application is shown. As described herein, a UAV can be an example of a movable object. The UAV 800 can include a propulsion system having four rotor rotors 1302, 1304, 1306, and 1308. Any number of rotor rotors can be provided (e.g., 1, 2, 3, 4, 5, 6, or more). The rotors can be embodiments of self-tightening rotors described elsewhere herein. The rotors, rotor assemblies, or other propulsion systems of the UAV can enable the UAV to hover / hold position, change orientation, and / or change position. The distance between the axes of the opposing rotors can be any suitable length 1310. For example, the length 1310 can be less than or equal to 2 m, or less than or equal to 5 m. In some embodiments, the length 1310 can be in a range from 40 cm to 1 m, from 10 cm to 2 m, or from 5 cm to 5 m. Any description herein of a drone can apply to a movable object, e.g., a movable object of a different type, and vice versa.

[0104] In some embodiments, a movable object can be configured to carry a payload. The payload can include one or more of a passenger, cargo, equipment, instrumentation, etc. The payload can be disposed within a housing. The housing can be separate from a housing of the movable object, or can be part of a housing for the movable object. Alternatively, a payload having a housing can be provided in the absence of a housing for the movable object. Alternatively, part or all of the payload can be absent a housing. The payload can be rigidly fixed relative to the movable object. Alternatively, the payload can be movable relative to the movable object (e.g., translatable or rotatable relative to the movable object).

[0105] In some embodiments, the payload includes a payload. The payload can be configured to perform no operation or function. Alternatively, the payload can be a payload configured to perform an operation or function, also referred to as a functional payload. For example, the payload can include one or more sensors for measuring one or more targets. Any suitable sensor can be incorporated into the payload, such as an image capture device (e.g., a camera), an audio capture device (e.g., a parabolic microphone), an infrared imaging device, or an ultraviolet imaging device. The sensor can provide static sensing data (e.g., a photograph) or dynamic sensing data (e.g., a video). In some embodiments, the sensor provides sensing data of a target of the payload. Alternatively or in combination, the payload can include one or more emitters for providing a signal to one or more targets. Any suitable emitter can be used, such as an illumination source or an acoustic source. In some embodiments, the payload includes one or more transceivers, such as for communicating with a module remote from the movable object. Alternatively, the payload can be configured to interact with the environment or a target. For example, the payload can include a tool, instrument, or mechanism capable of manipulating an object, such as a robotic arm.

[0106] Alternatively, the payload can include a carrier. The carrier can be provided for the payload, and the payload can be connected to the movable object directly (e.g., directly contact the movable object) or indirectly (e.g., without contacting the movable object) through the carrier. Conversely, the payload can be mounted on the movable object without the need for a carrier. The payload can be integrally formed with the carrier. Alternatively, the payload can be releasably connected to the carrier. In some embodiments, the payload can include one or more payload elements, and as described above, the one or more payload elements can be movable relative to the movable object and / or the carrier.

[0107] The carrier can be integrally formed with the movable object. Alternatively, the carrier can be releasably connected to the movable object. The carrier can be directly or indirectly connected to the movable object. The carrier can provide support for the payload (e.g., carry at least a portion of the weight of the payload). The carrier can include a suitable mounting structure (e.g., a gimbal platform) capable of stabilizing and / or guiding movement of the payload. In some embodiments, the carrier can be adapted to control a state (e.g., a position and / or an orientation) of the payload relative to the movable object. For example, the carrier can be configured to move relative to the movable object (e.g., relative to one, two, or three translational degrees of freedom and / or one, two, or three rotational degrees of freedom) such that the payload maintains its position and / or orientation relative to a suitable reference frame regardless of the movement of the movable object. The reference frame can be a fixed reference frame (e.g., the surrounding environment). Alternatively, the reference frame can be a moving reference frame (e.g., the movable object, a target of the payload).

[0108] In some embodiments, the carrier can be configured to allow movement of the payload relative to the carrier and / or the movable object. This movement can be translation with respect to up to three degrees of freedom (e.g., along one, two, or three axes) or rotation with respect to up to three degrees of freedom (e.g., about one, two, or three axes), or any suitable combination thereof.

[0109] In some cases, the carrier can include a carrier frame assembly and a carrier actuation assembly. The carrier frame assembly can provide structural support for the payload. The carrier frame assembly can include individual carrier frame components, some of which can be movable relative to one another. The carrier actuation assembly can include one or more actuators (e.g., motors) that actuate movement of the individual carrier frame components. The actuators can allow movement of multiple carrier frame components simultaneously, or can be configured to allow movement of a single carrier frame component at a time. Movement of the carrier frame components can result in corresponding movement of the payload. For example, the carrier actuation assembly can actuate rotation of one or more carrier frame components about one or more rotational axes (e.g., a roll axis, a pitch axis, or a yaw axis). Rotation of the one or more carrier frame components can result in rotation of the payload about the one or more rotational axes relative to the movable object. Alternatively or in combination, the carrier actuation assembly can actuate translation of one or more carrier frame components along one or more translational axes, and thereby result in translation of the payload along one or more corresponding axes relative to the movable object.

[0110] In some embodiments, movement of the movable object, the carrier, and the payload relative to a fixed frame of reference (e.g., the surrounding environment) and / or one another can be controlled by a terminal. The terminal can be a remote control device at a location remote from the movable object, the carrier, and / or the payload. The terminal can be disposed on or fixed to a support platform. Alternatively, the terminal can be a handheld device or a wearable device. For example, the terminal can include a smartphone, a tablet, a laptop, a computer, glasses, a glove, a headset, a microphone, or a suitable combination thereof. The terminal can include a user interface, such as a keyboard, a mouse, a joystick, a touchscreen, or a display. Any suitable user input can be used to interact with the terminal, such as manual input of commands, voice control, gesture control, or position control (e.g., through movement, position, or tilt of the terminal).

[0111] The terminal can be used to control any suitable state of the movable object, the carrier, and / or the payload. For example, the terminal can be used to control position and / or orientation of the movable object, the carrier, and / or the payload relative to a fixed frame of reference from and / or to one another. In some embodiments, the terminal can be used to control individual elements of the movable object, the carrier, and / or the payload, such as an actuation assembly of the carrier, a sensor of the payload, or an emitter of the payload. The terminal can include a wireless communication device suitable for communicating with one or more of the movable object, the carrier, or the payload.

[0112] The terminal can include a suitable display unit for viewing information of the movable object, the carrier, and / or the payload. For example, the terminal can be configured to display information of the movable object, the carrier, and / or the payload regarding position, translational velocity, translational acceleration, orientation, angular velocity, angular acceleration, or any suitable combination thereof. In some embodiments, the terminal can display information provided by the payload, such as data provided by a functional payload (e.g., images recorded by a camera or other image capture device).

[0113] Optionally, the same terminal can both control the movable object, the carrier, and / or the payload, and receive and / or display information from the movable object, the carrier, and / or the payload. For example, the terminal can control positioning of the payload relative to the environment while displaying image data captured by the payload or information regarding the position of the payload. Alternatively, different terminals can be used for different functions. For example, a first terminal can control movement or state of the movable object, the carrier, and / or the payload, while a second terminal can receive and / or display information from the movable object, the carrier, and / or the payload. For example, a first terminal can be used to control positioning of the payload relative to the environment, while a second terminal displays image data captured by the payload. Various communication modes can be utilized between the movable object and an integrated terminal that both controls the movable object and receives data from the movable object, or between the movable object and multiple terminals that both control the movable object and receive data from the movable object. For example, at least two different communication modes can be established between the movable object and the terminal that both controls the movable object and receives data from the movable object.

[0114] Figure 14 A movable object 900 including a carrier 1402 and a payload 1404 is shown in accordance with an embodiment. As previously described, although the movable object 1400 is described as an aircraft, such description is not intended to be limiting, and any suitable type of movable object can be used. Those skilled in the art will appreciate that any embodiments described herein in the context of an aircraft system can be applied to any suitable movable object (e.g., a drone). In some cases, the payload 1404 can be provided on the movable object 1400 without the carrier 1402. The movable object 1400 can include a propulsion mechanism 1406, a sensing system 1408, and a communication system 1410.

[0115] As previously mentioned, propulsion mechanisms 1406 can include one or more of rotors, propellers, blades, engines, motors, wheels, shafts, magnets, or nozzles. For example, propulsion mechanisms 1406 can be self-tightening rotors, rotor assemblies, or other rotating propulsion units as disclosed elsewhere herein. A movable object can have one or more, two or more, three or more, or four or more propulsion mechanisms. The propulsion mechanisms can all be of the same type. Alternatively, one or more of the propulsion mechanisms can be different types of propulsion mechanisms. Propulsion mechanisms 1406 can be mounted on movable object 1400 using any suitable means, such as support elements (e.g., drive shafts) as described elsewhere herein. Propulsion mechanisms 1406 can be mounted on any suitable portion of movable object 1400, such as on its top, bottom, front, back, sides, or suitable combinations thereof.

[0116] In some embodiments, propulsion mechanisms 1406 can enable movable object 1400 to take off vertically from a surface or land vertically on a surface without requiring any horizontal movement of movable object 1400 (e.g., without traveling along a runway). Alternatively, propulsion mechanisms 1406 can be operable to allow movable object 1400 to hover in a particular position and / or orientation in the air. One or more propulsion mechanisms 1400 can be controlled independently of the other propulsion mechanisms. Alternatively, propulsion mechanisms 1400 can be configured to be controlled simultaneously. For example, movable object 1400 can have multiple horizontally oriented rotors that can provide lift and / or thrust to the movable object. The multiple horizontally oriented rotors can be actuated to provide vertical takeoff, vertical landing, and hovering capabilities to movable object 1400. In some embodiments, one or more of the horizontally oriented rotors can rotate in a clockwise direction while one or more of the horizontally oriented rotors can rotate in a counterclockwise direction. For example, the number of clockwise rotors can be equal to the number of counterclockwise rotors. To control the lift and / or thrust generated by each rotor, the rotational speed of each horizontally oriented rotor can be varied independently, thereby adjusting the spatial disposition, velocity, and / or acceleration of movable object 1400 (e.g., with respect to up to three translational degrees of freedom and up to three rotational degrees of freedom).

[0117] The sensing system 1408 can include one or more sensors that can sense spatial disposition, velocity, and / or acceleration of the movable object 1400 (e.g., with respect to up to three translational degrees of freedom and up to three rotational degrees of freedom). The one or more sensors can include a global positioning system (GPS) sensor, a motion sensor, an inertial sensor, a proximity sensor, or an image sensor. Sensing data provided by the sensing system 1408 can be used to control the spatial disposition, velocity, and / or orientation of the movable object 1400 (e.g., by using a suitable processing unit and / or control module, as described below). Optionally, the sensing system 1408 can be used to provide data regarding the environment surrounding the movable object, e.g., weather conditions, proximity to potential obstacles, location of geographical features, location of man-made structures, etc.

[0118] The communication system 1410 enables communication with the terminal 1412 having a communication system 1414 via wireless signals 1416. The communication systems 1410, 1414 can include any number of transmitters, receivers, and / or transceivers suitable for wireless communication. The communication can be one-way communication such that data can be transmitted in only one direction. For example, one-way communication can involve only the movable object 1400 transmitting data to the terminal 1412, or vice versa. Data can be transmitted from one or more transmitters of the communication system 1410 to one or more receivers of the communication system 1412, or vice versa. Alternatively, the communication can be two-way communication such that data can be transmitted in both directions between the movable object 1400 and the terminal 1412. Two-way communication can involve transmitting data from one or more transmitters of the communication system to one or more receivers of the communication system 1414, and vice versa.

[0119] In some embodiments, the terminal 1412 can provide control data to one or more of the movable object 1400, the carrier 1402, and the payload 1404, and receive information from one or more of the movable object 1400, the carrier 1402, and the payload 1404 (e.g., position and / or motion information of the movable object, carrier, or payload; data sensed by the payload, such as image data captured by a payload camera). In some cases, the control data from the terminal can include instructions for relative position, motion, actuation, or control of the movable object, carrier, and / or payload. For example, the control data can cause a change in position and / or orientation of the movable object (e.g., by controlling the propulsion mechanism 1406), or motion of the payload relative to the movable object (e.g., by controlling the carrier 1402). The control data from the terminal can cause control of the payload, such as control of the operation of a camera or other image capture device (e.g., taking still or moving pictures, zooming in or out, turning on or off, switching imaging modes, changing image resolution, changing focus, changing depth of field, changing exposure time, changing angle of view or field of view). In some cases, the communications from the movable object, carrier, and / or payload can include information from one or more sensors (e.g., of the sensing system 1408 or the payload 1404). The communications can include sensing information from one or more different types of sensors (e.g., GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). This information can relate to the position (e.g., location, orientation), motion, or acceleration of the movable object, carrier, and / or payload. This information from the payload can include data captured by the payload or sensed state of the payload. The control data provided by the terminal 1412 for transmission can be configured to control the state of one or more of the movable object 1400, the carrier 1402, or the payload 1404. Alternatively or in combination, the carrier 1402 and the payload 1404 can each also include a communication module configured to communicate with the terminal 1412, such that the terminal can independently communicate with and control each of the movable object 1400, the carrier 1402, and the payload 1404.

[0120] In some embodiments, the movable object 1400 can be configured to communicate with another remote device other than or in addition to the terminal 1412. The terminal 1412 can also be configured to communicate with another remote device as well as the movable object 1400. For example, the movable object 1400 and / or the terminal 1412 can communicate with another movable object or a carrier or payload of another movable object. When desired, the remote device can be a second terminal or other computing device (e.g., a computer, a laptop, a tablet, a smartphone, or other mobile device). The remote device can be configured to send data to the movable object 1400, receive data from the movable object 1400, send data to the terminal 1412, and / or receive data from the terminal 1412. Optionally, the remote device can connect to the Internet or other telecommunications network, such that data received from the movable object 1400 and / or the terminal 1412 can be uploaded to a website or server.

[0121] The systems, devices, and methods described herein can be applied to a wide variety of movable objects. As previously mentioned, any description herein with respect to a drone can apply to any movable object. The movable objects of the present disclosure can be configured to move in any suitable environment, such as in the air (e.g., a fixed-wing aircraft, a rotary-wing aircraft, or an aircraft with neither fixed wings nor rotary wings), in water (e.g., a boat or a submarine), on the ground (e.g., a motorized vehicle, such as a car, a truck, a bus, a van, a motorcycle; a movable structure or frame, such as a stick, a fishing rod; or a train), under the ground (e.g., a subway), in space (e.g., a space vehicle, a satellite, or a probe), or any combination of these environments. The movable object can be a vehicle, such as the vehicles described elsewhere herein. The movable object can be a self-propelled unmanned vehicle that does not require human input. In some embodiments, the movable object can be mounted to a living being, such as a human or an animal. Suitable animals can include a cow, a dog, a cat, a horse, a cow, a sheep, a pig, a flea, a rodent, or an insect. In some embodiments, the movable object can be carried.

[0122] The movable object can be free to move in an environment with respect to six degrees of freedom, such as three degrees of freedom in translation and three degrees of freedom in rotation. Alternatively, the movement of the movable object can be constrained with respect to one or more degrees of freedom, such as by a predetermined path, trajectory, or direction. The movement can be actuated by any suitable actuation mechanism, such as an engine or a motor. The actuation mechanism of the movable object can be powered by any suitable energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. As described elsewhere herein, the movable object can be self-propelled by a propulsion system. The propulsion system can optionally use, for example, electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, the movable object can be carried by a living being.

[0123] In some instances, the movable object can be a vehicle. Suitable vehicles can include water-borne vehicles, aerial vehicles, space vehicles, or ground vehicles. For example, an aerial vehicle can be a fixed-wing aircraft (e.g., an airplane, a glider), a rotary-wing aircraft (e.g., a helicopter, a gyrocopter), an aircraft having both fixed- and rotary-wing, or an aircraft having neither fixed- nor rotary-wing (e.g., a dirigible, a hot air balloon). A vehicle can be self-propelled, for example, in air, on or in water, in space, or on or under a ground surface. A self-propelled vehicle can utilize a propulsion system, for example, including one or more engines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any suitable combination thereof. In some instances, a propulsion system can be used to enable the movable object to take off from a surface, land on a surface, maintain its current position and / or orientation (e.g., hover), change direction, and / or change location.

[0124] A movable object can be remotely controlled by a user or locally controlled by an occupant within or on the movable object. In some embodiments, the movable object is an unmanned movable object, for example, a drone. An unmanned movable object, for example, a drone, can not have an occupant on the movable object. A movable object can be controlled by a human or an autonomous control system (e.g., a computer control system) or any suitable combination thereof. A movable object can be an autonomous or semi-autonomous robot, for example, a robot configured with artificial intelligence.

[0125] Figure 15 A block diagram schematic of a system 1500 for controlling a movable object according to embodiments is shown. The system 1500 can be used in combination with any suitable embodiment of the systems, devices, and methods disclosed herein. The system 1500 can include a sensing module 1502, a processing unit 1504, a non-transitory computer-readable medium 1506, a control module 1508, and a communication module 1510.

[0126] The sensing module 1502 can utilize different types of sensors that collect movable object related information in different ways. Different types of sensors can sense different types of signals or signals from different sources. For example, a sensor can include an inertial sensor, a GPS sensor, a proximity sensor (e.g., a lidar), or a vision / image sensor (e.g., a camera). The sensing module 1502 can be operatively coupled to the processing unit 1504 having multiple processors. In some embodiments, the sensing module can be operatively coupled to a transmission module 1512 (e.g., a Wi-Fi image transmission module) configured to directly transmit sensing data to a suitable external device or system. For example, the transmission module 1512 can be used to transmit images captured by a camera of the sensing module 1502 to a remote terminal.

[0127] The processing unit 1504 can have one or more processors, such as programmable processors (e.g., central processing units (CPUs)). The processing unit 1504 can be operatively coupled to a non-transitory computer readable medium 1506. The non-transitory computer readable medium 1506 can store logic, code, and / or program instructions executable by the processing unit 1504 for performing one or more steps. The non-transitory computer readable medium can include one or more memory units (e.g., removable media such as an SD card or external memory such as random access memory (RAM)). In some embodiments, data from the sensing module 1502 can be directly transferred and stored within the memory unit of the non-transitory computer readable medium 1506. The memory unit of the non-transitory computer readable medium 1506 can store logic, code, and / or program instructions executable by the processing unit 1504 to perform any suitable embodiment of the methods described herein. For example, the processing unit 1504 can be configured to execute instructions that cause one or more processors of the processing unit 1504 to analyze sensing data generated by the sensing module. The memory unit can store sensing data from the sensing module to be processed by the processing unit 1504. In some embodiments, the memory unit of the non-transitory computer readable medium 1506 can be used to store processing results generated by the processing unit 1504.

[0128] In some embodiments, the processing unit 1504 can be operatively coupled to a control module 1508 configured to control a state of the movable object. For example, the control module 1508 can be configured to control a propulsion mechanism of the movable object to adjust a spatial disposition, velocity, and / or acceleration of the movable object with respect to six degrees of freedom. Alternatively or in combination, the control module 1508 can control one or more states of the carrier, payload, or sensing module.

[0129] The processing unit 1504 can be operatively coupled to a communication module 1510 configured to transmit and / or receive data from one or more external devices (e.g., a terminal, a display device, or other remote controller). Any suitable communication means can be used, such as wired or wireless communication. For example, the communication module 1510 can utilize one or more local area networks (LANs), wide area networks (WANs), infrared, radio, Wi-Fi, peer-to-peer (P2P) networks, telecommunication networks, cloud communication, etc. Optionally, a relay station such as a tower, satellite, or mobile station can be used. The wireless communication can or can not rely on close proximity. In some embodiments, the communication can or can not require line-of-sight. The communication module 1510 can transmit and / or receive one or more of sensing data from the sensing module 1502, processing results generated by the processing unit 1504, predetermined control data, user commands from a terminal or remote controller, etc.

[0130] The components of system 1500 can be arranged in any suitable configuration. For example, one or more components of system 1500 can be located on a movable object, a carrier, a piggybacked object, a terminal, a sensing system, or an additional external device in communication with one or more of the aforementioned. Additionally, although a single processing unit 1504 and a single non-transitory computer-readable medium 1506 are depicted, one of skill in the art will appreciate that this is not intended to be limiting, and that system 1500 can include multiple processing units and / or non-transitory computer-readable media. In some embodiments, one or more of the multiple processing units and / or non-transitory computer-readable media can be located in different locations, such as on a movable object, a carrier, a piggybacked object, a terminal, a sensing module, an additional external device in communication with one or more of the aforementioned, or a suitable combination thereof, such that any suitable aspect of the processing and / or storage functions performed by system 1500 can occur at one or more of the aforementioned locations. Figure 15 Although a single processing unit 1504 and a single non-transitory computer-readable medium 1506 are depicted, one of skill in the art will appreciate that this is not intended to be limiting, and that system 1500 can include multiple processing units and / or non-transitory computer-readable media. In some embodiments, one or more of the multiple processing units and / or non-transitory computer-readable media can be located in different locations, such as on a movable object, a carrier, a piggybacked object, a terminal, a sensing module, an additional external device in communication with one or more of the aforementioned, or a suitable combination thereof, such that any suitable aspect of the processing and / or storage functions performed by system 1500 can occur at one or more of the aforementioned locations.

[0131] As used herein, A and / or B includes one or more of A or B, and combinations thereof, e.g., A and B.

[0132] While preferred embodiments of the application have been shown and described herein, it will be apparent to those skilled in the art that many changes, modifications, and alternatives can be made thereto without departing from the application. It is to be understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application. It is intended to cover in the appended claims all such changes and modifications that fall within the scope of the application. The claims should not be read to pose a limitation on the scope of the application unlesssuch limitation is clearly recited in the claims.

Claims

1. A method of determining a flight path for an aerial vehicle, comprising: identifying, during a flight and by way of one or more processors, a loss or degradation of a signal transmitted between the aerial vehicle and a remote device occurring at a first location; in response to said identifying, selecting a first destination within a range proximate a second location, wherein a signal transmission status at the second location during a previous flight is different than a signal transmission status between the aerial vehicle and the remote device at the first location; determining a first flight path to the first destination, wherein the first flight path includes reachable locations detected by one or more sensors on one or more aerial vehicles, said sensors including one or more of: a location sensor, a vision sensor, a proximity or range sensor, an inertial sensor, an altitude sensor, an attitude sensor, a pressure sensor, an audio sensor, a field sensor; determining a second flight path to a second destination when the signal transmission status at the first destination returns to a normal status at the first destination, the second flight path including locations not detected by any of said one or more sensors on said one or more aerial vehicles during the previous flight; evaluating the signal transmission status at the first destination in real time when the aerial vehicle reaches the first destination.

2. The method of claim 1, wherein, The first destination is at the second location.

3. The method of claim 1, wherein, The remote device is a remote controller or a satellite.

4. The method of claim 1, wherein, The second location is a location detected by said one or more sensors on said one or more aerial vehicles during the previous flight.

5. The method of claim 1, wherein, The second location is the last point of successful signal transmission between the aerial vehicle and the remote device during the previous flight.

6. The method of claim 1, wherein, The first flight path is a reverse of a last flight path for the aerial vehicle.

7. The method of claim 1, wherein, The first flight path does not include locations not detected by said one or more sensors during the previous flight.

8. The method of claim 1, wherein, The second destination is a location not detected by said one or more sensors during the previous flight; or, the second destination is a predetermined location reached by the aerial vehicle prior to identifying the change in signal transmission status occurring at the first location.

9. The method of claim 1, wherein, The second destination is a starting point of the current flight.

10. The method of claim 1, wherein, Selecting the second flight path is further dependent on a user input from the remote device.

Images