Multi-sensor environment map construction

Abstract

Systems and methods for controlling a movable object within an environment are provided. In one aspect, a method can include determining an initial position of the movable object using at least one sensor of a plurality of sensors carried by the movable object; generating a first signal to cause the movable object to navigate within the environment; receiving, using the at least one sensor of the plurality of sensors, sensory data regarding the environment; generating, based on the sensory data, an environmental map representing at least a portion of the environment; receiving an instruction to return to the initial position; and generating, based on the environmental map, a second signal to cause the movable object to return to the initial position.

Description

[0001] This application is a divisional application of a patent application with application number 201811161897.0, filed on September 30, 2018, and titled "Multi-sensor environmental map construction," and the divisional application with application number 201811161897.0 is a divisional application of a patent application with application number 201480031173.8, filed on September 5, 2014, and titled "A system and method for controlling movable objects in an environment." Background Art

[0002] Unmanned vehicles, such as unmanned aerial vehicles (UAVs), can be used to perform surveillance, reconnaissance, and exploration missions in a variety of environments for both military and civilian applications. UAVs can be manually controlled by a remote user or can operate in a semi-autonomous or fully autonomous manner. Such UAVs can include sensors configured to collect data from their surroundings.

[0003] In some cases, existing methods for acquiring environmental data may not be optimal. For example, the accuracy of environmental data may be limited based on the capabilities of the specific sensor type used to collect the data. Inaccurate environmental data may adversely affect UAV functionality. Summary of the Invention

[0004] Embodiments disclosed herein provide improved methods for controlling movable objects such as UAVs within an environment. In many embodiments, the UAV includes a variety of different sensor types for collecting information about the surrounding environment. The data obtained from each of the different types of sensors can be combined to generate a representation of the surrounding environment, such as a two-dimensional (2D) or three-dimensional (3D) environmental map, and the data can also be used to facilitate navigation, object recognition, and obstacle avoidance. Advantageously, the methods described herein can be used to improve UAV functionality under diverse environmental types and operating conditions.

[0005] Thus, in one aspect, a method for controlling a movable object within an environment is provided. The method includes: determining an initial position of the movable object using at least one sensor from a plurality of sensors carried by the movable object; generating a first signal to cause the movable object to navigate within the environment; receiving sensory data about the environment using the at least one sensor from the plurality of sensors; generating an environment map representing at least a portion of the environment based on the sensory data; receiving an instruction to return to the initial position; and generating a second signal to cause the movable object to return to the initial position based on the environment map.

[0006] In some embodiments, the movable object is an unmanned aerial vehicle (UAV). The UAV may weigh no more than 10 kg. The maximum dimension of the UAV may be no more than 1.5 m. The UAV may be configured to fly at an altitude of no more than 400 m. Optionally, the UAV may be configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. The restricted flight area may be an airport. The UAV may be a multirotor aircraft.

[0007] In some embodiments, the environment may be an indoor environment or a low-altitude environment.

[0008] In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a visual sensor, or a distance sensor. The distance sensor may include at least one of a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor. The plurality of sensors may include a variety of different sensor types.

[0009] In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following: a point cloud, a 3D grid map, a 2D grid map, or a 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

[0010] In some embodiments, generating the second signal includes: determining the current position of the movable object using at least one of the plurality of sensors; determining a path from the current position to the initial position based on the environment map; and generating a signal to cause the movable object to move along the path to return to the initial position. Determining the path may include determining the shortest path from the current position to the initial position. Alternatively or in combination, determining the path may include determining a path from the current position to the initial position that avoids one or more obstacles within the environment. The path may include one or more portions previously traveled by the movable object. The path may be different from a path previously traveled by the movable object. Optionally, the path may be a flight path of the movable object. The path may include the spatial position and direction of the movable object.

[0011] In some embodiments, the path includes a plurality of waypoints corresponding to locations previously traveled by the movable object, the plurality of waypoints being recorded as the movable object navigates within the environment. The plurality of waypoints may be recorded in real time as the movable object navigates within the environment. Alternatively, the plurality of waypoints may be recorded at predetermined time intervals as the movable object navigates within the environment. The plurality of waypoints may be stored in a list data structure.

[0012] In some embodiments, generating the signal to cause the movable object to move along the path may include: detecting an obstacle in the environment located along the path; modifying the path to avoid the obstacle; and generating the signal to cause the movable object to move along the modified path.

[0013] In another aspect, a system for controlling a movable object within an environment is provided. The system may include: a plurality of sensors carried by the movable object and one or more processors. The one or more processors may be configured, individually or collectively, to: determine an initial position of the movable object using at least one of the plurality of sensors; generate a first signal that causes the movable object to navigate within the environment; receive sensory data about the environment using the at least one of the plurality of sensors; generate an environment map representing at least a portion of the environment based on the sensory data; receive an instruction to return to the initial position; and generate a second signal based on the environment map that causes the movable object to navigate to return to the initial position.

[0014] In another aspect, a method for controlling an unmanned aerial vehicle (UAV) within an environment is provided. The method includes generating a first signal to cause the UAV to navigate within the environment; using a plurality of sensors carried by the UAV to receive sensory data regarding at least a portion of the environment; generating an environment map representing the at least a portion of the environment based on the sensory data; using the environment map to detect one or more obstacles located in the portion of the environment; and using the environment map to generate a second signal to cause the UAV to navigate to avoid the one or more obstacles.

[0015] In some embodiments, the UAV is a rotorcraft. The UAV may weigh no more than 10 kg. The maximum dimension of the UAV may be no more than 1.5 m. The UAV may be configured to fly at an altitude of no more than 400 m. Optionally, the UAV may be configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. The restricted flight area may be an airport.

[0016] In some embodiments, the environment may be an indoor environment or a low-altitude environment.

[0017] In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a visual sensor, or a distance sensor. The distance sensor may include at least one of a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor. The plurality of sensors may include a plurality of different sensor types. The sensed data may include data relative to a plurality of different coordinate systems, and generating the environment map includes mapping the data onto a single coordinate system.

[0018] In some embodiments, the environment map may include a topological map or a scale map. The scale map may include at least one of the following: a point cloud, a 3D grid map, a 2D grid map, or a 2.5D grid map. Optionally, the scale map may include an occupancy grid map.

[0019] In some embodiments, the first signal is generated based on an instruction received from a remote terminal in communication with the UAV. The instruction may be input into the remote terminal by a user. Alternatively, the first signal may be generated autonomously by the UAV.

[0020] In some embodiments, the sensory data includes data relative to a plurality of different coordinate systems, and generating the environmental map includes mapping the data onto a single coordinate system. Generating the first signal may include generating a flight path for the UAV, and generating the second signal may include modifying the flight path based on the environmental map to avoid the one or more obstacles. The flight path may be configured to guide the UAV from a current location to a previous location.

[0021] In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system includes a plurality of sensors carried by the unmanned aerial vehicle and one or more processors. The one or more processors can be configured, individually or collectively, to: generate a first signal to cause the unmanned aerial vehicle to navigate within the environment; receive sensory data regarding at least a portion of the environment using the plurality of sensors carried by the unmanned aerial vehicle; generate an environmental map representing the at least a portion of the environment based on the sensory data; detect one or more obstacles located in the portion of the environment using the environmental map; and generate a second signal to cause the unmanned aerial vehicle to navigate to avoid the one or more obstacles using the environmental map.

[0022] In another aspect, a method for controlling an unmanned aerial vehicle within an environment is provided. The method includes: receiving a first sensing signal about the environment from a first sensor and receiving a second sensing signal about the environment from a second sensor, wherein the first sensor and the second sensor are of different sensor types and wherein the first sensor and the second sensor are carried by the unmanned aerial vehicle; generating a first environment map using the first sensing signal and generating a second environment map using the second sensing signal, wherein the first environment map and the second environment map each include obstacle occupancy information of the environment; and combining the first environment map and the second environment map to generate a final environment map including the obstacle occupancy information of the environment.

[0023] In some embodiments, the method further includes generating a signal to cause the UAV to navigate within the environment based at least in part on obstacle occupancy information in the final map of the environment.

[0024] In some embodiments, the UAV is a rotorcraft. The UAV may weigh no more than 10 kg. The maximum dimension of the UAV may be no more than 1.5 m. The UAV may be configured to fly at an altitude of no more than 400 m. Optionally, the UAV may be configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. The restricted flight area may be an airport.

[0025] In some embodiments, the environment may be an indoor environment or a low-altitude environment.

[0026] In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a visual sensor, or a distance sensor. The distance sensor may include at least one of a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor. The plurality of sensors may include a variety of different sensor types.

[0027] In some embodiments, the environment map may include a topological map or a scale map. The scale map may include at least one of the following: a point cloud, a 3D grid map, a 2D grid map, or a 2.5D grid map. Optionally, the scale map may include an occupancy grid map.

[0028] In some embodiments, the first environment map is provided relative to a first coordinate system and the second environment map is provided relative to a second coordinate system different from the first coordinate system. The first coordinate system may be a global coordinate system and the second coordinate system may be a local coordinate system. Combining the first and second environment maps may include converting the first and second environment maps to a single coordinate system.

[0029] In some embodiments, a sensing range of the first sensor is different from a sensing range of the second sensor.

[0030] In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system includes: a first sensor carried by the unmanned aerial vehicle and configured to generate a first sensing signal about the environment; a second sensor carried by the unmanned aerial vehicle and configured to generate a second sensing signal about the environment, the second sensor being of a different sensor type than the first sensor; and one or more processors. The one or more processors can be configured individually or collectively to: receive the first sensing signal and the second sensing signal; use the first sensing signal to generate a first environment map and use the second sensing signal to generate a second environment map, each of the first environment map and the second environment map containing obstacle occupancy information of the environment; and combine the first environment map and the second environment map to generate a final environment map containing obstacle occupancy information of the environment.

[0031] In another aspect, a method for controlling an unmanned aerial vehicle within an environment is provided. The method includes determining an initial position of the unmanned aerial vehicle using at least one sensor from a plurality of sensors carried by the unmanned aerial vehicle; generating a first signal to cause the unmanned aerial vehicle to navigate within the environment; receiving sensory data about the environment using the at least one sensor from the plurality of sensors; receiving an instruction to return to the initial position; and generating a second signal to cause the unmanned aerial vehicle to return to the initial position along a path, wherein when the at least one sensor from the plurality of sensors detects an obstacle in the environment along the path, the path is modified to avoid the obstacle.

[0032] In some embodiments, the UAV is a rotorcraft. The UAV may weigh no more than 10 kg. The maximum dimension of the UAV may be no more than 1.5 m. The UAV may be configured to fly at an altitude of no more than 400 m. Optionally, the UAV may be configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. The restricted flight area may be an airport.

[0033] In some embodiments, the environment may be an indoor environment or a low-altitude environment.

[0034] In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a visual sensor, or a distance sensor. The distance sensor may include at least one of a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor. The plurality of sensors may include a variety of different sensor types.

[0035] In some embodiments, the environment map may include a topological map or a scale map. The scale map may include at least one of the following: a point cloud, a 3D grid map, a 2D grid map, or a 2.5D grid map. Optionally, the scale map may include an occupancy grid map.

[0036] In some embodiments, the path includes one or more portions previously traveled by the movable object. The path may be different from a path previously traveled by the movable object. Alternatively, the path may be a flight path of the movable object. The path may include the spatial position and orientation of the movable object.

[0037] In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system may include: a plurality of sensors carried by the movable object and one or more processors. The one or more processors may be configured, individually or collectively, to: determine an initial position of the movable object using at least one of the plurality of sensors carried by the unmanned aerial vehicle; generate a first signal to cause the unmanned aerial vehicle to navigate within the environment; receive sensed data about the environment using the at least one of the plurality of sensors; receive an instruction to return to the initial position; and generate a second signal to cause the unmanned aerial vehicle to return to the initial position along a path, wherein when the at least one of the plurality of sensors detects an obstacle in the environment located along the path, the path is modified to avoid the obstacle.

[0038] In another aspect, a method for generating a map of an environment is provided. The method includes: receiving first sensory data from one or more visual sensors carried by an unmanned aerial vehicle, the first sensory data including depth information of the environment; receiving second sensory data from one or more range sensors carried by the unmanned aerial vehicle, the second sensory data including depth information of the environment; and generating an environment map including the depth information of the environment using the first sensory data and the second sensory data.

[0039] In some embodiments, the first and second sensory data each include at least one image having a plurality of pixels, each of the plurality of pixels being associated with a two-dimensional image coordinate and a depth value. Each of the plurality of pixels may be associated with a color value. The first and second sensory data may each include contour information of one or more objects in the environment.

[0040] In some embodiments, the method further includes generating a signal to enable the UAV to navigate within the environment based at least in part on the depth information in the map of the environment.

[0041] In some embodiments, the UAV is a rotorcraft. The UAV may weigh no more than 10 kg. The maximum dimension of the UAV may be no more than 1.5 m. The UAV may be configured to fly at an altitude of no more than 400 m. Optionally, the UAV may be configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. The restricted flight area may be an airport.

[0042] In some embodiments, the environment may be an indoor environment or a low-altitude environment.

[0043] In some embodiments, the one or more visual sensors include only one camera. Alternatively, the one or more visual sensors may include two or more cameras. The one or more distance sensors may include at least one ultrasonic sensor or at least one lidar sensor. The first sensing data may include a first set of depth images and the second sensing data may include a second set of depth images. Generating the environment map may include: identifying a first plurality of feature points present in the first set of depth images; identifying a second plurality of feature points present in the second set of depth images, each feature point in the second plurality of feature points corresponding to a feature point in the first plurality of feature points; determining a correspondence between the first plurality of feature points and the second plurality of feature points; and generating the environment map by combining the first set of depth images and the second set of depth images based on the correspondence.

[0044] In some embodiments, the first sensory data is provided relative to a first coordinate system and the second sensory data is provided relative to a second coordinate system different from the first coordinate system. Generating the environment map may include representing the first sensory data and the second sensory data relative to a third coordinate system. The third coordinate system may be the first coordinate system or the second coordinate system. Alternatively, the third coordinate system may be different from the first coordinate system and the second coordinate system.

[0045] In some embodiments, the environment map may include a topological map or a scale map. The scale map may include at least one of the following: a point cloud, a 3D grid map, a 2D grid map, or a 2.5D grid map. Optionally, the scale map may include an occupancy grid map.

[0046] In another aspect, a system for generating a map of an environment is provided. The system may include: one or more visual sensors carried by an unmanned aerial vehicle and configured to generate first sensory data containing depth information of the environment; one or more range sensors carried by the unmanned aerial vehicle and configured to generate second sensory data containing depth information of the environment; and one or more processors. The one or more processors may be configured individually or collectively to receive the first sensory data and the second sensory data; and use the first sensory data and the second sensory data to generate a map of the environment containing depth information of the environment.

[0047] It should be understood that the different aspects of the present invention can be understood individually, collectively, or in combination with each other. The various aspects of the invention described herein may be applicable to any specific application set forth below or to any other type of movable object. Any description of an aircraft herein may be applicable to and used for any movable object, such as any vehicle. In addition, the systems, devices, and methods disclosed herein in the context of aerial motion (e.g., flight) may also be applicable to other types of motion, such as movement on the ground or on water, underwater motion, or motion in space. In addition, any description of a rotor or rotor assembly herein may be applicable to and used for any propulsion system, device, or mechanism (e.g., propeller, wheel, axle) configured to generate propulsion by rotation.

[0048] Other objects and features of the present invention will become apparent from an examination of the specification, claims and drawings.

[0049] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The novel features of the present invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description and accompanying drawings which illustrate illustrative embodiments in which the principles of the invention are utilized; in which:

[0051] Figure 1A illustrates a UAV operating in an outdoor environment according to an embodiment;

[0052] Figure 1B illustrates a UAV operating in an indoor environment according to an embodiment;

[0053] Figure 2A illustrates a scheme for estimating UAV pose using sensor fusion according to an embodiment;

[0054] Figure 2B illustrates a scheme for estimating UAV position and velocity using sensor fusion, according to an embodiment;

[0055] Figure 3A illustrates a scheme for using sensor fusion to build an environment map according to an embodiment;

[0056] Figure 3B illustrates a method for generating an environment map using sensor fusion according to an embodiment;

[0057] Figure 4 illustrates a method for using different sensor types to construct an environment map according to an embodiment;

[0058] Figure 5 A method for controlling a UAV to avoid obstacles according to an embodiment is illustrated;

[0059] Figure 6 A method for controlling a UAV to return to an initial position according to an embodiment is illustrated;

[0060] Figure 7 A method for controlling a UAV to return to an initial position while avoiding obstacles according to an embodiment is illustrated;

[0061] Figure 8A and Figure 8B An algorithm for controlling a UAV to return to an initial position using waypoints according to an embodiment is illustrated;

[0062] Figure 9A and Figure 9B An algorithm for controlling a UAV to return to a target location using a topological map according to an embodiment is illustrated;

[0063] Figure 10 A UAV according to an embodiment is illustrated;

[0064] Figure 11 illustrates a movable object including a carrier and a payload according to an embodiment; and

[0065] Figure 12 A system for controlling a movable object according to an embodiment is illustrated. DETAILED DESCRIPTION

[0066] The present disclosure provides systems and methods for controlling a movable object, such as an unmanned aerial vehicle (UAV). In some embodiments, the UAV may be adapted to carry multiple sensors configured to collect environmental data. Some of these sensors may be of different types (e.g., a visual sensor used in combination with a range sensor). The data acquired by the multiple sensors may be combined to generate an environmental map representing the surrounding environment. In some embodiments, the environmental map may include information about the location of objects (such as objects or obstacles) in the environment. The UAV may use the generated map to perform various operations, some of which may be semi-automated or fully automated. For example, in some embodiments, the environmental map may be used to automatically determine a flight path for the UAV to navigate from its current location to a target location. As another example, the environmental map may be used to determine the spatial arrangement of one or more obstacles, thereby enabling the UAV to perform obstacle avoidance maneuvers. Advantageously, the use of multiple sensor types for collecting environmental data, as disclosed herein, can improve the accuracy of environmental map construction, even in diverse environments and operating conditions, thereby enhancing the robustness and flexibility of UAV functionality such as navigation and obstacle avoidance.

[0067] The embodiments provided herein are applicable to various types of UAVs. For example, the UAV can be a small UAV that weighs no more than 10 kg and / or has a maximum dimension of no more than 1.5 m. In some embodiments, the UAV can be a rotorcraft, such as a multi-rotor aircraft (e.g., a quadcopter) that is propelled by multiple propellers to move in the air. Additional examples of UAVs and other movable objects suitable for use with the embodiments described herein are further described below.

[0068] The UAV described herein can be operated fully autonomously (e.g., by a suitable computing system such as an onboard controller), semi-autonomously, or manually (e.g., by a human user). The UAV can receive commands from a suitable entity (e.g., a human user or an autonomous control system) and respond to such commands by performing one or more actions. For example, the UAV can be controlled to take off from the ground, move in the air (e.g., with up to three translational degrees of freedom and up to three rotational degrees of freedom), move to a target position or a series of target positions, hover in the air, land on the ground, etc. For another example, the UAV can be controlled to move at a specified speed and / or acceleration (e.g., with up to three translational degrees of freedom and up to three rotational degrees of freedom) or along a specified movement path. In addition, the commands can be used to control one or more UAV components, such as components described herein (e.g., sensors, actuators, propulsion units, loads, etc.). For example, some commands can be used to control the position, direction, and / or operation of UAV loads such as cameras. Alternatively, the UAV can be configured to operate according to one or more predetermined operating rules. The operating rules can be used to control any suitable aspect of the UAV, such as the position (e.g., latitude, longitude, altitude), direction (e.g., roll, pitch, yaw), speed (e.g., translation and / or angle), and / or acceleration (e.g., translation and / or angle) of the UAV. For example, the operating rules can be designed so that the UAV is not allowed to fly beyond a threshold altitude, for example, the UAV can be configured to fly at an altitude of no more than 400 meters above the ground. In some embodiments, the operating rules can be adapted to provide an automated mechanism for improving the safety of the UAV and preventing safety incidents. For example, the UAV can be configured to detect a restricted flight area (e.g., an airport) and not fly within a predetermined distance of the restricted flight area, thereby avoiding potential collisions with aircraft and other obstacles.

[0069] Turning now to the accompanying drawings, Figure 1A The diagram illustrates a UAV 102 operating in an outdoor environment 100, according to an embodiment. The outdoor environment 100 can be an urban, suburban, or rural environment, or any other environment that is not at least partially located within a building. The UAV 102 can operate relatively close to the ground 104 (e.g., at low altitude) or relatively far from the ground 104 (e.g., at high altitude). For example, a UAV 102 operating at less than or equal to about 10 meters above the ground can be considered to be at low altitude, while a UAV 102 operating at greater than or equal to about 10 meters above the ground can be considered to be at high altitude.

[0070] In some embodiments, the outdoor environment 100 includes one or more obstacles 108a, 108d. Obstacles can include any object or entity that can hinder the movement of the UAV 102. Some obstacles (e.g., obstacles 108a, 108d) may be located on the ground 104, such as buildings, ground vehicles (e.g., cars, motorcycles, trucks, bicycles), humans, animals, plants (e.g., trees, bushes), and other man-made or natural structures. Some obstacles may be in contact with and / or supported by the ground 104, water, man-made structures, or natural structures. Alternatively, some obstacles (e.g., obstacles 108b, 108c) may be completely located in the air 106, including aircraft (e.g., airplanes, helicopters, hot air balloons, other UAVs) or birds. Aerial obstacles may not be supported by the ground 104, by water, or by any natural or man-made structures. Obstacles located on the ground 104 may include portions that extend significantly into the air 106 (eg, tall structures such as towers, skyscrapers, lampposts, radio towers, power lines, trees, etc.).

[0071] Figure 1B A UAV 152 is illustrated operating in an indoor environment 150 according to an embodiment. The indoor environment 150 is located in the interior of a building 154 having a floor 156, one or more walls 158, and / or a ceiling or roof 160. Exemplary buildings include residential, commercial, or industrial buildings, such as houses, apartments, office buildings, manufacturing facilities, storage facilities, and the like. The interior of the building 154 may be completely enclosed by the floor 156, walls 158, and ceiling 160, such that the UAV 152 is constrained to the indoor space. Conversely, at least one of the floor 156, walls 158, or ceiling 160 may not be present, thereby enabling the UAV 152 to fly from the inside to the outside, or vice versa. Alternatively, or in combination, one or more openings 164 (e.g., doors, windows, skylights) may be formed in the floor 156, walls 158, or ceiling 160.

[0072] Similar to the outdoor environment 100, the indoor environment 150 may include one or more obstacles 162a-162d. Some obstacles may be located on the floor 156 (e.g., obstacle 162a), such as furniture, appliances, humans, animals, plants, or other man-made or natural objects. Conversely, some obstacles may be located in the air (e.g., obstacle 162b), such as birds or other UAVs. Some obstacles in the indoor environment 150 may be supported by other structures or objects. Obstacles may also be attached to the ceiling 160 (e.g., obstacle 162c), such as light fixtures, ceiling fans, roof beams, or other ceiling-mounted appliances or structures. In some embodiments, obstacles may be attached to the walls 158 (e.g., obstacle 162d), such as light fixtures, shelves, cabinets, or other wall-mounted appliances or structures. It is worth noting that the structural components of the building 154, including the floor 156, walls 158, and ceiling 160, may also be considered obstacles.

[0073] The obstacles described herein may be substantially stationary (e.g., buildings, plants, structures) or substantially mobile (e.g., humans, animals, vehicles, or other objects capable of movement). Some obstacles may include a combination of stationary and mobile components (e.g., a windmill). Mobile obstacles or obstacle assemblies may move according to a predetermined or predictable path or pattern. For example, the movement of a car may be relatively predictable (e.g., based on the shape of the road). Alternatively, some mobile obstacles or obstacle assemblies may move along random or otherwise unpredictable trajectories. For example, organisms such as animals may move in a relatively unpredictable manner.

[0074] To ensure safe and efficient operation, it may be beneficial to provide UAVs with mechanisms for detecting and distinguishing environmental objects, such as obstacles. Furthermore, the recognition of environmental objects, such as landmarks and features, can facilitate navigation, particularly when the UAV is operating in a semi-autonomous or fully autonomous manner. Furthermore, knowledge of the UAV's precise location within the environment and its spatial relationship to surrounding environmental objects can be valuable for a variety of UAV functionalities.

[0075] Thus, the UAV described herein may include one or more sensors configured to collect relevant data, such as information about the state of the UAV, the surrounding environment, or objects within the environment. Exemplary sensors suitable for use in the embodiments disclosed herein include position sensors (e.g., global positioning system (GPS) sensors, mobile device transmitters that support position triangulation), visual sensors (e.g., imaging devices capable of detecting visible light, infrared light, or ultraviolet light, such as cameras), distance or range sensors (e.g., ultrasonic sensors, lidars, time-of-flight cameras), inertial sensors (e.g., accelerometers, gyroscopes, inertial measurement units (IMUs)), altitude sensors, attitude sensors (compasses), pressure sensors (e.g., barometers), audio sensors (e.g., microphones), or field sensors (e.g., magnetometers, electromagnetic sensors). Any suitable number and combination of sensors may be used, such as one, two, three, four, five, or more sensors. Alternatively, data may be received from sensors of different types (e.g., two, three, four, five, or more types). Different types of sensors may measure different types of signals or information (e.g., position, orientation, velocity, acceleration, distance, pressure, etc.) and / or utilize different types of measurement techniques to acquire data. For example, sensors may include any suitable combination of active sensors (e.g., sensors that generate and measure energy from their own energy sources) and passive sensors (e.g., sensors that detect available energy). For another example, some sensors may generate absolute measurements provided in accordance with a global coordinate system (e.g., position data provided by a GPS sensor, attitude data provided by a compass or magnetometer), while other sensors may generate relative measurements provided in accordance with a local coordinate system (e.g., relative angular velocity provided by a gyroscope; relative translational acceleration provided by an accelerometer; relative attitude information provided by a vision sensor; relative distance information provided by an ultrasonic sensor, lidar, or time-of-flight camera). In some cases, the local coordinate system may be a body coordinate system defined relative to the UAV.

[0076] The sensors described herein may be carried by a UAV. The sensors may be located in any suitable part of the UAV, such as on the top, bottom, one or more sides, or within the body of the UAV. Some sensors may be mechanically coupled to the UAV so that the spatial arrangement and / or movement of the UAV corresponds to the spatial arrangement and / or movement of the sensor. The sensor may be coupled to the UAV via a rigid coupling so that the sensor does not move relative to the part of the UAV to which it is attached. Alternatively, the coupling between the sensor and the UAV may allow the sensor to move relative to the UAV. The coupling may be a permanent coupling or a non-permanent (e.g., detachable) coupling. Suitable coupling methods may include adhesives, bonding, welding, and / or fasteners (e.g., screws, nails, pins, etc.). Alternatively, the sensor may be integrally formed with a portion of the UAV. In addition, the sensor may be electrically coupled to a portion of the UAV (e.g., a processing unit, a control system, data storage) so that the data collected by the sensor can be used for various functions of the UAV (e.g., navigation, control, propulsion, communication with a user or other device, etc.), such as the embodiments discussed herein.

[0077] Sensors may be configured to collect various types of data, such as data related to the UAV, the surrounding environment, or objects within the environment. For example, at least some sensors may be configured to provide data about the state of the UAV. The state information provided by the sensors may include information about the spatial arrangement of the UAV (e.g., location or position information such as longitude, latitude, and / or altitude; direction or attitude information such as roll, pitch, and / or yaw). The state information may also include information about the motion of the UAV (e.g., translational velocity, translational acceleration, angular velocity, angular acceleration, etc.). For example, the sensors may be configured to determine the spatial arrangement and / or motion of the UAV with respect to up to six degrees of freedom (e.g., three positions and / or translational degrees of freedom, three directions, and / or rotational degrees of freedom). State information may be provided relative to a global coordinate system or relative to a local coordinate system (e.g., relative to the UAV or another entity). For example, the sensors may be configured to determine the distance between the UAV and the user controlling the UAV, or the distance between the UAV and the flight starting point of the UAV.

[0078] The data acquired by the sensors can provide various types of environmental information. For example, the sensor data can be an indication of the type of environment, such as an indoor environment, an outdoor environment, a low-altitude environment, or a high-altitude environment. The sensor data can also provide information about the current environmental conditions, including weather (e.g., sunny, rainy, snowy), visibility conditions, wind speed, time of day, etc. In addition, the environmental information collected by the sensors can include information about objects in the environment (such as the obstacles described herein). Obstacle information can include information about the number, density, geometry, and / or spatial arrangement of obstacles in the environment.

[0079] In some embodiments, a sensing result is generated by combining sensor data acquired by multiple sensors (also known as "sensor fusion"). For example, sensor fusion can be used to combine sensing data acquired by different sensor types, including GPS sensors, inertial sensors, visual sensors, lidars, ultrasonic sensors, etc. For another example, sensor fusion can be used to combine different types of sensing data, such as absolute measurement data (e.g., data provided relative to a global coordinate system, such as GPS data) and relative measurement data (e.g., data provided relative to a local coordinate system, such as visual sensing data, lidar data, or ultrasonic sensing data). Sensor fusion can be used to compensate for limitations or errors associated with individual sensor types, thereby improving the accuracy and reliability of the final sensing result.

[0080] Figure 2A A scheme 200 for estimating UAV pose using sensor fusion is illustrated in accordance with many embodiments. Figure 2A The embodiments of scheme 200 relate to estimating the yaw angle of a UAV, but it should be understood that the method described in scheme 200 can also be applied to estimating the roll angle or pitch angle of a UAV. Scheme 200 utilizes an IMU 202, at least one relative orientation sensor 204, and a magnetometer 206. The IMU 202 and the magnetometer 206 can be used to provide respective absolute estimates of the UAV yaw angles 208, 210. The relative orientation sensor 204 can be any sensor that provides attitude information with respect to a local coordinate system (e.g., the UAV body coordinate system) rather than a global coordinate system. Exemplary relative orientation sensors include visual sensors, lidar, ultrasonic sensors, and time-of-flight or depth cameras. The relative orientation sensor data can be analyzed to provide estimates of yaw rate 212 and relative yaw angle 214.

[0081] In embodiments where the relative orientation sensor 204 is a visual sensor configured to capture a sequence of images ("frames"), the yaw rate 212 and the yaw angle 214 may be determined using one or more images ("keyframes") selected from the sequence. Any suitable method may be used to select the keyframes. For example, the keyframes may be selected at predetermined intervals, such as at predetermined time intervals, predetermined distance intervals, predetermined displacement intervals, predetermined attitude intervals, or predetermined average disparity intervals between keyframes. As another example, the keyframes may be selected based on a relationship between consecutive keyframes (e.g., the amount of overlapping area between keyframes). Alternatively, the keyframes may be selected based on a combination of parameters provided herein. In some embodiments, at least some of the keyframes may include consecutive image frames. Alternatively, the keyframes may not include any consecutive image frames. The latter approach may be advantageous in reducing the propagation of estimation errors present in consecutive keyframes.

[0082] Any suitable number of keyframes, such as one, two, three, four, or more keyframes, can be used to estimate the yaw rate 212 and yaw angle 214 at a given time. Any suitable method, such as image analysis, can be used to determine the yaw rate 212 and yaw angle 214. In some embodiments, it can be assumed that two keyframes capturing the same scene at different yaw angles will differ. The yaw angles between different keyframes can be determined using mathematical modeling techniques. For example, the difference between keyframes can be determined by identifying and matching feature points present in both keyframes. Based on the coordinates of the feature points in both keyframes, a similarity transformation matrix can be determined for the two keyframes, and the yaw angle 212 can be obtained from the similarity transformation matrix. The yaw angle 214 and the time interval between the two keyframes can then be used to calculate the yaw rate 212.

[0083] The absolute and relative estimates provided by the sensors 202, 204, and 206 may be fused (e.g., using an extended Kalman filter 216 or other type of Kalman filter) to provide a final yaw angle result 218. Any suitable method may be implemented to fuse the estimates. For example, the integral of the yaw rate estimate 212 may be fused with the relative yaw angle estimate 214 to update the relative yaw angle estimate 214. The relative yaw angle estimate 214 may be fused with the absolute yaw angle estimates 208 and 210 from the IMU 202 and magnetometer 206, respectively, to determine the yaw angle of the UAV. The final yaw angle result 218 may be provided as a relative yaw angle (e.g., relative to the UAV's body coordinate system) or as an absolute yaw angle. As previously described, the combination of different types of measurement data may improve the accuracy of the final result 218. For example, in situations where the data provided by magnetometer 206 does not meet requirements (eg, due to external magnetic fields), using data from IMU 202 and / or relative positioning sensor 204 may reduce the extent to which magnetometer errors affect the final result.

[0084] Figure 2B A scheme 250 for estimating UAV position and velocity using sensor fusion is illustrated in accordance with an embodiment. The scheme 250 utilizes an IMU 252, at least one relative position and / or velocity sensor 254, and a GPS sensor 256. The IMU 252 may provide an estimate of UAV acceleration 258. The GPS sensor 256 may provide an estimate of the absolute position of the UAV 260. Estimates of UAV velocity 262 and relative UAV position 264 may be obtained using a relative position and / or velocity sensor 254 (e.g., a vision sensor, a lidar, an ultrasonic sensor, a time of flight or depth camera, or any other sensor that provides relative position and / or velocity information). Similar to Figure 2AIn embodiments where the relative position and / or velocity sensor 254 is a vision sensor, velocity and relative position estimates 262, 264 may be determined based on one or more keyframes captured by the vision sensor. For example, image analysis and mathematical modeling may be used to evaluate differences between consecutive keyframes and thereby determine the translational movement of the UAV that would have produced these differences. The UAV velocity may then be estimated based on the time elapsed between the keyframes.

[0085] An extended Kalman filter 266 or other suitable Kalman filter type can be used to fuse the estimates provided by the sensors 252, 254, and 256 to obtain a final position and velocity result 268 for the UAV. For example, the integral of the acceleration estimate 258 can be fused with the UAV velocity estimate 262 to determine the UAV velocity. The integral of the determined UAV velocity can be fused with the relative position estimate 264 and the absolute position estimate 260 to determine the UAV position. The position and velocity result 268 can be expressed relative to a local coordinate system or a global coordinate system.

[0086] Optionally, sensor fusion using Kalman filter 266 can also be used to determine calibration data for converting between the local coordinate system of relative speed and / or position sensor 254 and the global coordinate system of IMU252 and GPS256. Calibration data can be used to map the absolute sensing data acquired by IMU252 and / or GPS256 to the relative sensing data acquired by sensor 254, or vice versa. For example, IMU252 and / or GPS256 can be used to determine the path of the UAV in absolute coordinates, while relative sensor 254 can be used to acquire sensing data indicating the path of the UAV in relative coordinates. The sensor fusion method provided herein can be applied to determine the calibration parameters (e.g., scale factor, rotation, translation) required to project absolute path data onto relative path data, or vice versa. For example, the relationship between calibration parameters, absolute path data, and relative path data can be represented by mathematical modeling. The absolute path data and relative path data acquired by the sensor can then be input into the model to solve for calibration parameters. The combination of absolute and relative path data described herein may be used to increase the accuracy of path determination, thereby improving UAV navigation.

[0087] The sensor fusion methods described herein may be adapted to provide more accurate estimates of the UAV state and environment information. In some embodiments, the estimates of the UAV state information and environment information may be interdependent such that the UAV state information is used to estimate the environment information, and vice versa. For example, some sensors may be used to obtain absolute environment information. Other sensors can be used to obtain relative environmental information Can be based on relationships The absolute environment information is projected onto the relative environment information (or vice versa), R represents the rotation matrix between the global coordinate system and the local coordinate system, and T represents the translation vector from the origin of the global coordinate system to the origin of the local coordinate system. In some embodiments, the coordinate system of the UAV is provided with respect to the body of the UAV. So that T and R can be determined based on the UAV's position and orientation, respectively. Thus, the UAV's position and orientation information can be used to combine absolute and relative environmental data to generate an estimate of environmental information. Conversely, the absolute and relative environmental data can be used to estimate the UAV's position and orientation using the relationships described above. This method can be iterated to provide updated estimates of the UAV's state and environmental information.

[0088] In some embodiments, sensor data can be used to generate a representation of the environment or at least a portion thereof. Such a representation may be referred to herein as an "environmental map." The environmental map may represent a portion of the environment immediately adjacent to the UAV (a local map), or may also represent a portion relatively far from the UAV (a global map). For example, a local map may represent a portion of the environment within a radius of approximately 2m, 5m, 10m, 15m, 20m, 25m, or 50m from the UAV. A global map may represent a portion within a radius of approximately 25m, 50m, 75m, 100m, 125m, 150m, 200m, 225m, 250m, 300m, 400m, 500m, 1000m, 2000m, or 5000m from the UAV. The size of the environmental map may be determined based on the effective range of the sensor used to generate the map. The effective range of the sensor may vary depending on the sensor type. For example, a visual sensor may be used to detect objects within a radius of approximately 10m, 15m, 20m, or 25m from the UAV. A lidar sensor can be used to detect objects within a radius of approximately 2m, 5m, 8m, 10m, or 15m from the UAV. An ultrasonic sensor can be used to detect objects within a radius of approximately 2m, 5m, 8m, 10m, or 15m from the UAV. In some embodiments, the environment map may only represent the portion of the environment that the UAV has previously traveled. Alternatively, in other embodiments, the environment map may also represent portions of the environment that the UAV has not yet traveled.

[0089] Environmental maps can be used to represent various types of environmental information. For example, a map can provide information indicating the geometry (e.g., length, width, height, thickness, shape, surface area, volume), spatial arrangement (e.g., position, orientation), and type of environmental objects such as obstacles, structures, landmarks, or features. For another example, a map can provide information indicating which portions of the environment are blocked (e.g., not traversable by the UAV) and which portions are unblocked (e.g., traversable by the UAV). In addition, a map can provide information about the current position of the UAV relative to various environmental objects and / or their spatial arrangement.

[0090] Any suitable type of environment map can be used, such as a scale map or a topological map. A topological map can depict the connectivity between locations within the environment, while a scale map can depict the geometry of objects within the environment. Alternatively, the environment map can include a combination of scale information and topological information. For example, the environment map can depict the connectivity and absolute spatial relationships (e.g., distances) between environmental objects and / or locations. Alternatively, or in combination, some portions of the map can be depicted topologically, while other portions can be depicted scale-wise.

[0091] The environment map can be provided in any suitable format. For example, a topological environment map can be provided as a graph having vertices representing locations and edges representing paths between locations. The edges of the topological environment map can be associated with distance information of the corresponding paths. The scale environment map can be any representation of the spatial coordinates of the environment locations and objects, such as a point cloud, a topological map, a 2D grid map, a 2.5D grid map, or a 3D grid map. The spatial coordinates can be 3D coordinates (e.g., x, y, and z coordinates) that represent the spatial position of a point on the surface of an environment object. In some embodiments, the scale environment map can be an occupancy grid map. The occupancy grid map can represent the environment as multiple volumes. The volumes can have the same size or different sizes. The occupancy grid map can indicate for each volume whether the volume is substantially occupied by obstacles, thereby providing an accurate representation of blocked and unblocked spaces within the environment.

[0092] Any suitable method may be used to generate the environment maps described herein. For example, one or more portions of the environment map may be generated during UAV operation (e.g., during flight), such as by using simultaneous localization and mapping (SLAM) or other robotic map building techniques. Alternatively, one or more portions of the map may be generated prior to UAV operation and provided to the UAV before or during flight (e.g., transmitted from a remote computing system). Such a map may be modified based on data acquired by the UAV during flight, thereby providing further refinement of the map data.

[0093] In some embodiments, environmental mapping can be performed using sensor fusion to combine environmental information collected by multiple sensor types. This approach can be helpful in compensating for the limitations of a single sensor type. For example, when the UAV is in an indoor environment, GPS sensing data may be inaccurate or unavailable. Vision sensors may not be optimized for detecting transparent objects (e.g., glass) or for relatively dark environments (e.g., at night). In some embodiments, lidar sensors may have a relatively short detection range compared to other sensor types. The use of multi-sensor fusion described herein can provide accurate mapping in diverse environmental types and operating conditions, thereby improving the robustness and flexibility of UAV operations. Figure 3A A scheme 300 for using sensor fusion to construct an environment map according to an embodiment is illustrated. Scheme 300 can be used to generate any embodiment of the environment map described herein. In scheme 300, sensory data is received from multiple sensors 302. The multiple sensors 302 can include one or more different sensor types. The sensors 302 can be carried by a UAV, for example, coupled to the body of the UAV. Optionally, the sensory data from each sensor 302 can be pre-processed to improve data quality through filtering, noise reduction, image distortion correction, etc. In some embodiments, the sensory data provided by each sensor 302 can be expressed relative to a corresponding coordinate system (e.g., based on the sensor's position and orientation relative to the UAV body). Therefore, the sensory data from each sensor can be combined by converting all of the sensor data into a single coordinate system 304. For example, sensory data provided relative to a local coordinate system can be converted to a global coordinate system, or vice versa. The coordinate system conversion 304 can be performed based on sensor calibration data 306. Sensor calibration can be performed using any suitable technique and can be performed prior to operation of the UAV (offline calibration) or during operation of the UAV (online calibration). In some embodiments, sensor calibration involves determining external parameters of the sensors 302, such as the spatial relationship (e.g., relative position and orientation) between each sensor 302. A transformation calculation for converting the sensed data into a single coordinate system can then be determined based on the determined sensor parameters.

[0094] After the coordinate system transformation 304, sensor fusion 308 can then be used to combine the transformed sensed data to obtain a single sensed result. Various techniques can be used to perform sensor fusion, such as Kalman filtering (e.g., Kalman filter, extended Kalman filter, unscented Kalman filter), particle filtering, or other filtering techniques known to those skilled in the art. The method used can vary depending on the specific combination and type of sensors used. In some embodiments, sensor fusion 308 can utilize sensed data from all sensors 302. Conversely, sensor fusion 308 can utilize data from only a subset of sensors 302. The latter approach can be advantageous in ignoring insufficient or unreliable sensor data (e.g., GPS sensed data when the UAV is indoors). The fused sensed data can then be used for environment map generation 310.

[0095] Environmental mapping can be performed based on sensory data from any suitable combination of sensors. For example, in some embodiments, the map building method described herein is performed by a UAV carrying a lidar sensor and a vision sensor (e.g., a monocular vision sensor such as a single camera). The lidar sensor can be used to obtain distance data of environmental objects relative to the UAV, while the vision sensor can be used to capture image data of surrounding environmental objects. Sensor calibration data for the lidar and vision sensors may include sensor parameters that indicate the spatial relationship between the lidar and vision sensors. For example, for an object determined by the lidar sensor to be at a distance M from the UAV, i , and corresponds to the coordinate m in the image data captured by the vision sensor i Environmental objects at m i With M i The relationship between can be expressed by the following formula:

[0096] s i m i =(R,t)M i

[0097] Where R is the rotation matrix, t is the transformation matrix, s i is an unknown scalar, and K is the internal matrix of the vision sensor (internal parameters determined by previous calibration). In the above equation, M i 、m i and K are known, while R and t are unknown. If N measurements have been made, then since R and t are constant (the relative positions and orientations of the lidar sensor and the vision sensor are fixed), the problem is an n-point perspective (PNP) problem and can be solved using techniques known to those skilled in the art to obtain the sensor parameters R and t.

[0098] Once the sensor calibration data has been acquired, the lidar and vision data can be transformed into the same coordinate system (the "world" coordinate system):

[0099]

[0100] X W is the data relative to the world coordinate system (which corresponds to the visual sensor coordinate system in this embodiment), is the visual sensing data relative to the world coordinate system, is the visual sensing data relative to the visual sensor coordinate system, is the lidar sensing data relative to the world coordinate system, is the lidar sensing data relative to the lidar sensor coordinate system, and R WL and t WL is the rotation matrix and transformation matrix used to convert between the visual sensor coordinate system and the lidar sensor coordinate system (corresponding to R and t above). E is the diagonal matrix

[0101]

[0102] Where s is the unknown scaling factor.

[0103] Sensor fusion techniques can then be used to combine the transformed sensory data. For example, the visual sensory data relative to the visual coordinate system Visual perception data relative to the world coordinate system The relationship between can be expressed by the following formula:

[0104]

[0105] Using the lidar sensing data, you can get the sensing direction along the lidar sensor The point closest to the UAV on the y-axis (denoted as point i). Therefore, due to

[0106]

[0107] and and It is known (due to ), the value of s can be determined. Therefore, the local sensor data and Can be converted into corresponding world coordinates and and fused to generate a single result X W The fused data can then be used to generate a map of the UAV's surroundings.

[0108] Figure 3BA method 350 for generating a map of an environment using sensor fusion is illustrated, according to an embodiment. The method 350 may be performed by any of the systems and devices provided herein, such as by one or more processors of a UAV.

[0109] In step 360, first sensing data is received from one or more visual sensors, wherein the first sensing data includes depth information of the environment. For example, the visual sensor may include only one camera (monocular visual sensor). Alternatively, the visual sensor may include two (binocular visual sensor) or more cameras. The visual sensor may be carried by the UAV, such as by the UAV fuselage. In an embodiment using multiple visual sensors, each sensor may be located on a different part of the UAV, and the parallax between the image data collected by each sensor may be used to provide depth information of the environment. Depth information may be used herein to refer to information about the distance of one or more objects from the UAV and / or sensor. In an embodiment using a single visual sensor, depth information may be obtained by capturing image data for multiple different positions and directions of the visual sensor and then reconstructing the depth information using a suitable image analysis technique (e.g., structure from motion).

[0110] In step 370, second sensory data is received from one or more distance sensors, the second sensory data including depth information of the environment. The distance sensor may include at least one ultrasonic sensor (e.g., a wide-angle sensor, an array sensor) and / or at least one lidar sensor. In some embodiments, an ultrasonic array sensor may provide better detection accuracy than other types of ultrasonic sensors. The distance sensor may also be carried by the UAV. The distance sensor may be located near the visual sensor. Alternatively, the distance sensor may be located on a portion of the UAV different from the portion used to carry the visual sensor.

[0111] In step 380, an environment map including depth information of the environment is generated using the first sensory data and the second sensory data. As described herein, the first sensory data and the second sensory data may each include depth information of the environment. Optionally, the first sensory data and the second sensory data may also include outline information of one or more objects in the environment. As used herein, outline information refers to information about the shape, outline, or edge of an object in the environment. The sensory data generated by one or more range sensors may be used as a pixel depth map, while depth information may be extracted from image data collected by one or more visual sensors using techniques such as structure from motion, structured light, light sheets, time of flight, or stereoscopic parallax mapping. In some embodiments, the first sensory data and the second sensory data each include at least one image having a plurality of pixels, each pixel being associated with a 2D image coordinate (e.g., x and y coordinate), a depth value (e.g., the distance between the environmental object corresponding to the pixel and the UAV and / or sensor), and / or a color value (e.g., RGB color value). Such an image may be referred to herein as a depth image.

[0112] Taking into account the relative reliability and accuracy of each type of sensory data during the fusion process, the depth information associated with each set of sensory data can be spatially aligned and combined (e.g., using a suitable sensor fusion method, such as Kalman filtering) to generate a map including depth information (e.g., a 3D environment representation, such as an occupancy grid map). In some embodiments, generating the environment map can involve identifying a plurality of feature points present in a set of depth images provided by a visual sensor, identifying a corresponding plurality of feature points present in a set of depth images provided by a range sensor, and determining correspondences between the plurality of feature points. The correspondences can include information about one or more transformations (e.g., translation, rotation, scaling) applicable to mapping the range depth images onto the visual depth images, or vice versa. The depth images can then be combined based on such correspondences to generate the environment map. Alternatively, or in combination, the first and second sensory data can be provided relative to different coordinate systems, and the environment map can be generated by representing the first and second sensory data in the same coordinate system. This coordinate system can be the coordinate system associated with the first sensory data, the coordinate system associated with the second sensory data, or an entirely different coordinate system. Once a map of the environment has been generated, the UAV may be navigated within the environment based on the depth information contained in the map of the environment.

[0113] The combination of distance sensing and visual sensing described herein can compensate for the limitations of individual sensor types, thereby improving map generation accuracy. For example, a visual sensor can produce relatively high-resolution color images, but it can be relatively difficult to obtain accurate depth data from the image data when using a monocular camera or when the distance between the binocular cameras is relatively small (such as when the cameras are mounted on a small UAV). In addition, visual sensors may not provide satisfactory image data when the lighting is bright or has high contrast, or in adverse environmental conditions such as rain, fog, or smoke. Conversely, distance sensors such as ultrasonic sensors can provide accurate depth data, but may have lower resolution than visual sensors. Furthermore, in some cases, ultrasonic sensors and other distance sensor types may not be able to detect objects with small reflective surfaces (e.g., twigs, corners, railings) or absorbent objects (e.g., carpet), or may not be able to distinguish distances in complex environments with many objects (e.g., indoor environments). However, visual sensing data can generally complement distance sensing data because visual sensors can produce reliable data in situations where distance sensors are generating less than optimal data, and vice versa. Therefore, the combined use of vision sensors and range sensors can be used to generate accurate environmental maps under a variety of operating conditions and for a wide variety of environments.

[0114] Figure 4 A method 400 for mapping an environment using different sensor types, according to an embodiment, is illustrated. As with all methods disclosed herein, method 400 can be implemented using any embodiment of the systems and devices described herein, such as one or more processors carried onboard a UAV. Furthermore, as with all methods disclosed herein, any step of method 400 can be combined with or replaced by any step of any other method described herein.

[0115] In step 410, a first sensing signal is received from a first sensor carried by the UAV. The first sensing signal may include information about the environment in which the UAV is operating, such as environmental data indicating the location and geometry of environmental objects (e.g., obstacles). Similarly, in step 420, a second signal is received from a second sensor carried by the UAV. In some embodiments, the first sensor and the second sensor may be different sensor types, including any of the sensor types described above (e.g., a lidar sensor and a visual sensor, an ultrasonic sensor and a visual sensor, etc.).

[0116] In step 430, the first sensory signal is used to generate a first environment map, the first environment map including occupancy information of the environment. In step 440, the second sensory signal is used to generate a second environment map, the second environment map including occupancy information of the environment. The first environment map and the second environment map may each include obstacle occupancy information of the environment, so that the environment maps can be used to determine the location of obstacles relative to the UAV. For example, the first environment map and the second environment map may be occupancy grid maps or any other map type that includes information about blocked and unblocked spaces within the environment.

[0117] Alternatively, the first environment map may represent a different portion of the environment than the second environment map. For example, the first environment map may represent a portion of the environment relatively close to the UAV, while the second environment map may represent a portion of the environment relatively far from the UAV. In some embodiments, different sensing signals may be used to generate maps spanning different portions of the environment. The selection of the signals to be used to generate the different maps may be based on any suitable criteria, such as the relative signal quality and / or accuracy of the first and second sensing signals for a particular portion of the environment. The quality and accuracy of the sensing data may depend on the specific characteristics of each sensor and may vary depending on the type of environment (e.g., indoors, outdoors, low altitude, high altitude), weather conditions (e.g., clear, rainy, foggy), the relative position of sensed environmental objects (e.g., close range, far range), and the properties of the sensed environmental objects (e.g., transparency, reflectivity, absorptivity, shape, size, material, mobility, etc.). For example, a first sensor may be more accurate than a second sensor at close range, while a second sensor may be more accurate than the first sensor at long range. Thus, a first sensor can be used to generate a map of a portion of the environment relatively close to the UAV, while a second sensor can be used to generate a map of a portion of the environment relatively far from the UAV. Alternatively, or in combination, the selection of a signal can be based on whether the portion of the environment is within the sensing range of the corresponding sensor. This method may be advantageous in embodiments where the first and second sensors have different sensing ranges. For example, a short-range sensor can be used to generate a map of a portion of the environment relatively close to the UAV, while a long-range sensor can be used to generate a map of a portion relatively far from the UAV that is outside the range of the short-range sensor.

[0118] In step 450, the first environment map and the second environment map are combined to generate a final environment map containing occupancy information of the environment. The environment maps can be combined using the sensor fusion technology described herein. In an embodiment where sensing signals are provided relative to different coordinate systems (e.g., a local coordinate system and a global coordinate system), the generation of the map portion can involve converting the two sensing signals into a single coordinate system to align the corresponding environment data, thereby generating a final environment map. Subsequently, the UAV can navigate within the environment based on the final environment map. For example, the UAV can navigate at least in part based on the obstacle occupancy information represented in the final environment map to avoid collisions. The UAV can be navigated by a user, by an automated control system, or a suitable combination thereof. Additional examples of UAV navigation based on environment map data are further described below.

[0119] While the above steps illustrate a method 400 for constructing an environment map according to an embodiment, those skilled in the art will recognize many variations based on the teachings described herein. Some steps may include sub-steps. In some embodiments, the steps of method 400 may be repeated as needed (e.g., continuously or at predetermined time intervals) so that the generated environment map is updated and refined as the UAV navigates within the environment. Such a real-time map construction method can enable the UAV to quickly detect environmental objects and adapt to changing operating conditions.

[0120] The sensor fusion method described herein can be applied to various types of UAV functions, including navigation, object recognition, and obstacle avoidance. In some embodiments, the environmental data obtained using the sensor fusion results can be used to improve the robustness, safety, and flexibility of UAV operations by providing accurate location information and information about potential obstacles. The environmental data can be provided to the user (e.g., via a remote control or terminal, mobile device, or other user device) to provide information for the user's manual control of the UAV. Alternatively or in combination, the environmental data can be used in a semi-autonomous or fully autonomous control system to guide the automated flight of the UAV.

[0121] For example, the embodiments disclosed herein can be used to perform obstacle avoidance maneuvers to prevent a UAV from colliding with environmental objects. In some embodiments, obstacle detection and avoidance can be automated, thereby improving the safety of the UAV and reducing the user's responsibility for avoiding collisions. Such an approach can be advantageous for inexperienced operators and in situations where the user cannot easily perceive the presence of obstacles near the UAV. In addition, the implementation of automated obstacle avoidance can reduce the safety risks associated with semi-autonomous or fully autonomous UAV navigation. Furthermore, the multi-sensor fusion techniques described herein can be used to generate a more accurate representation of the environment, thereby improving the reliability of such automated collision avoidance mechanisms.

[0122] Figure 5 A method 500 for controlling a UAV to avoid obstacles is illustrated, according to an embodiment. The method 500 may be implemented by one or more processors carried by the UAV. As previously described, the method 500 may be fully automated or at least partially automated, thereby providing automatic obstacle detection and avoidance capabilities.

[0123] In step 510, a first signal is generated to enable the UAV to navigate within the environment. The first signal may include a control signal for a propulsion system (e.g., rotors) of the UAV. The first signal may be generated based on a user command input into a remote terminal or other user device and subsequently transmitted to the UAV. Alternatively, the first signal may be generated autonomously by the UAV (e.g., an automated onboard controller). In some cases, the first signal may be generated semi-autonomously with contributions from user input and an automated path determination mechanism. For example, a user may indicate a series of waypoints for the UAV, and the UAV may automatically calculate a flight path to pass through all of the waypoints.

[0124] In some embodiments, the flight path can indicate a series of desired positions and / or directions (e.g., with respect to up to six degrees of freedom) for the UAV. For example, the flight path can include at least an initial position and a target position of the UAV. Alternatively, the flight path can be configured to guide the UAV from a current position to a previous position. For another example, the flight path can include a target flight direction for the UAV. In some embodiments, the instructions can specify a velocity and / or acceleration (e.g., with respect to up to six degrees of freedom) for the UAV to move along the flight path.

[0125] In step 520, a plurality of sensors are used to receive sensory data about at least a portion of the environment. The plurality of sensors may include different types of sensors (e.g., vision, lidar, ultrasound, GPS, etc.). The sensory data may include information about the location and characteristics of obstacles and / or other environmental objects. For example, the sensory data may include distance information indicating the distance from the UAV to a nearby obstacle. Alternatively, the sensory data may include information about portions of the environment along the flight path of the UAV, such as portions overlapping or proximate to the flight path.

[0126] In step 530, an environment map representing at least a portion of the environment is generated based on the sensory data. The environment map can be a local map representing the portion of the environment immediately surrounding the UAV (e.g., within a radius of approximately 2m, 5m, 10m, 15m, 20m, 25m, or 50m from the UAV). Alternatively, the environment map can be a global map that also represents the portion of the environment relatively far from the UAV (e.g., within a radius of approximately 25m, 50m, 75m, 100m, 125m, 150m, 200m, 225m, 250m, 300m, 400m, 500m, 1000m, 2000m, or 5000m from the UAV). As previously described, the size of the environment map can be determined based on the effective range of the sensors used in step 510. In some embodiments, the environment map can be generated using the sensor fusion-based methods described above (e.g., scheme 300 and / or method 400). For example, the sensory data may include data relative to multiple different coordinate systems, and environment map generation may involve mapping the data onto a single coordinate system in order to facilitate fusion of the sensory data.

[0127] In step 540, the environment map is used to detect one or more obstacles located in the portion of the environment. Obstacle detection from the map information can be performed using various strategies, such as through feature extraction or pattern recognition techniques. Optionally, a suitable machine learning algorithm can be implemented to perform obstacle detection. In some embodiments, if the environment map is an occupancy grid map, obstacles can be identified by detecting continuously occupied volumes in the occupancy grid map. The obstacle detection results can provide information about the location, direction, size, distance and / or type of each obstacle, as well as corresponding confidence information for the results. The map can be analyzed to identify obstacles that pose a collision risk to the UAV (e.g., obstacles located along or near the flight path).

[0128] In step 550, a second signal is generated using the environmental map to direct the UAV to navigate to avoid the one or more obstacles. The environmental map can be analyzed to determine the position of the one or more obstacles relative to the UAV, as well as the position of any unobstructed space available for the UAV to move, in order to avoid collision with the obstacles. The second signal can thus provide appropriate control signals (e.g., to the UAV's propulsion system) to direct the UAV to navigate through the unobstructed space. In embodiments where the UAV is navigated according to a flight path, the flight path can be modified based on the environmental map to avoid the one or more obstacles, and the UAV can navigate according to the modified flight path. The flight path modification can direct the UAV to traverse only through unobstructed space. For example, the flight path can be modified to cause the UAV to fly around an obstacle (e.g., above, below, or to the side), fly away from the obstacle, or maintain a specified distance from the obstacle. Where multiple modified flight paths are possible, a preferred flight path can be selected based on any suitable criteria, such as minimizing travel distance, minimizing travel time, minimizing the amount of energy required to travel the path, minimizing deviation from the original flight path, and the like. Alternatively, if a suitable flight path cannot be determined, the UAV may simply hover in position and wait for obstacles to move out of the way, or for a user to take manual control.

[0129] In another exemplary application of multi-sensor fusion for UAV operations, the embodiments introduced in the present disclosure may be implemented as part of an "auto return home" functionality, wherein the UAV will automatically navigate from its current position to a "return home" position under certain circumstances. The return home position may be an initial position that the UAV has previously traveled to, such as the position where the UAV initially took off and began flying. Alternatively, the return home position may not be a position that the UAV has previously traveled to. The return home position may be automatically determined or specified by the user. Examples of situations that may trigger the auto return home function include receiving a return home instruction from the user, loss of communication with the user's remote control or other indication that the user can no longer control the UAV, the UAV battery being low, or detecting a UAV malfunction or other emergency situation.

[0130] Figure 6 A method 600 for controlling a UAV to return to an initial position according to an embodiment is illustrated. In some embodiments, the method 600 is implemented by one or more processors associated with an automated control system so that little or no user input is required to perform the method 600.

[0131] In step 610, an initial position of the UAV is determined using at least one of the plurality of sensors. In some embodiments, the position may be determined using a plurality of sensors having different sensor types. The position may be determined relative to a global coordinate system (e.g., GPS coordinates) or relative to a local coordinate system (e.g., relative to a local environmental landmark or feature).

[0132] In step 620, a first signal is generated to cause the UAV to navigate within the environment. The first signal can be generated manually, semi-autonomously, or fully autonomously. For example, a user can input a command into a remote control, which is transmitted to the UAV to control the movement of the UAV. For another example, the UAV can move according to a predetermined flight path or travel to a predetermined location. The UAV can be navigated to a location within the environment that is different from the initial location (e.g., with respect to longitude, latitude, or altitude).

[0133] In step 630, sensed data about the environment is received from at least one of the plurality of sensors. The sensed data may be received from the same sensor used in step 610 or from a different sensor. Similarly, sensed data may be acquired from a variety of different sensor types. As previously described, such sensed data may include information about the location and characteristics of environmental objects.

[0134] In step 640, an environment map representing at least a portion of the environment is generated based on the sensory data. In embodiments where the sensory data includes data from multiple sensor types, the environment map may be generated using the sensor fusion-based methods described above (e.g., scheme 300 and / or method 400). The resulting map may be provided in any suitable format and may include information about obstacle occupancy (e.g., an obstacle grid map).

[0135] In step 650, a second signal is generated based on the environmental map to return the UAV to the initial position. The second signal may be generated in response to an automatic return instruction. The automatic return instruction may be input by a user or may be automatically generated, for example, in response to an emergency such as loss of communication with a remote control controlling the UAV. In some embodiments, once the UAV has received an automatic return instruction, the UAV may use the environmental map to determine its current position, the spatial relationship between the current position and the initial position, and / or the location of any environmental obstacles. Based on the obstacle occupancy information in the environmental map, the UAV may then determine an appropriate path (e.g., a flight path) to navigate from the current position to the initial position. Various methods may be used to determine a suitable path for the UAV. For example, as described herein with respect to Figure 9A and Figure 9BAs further detailed, a map of the environment, such as a topological map, may be used to determine the path. In some embodiments, the path may be configured to avoid one or more environmental obstacles that may hinder the flight of the UAV. Alternatively, or in combination, the path may be the shortest path between the current location and the initial location. The path may include one or more portions that the UAV has previously traveled, and one or more portions that are different from the path that the UAV has previously traveled. For example, the entire path may have been previously traveled by the UAV. For another example, as described herein with respect to Figure 8A and Figure 8B As further detailed, the path may include multiple waypoints corresponding to locations previously traveled by the UAV. Conversely, the entire path may not have been previously traveled by the UAV. In some embodiments, multiple potential paths may be generated, and a path may be selected from the multiple paths based on appropriate criteria (e.g., minimizing total flight time, minimizing total flight distance, minimizing energy consumption, minimizing the number of obstacles encountered, maintaining a predetermined distance from obstacles, maintaining a predetermined altitude range, etc.). The UAV's flight path information (e.g., previously traveled paths, potential paths, selected path) may be included in the environment map.

[0136] Once a path has been determined, the path information can be processed to generate instructions for controlling the propulsion system of the UAV so that it travels along the path to the initial location. For example, the path can include a series of spatial position and orientation information for the UAV, which can be converted into control signals to guide the flight of the UAV from the current location to the initial location. In some embodiments, as the UAV navigates along the path, it can detect one or more obstacles along the path that impede its flight. In such cases, the path can be modified to avoid the obstacles so as to allow the UAV to continue navigating to the initial location, for example, using the techniques described above with respect to method 500.

[0137] Figure 7 A method 700 for controlling a UAV to return to a home position while avoiding obstacles is illustrated in accordance with many embodiments. Similar to method 500, method 700 can be performed in a semi-automated or fully automated manner as part of an automatic return-to-home function of the UAV. At least some steps of method 700 can be performed by one or more processors associated with the UAV.

[0138] In step 710, an initial position of the UAV is determined using at least one of the plurality of sensors. Alternatively, a plurality of different sensor types may be used to determine the initial position. The position information may include information regarding the position (e.g., altitude, latitude, longitude) and / or orientation (e.g., roll, pitch, yaw) of the UAV. The position information may be provided relative to a global coordinate system (e.g., a geographic coordinate system) or a local coordinate system (e.g., relative to the UAV).

[0139] In step 720, a first signal is generated to cause the UAV to navigate within the environment, for example, to a location different from the initial location. As previously described, the first signal may be generated based on user input, instructions provided by the autonomous control system, or a suitable combination thereof.

[0140] In step 730, at least one of the plurality of sensors is used to receive sensor data regarding the environment. Optionally, the sensor data may be received from a different sensor than the sensor used in step 710. As described above, the sensor signals from different sensor types may be combined using a suitable sensor fusion-based method to generate the sensor data. In some embodiments, the fused sensor data may be used to generate a representative map of the environment, such as in the embodiments discussed above.

[0141] In step 740, a command to return to the initial position is received. The return command may be input by the user into the remote control and subsequently transmitted to the UAV. Alternatively, the command may be automatically generated by the UAV, such as by an onboard processor and / or controller. In some embodiments, the command may be generated independently of any user input, for example, when the UAV detects that communication between the user device and the UAV has been lost, when a malfunction or emergency has occurred, etc.

[0142] In step 750, a second signal is generated to cause the UAV to return to the initial position along a path, wherein when at least one of the plurality of sensors detects an obstacle along the path, the path is modified to avoid the obstacle. Optionally, the sensor used to detect the obstacle may be different from the sensor used in steps 710 and / or 730. Obstacles may be detected based on sensory data generated by a variety of different sensor types. In some embodiments, an environment map (e.g., an occupancy grid map) generated based on the sensor data may be used to detect obstacles. The environment map may be previously generated (e.g., based on the sensory data collected in step 730) or may be generated in real time as the UAV navigates along the path. Various methods may be used to modify the path to prevent the UAV from colliding with detected obstacles (such as those described above with respect to method 500). Determination of the modified path may be performed based on the acquired sensory data and / or obstacle information represented in the environment map.

[0143] Figure 8A and Figure 8B The diagram illustrates an algorithm 800 for controlling a UAV to return to an initial location using waypoints according to an embodiment. The algorithm 800 can be executed by a suitable processor and / or controller for controlling the operation of the UAV. One or more steps of the algorithm 800 can be automatically executed to provide an automatic return-to-home function for the UAV.

[0144] After the UAV flight begins (e.g., takes off) in step 802, a determination is made in step 804 as to whether the UAV flight has ended. This determination may be based on whether the UAV has landed (e.g., on a surface such as the ground), whether the UAV's propulsion system has been shut down, whether the user has provided instructions to end the flight, etc. If the flight has ended, algorithm 800 ends at step 806. Otherwise, algorithm 800 continues to step 808 to determine the UAV's current position (e.g., latitude, longitude, altitude). As previously described, one or more sensors (at least some of which are of different types) may be used to determine the current position. For example, the UAV may include a GPS sensor and another sensor type, such as a visual sensor. Visual sensor data may be used in conjunction with or in place of GPS sensor data to compensate for situations where the GPS sensor is unable to provide reliable sensor data (e.g., in indoor and / or low-altitude environments where communication with GPS satellites may be less than ideal, or at the beginning of a flight when the GPS sensor has not yet established communication with GPS satellites). Therefore, the combined use of GPS sensor data and visual sensor data can improve positioning accuracy. Alternatively, an environmental map that supports positioning the UAV relative to a global coordinate system or a local coordinate system may be used to determine the current position. As described above, a sensor fusion technique may be used to generate an environmental map.

[0145] In step 810, the current position information is stored in a suitable data structure, such as a position information list. The position list can be used to store a set of locations previously traveled by the UAV, which may be referred to herein as "waypoints." The waypoints can be stored in sequence, thereby providing a representation of the flight path traveled by the UAV. In such an embodiment, the first waypoint stored in the position list corresponds to the initial position of the UAV, and the last waypoint in the position list corresponds to the most recent position of the UAV. Therefore, as described below, the stored waypoints can be traversed in reverse order (from the last waypoint to the first waypoint) in order to return the UAV to the initial position.

[0146] In step 812, a determination is made as to whether the UAV should initiate an automatic return function in order to return to an initial position corresponding to the first waypoint stored in the position list. As previously described, the automatic return decision can be made automatically or based on user input. If the automatic return function is not initiated, the algorithm 800 returns to step 804 to begin the next iteration of waypoint determination and storage. In some embodiments, steps 804, 808, 810, and 812 can be repeated in sequence at a desired frequency (e.g., continuously, at predetermined time intervals, such as once every 0.5s, 1s, 2s, 5s, 10s, 20s, or 30s) to generate and store a series of previously traveled waypoints during the flight of the UAV. As described below with respect to Figure 8B As further detailed, once it is determined in step 812 that the UAV should automatically return home, the automatic return home function is executed in step 814. After the automatic return home function is completed, the algorithm 800 ends at step 806.

[0147] Figure 8B The sub-steps of step 814 of algorithm 800 according to an embodiment are illustrated. After the automatic return-to-home function is initiated at step 816, a determination is made in step 818 as to whether the location list (waypoint list) is empty. If the list is empty, this indicates that the UAV has reached the initial location and the automatic return-to-home function is completed in step 820. If the list is not empty, this indicates that the UAV has not yet reached the initial location. In this case, the next target location (waypoint) is then retrieved from the location list in step 822.

[0148] The current location of the UAV is then determined in step 824. The current location can be determined using any of the methods described herein, such as using multi-sensor fusion and / or environmental mapping. In some embodiments, a spatial relationship between the current location and the retrieved target location is determined to generate a flight path between the current location and the target location. As previously described, the flight path can be generated based on environmental mapping data to avoid potential flight obstacles.

[0149] In step 826, the UAV is navigated from its current location to the target location, for example, using the flight path generated in step 824. As the UAV moves toward the target location, it may acquire and process sensor data to determine whether there are any obstacles that could impede its flight. For example, the sensor data may be used to provide a local environment map containing obstacle occupancy information as discussed above. Exemplary sensors that may be used to provide obstacle information include visual sensors, lidar sensors, ultrasonic sensors, and the like. Based on the sensor data, a determination is made in step 828 as to whether an obstacle has been detected. If no obstacle is detected, the UAV completes its navigation to the target location, and the automatic return-to-home function continues to step 818 to begin the next iteration. If an obstacle has been detected, an appropriate obstacle avoidance maneuver is determined and executed, as indicated in step 830. For example, the flight path may be modified (e.g., based on the environment map data) so that the UAV navigates to the target location without colliding with a detected obstacle.

[0150] Figure 9A and Figure 9BAn algorithm 900 for controlling a UAV to return to a target location using a topological map is illustrated according to an embodiment. Similar to algorithm 800, algorithm 900 can be executed by a suitable processor and / or controller for controlling the operation of the UAV. One or more steps of algorithm 900 can be automatically executed to provide an automatic return-to-home function for the UAV.

[0151] Similar to step 802 of algorithm 800, after the UAV flight begins (e.g., takes off) in step 902, a determination is made in step 904 as to whether the UAV flight has ended. If the flight has ended, then algorithm 900 ends at step 906. Otherwise, algorithm 900 proceeds to step 908 to determine the current position of the UAV and environmental information about the environment surrounding the UAV. In some embodiments, as described above, environmental information includes information about the spatial arrangement, geometry, and / or characteristics of one or more environmental objects. The current position and / or environmental information can be determined using sensory data obtained from one or more sensors. Optionally, sensor fusion technology can be used to process and combine sensory data from different sensor types to improve the reliability and accuracy of the sensing results. For example, sensory data can be obtained from a GPS sensor and at least one other sensor type (e.g., a visual sensor, a lidar sensor, and / or an ultrasonic sensor).

[0152] In step 910, current position and environmental information are added to a topological map representing the operating environment. In alternative embodiments, other types of maps (e.g., scale maps such as grid maps) can also be used. The topological map can be used to store the position information corresponding to the position previously traveled by the UAV, thereby providing a record of the flight path of the UAV. In addition, according to the embodiments described above, the topological map can be used to depict the connectivity between each environmental location. Optionally, the topological map can also include scale information, such as information indicating the distance between the represented positions, and information indicating the spatial position and geometric structure of environmental objects. At least some parts of the topological map can be generated during the UAV flight, for example, using a multi-sensor map to build. Alternatively or in combination, one or more parts of the topological map can be generated before the operation of the UAV and provided to the UAV (e.g., transmitted from a remote controller and / or stored in an onboard memory). In a latter embodiment, the topological map can be refined or modified based on the sensor data collected during the UAV operation to reflect current environmental information.

[0153] Similar to step 812 of algorithm 800, in step 912, a determination is made as to whether the UAV should activate the automatic return function in order to return to the target location. If the automatic return function is not activated, algorithm 900 returns to step 904. Steps 904, 908, 910, and 912 may be repeated in sequence at a desired frequency (e.g., continuously, at predetermined time intervals, such as once every 0.5 s, 1 s, 2 s, 5 s, 10 s, 20 s, or 30 s) to update the topological map with the latest position and environmental information as the UAV navigates within the environment. As described below with reference to Figure 9B As further detailed, once it is determined in step 912 that the UAV should automatically return home, the automatic return home function is executed in step 914. After the automatic return home function is completed, the algorithm 900 ends in step 906.

[0154] Figure 9B The sub-steps of step 914 of algorithm 900 according to an embodiment are illustrated. After the automatic return function is initiated at step 916, a determination is made in step 918 as to whether the UAV has reached the target location. The target location can be a location specified by the user or automatically determined by the UAV. The target location can be a location that the UAV has previously traveled to, such as the initial location of the UAV at the beginning of the flight (e.g., the location of takeoff). Alternatively, the target location can be a location that the UAV has not previously traveled to. If it is determined that the UAV has reached the target location, the automatic return function is completed in step 920. Otherwise, the automatic return continues to step 924.

[0155] In step 924, the current position of the UAV is determined using any of the methods described herein, such as multi-sensor fusion and / or environmental map construction. In step 926, a topological map position corresponding to the current UAV position is determined, thereby localizing the UAV relative to the environmental information represented in the topological map. Based on the topological map, a flight path from the current position to the target location is determined in step 926. In some embodiments, one or more portions of the return flight path may correspond to a flight path previously traveled by the UAV. Alternatively, one or more portions of the flight path may differ from the UAV's previous flight path. The flight path can be determined based on any suitable criteria. For example, the flight path can be configured to avoid environmental obstacles. In another example, the flight path can be configured to minimize the total distance traveled by the UAV to reach the target location. Subsequently, in step 928, the UAV flight controls are adjusted based on the determined flight path to move the UAV along the flight path. For example, the flight path can specify a series of positions and / or directions for the UAV, and appropriate control signals can be generated and transmitted to the UAV propulsion system to cause the UAV to assume the specified positions and / or directions.

[0156] As described in steps 930 and 932, as the UAV moves toward the target location, it can detect obstacles and perform obstacle avoidance maneuvers as appropriate. The techniques for performing obstacle detection and obstacle avoidance can be similar to those described with respect to steps 828 and 830 of algorithm 800, respectively. Optionally, obstacle detection and / or obstacle avoidance can be based on a generated topological map. For example, once an obstacle has been identified, the environmental information represented in the topological map can be used to determine appropriate modifications to the flight path to avoid potential collisions.

[0157] The systems, devices and methods described herein can be applicable to a variety of movable objects. As mentioned above, any description of aircraft herein can be applicable to and used for any movable object. The movable object of the present invention can be configured to move in any suitable environment, such as in the air (e.g., fixed-wing aircraft, rotorcraft or aircraft that has neither fixed wings nor rotors), in water (e.g., ships or submarines), on the ground (e.g., motor vehicles, such as cars, trucks, buses, vans, motorcycles; movable structures or frames, such as sticks, fishing rods; or trains), underground (e.g., subways), in space (e.g., space shuttles, satellites or probes), or any combination of these environments. The movable object can be a vehicle, such as the vehicle described elsewhere herein. In some embodiments, the movable object can be mounted on a living body, such as a human or an animal. Suitable animals can include birds, dogs, cats, horses, cattle, sheep, pigs, dolphins, rodents or insects.

[0158] The movable object may be able to move freely within the environment about six degrees of freedom (e.g., three translational degrees of freedom and three rotational degrees of freedom). Alternatively, the movement of the movable object may be constrained about one or more degrees of freedom, such as by a predetermined path, trajectory, or direction. The movement may be actuated by any suitable actuating mechanism, such as an engine or motor. The actuating mechanism of the movable object may be powered by any suitable energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. The movable object may be self-propelled via a propulsion system as described elsewhere herein. The propulsion system may optionally operate on an energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, the movable object may be carried by a living being.

[0159] In some cases, the movable object can be a vehicle. Suitable vehicles can include water vehicles, aircraft, space vehicles or ground vehicles. For example, the aircraft can be a fixed-wing aircraft (for example, airplane, glider), a rotorcraft (for example, a helicopter, a gyroplane), an aircraft with fixed-wing and rotor simultaneously or an aircraft (for example, an airship, a hot air balloon) without fixed-wing and rotor. The vehicle can be self-propelled, such as self-propelled in the air, on water or in water, in space or on the ground or underground. Self-propelled vehicles can utilize a propulsion system, such as a propulsion system comprising one or more engines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles or any suitable combination thereof. In some cases, a propulsion system can be used to enable the movable object to take off from a surface, land on a surface, maintain its current position and / or direction (for example, hover), change orientation and / or change position.

[0160] The movable object may be remotely controlled by a user or locally controlled by an occupant within or on the movable object. In some embodiments, the movable object is an unmanned movable object, such as a UAV. An unmanned movable object (such as a UAV) may not have an operator on board the movable object. The movable object may be controlled by a human or an autonomous control system (e.g., a computer control system), or any suitable combination thereof. The movable object may be an autonomous or semi-autonomous robot, such as a robot equipped with artificial intelligence.

[0161] The movable object may have any suitable size and / or dimensions. In some embodiments, the movable object may have a size and / or dimensions that can accommodate a human occupant in or on the vehicle. Alternatively, the movable object may have a size and / or dimensions that is smaller than a size and / or dimensions that can accommodate a human occupant in or on the vehicle. The movable object may have a size and / or dimensions that is suitable for being carried or carried by a human. Alternatively, the movable object may be larger than a size and / or dimensions that is suitable for being carried or carried by a human. In some cases, the movable object may have a maximum dimension (e.g., length, width, height, diameter, diagonal) that is less than or equal to approximately: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. The maximum dimension may be greater than or equal to approximately: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance between the axes of the opposing rotors of the movable object can be less than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Alternatively, the distance between the axes of the opposing rotors can be greater than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.

[0162] In some embodiments, the movable object may have a volume less than 100 cm x 100 cm x 100 cm, less than 50 cm x 50 cm x 30 cm, or less than 5 cm x 5 cm x 3 cm. The total volume of the movable object may be less than or equal to approximately: 1 cm 3 , 2cm 3 , 5cm 3 , 10cm 3 , 20cm 3 , 30cm 3 , 40cm 3 , 50cm 3 、60cm 3 , 70cm 3 , 80cm 3 , 90cm 3 , 100cm 3 , 150cm 3 , 200cm 3 , 300cm 3 , 500cm 3 , 750cm 3 , 1000cm 3 , 5000cm 3 , 10,000cm 3 , 100,000cm 3 , 1m 3 or 10m 3 Conversely, the total volume of the movable object may be greater than or equal to approximately: 1 cm 3 , 2cm 3 , 5cm 3 , 10cm 3 , 20cm 3 , 30cm 3 , 40cm 3 , 50cm 3 、60cm 3 , 70cm 3 , 80cm 3 , 90cm 3 , 100cm 3 , 150cm 3 , 200cm 3 , 300cm 3 , 500cm 3 , 750cm 3 , 1000cm 3 , 5000cm 3 , 10,000cm 3 , 100,000cm 3 , 1m 3 or 10m3 .

[0163] In some embodiments, a movable object may have a footprint (which may refer to the cross-sectional area enclosed by the movable object) that is less than or equal to approximately: 32,000 cm 2 , 20,000cm 2 , 10,000cm 2 1,000cm 2 , 500cm 2 , 100cm 2 , 50cm 2 , 10cm 2 or 5cm 2 Conversely, the footprint may be greater than or equal to approximately: 32,000 cm 2 , 20,000cm 2 , 10,000cm 2 1,000cm 2 , 500cm 2 , 100cm 2 , 50cm 2 , 10cm 2 or 5cm 2 .

[0164] In some cases, the movable object may weigh no more than 1000 kg. The movable object may weigh less than or equal to approximately: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg. Conversely, the weight may be greater than or equal to approximately: 1000kg, 750kg, 500kg, 200kg, 150kg, 100kg, 80kg, 70kg, 60kg, 50kg, 45kg, 40kg, 35kg, 30kg, 25kg, 20kg, 15kg, 12kg, 10kg, 9kg, 8kg, 7kg, 6kg, 5kg, 4kg, 3kg, 2kg, 1kg, 0.5kg, 0.1kg, 0.05kg or 0.01kg.

[0165] In some embodiments, the movable object can be small relative to the load carried by the movable object. As further described below, the load can include a payload and / or a carrier. In some examples, the ratio of the weight of the movable object to the weight of the payload can be greater than, less than, or equal to approximately 1:1. In some cases, the ratio of the weight of the movable object to the weight of the payload can be greater than, less than, or equal to approximately 1:1. Alternatively, the ratio of the weight of the carrier to the weight of the payload can be greater than, less than, or equal to approximately 1:1. When desired, the ratio of the weight of the movable object to the weight of the payload can be less than or equal to: 1:2, 1:3, 1:4, 1:5, 1:10, or even less. Conversely, the ratio of the weight of the movable object to the weight of the payload can also be greater than or equal to: 2:1, 3:1, 4:1, 5:1, 10:1, or even greater.

[0166] In some embodiments, the movable object can have low energy consumption. For example, the movable object can use less than about: 5W / h, 4W / h, 3W / h, 2W / h, 1W / h, or less. In some cases, the carrier of the movable object can have low energy consumption. For example, the carrier can use less than about: 5W / h, 4W / h, 3W / h, 2W / h, 1W / h, or less. Alternatively, the payload of the movable object can have low energy consumption, such as less than about: 5W / h, 4W / h, 3W / h, 2W / h, 1W / h, or less.

[0167] Figure 10 A UAV 1000 according to an embodiment of the present invention is illustrated. The UAV may be an example of a movable object as described herein. UAV 1000 may include a propulsion system having four rotors 1002, 1004, 1006, and 1008. Any number of rotors may be provided (e.g., one, two, three, four, five, six, or more). The rotors may be embodiments of self-tightening rotors as described elsewhere herein. The rotors, rotor assembly, or other propulsion system of the UAV may enable the UAV to hover / maintain position, change orientation, and / or change position. The distance between the axes of the opposing rotors may be any suitable length 1010. For example, length 1010 may be less than or equal to 2 meters, or less than or equal to 5 meters. In some embodiments, length 1010 may range from 40 cm to 1 meter, from 10 cm to 2 meters, or from 5 cm to 5 meters. Any description herein of a UAV may apply to a movable object, such as a different type of movable object, and vice versa.

[0168] In some embodiments, the movable object may be configured to carry a load. The load may include one or more of passengers, cargo, equipment, instruments, and the like. The load may be provided within a housing. The housing may be separate from the housing of the movable object, or may be part of the housing of the movable object. Alternatively, the load may have a housing when the movable object does not have a housing. Alternatively, a portion of the load or the entire load may not have a housing. The load may be rigidly fixed relative to the movable object. Alternatively, the load may be movable relative to the movable object (e.g., may translate or rotate relative to the movable object).

[0169] In some embodiments, the payload includes a payload. The payload may be configured not to perform any operation or function. Alternatively, the payload may be configured to perform an operation or function, also known as a functional payload. For example, the payload may include one or more sensors for surveying one or more targets. Any suitable sensor may be incorporated into the payload, such as an image capture device (e.g., a camera), an audio capture device (e.g., a parabolic microphone), an infrared imaging device, or an ultraviolet imaging device. The sensor may provide static sensing data (e.g., a photo) or dynamic sensing data (e.g., a video). In some embodiments, the sensor provides sensing data of the target of the payload. Alternatively or in combination, the payload may include one or more transmitters for providing signals to one or more targets. Any suitable transmitter may be used, such as an illumination source or a sound source. In some embodiments, the payload includes one or more transceivers, such as for communicating with a module away from a movable object. Alternatively, the payload may be configured to interact with the environment or target. For example, the payload may include a tool, instrument, or mechanism capable of manipulating an object, such as a robotic arm.

[0170] Optionally, the load may include a carrier. A carrier may be provided for the load, and the load may be coupled to the movable object via the carrier directly (e.g., directly contacting the movable object) or indirectly (e.g., not contacting the movable object). Conversely, the load may be mounted on the movable object without the need for a carrier. The load may be integrally formed with the carrier. Alternatively, the load may be detachably coupled to the carrier. In some embodiments, the load may include one or more load elements, and one or more of the load elements may be movable relative to the movable object and / or the carrier as described above.

[0171] The carrier may be integrally formed with the movable object. Alternatively, the carrier may be removably coupled to the movable object. The carrier may be coupled to the movable object directly or indirectly. The carrier may provide support to the load (e.g., carry at least a portion of the weight of the load). The carrier may include a suitable mounting structure (e.g., a pan-tilt platform) that is capable of stabilizing and / or guiding the movement of the load. In some embodiments, the carrier may be suitable for controlling the state (e.g., position and / or orientation) of the load relative to the movable object. For example, the carrier may be configured to move relative to the movable object (e.g., with respect to one, two, or three degrees of translation and / or one, two, or three degrees of rotation) so that the load maintains its position and / or orientation relative to a suitable reference frame, independent of the movement of the movable object. The reference frame may be a fixed reference frame (e.g., the surrounding environment). Alternatively, the reference frame may be a moving reference frame (e.g., the movable object, the load target).

[0172] In some embodiments, the carrier can be configured to allow movement of the payload relative to the carrier and / or the movable object. The movement can be translation with up to three degrees of freedom (e.g., along one, two, or three axes) or rotation with up to three degrees of freedom (e.g., about one, two, or three axes), or any suitable combination thereof.

[0173] In some cases, the carrier may include a carrier frame assembly and a carrier actuation assembly. The carrier frame assembly may provide structural support to the load. The carrier frame assembly may include individual carrier frame elements, some of which may be movable relative to each other. The carrier actuation assembly may include one or more actuators (e.g., motors) that actuate the movement of individual carrier frame elements. The actuators may allow multiple carrier frame elements to move simultaneously, or may be configured to allow movement of a single carrier frame element at a time. The movement of the carrier frame elements may produce corresponding movement of the load. For example, the carrier actuation assembly may actuate the rotation of one or more carrier frame elements around one or more rotational axes (e.g., roll, pitch, or yaw). The rotation of the one or more carrier frame elements may cause the load to rotate relative to the movable object around one or more rotational axes. Alternatively or in combination, the carrier actuation assembly may actuate the translation of one or more carrier frame elements along one or more translation axes, thereby producing translation of the load relative to the movable object along one or more corresponding axes.

[0174] In some embodiments, the movement of the movable object, carrier, and payload relative to a fixed reference frame (e.g., the surrounding environment) and / or relative to each other can be controlled by a terminal. The terminal can be a remote control device at a location away from the movable object, carrier, and / or payload. The terminal can be placed on or fixed to a support platform. Alternatively, the terminal can be a handheld or wearable device. For example, the terminal can include a smart phone, a tablet computer, a laptop computer, a computer, glasses, gloves, a helmet, a microphone, or a suitable combination thereof. The terminal can include a user interface such as a keyboard, a mouse, a joystick, a touch screen, or a display. Any suitable user input can be used to interact with the terminal, such as manually inputting instructions, voice control, gesture control, or position control (e.g., via movement, positioning, or tilting of the terminal).

[0175] The terminal can be used to control any suitable state of the movable object, carrier, and / or payload. For example, the terminal can be used to control the position and / or orientation of the movable object, carrier, and / or payload relative to and / or toward each other with respect to a fixed reference. In some embodiments, the terminal can be used to control individual elements of the movable object, carrier, and / or payload, such as an actuation assembly of the carrier, a sensor of the payload, or a transmitter of the payload. The terminal can include a wireless communication device suitable for communicating with one or more of the movable object, carrier, or payload.

[0176] The terminal may include a suitable display unit for viewing information about the movable object, carrier, and / or payload. For example, the terminal may be configured to display information about the movable object, carrier, and / or payload, such information relating to position, translational velocity, translational acceleration, direction, angular velocity, angular acceleration, or any suitable combination thereof. In some embodiments, the terminal may display information provided by the payload, such as data provided by the functional payload (e.g., images recorded by a camera or other image capture device).

[0177] Optionally, the same terminal can simultaneously control the movable object, carrier, and / or payload, or the state of the movable object, carrier, and / or payload, as well as receive and / or display information from the movable object, carrier, and / or payload. For example, a terminal can control the positioning of a payload relative to its environment while simultaneously displaying image data captured by the payload or information regarding its location. Alternatively, different terminals can be used for different functions. For example, a first terminal can control the movement or state of a movable object, carrier, and / or payload, while a second terminal can receive and / or display information from the movable object, carrier, and / or payload. For example, a first terminal can control the positioning of a payload relative to its environment while a second terminal displays image data captured by the payload. Multiple communication modes can be used between a movable object and an integrated terminal that both controls the movable object and receives data, or multiple communication modes can be used between the movable object and multiple terminals that both control the movable object and receive data. For example, at least two different communication modes can be established between a movable object and a terminal that both controls the movable object and receives data from the movable object.

[0178] Figure 11 The diagram illustrates a movable object 1100 including a carrier 1102 and a payload 1104, according to an embodiment. Although the movable object 1100 is depicted as an aircraft, such depiction is not limiting, and as previously described, any suitable type of movable object may be used. It will be understood by those skilled in the art that any embodiment described herein in the context of an aircraft system may be applied to any suitable movable object (e.g., a UAV). In some cases, the payload 1104 may be provided on the movable object 1100 without the carrier 1102. The movable object 1100 may include a propulsion mechanism 1106, a sensing system 1108, and a communication system 1110.

[0179] As previously described, the propulsion mechanism 1106 may include one or more of a rotor, a propeller, a blade, an engine, a motor, a wheel, a shaft, a magnet, or a nozzle. For example, the propulsion mechanism 1106 may be a self-tightening rotor, a rotor assembly, or other rotating propulsion unit as disclosed elsewhere herein. The movable object may have one or more, two or more, three or more, or four or more propulsion mechanisms. The propulsion mechanisms may all be of the same type. Alternatively, one or more propulsion mechanisms may be propulsion mechanisms of different types. The propulsion mechanism 1106 may be mounted on the movable object 1100 using any suitable device, such as a support element (e.g., a drive shaft) described elsewhere herein. The propulsion mechanism 1106 may be mounted on any suitable portion of the movable object 1100, such as the top, bottom, front, back, side, or a suitable combination thereof.

[0180] In some embodiments, the propulsion mechanism 1106 can enable the movable object 1100 to take off vertically from a surface or land vertically on a surface without any horizontal movement of the movable object 1100 (e.g., without traveling along a runway). Optionally, the propulsion mechanism 1106 can be operable to allow the movable object 1100 to hover in the air at a specified position and / or orientation. One or more propulsion mechanisms 1100 can be controlled independently of other propulsion mechanisms. Alternatively, the propulsion mechanisms 1100 can be configured to be controlled simultaneously. For example, the movable object 1100 can have multiple horizontally oriented rotors that can provide lift and / or thrust to the movable object. The multiple horizontally oriented rotors can be actuated to provide the movable object 1100 with vertical takeoff, vertical landing, and hovering capabilities. In some embodiments, one or more of the horizontally oriented rotors can rotate in a clockwise direction, while one or more of the horizontal rotors can rotate in a counterclockwise direction. For example, the number of clockwise rotors can be equal to the number of counterclockwise rotors. The rotation rate of each horizontally-oriented rotor can be independently varied to control the lift and / or thrust generated by each rotor and thereby adjust the spatial arrangement, velocity and / or acceleration of movable object 1100 (e.g., with respect to up to three translational degrees of freedom and up to three rotational degrees of freedom).

[0181] The sensing system 1108 may include one or more sensors that can sense the spatial alignment, velocity, and / or acceleration of the movable object 1100 (e.g., with respect to up to three translational degrees of freedom and up to three rotational degrees of freedom). The one or more sensors may include a global positioning system (GPS) sensor, a motion sensor, an inertial sensor, a distance sensor, or an image sensor. The sensed data provided by the sensing system 1208 can be used to control the spatial alignment, velocity, and / or direction of the movable object 1100 (e.g., using a suitable processing unit and / or control module, as described below). Alternatively, the sensing system 1108 can be used to provide data about the environment surrounding the movable object, such as weather conditions, distance to potential obstacles, location of geographic features, location of man-made structures, etc.

[0182] Communication system 1110 supports communication with a terminal 1112 having a communication system 1114 via wireless signals 1116. Communication systems 1110 and 1114 may include any number of transmitters, receivers, and / or transceivers suitable for wireless communication. The communication may be one-way communication, such that data can only be transmitted in one direction. For example, one-way communication may involve only the transmission of data from movable object 1100 to terminal 1112, or vice versa. Data may be transmitted from one or more transmitters of communication system 1110 to one or more receivers of communication system 1112, or vice versa. Alternatively, the communication may be two-way communication, such that data can be transmitted in both directions between movable object 1100 and terminal 1112. Two-way communication may involve the transmission of data from one or more transmitters of communication system 1110 to one or more receivers of communication system 1114, or vice versa.

[0183] In some embodiments, the terminal 1112 can provide control data to one or more of the movable object 1100, the carrier 1102, and the payload 1104, and receive information from one or more of the movable object 1100, the carrier 1102, and the payload 1104 (e.g., position and / or motion information of the movable object, the carrier, or the payload; data sensed by the payload, such as image data captured by the payload camera). In some cases, the control data from the terminal can include instructions for the relative position, movement, actuation, or control of the movable object, the carrier, and / or the payload. For example, the control data can result in a modification of the position and / or orientation of the movable object (e.g., via control of the propulsion mechanism 1106), or movement of the payload relative to the movable object (e.g., via control of the carrier 1102). The control data from the terminal can result in control of the payload, such as control of the operation of a camera or other image capture device (e.g., taking still or moving pictures, zooming in or out, turning on or off, switching imaging modes, changing image resolution, changing focus, changing depth of field, changing exposure time, changing viewing angle or field of view). In some cases, communications from the movable object, carrier, and / or payload may include information from one or more sensors (e.g., sensors of sensing system 1108 or payload 1104). The communications may include sensed information from one or more different types of sensors (e.g., GPS sensors, motion sensors, inertial sensors, distance sensors, or image sensors). Such information may relate to the position (e.g., location, orientation), movement, or acceleration of the movable object, carrier, and / or payload. Such information from the payload may include data captured by the payload or a sensed state of the payload. The control data provided and transmitted by terminal 1112 may be configured to control the state of one or more of movable object 1100, carrier 1102, or payload 1104. Alternatively, or in combination, carrier 1102 and payload 1104 may each include a communication module configured to communicate with terminal 1112, so that the terminal can independently communicate with and control each of movable object 1100, carrier 1102, and payload 1104.

[0184] In some embodiments, the movable object 1100 may be configured to communicate with another remote device—in addition to or in place of the terminal 1112. The terminal 1112 may also be configured to communicate with another remote device as well as the movable object 1100. For example, the movable object 1100 and / or the terminal 1112 may communicate with another movable object or the carrier or payload of another movable object. When desired, the remote device may be a second terminal or other computing device (e.g., a computer, laptop, tablet, smartphone, or other mobile device). The remote device may be configured to transmit data to the movable object 1100, receive data from the movable object 1100, transmit data to the terminal 1112, and / or receive data from the terminal 1112. Optionally, the remote device may be connected to the Internet or other telecommunications network so that data received from the movable object 1100 and / or the terminal 1112 may be uploaded to a website or server.

[0185] Figure 12 FIG2 is a block diagram illustrating a system 1200 for controlling a movable object according to an embodiment. The system 1200 can be used in conjunction with any suitable embodiment of the systems, devices, and methods disclosed herein. The system 1200 may include a sensing module 1202, a processing unit 1204, a non-transitory computer-readable medium 1206, a control module 1208, and a communication module 1210.

[0186] The sensing module 1202 can utilize different types of sensors to collect information related to the movable object in different ways. Different types of sensors can sense different types of signals or signals from different sources. For example, the sensors may include inertial sensors, GPS sensors, distance sensors (e.g., lidar), or visual / image sensors (e.g., cameras). The sensing module 1202 can be effectively coupled to a processing unit 1204 having multiple processors. In some embodiments, the sensing module can be effectively coupled to a transmission module 1212 (e.g., a Wi-Fi image transmission module), which is configured to directly transmit sensing data to a suitable external device or system. For example, the transmission module 1212 can be used to transmit an image captured by a camera of the sensing module 1202 to a remote terminal.

[0187] The processing unit 1204 may have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)). The processing unit 1204 may be operatively coupled to a non-volatile computer-readable medium 1206. The non-volatile computer-readable medium 1206 may store logic, code, and / or program instructions executable by the processing unit 1204 to perform one or more steps. The non-volatile computer-readable medium may include one or more memory units (e.g., removable media or external memory, such as an SD card or random access memory (RAM)). In some embodiments, data from the sensing module 1202 may be directly transmitted to and stored in the memory units of the non-volatile computer-readable medium 1206. The memory units of the non-volatile computer-readable medium 1206 may store logic, code, and / or program instructions executable by the processing unit 1204 to perform any suitable embodiment of the method described herein. For example, the processing unit 1204 may be configured to execute instructions, thereby causing one or more processors of the processing unit 1204 to analyze the sensed data generated by the sensing module. The memory unit can store sensing data from the sensing module to be processed by the processing unit 1204. In some embodiments, the storage unit of the non-volatile computer-readable medium 1206 can be used to store processing results generated by the processing unit 1204.

[0188] In some embodiments, the processing unit 1204 can be operatively coupled to a control module 1208 configured to control the state of the movable object. For example, the control module 1208 can be configured to control a propulsion mechanism of the movable object to adjust the spatial alignment, velocity, and / or acceleration of the movable object with respect to six degrees of freedom. Alternatively, or in combination, the control module 1208 can control one or more of the states of the carrier, payload, or sensing module.

[0189] The processing unit 1204 can be operatively coupled to a communication module 1210, which is configured to transmit and / or receive data from one or more external devices (e.g., a terminal, display device, or other remote control). Any suitable communication means can be used, such as wired or wireless communication. For example, the communication module 1210 can utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, and the like. Alternatively, a relay station such as a tower, satellite, or mobile station can be used. Wireless communication can be distance-dependent or distance-independent. In some embodiments, communication may or may not require line of sight. The communication module 1210 can transmit and / or receive one or more of sensory data from the sensing module 1202, processing results generated by the processing unit 1204, predetermined control data, user commands from a terminal or remote control, and the like.

[0190] The components of system 1200 may be arranged in any suitable configuration. For example, one or more components of system 1200 may be located on a movable object, a carrier, a payload, a terminal, a sensing system, or an additional external device that communicates with one or more of the above. Figure 12 A single processing unit 1204 and a single non-volatile computer-readable medium 1206 are depicted, but those skilled in the art will appreciate that this is not so limiting and that the system 1200 may include multiple processing units and / or non-volatile computer-readable media. In some embodiments, one or more of the multiple processing units and / or non-volatile computer-readable media may be located at different locations, such as on a movable object, a carrier, a payload, a terminal, a sensing module, an attached external device in communication with one or more of the foregoing, or a suitable combination thereof, such that any suitable aspect of the processing and / or storage functions performed by the system 1200 may occur at one or more of the foregoing locations.

[0191] A and / or B used herein includes one or more of A or B and combinations thereof, such as A and B.

[0192] Although preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will now appreciate that many modifications, variations, and substitutions can occur without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein can be employed in practicing the present invention. The following claims are intended to define the scope of the present invention and are therefore intended to cover methods, structures, and their equivalents within the scope of these claims.

Claims

1. A method for controlling a movable object within an environment, the method comprising: determining an initial position of the movable object using at least one sensor of a plurality of sensors carried by the movable object; generating a first signal to cause the movable object to navigate within the environment; receiving sensed data about the environment using the at least one sensor of the plurality of sensors; generating an environment map representing at least a portion of the environment based on the sensory data; receiving an instruction to return to the initial position; determining a current position of the movable object using at least one sensor of the plurality of sensors; determining, based on the map of the environment, a path from the current location to the initial location that avoids one or more obstacles within the environment; and generating a second signal to cause the movable object to move along the path to return to the initial position; The step of generating the second signal to cause the movable object to move along the path to return to the initial position includes: detecting an obstacle in the environment positioned along the path using the at least one sensor of the plurality of sensors; modifying the path to avoid the obstacle; and The second signal is generated to cause the movable object to move along the modified path.

2. The method of claim 1, wherein the movable object is configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area. 3 . The method of claim 1 , wherein the plurality of sensors comprises one or more of a Global Positioning System (GPS) sensor and / or a vision sensor. 4 . The method of claim 1 , wherein the plurality of sensors comprises a range sensor, wherein the range sensor comprises at least one of: a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor. The method of claim 1 , wherein determining the path comprises determining a shortest path from the current location to the initial location. The method of claim 1 , wherein the path includes one or more sections previously traveled by the movable object. The method of claim 1 , wherein the path is different from a path previously traveled by the movable object. The method of claim 1 , wherein the path is a flight path of the movable object.

9. The method of claim 1, wherein the path comprises a spatial position and orientation of the movable object.

10. The method of claim 1, wherein the path includes a plurality of waypoints corresponding to locations previously traveled by the movable object, the plurality of waypoints being recorded as the movable object navigates within the environment.

11. A method for controlling a movable object within an environment, the method comprising: determining an initial position of the movable object using at least one sensor of a plurality of sensors carried by the movable object; generating a first signal to cause the movable object to navigate within the environment; receiving sensed data about the environment using the at least one sensor of the plurality of sensors; receiving an instruction to return to the initial position; determining a current position of the movable object using at least one sensor of the plurality of sensors; determining, based on a map of the environment, a path from the current location to the initial location that avoids one or more obstacles within the environment; and generating a second signal to cause the movable object to return along the path to the initial position, wherein when at least one of the plurality of sensors detects an obstacle located along the path, the path is modified to avoid the obstacle, and the second signal is generated to cause the movable object to move along the modified path.

12. A system for controlling an unmanned aerial vehicle within an environment, the system comprising: a plurality of sensors carried by the UAV; as well as One or more processors, individually or collectively configured to: generating a first signal to cause the UAV to navigate within the environment; receiving sensory data regarding at least a portion of the environment using a plurality of sensors carried by the UAV; generating an environment map representing the at least a portion of the environment based on the sensory data; Receive instructions to return to the initial position; determining a current position of the UAV using at least one sensor from the plurality of sensors; determining, based on the map of the environment, a path from the current location to the initial location that avoids one or more obstacles within the environment; and generating a second signal to cause the UAV to move along the path to return to the initial position; wherein the one or more processors are individually or collectively configured to: detecting an obstacle in the environment positioned along the path using the at least one sensor of the plurality of sensors; modifying the path to avoid the obstacle; and The second signal is generated to cause the UAV to move along the modified path.

13. A method for controlling an unmanned aerial vehicle within an environment, the method comprising: determining an initial position of the UAV using at least one sensor from a plurality of sensors carried by the UAV; generating a first signal to cause the UAV to navigate within the environment; receiving sensory data about the environment using the plurality of sensors to form a map of the environment; The environment map includes a first environment map generated based on first sensing data and a second environment map generated based on second sensing data, wherein the first sensing data is acquired by a first sensor and the second sensing data is acquired by a second sensor; receiving an instruction to return to the initial position; as well as generating a second signal to cause the UAV to return along a path to the initial position, wherein when the sensory data identifies an obstacle in the environment along the path, the path is modified to avoid the obstacle; The step of generating the second signal to enable the UAV to return to the initial position along the path includes: detecting an obstacle in the environment positioned along the path using the at least one sensor of the plurality of sensors; modifying the path to avoid the obstacle; and The second signal is generated to cause the UAV to move along the modified path.

14. The method of claim 13, wherein the UAV is configured to detect the presence of a restricted flight area and not fly within a predetermined distance of the restricted flight area.

15. The method of claim 13, wherein the plurality of sensors comprises a global positioning system (GPS) sensor; and / or, the plurality of sensors comprises a vision sensor; and / or, the plurality of sensors comprises a distance sensor.

16. The method of claim 13, wherein the path includes one or more segments previously traveled by the UAV.

17. The method of claim 13, wherein the path is different from a path previously traveled by the UAV.

18. The method of claim 13, wherein the path includes a spatial position and orientation of the UAV.

Images