Method for controlling unmanned aerial vehicle to avoid obstacles

Abstract

A computer-implemented method (1000) includes receiving (1005), by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV), where one or more pixels of the depth image are associated with corresponding depth values indicative of a distance of one or more objects to the stereo camera. The image processing system determines (1010) that one or more pixels of the depth image are associated with invalid depth values. The image processing system infers (1015) the presence of a potential obstacle in the environment of the UAV based on a distribution of one or more pixels of the depth image associated with invalid depth values. The UAV is controlled (1020) based on the inferred presence of the potential obstacle.

Description

Background Art

[0001] This application claims priority to U.S. Patent Application No. 17 / 710,503, filed on March 31, 2022, the entire contents of which are incorporated herein by reference.

[0002] An unmanned vehicle (which may also be referred to as an autonomous vehicle) is a vehicle that is capable of traveling without an on-site human operator. An unmanned vehicle may operate in a remote control mode, an autonomous mode, or a partially autonomous mode.

[0003] When an unmanned vehicle is operated in a remote control mode, a pilot or driver at a remote location can control the unmanned vehicle by sending commands to the unmanned vehicle via a wireless link. When an unmanned vehicle is operated in an autonomous mode, the unmanned vehicle is usually moved based on pre-programmed navigation waypoints, a dynamic automation system, or a combination of these. In addition, some unmanned vehicles can operate in both remote control mode and autonomous mode, and in some instances can do so simultaneously. For example, a remote pilot or driver may wish to leave navigation to an autonomous system while manually performing another task, such as, for example, operating a mechanical system for picking up an object.

[0004] There are various types of unmanned vehicles for various different environments. For example, there are unmanned vehicles for operation in the air, on the ground, underwater, and in space. Examples include quadcopters and stand-up landing UAVs, among others. There are also unmanned vehicles for hybrid operations where multi-environment operations are possible. Examples of hybrid unmanned vehicles include amphibians that can operate on land as well as on water or floatplanes that can land on water as well as on land. Other examples are also possible. Summary of the invention

[0005] Various examples of UAVs and support systems are disclosed herein that collaborate to facilitate avoiding obstacles in a delivery area or other environment of the UAV. In particular, various examples herein utilize stereo / depth cameras that help determine the distance between the UAV and various objects in the delivery area. When the depth of an object cannot be determined, the stereo camera reports invalid depth data. Invalid depth data helps identify objects that may interfere with the operation of the UAV.

[0006] In a first aspect, a computer-implemented method includes receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV). One or more pixels of the depth image are associated with corresponding depth values ​​indicating a distance of one or more objects to the stereo camera. The image processing system determines that one or more pixels of the depth image are associated with an invalid depth value. The image processing system infers the presence of a potential obstacle in an environment of the UAV based on a distribution of one or more pixels of the depth image associated with the invalid depth value. The UAV is controlled based on the inferred presence of the potential obstacle.

[0007] In a second aspect, an unmanned aerial vehicle (UAV) includes a camera and a control system. The control system is configured to perform operations including receiving, by an image processing system, a depth image captured by a stereo camera on the unmanned aerial vehicle (UAV). One or more pixels of the depth image are associated with corresponding depth values ​​indicating the distance of one or more objects to the stereo camera. The control system determines that one or more pixels of the depth image are associated with invalid depth values. The control system infers the presence of potential obstacles in the environment of the UAV based on the distribution of one or more pixels of the depth image associated with the invalid depth values. The control system controls the UAV based on the inferred presence of the potential obstacles.

[0008] In a third aspect, a non-transitory computer-readable medium includes receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV). One or more pixels of the depth image are associated with corresponding depth values ​​indicating a distance of one or more objects to the stereo camera. The image processing system determines that one or more pixels of the depth image are associated with invalid depth values. The image processing system infers the presence of a potential obstacle in an environment of the UAV based on a distribution of one or more pixels of the depth image associated with the invalid depth values. The UAV is controlled based on the inferred presence of the potential obstacle.

[0009] These and other aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with appropriate reference to the accompanying drawings. In addition, it should be understood that the descriptions provided in the Summary of the Invention section and elsewhere in this document are intended to illustrate the claimed subject matter by way of example and not limitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a simplified illustration of a drone in accordance with an example embodiment.

[0011] Figure 1B is a simplified illustration of a drone in accordance with an example embodiment.

[0012] Figure 1Cis a simplified illustration of a drone in accordance with an example embodiment.

[0013] Figure 1D is a simplified illustration of a drone in accordance with an example embodiment.

[0014] Figure 1E is a simplified illustration of a drone in accordance with an example embodiment.

[0015] Figure 2 is a simplified block diagram illustrating components of a drone according to an example embodiment.

[0016] Figure 3 is a simplified block diagram illustrating a UAV system according to an example embodiment.

[0017] Figure 4 Operations performed by one or more of the devices described herein are illustrated according to example embodiments.

[0018] Figure 5 An example of a depth image according to example embodiments is shown.

[0019] Figure 6 An example of a portion of a depth mask according to example embodiments is shown.

[0020] Figure 7 An example of an invalid pixel mask according to example embodiments is shown.

[0021] Figure 8 An example of obstacle detection logic according to an example embodiment is shown.

[0022] Fig. 9 An example of an adjusted delivery zone is shown according to an example embodiment.

[0023] Fig.10 Operations performed by one or more of the devices described herein are shown according to example embodiments. DETAILED DESCRIPTION

[0024] Exemplary methods and systems are described herein. It should be understood that the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation or feature described herein as "exemplary" or "illustrative" is not necessarily to be construed as preferred or advantageous over other implementations or features. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. The example implementations described herein are not meant to be limiting. It will be readily understood that the various aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0025] I. Overview

[0026] As described above, some UAVs include stereo cameras that can be used to determine the distance to objects in the delivery area. However, it is not always possible to determine the distance to some objects such as power lines. Such objects may pose a hazard to the UAV, especially when the UAV descends to deliver a payload. This problem and other problems are improved by various examples of UAVs described herein.

[0027] In particular, some examples of UAVs include an image processing system configured to receive a depth image captured by an onboard stereo camera. One or more pixels of the depth image are associated with corresponding depth values ​​indicating the distance of one or more objects to the stereo camera. The image processing system determines pixels of the depth image associated with invalid depth values, and infers the presence of potential obstacles in the environment of the UAV based on the distribution of the pixels. The UAV is then controlled (e.g., to avoid the obstacle) based on the inferred presence of the potential obstacle.

[0028] Some examples of image processing systems are configured to generate an invalid pixel mask associated with a depth image. Pixels in the invalid pixel mask associated with pixels having corresponding depth values ​​in the depth image are set to a first value, and pixels associated with pixels not having corresponding depth values ​​in the depth image are set to a second value different from the first value. In some examples, the invalid pixel mask is evaluated to identify elongated objects that define lines (e.g., power lines) in the environment. In some examples, this is achieved via a Hough transform.

[0029] In some examples, the above operations are not performed until the UAV descends to a first predetermined height above ground level. In addition, in some examples, the operations stop when the UAV descends to a second / lower predetermined height above ground level.

[0030] II. Example Unmanned Vehicles

[0031] As used herein, the terms "drone" and "UAV" refer to any autonomous or semi-autonomous vehicle capable of performing some function without the physical presence of a human pilot.

[0032] UAVs can take various forms. For example, UAVs can take the form of fixed-wing aircraft, gliders, stand-up aircraft, jet aircraft, ducted fan aircraft, lighter-than-air airships (such as blimps or steerable balloons), rotorcraft (such as helicopters or multirotors), and / or flapping-wing aircraft, among other possibilities. In addition, the terms "drone," "unmanned aerial vehicle system" (UAVS), or "unmanned aerial system" (UAS) may also be used to refer to UAVs.

[0033] Figure 1A 1 is an isometric view of an example UAV 100. The UAV 100 includes wings 102, booms 104, and a fuselage 106. The wings 102 can be stationary and can generate lift based on the wing shape and the forward airspeed of the UAV. For example, the two wings 102 can have an airfoil-shaped cross-section to generate aerodynamic forces on the UAV 100. In some embodiments, the wings 102 can carry horizontal propulsion units 108, and the booms 104 can carry vertical propulsion units 110. In operation, power for the propulsion units can be provided from a battery bay 112 of the fuselage 106. In some embodiments, the fuselage 106 also includes a delivery unit (not shown, such as a winch system) for handling payloads, such as an avionics bay 114, an additional battery bay (not shown), and / or a payload. In some embodiments, the fuselage 106 is modular, and two or more bays (e.g., battery bay 112, avionics bay 114, other payload and delivery bays) are detachable from one another and can be secured to one another (e.g., mechanically, magnetically, or otherwise) to continuously form at least a portion of the fuselage 106.

[0034] In some embodiments, boom 104 terminates in rudder 116 for improved yaw control of UAV 100. Additionally, wings 102 may terminate in wingtips 117 for improved control of lift of the UAV.

[0035] In the illustrated configuration, the UAV 100 includes a structural frame. The structural frame may be referred to as a "structural H-frame" or "H-frame" (not shown) of the UAV. The H-frame may include a wing spar (not shown) within the wing 102 and a boom bracket (not shown) within the boom 104. In some embodiments, the wing spar and boom bracket may be made of carbon fiber, hard plastic, aluminum, light metal alloy, or other material. The wing spar and boom bracket may be connected with a clamp. The wing spar may include pre-drilled holes for the horizontal propulsion unit 108, and the boom bracket may include pre-drilled holes for the vertical propulsion unit 110.

[0036] In some embodiments, the fuselage 106 can be removably attached to the H-frame (e.g., attached to the wing spar by a clamp, configured with grooves, protrusions or other features to cooperate with corresponding H-frame features, etc.). In other embodiments, the fuselage 106 can similarly be removably attached to the wing 102. The removable attachment of the fuselage 106 can improve the quality and / or modularity of the UAV 100. For example, the electrical / mechanical components and / or subsystems of the fuselage 106 can be tested separately from the H-frame before being attached to the H-frame. Similarly, the printed circuit boards (PCBs) 118 can be tested separately from the boom bracket before being attached to the boom bracket, thus eliminating defective components / subassemblies before completing the UAV. For example, components of the fuselage 106 (e.g., avionics, battery units, delivery units, additional battery compartments, etc.) can be electrically tested before the fuselage 106 is installed on the H-frame. In addition, the motors and electronic devices of the PCB 118 can also be electrically tested before final assembly. Typically, identifying defective parts and subassemblies early in the assembly process reduces the overall cost and delivery time of the UAV. In addition, different types / models of fuselages 106 can be attached to the H-frame, thus improving the modularity of the design. This modularity allows these various parts of the UAV 100 to be upgraded without a substantial overhaul of the manufacturing process.

[0037] In some embodiments, the wing shells and boom shells can be attached to the H-frame by adhesive elements (e.g., adhesive tape, double-sided adhesive tape, glue, etc.). Thus, multiple shells can be attached to the H-frame instead of having a monolithic body sprayed onto the H-frame. In some embodiments, the presence of multiple shells reduces stress caused by the coefficient of thermal expansion of the structural frame of the UAV. As a result, the UAV can have better dimensional accuracy and / or improved reliability.

[0038] Furthermore, in at least some embodiments, the same H-frame can be used with wing shells and / or boom shells of different sizes and / or designs, thereby increasing the modularity and versatility of the UAV design. The wing shells and / or boom shells can be made of a relatively light polymer (e.g., closed-cell foam) covered by a harder but relatively thin plastic skin.

[0039] Power and / or control signals from the fuselage 106 can be routed to the PCB 118 via cables passing through the fuselage 106, the wings 102, and the boom 104. In the illustrated embodiment, the UAV 100 has four PCBs, but other numbers of PCBs are possible. For example, the UAV 100 can include two PCBs, one for each boom. The PCBs carry electronic components 119, including, for example, power converters, controllers, memory, passive components, etc. In operation, the propulsion units 108 and 110 of the UAV 100 are electrically connected to the PCBs.

[0040] Many variations of the UAV shown are possible. For example, a fixed-wing UAV may include more or fewer rotor units (vertical or horizontal), and / or may utilize a ducted fan or multiple ducted fans for propulsion. In addition, a UAV with more wings (e.g., an "x-wing" configuration with four wings) is also possible. Although FIG. 1 shows two wings 102, two booms 104, two horizontal propulsion units 108, and six vertical propulsion units 110 per boom 104, it should be understood that other variations of UAV 100 may be implemented with more or less of these components. For example, UAV 100 may include four wings 102, four booms 104, and more or fewer propulsion units (horizontal or vertical).

[0041] Similarly, Figure 1B Another example of a fixed-wing UAV 120 is shown. The fixed-wing UAV 120 includes a fuselage 122, two wings 124 with airfoil cross-sections that provide lift for the UAV 120, vertical stabilizers 126 (or fins) that stabilize the aircraft in yaw (turning left or right), horizontal stabilizers 128 (also called elevators or horizontal tails) that stabilize pitch (tilting up or down), landing gear 130, and a propulsion unit 132, which may include a motor, shaft, and propeller.

[0042] Figure 1C An example of a UAV 140 having propellers in a pusher configuration is shown. The term "pusher" refers to the fact that the propulsion unit 142 is mounted at the rear of the UAV and "pushes" the vehicle forward, as opposed to a propulsion unit mounted at the front of the UAV. Figure 1A and 1B Provide a description, Figure 1C A common structure used in a pusher aircraft is depicted, including a fuselage 144, two wings 146, a vertical stabilizer 148, and a propulsion unit 142, which may include a motor, shaft, and propeller.

[0043] Figure 1DAn example of a vertical take-off and landing UAV 160 is shown. In the example shown, the vertical take-off and landing UAV 160 has fixed wings 162 to provide lift and allow the UAV 160 to glide horizontally (e.g., along the x-axis, approximately perpendicular to the Figure 1D However, fixed wings 162 also allow stand-up / landing UAV 160 to take off and land vertically on its own.

[0044] For example, at a launch site, a vertical take-off and landing UAV 160 can be positioned vertically (as shown), with its fins 164 and / or wings 162 resting on the ground and stabilizing the UAV 160 in a vertical position. Then, the vertical take-off and landing UAV 160 can take off by operating its propellers 166 to generate an upward thrust (e.g., thrust roughly along the y-axis). Once at a suitable height, the vertical take-off and landing UAV 160 can use its flaps 168 to redirect itself in a horizontal position so that its fuselage 170 is closer to being aligned with the x-axis rather than aligned with the y-axis. The horizontally positioned propellers 166 can provide forward thrust so that the vertical take-off and landing UAV 160 can fly in a manner similar to a typical aircraft.

[0045] Many variations of the fixed-wing UAV shown are possible. For example, the fixed-wing UAV may include more or fewer propellers, and / or may utilize a ducted fan or multiple ducted fans for propulsion. In addition, UAVs with more wings (e.g., an "x-wing" configuration with four wings), fewer wings, or even no wings are also possible.

[0046] As described above, some embodiments may involve other types of UAVs in addition to or in lieu of fixed-wing UAVs. For example, Figure 1E An example of a rotorcraft commonly referred to as a multirotor 180 is shown. Multirotor 180 may also be referred to as a quadrotor because it includes four rotors 182. It should be understood that example embodiments may relate to rotorcraft having more or fewer rotors than multirotor 180. For example, a helicopter typically has two rotors. Other examples with three or more rotors are also possible. As used herein, the term "multirotor" refers to any rotorcraft having more than two rotors, and the term "helicopter" refers to a rotorcraft having two rotors.

[0047] Referring to multirotor 180 in more detail, four rotors 182 provide propulsion and maneuverability for multirotor 180. More specifically, each rotor 182 includes a blade attached to a motor 184. So configured, rotors 182 can allow multirotor 180 to take off and land vertically, maneuver in any direction, and / or hover. In addition, the pitch of the blades can be adjusted as a group and / or adjusted differentially, and can allow multirotor 180 to control its pitch, roll, yaw, and / or altitude.

[0048] It should be understood that references to "unmanned" aerial vehicles or UAVs herein may apply equally to autonomous and semi-autonomous aerial vehicles. In autonomous implementations, all functions of the aerial vehicle are automated; for example, pre-programmed or controlled via real-time computer functions that respond to inputs and / or predetermined information from various sensors. In semi-autonomous implementations, some functions of the aerial vehicle may be controlled by a human operator, while other functions are performed autonomously. In addition, in some embodiments, the UAV may be configured to allow a remote operator to take over functions that would otherwise be autonomously controlled by the UAV. In addition, a given type of function may be remotely controlled at one level of abstraction and autonomously performed at another level of abstraction. For example, a remote operator may control high-level navigation decisions of the UAV, such as by specifying that the UAV should travel from one location to another (e.g., from a warehouse in the suburbs to a delivery address in a nearby city), while the UAV's navigation system autonomously controls more fine-grained navigation decisions, such as a specific route taken between two locations, specific flight controls to achieve the route and avoid obstacles while navigating the route, and the like.

[0049] More generally, it should be understood that the example UAVs described herein are not intended to be limiting. Example embodiments may be directed to, implemented within, or take the form of any type of unmanned aerial vehicle.

[0050] III. Illustrative UAV Components

[0051] Figure 2 2 is a simplified block diagram showing components of UAV 200 according to an example embodiment. UAV 200 may take the form of a reference Figures 1A-1E The UAV 200 may be in the form of one of the UAVs 100, 120, 140, 160, and 180, or a form similar thereto. However, the UAV 200 may also take other forms.

[0052] UAV 200 may include various types of sensors and may include a computing system configured to provide the functionality described herein. In the illustrated embodiment, the sensors of UAV 200 include an inertial measurement unit (IMU) 202, an ultrasonic sensor 204, and a GPS 206, among other possible sensors and sensing systems.

[0053] In the illustrated embodiment, the UAV 200 also includes one or more processors 208. The processor 208 may be a general purpose processor or a special purpose processor (e.g., a digital signal processor, an application specific integrated circuit, etc.). The one or more processors 208 may be configured to execute computer readable program instructions 212 stored in the data memory 210 and executable to provide the functionality of the UAV described herein.

[0054] The data storage 210 may include or take the form of one or more computer-readable storage media that may be read or accessed by the at least one processor 208. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other storage or disk storage, which may be integrated in whole or in part with at least one of the one or more processors 208. In some embodiments, the data storage 210 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other storage or disk storage unit), while in other embodiments, the data storage 210 may be implemented using two or more physical devices.

[0055] As described above, the data storage 210 may include computer-readable program instructions 212 and possibly additional data, such as diagnostic data of the UAV 200. Thus, the data storage 210 may include program instructions 212 to perform or facilitate some or all of the UAV functions described herein. For example, in the illustrated embodiment, the program instructions 212 include a navigation module 214 and a cable control module 216.

[0056] A. Sensor

[0057] In an illustrative embodiment, the IMU 202 may include both an accelerometer and a gyroscope, which may be used together to determine the orientation of the UAV 200. In particular, the accelerometer may measure the orientation of the vehicle relative to the earth, while the gyroscope measures the rate of rotation about an axis. IMUs are commercially available in low-cost, low-power packages. For example, the IMU 202 may take the form of or include a micro-electromechanical system (MEMS) or a nano-electromechanical system (NEMS). Other types of IMUs may also be utilized.

[0058] In addition to the accelerometer and gyroscope, the IMU 202 may also include other sensors that may help better determine the position and / or help increase the autonomy of the UAV 200. Two examples of such sensors are magnetometers and pressure sensors. In some embodiments, the UAV may include a low-power digital 3-axis magnetometer that may be used to implement an orientation-independent electronic compass to obtain accurate heading information. However, other types of magnetometers may also be utilized. Other examples are also possible. In addition, note that the UAV may include some or all of the above-mentioned inertial sensors as components separate from the IMU.

[0059] UAV 200 may also include a pressure sensor or barometer, which may be used to determine the altitude of UAV 200. Alternatively, other sensors such as a sonic altimeter or a radar altimeter may be used to provide an indication of altitude, which may help improve the accuracy of the IMU and / or prevent drift of the IMU.

[0060] On the other hand, UAV 200 may include one or more sensors that allow the UAV to sense objects in the environment. For example, in the illustrated embodiment, UAV 200 includes an ultrasonic sensor 204. Ultrasonic sensor 204 can determine the distance to an object by generating sound waves and determining the time interval between the transmission of the wave and the reception of the corresponding echo from the object. Typical applications of ultrasonic sensors for unmanned vehicles or IMUs are low-altitude height control and obstacle avoidance. Ultrasonic sensors can also be used for vehicles that need to hover at a certain height or need to be able to detect obstacles. Other systems can be used to determine nearby objects, sense the presence of nearby objects, and / or determine the distance to nearby objects, such as light detection and ranging (LIDAR) systems, laser detection and ranging (LADAR) systems, and / or infrared or forward-looking infrared (FLIR) systems, as well as other possibilities.

[0061] In some embodiments, UAV 200 may also include one or more imaging systems. For example, UAV 200 may utilize one or more still cameras and / or video cameras to capture image data from the UAV's environment. As specific examples, a charge-coupled device (CCD) camera or a complementary metal-oxide-semiconductor (CMOS) camera may be used with an unmanned vehicle. Such imaging sensors have many possible applications, such as obstacle avoidance, positioning techniques, ground tracking for more accurate navigation (e.g., by applying optical flow techniques to images), video feedback, and / or image recognition and processing, among others.

[0062] The UAV 200 may also include a GPS receiver 206. The GPS receiver 206 may be configured to provide typical data of the well-known GPS system, such as the GPS coordinates of the UAV 200. Such GPS data may be used by the UAV 200 for various functions. Thus, the UAV may use its GPS receiver 206 to help navigate to the location of the caller, as indicated at least in part by the GPS coordinates provided by their mobile device. Other examples are also possible.

[0063] B. Navigation and Position Determination

[0064] The navigation module 214 may provide functionality that allows the UAV 200 to, for example, move within its environment and reach a desired location. To this end, the navigation module 214 may control the altitude and / or direction of flight by controlling the mechanical features of the UAV that affect flight (e.g., its rudder, elevators, ailerons, and / or the speed of its propellers).

[0065] In order to navigate the UAV 200 to the target location, the navigation module 214 can implement various navigation techniques, such as map-based navigation and positioning-based navigation. With map-based navigation, a map of its environment can be provided to the UAV 200, which can then be used to navigate to a specific location on the map. With positioning-based navigation, the UAV 200 can use localization to navigate in an unknown environment. Navigation based on positioning can involve the UAV 200 building its own environmental map and calculating its position within the map and / or the position of the object in the environment. For example, when the UAV 200 moves throughout its environment, the UAV 200 can continuously use positioning to update its environmental map. This continuous mapping process can be referred to as simultaneous localization and mapping (SLAM). Other navigation techniques can also be used.

[0066] In some embodiments, the navigation module 214 can use a technique that relies on waypoints to navigate. In particular, a waypoint is a set of coordinates that identifies a point in physical space. For example, an air navigation waypoint can be defined by a certain latitude, longitude, and altitude. Therefore, the navigation module 214 can cause the UAV 200 to move from waypoint to waypoint in order to finally travel to a final destination (e.g., the final waypoint in a sequence of waypoints).

[0067] In another aspect, the navigation module 214 and / or other components and systems of the UAV 200 may be configured for a scenario of "localization" to more precisely navigate to a target location. More specifically, in some cases, it may be desirable for the UAV to be within a threshold distance of a target location to which the UAV is delivering a payload 228 (e.g., within a few feet of the target destination). To this end, the UAV may use a two-tier approach, in which the UAV uses a more general location determination technique to navigate to a general area associated with the target location, and then uses a more refined location determination technique to identify and / or navigate to the target location within the general area.

[0068] For example, the UAV 200 can use waypoint and / or map-based navigation to navigate to the general area of ​​the target destination where the payload 228 is being delivered. The UAV can then switch to a mode in which it uses a positioning process to locate and travel to a more specific location. For example, if the UAV 200 is to deliver a payload to a user's home, the UAV 200 may need to be substantially close to the target location in order to avoid delivering the payload to an undesirable area (e.g., onto a rooftop, into a swimming pool, onto a neighbor's property, etc.). However, a GPS signal may only allow the UAV 200 to reach such a distance (e.g., within the block of the user's home). A more precise location determination technique can then be used to find the specific target location.

[0069] Once the UAV 200 has navigated to the general area of ​​the target delivery location, various types of position determination techniques may be used to achieve localization of the target delivery location. For example, the UAV 200 may be equipped with one or more sensor systems, such as ultrasonic sensors 204, infrared sensors (not shown), and / or other sensors, which may provide inputs that the navigation module 214 uses to autonomously or semi-autonomously navigate to a specific target location.

[0070] As another example, once the UAV 200 reaches the general area of ​​a target delivery location (or a mobile object such as a person or their mobile device), the UAV 200 may switch to a "fly-by-wire" mode, in which the UAV 200 is at least partially controlled by a remote operator who may navigate the UAV 200 to a specific target location. To this end, sensory data from the UAV 200 may be sent to the remote operator to assist them in navigating the UAV 200 to a specific location.

[0071] As yet another example, the UAV 200 may include a module capable of signaling to passersby for assistance in reaching a particular target delivery location; for example, the UAV 200 may display a visual message in a graphical display requesting such assistance, play an audio message or tone through a speaker to indicate that such assistance is needed, among other possibilities. Such a visual or audio message may indicate that assistance is needed in delivering the UAV 200 to a particular person or a particular location, and may provide information to assist passersby in delivering the UAV 200 to the person or location (e.g., a description or picture of the person or location, and / or the name of the person or location), among other possibilities. Such a feature may be useful in scenarios where the UAV cannot use sensing functionality or another location determination technology to reach a specific target location. However, the feature is not limited to such scenarios.

[0072] In some embodiments, once UAV 200 arrives at the approximate area of ​​the target delivery location, UAV 200 can use a beacon from a user's remote device (e.g., a user's mobile phone) to locate the person. Such a beacon can take various forms. As an example, consider a scenario in which a remote device (such as a mobile phone of a person requesting UAV delivery) is able to send a directional signal (e.g., via an RF signal, an optical signal, and / or an audio signal). In this scenario, UAV 200 can be configured to navigate by "tracing back" such a directional signal, in other words, by determining where the signal is strongest and navigating accordingly. As another example, a mobile device can transmit a frequency within or outside the human range, and UAV 200 can listen to the frequency and navigate accordingly. As a related example, if UAV 200 is listening for verbal commands, UAV 200 can use verbal statements, such as "I'm here!" to trace the specific location of the person requesting the delivery of the payload.

[0073] In an alternative arrangement, the navigation module may be implemented at a remote computing device that is in wireless communication with the UAV 200. The remote computing device may receive data indicating the operational state of the UAV 200, sensor data from the UAV 200 that allows it to assess the environmental conditions that the UAV 200 is experiencing, and / or location information of the UAV 200. Provided with such information, the remote computing device may determine the latitude and / or directional adjustments that the UAV 200 should make and / or may determine how the UAV 200 should adjust its mechanical features (e.g., the speed of its rudder, elevator, aileron, and / or its propeller) in order to achieve such movement. The remote computing system may then transmit such adjustments to the UAV 200 so that it may move in the determined manner.

[0074] C. Communication system

[0075] On the other hand, UAV 200 includes one or more communication systems 218. Communication system 218 may include one or more wireless interfaces and / or one or more wired interfaces that allow UAV 200 to communicate via one or more networks. Such wireless interfaces may ensure communication under one or more wireless communication protocols, such as Bluetooth, WiFi (e.g., IEEE 802.11 protocol), Long-Term Evolution (LTE), WiMAX (e.g., IEEE802.16 standard), radio-frequency ID (RFID) protocol, near-field communication (NFC) and / or other wireless communication protocols. Such wired interfaces may include Ethernet interfaces, Universal Serial Bus (USB) interfaces or similar interfaces to communicate via wires, twisted pairs, coaxial cables, optical links, fiber optic links or other physical connections to a wired network.

[0076] In some embodiments, UAV 200 may include a communication system 218 that allows for both short-range and long-range communications. For example, UAV 200 may be configured for short-range communications using Bluetooth and long-range communications under a CDMA protocol. In such an embodiment, UAV 200 may be configured to function as a "hotspot"; or in other words, as a gateway or proxy between a remote support device and one or more data networks (such as a cellular network and / or the Internet). So configured, UAV 200 may facilitate data communications that a remote support device would not otherwise be able to perform on its own.

[0077] For example, UAV 200 can provide a WiFi connection to a remote device and act as a proxy or gateway to a cellular service provider's data network, such as to which the UAV can connect under LTE or 3G protocols. UAV 200 can also act as a proxy or gateway to high altitude balloon networks, satellite networks, or a combination of these networks, etc., which the remote device may not otherwise be able to access.

[0078] D. Power system

[0079] In another aspect, UAV 200 may include power system 220. Power system 220 may include one or more batteries for providing power to UAV 200. In one example, one or more batteries may be rechargeable, and each battery may be recharged via a wired connection between the battery and a power source and / or via a wireless charging system (such as an inductive charging system that applies an external time-varying magnetic field to an internal battery).

[0080] E. Payload Delivery

[0081] UAV 200 can adopt various systems and configurations in order to transport and deliver payload 228. In some implementations, the payload 228 of a given UAV 200 may include or take the form of a "package" designed to transport various goods to a target delivery location. For example, UAV 200 may include a cabin in which one or more items may be transported. Such a package may include one or more foods, purchased goods, medical items, or any other object of a size and weight suitable for transportation between two locations by a UAV. In other embodiments, payload 228 may simply be one or more items being delivered (e.g., without any packaging to hold these items).

[0082] In some embodiments, the payload 228 may be attached to the UAV and substantially located outside the UAV during some or all of the flight of the UAV. For example, the package may be tethered or otherwise releasably attached below the UAV during flight to a target location. In some embodiments, the package may include various features that protect its contents from environmental influences, reduce aerodynamic drag on the system, and prevent the contents of the package from shifting during flight of the UAV. In other embodiments, the package may be a standard shipping package that is not specifically customized for UAV flight.

[0083] To deliver a payload, the UAV may include a winch system 221 controlled by a cable control module 216 to lower a payload 228 to the ground while the UAV is hovering above. Figure 2 As shown, winch system 221 may include cable 224, and cable 224 may be coupled to payload 228 via payload retriever 226. Cable 224 may be wound around a spool of motor 222 coupled to UAV. Motor 222 may take the form of a DC motor (e.g., servomotor), which may be actively controlled by a speed controller. Cable control module 216 may control a speed controller to rotate the spool with motor 222, thereby unfolding or retracting cable 224 and lowering or raising payload retriever 226. In practice, a speed controller may output a desired operating rate (e.g., desired RPM) of the spool, which may correspond to a speed at which cable 224 and payload 228 should be lowered toward the ground. Then, motor 222 may rotate the spool so that it maintains a desired operating rate.

[0084] In order to control the motor 222 via the speed controller, the cable control module 216 can receive data from a speed sensor (e.g., encoder) that is configured to convert a mechanical position into a representative analog or digital signal. In particular, the speed sensor can include a rotary encoder that can provide information and other possibilities related to the rotational position (and / or rotational movement) of the shaft of the motor or the bobbin connected to the motor. In addition, the speed sensor can take the form of an absolute encoder and / or an incremental encoder, etc. Therefore, in an example embodiment, when the motor 222 causes the rotation of the bobbin, a rotary encoder can be used to measure the rotation. In doing so, the rotary encoder can be used to convert the rotational position into an analog or digital electronic signal used by the cable control module 216 to determine the amount of rotation of the bobbin from a fixed reference angle and / or to an analog or digital electronic signal representing a new rotational position, as well as other options. Other examples are also possible.

[0085] Based on the data from the speed sensor, the cable control module 216 can determine the rotational speed of the motor 222 and / or the spool and responsively control the motor 222 (e.g., by increasing or decreasing the current provided to the motor 222) to match the rotational speed of the motor 222 to the desired speed. When adjusting the motor current, the magnitude of the current adjustment can be based on a proportional-integral-derivative (PID) calculation using the determined and desired speeds of the motor 222. For example, the magnitude of the current adjustment can be based on the current difference between the determined speed of the spool and the desired speed, the past difference (based on the accumulated error over time), and the future difference (based on the current rate of change).

[0086] In some embodiments, the cable control module 216 can change the rate at which the cable 224 and the payload 228 are lowered to the ground. For example, the speed controller can change the desired operating rate according to the variable deployment rate curve and / or in response to other factors, so as to change the rate at which the payload 228 descends toward the ground. To this end, the cable control module 216 can adjust the amount of braking or friction applied to the cable 224. For example, in order to change the cable deployment rate, the UAV 200 may include a friction pad that can apply a variable amount of pressure to the cable 224. As another example, the UAV 200 may include a motorized braking system that changes the rate at which the spool releases the cable 224. This braking system can take the form of an electromechanical system, in which the motor 222 operates to slow down the rate at which the spool releases the cable 224. In addition, the motor 222 can change the amount of its speed (e.g., RPM) of the regulating spool, so the deployment rate of the cable 224 can be changed. Other examples are also possible.

[0087] In some embodiments, the cable control module 216 can be configured to limit the motor current supplied to the motor 222 to a maximum value. Due to such a limit imposed on the motor current, there may be a situation where the motor 222 cannot operate at the desired operation specified by the speed controller. For example, as discussed in more detail below, there may be a situation where the speed controller specifies the desired operating rate at which the motor 222 should retract the cable 224 toward the UAV 200, but the motor current may be limited so that a sufficiently large downward force on the cable 224 will offset the retraction force of the motor 222 and cause the cable 224 to unfold. And as further discussed below, the limit on the motor current can be applied and / or changed according to the operating state of the UAV 200.

[0088] In some embodiments, the cable control module 216 can be configured to determine the state of the cable 224 and / or the payload 228 based on the amount of current supplied to the motor 222. For example, if a downward force is applied to the cable 224 (e.g., if the payload 228 is attached to the cable 224 or if the cable 224 is hooked on an object when retracting toward the UAV 200), the cable control module 216 may need to increase the motor current in order to match the determined rotational speed of the motor 222 and / or the spool to the desired speed. Similarly, when the downward force is removed from the cable 224 (e.g., when delivering the payload 228 or removing the cable hook), the cable control module 216 may need to reduce the motor current in order to match the determined rotational speed of the motor 222 and / or the spool to the desired speed. In this way, the cable control module 216 can be configured to monitor the current supplied to the motor 222. For example, the cable control module 216 can determine the motor current based on sensor data received from the current sensor of the motor or the current sensor of the power system 220. In any case, based on the current supplied to the motor 222, it is determined whether the payload 228 is attached to the cable 224, whether someone or something is pulling on the cable 224, and / or whether the payload retriever 226 is pressed against the UAV 200 after retracting the cable 224. Other examples are possible.

[0089] During delivery of payload 228, payload retriever 226 may be configured to secure payload 228 while being lowered from UAV via cable 224, and may be further configured to release payload 228 upon reaching ground level. Payload retriever 226 may then be retracted into UAV by reeling in cable 224 using motor 222.

[0090] In some implementations, once the payload 228 is lowered to the ground, the payload 228 can be passively released. For example, the passive release mechanism can include one or more swing arms adapted to be retracted into and extended from the housing. The extended swing arm can form a hook to which the payload 228 can be attached. When the release mechanism and the payload 228 are lowered to the ground by the cable, the gravity on the release mechanism and the downward inertial force can separate the payload 228 from the hook, thereby allowing the release mechanism to rise upward toward the UAV. The release mechanism can further include a spring mechanism that biases the swing arm to retract into the housing when there is no other external force on the swing arm. For example, a spring can apply a force on the swing arm that pushes or pulls the swing arm toward the housing so that once the weight of the payload 228 no longer forces the swing arm to extend from the housing, the swing arm retracts into the housing. Retracting the swing arm into the housing can reduce the possibility of the release mechanism hooking the payload 228 or other nearby objects when the release mechanism is raised toward the UAV when delivering the payload 228.

[0091] Active payload release mechanisms are also possible. For example, sensors such as altimeters and / or accelerometers based on barometric pressure can help detect the position of the release mechanism (and payload) relative to the ground. Data from the sensors can be transmitted back to the UAV and / or control system via a wireless link and used to help determine when the release mechanism reaches ground level (e.g., by detecting measurements with accelerometers that have features of ground impact). In other examples, the UAV can determine that the payload has reached the ground based on a weight sensor detecting a low threshold downward force on the cable and / or based on a low threshold measurement of the power drawn by the winch when lowering the payload.

[0092] In addition to or as an alternative to the cable delivery system, other systems and techniques for delivering the payload are possible. For example, the UAV 200 may include an airbag delivery system or a parachute delivery system. Alternatively, the UAV 200 carrying the payload may simply land on the ground at the delivery location. Other examples are also possible.

[0093] IV. Illustrative UAV Deployment System

[0094] A UAV system may be implemented to provide a variety of UAV-related services. In particular, UAVs may be provided at multiple different launch sites that may communicate with a regional and / or central control system. Such a distributed UAV system may allow UAVs to be rapidly deployed to provide services over a large geographic area (e.g., much larger than the flight range of any single UAV). For example, UAVs capable of carrying payloads may be distributed over multiple launch sites over a large geographic area (perhaps even throughout a country or even the world) to provide on-demand transportation of various items to locations throughout a geographic area. Figure 3 is a simplified block diagram illustrating a distributed UAV system 300 according to an example embodiment.

[0095] In the illustrative UAV system 300, the access system 302 may allow for interaction with, control of, and / or utilization of the network of the UAV 304. In some embodiments, the access system 302 may be a computing system that allows for human control scheduling of the UAV 304. As such, the control system may include or otherwise provide a user interface through which a user may access and / or control the UAV 304.

[0096] In some embodiments, the dispatch of UAVs 304 may additionally or alternatively be accomplished via one or more automated processes. For example, the access system 302 may dispatch one of the UAVs 304 to transport a payload to a target location, and the UAV may autonomously navigate to the target location by utilizing various onboard sensors (such as a GPS receiver and / or other various navigation sensors).

[0097] In addition, the access system 302 can provide remote operation of the UAV. For example, the access system 302 can allow an operator to control the flight of the UAV via its user interface. As a specific example, the operator can use the access system 302 to dispatch the UAV 304 to a target location. The UAV 304 can then autonomously navigate to the approximate area of ​​the target location. At this point, the operator can use the access system 302 to control the UAV 304 and navigate the UAV to the target location (e.g., to a specific person to whom the payload is transported). Other examples of remote operation of the UAV are also possible.

[0098] In the illustrative embodiment, UAVs 304 may take various forms. For example, each of UAVs 304 may be a Figures 1A-1E. However, the UAV system 300 may also utilize other types of UAVs without departing from the scope of the present invention. In some implementations, all UAVs 304 may have the same or similar configuration. However, in other implementations, the UAVs 304 may include a plurality of different types of UAVs. For example, the UAVs 304 may include a plurality of types of UAVs, each type of UAV being configured for one or more different types of payload delivery capabilities.

[0099] The UAV system 300 may also include a remote device 306, which may take various forms. Typically, the remote device 306 may be any device through which a direct or indirect request for scheduling a UAV may be made. (Note that an indirect request may involve any communication that may be responded to by dispatching a UAV, such as requesting a package delivery). In an example embodiment, the remote device 306 may be a mobile phone, a tablet computer, a laptop computer, a personal computer, or any network-connected computing device. In addition, in some cases, the remote device 306 may not be a computing device. As an example, a standard telephone that allows communication via plain old telephone service (POTS) may be used as the remote device 306. Other types of remote devices are also possible.

[0100] Additionally, the remote device 306 can be configured to communicate with the access system 302 via one or more types of communication networks 308. For example, the remote device 306 can communicate with the access system 302 (or a human operator of the access system 302) by communicating over a POTS network, a cellular network, and / or a data network such as the Internet. Other types of networks can also be utilized.

[0101] In some embodiments, the remote device 306 can be configured to allow the user to request that one or more items be delivered to a desired location. For example, a user can request the UAV to deliver a package to their home via their mobile phone, tablet computer, or laptop computer. As another example, a user can request dynamic delivery to anywhere they are at the time of delivery. In order to provide such dynamic delivery, the UAV system 300 can receive location information (e.g., GPS coordinates, etc.) from the user's mobile phone or any other device on the user so that the UAV can navigate to the user's location (as indicated by their mobile phone).

[0102] In the illustrative arrangement, the central dispatch system 310 can be a server or group of servers that is configured to receive dispatch message requests and / or dispatch instructions from the access system 302. Such dispatch messages can request or instruct the central dispatch system 310 to coordinate the deployment of UAVs to various target locations. The central dispatch system 310 can also be configured to route such requests or instructions to one or more local dispatch systems 312. To provide such functionality, the central dispatch system 310 can communicate with the access system 302 via a data network, such as the Internet or a private network established for communication between the access system and the automated dispatch system.

[0103] In the illustrated configuration, the central dispatch system 310 may be configured to coordinate the dispatch of UAVs 304 from a plurality of different local dispatch systems 312. In this way, the central dispatch system 310 may track which UAVs 304 are located at which local dispatch systems 312, which UAVs 304 are currently available for deployment, and / or which services or operations each UAV 304 is configured for (in the case where the UAV fleet includes multiple types of UAVs configured for different services and / or operations). Additionally or alternatively, each local dispatch system 312 may be configured to track which of its associated UAVs 304 are currently available for deployment and / or are currently in the middle of transporting an item.

[0104] In some cases, when the central dispatch system 310 receives a request for a UAV-related service (e.g., transportation of an item) from the access system 302, the central dispatch system 310 may select a specific UAV 304 to dispatch. The central dispatch system 310 may accordingly instruct the local dispatch system 312 associated with the selected UAV to dispatch the selected UAV. The local dispatch system 312 may then operate its associated deployment system 314 to launch the selected UAV. In other cases, the central dispatch system 310 may forward the request for the UAV-related service to a local dispatch system 312 near the location where support is requested, and leave the selection of the specific UAV 304 to the local dispatch system 312.

[0105] In an example configuration, the local dispatch system 312 may be implemented as a computing system that is co-located with the deployment systems 314 that it controls. For example, the local dispatch system 312 may be implemented by a computing system installed at a building, such as a warehouse, where the deployment systems 314 and UAVs 304 associated with a particular local dispatch system 312 are also located. In other embodiments, the local dispatch system 312 may be implemented at a location remote from the deployment systems 314 and UAVs 304 to which it is associated.

[0106] Many variations and alternatives to the illustrated configuration of the UAV system 300 are possible. For example, in some embodiments, a user of a remote device 306 can request a package be delivered directly from the central dispatch system 310. To this end, an application can be implemented on the remote device 306 that allows the user to provide information about the requested delivery, and generates and sends a data message to request the UAV system 300 to provide the delivery. In such embodiments, the central dispatch system 310 may include automated functionality to process requests generated by such an application, evaluate such requests, and, if appropriate, coordinate with the appropriate local dispatch system 312 to deploy the UAV.

[0107] In addition, some or all of the functionality attributed herein to the central dispatch system 310, the local dispatch system 312, the access system 302 and / or the deployment system 314 may be combined in a single system, implemented in a more complex system, and / or reallocated among the central dispatch system 310, the local dispatch system 312, the access system 302 and / or the deployment system 314 in various ways.

[0108] Furthermore, while each local dispatch system 312 is shown as having two associated deployment systems 314, a given local dispatch system 312 may alternatively have more or fewer associated deployment systems 314. Similarly, while the central dispatch system 310 is shown as communicating with two local dispatch systems 312, the central dispatch system 310 may alternatively communicate with more or fewer local dispatch systems 312.

[0109] In another aspect, the deployment system 314 can take a variety of forms. In general, the deployment system 314 can take the form of or include a system for physically launching one or more of the UAVs 304. Such a launch system can include features that provide for automatic UAV launches and / or features that allow for human-assisted UAV launches. In addition, the deployment systems 314 can each be configured to launch a specific UAV 304, or to launch multiple UAVs 304.

[0110] The deployment system 314 can also be configured to provide additional functionality, including, for example, diagnostic-related functionality, such as verifying system functionality of the UAV, verifying functionality of equipment housed within the UAV (e.g., a payload delivery device), and / or maintaining equipment or other items housed in the UAV (e.g., by monitoring the status of the payload, such as its temperature, weight, etc.).

[0111] In some embodiments, the deployment system 314 and its corresponding UAV 304 (and possibly associated local dispatch system 312) can be strategically distributed throughout an area such as a city. For example, the deployment system 314 can be strategically distributed so that each deployment system 314 is close to one or more payload pickup locations (e.g., close to a restaurant, store, or warehouse). However, the deployment system 314 (and possible local dispatch system 312) can be distributed in other ways, depending on the specific implementation. As another example, a self-service terminal that allows a user to transport a package via a UAV can be installed in various locations. Such a self-service terminal can include a UAV launch system, and can allow users to provide their packages to be loaded onto the UAV and pay for UAV shipping services, etc. Other examples are also possible.

[0112] In another aspect, the UAV system 300 may include or have access to a user account database 316. The user account database 316 may include data for a plurality of user accounts, and each user account may be associated with one or more persons. For a given user account, the user account database 316 may include data relevant or useful for providing UAV-related services. Typically, the user data associated with each user account may be optionally provided by the associated user and / or collected with the associated user's permission.

[0113] Furthermore, in some embodiments, a person may be required to register a user account with the UAV system 300 if they wish to have UAV-related services provided to them by a UAV 304 from the UAV system 300. As such, the user account database 316 may include authorization information (e.g., a username and password) for a given user account and / or other information that may be used to authorize access to a user account.

[0114] In some embodiments, a person may associate one or more of their devices with their user account so that they may access the services of the UAV system 300. For example, when a person uses an associated mobile phone, for example, to make a call to an operator of the access system 302 or to send a message to a dispatch system requesting UAV-related services, the phone may be identified by a unique device identification number, and the call or message may then be attributed to the associated user account. Other examples are possible.

[0115] V. Example Systems and Methods for Controlling a UAV to Avoid Obstacles

[0116] Figure 4An example of operations 400 performed by some examples of the above devices (e.g., UAV, deployment system 314, local dispatch system 312, central dispatch system 310, etc.) when the UAV is traveling (e.g., traveling to deliver a package) is shown. In some examples, one or more of these operations are implemented via instruction code stored in corresponding data memories of these devices. Execution of the instruction code by the corresponding processor of the device causes these devices, alone or in combination with other devices, to perform these operations 400. Reference Figure 5-10 A clearer understanding Figure 4 Operation 400.

[0117] The operation at box 405 involves performing a UAV descent operation. For example, after the UAV arrives at the destination, the UAV begins to descend toward the delivery area.

[0118] The operations at block 410 involve capturing a depth image. Figure 5 An example of a depth image 500 is shown. As described above, some examples of a UAV include a camera that facilitates capturing an image depicting features 505 of the environment below the UAV. For example, some examples of the image depict a lane 505A, a building 505B, vegetation 505C, a fence 505D, a vehicle 505E, power lines 505F, 505G, etc. in the area below the UAV.

[0119] Some examples of cameras correspond to stereo / depth cameras that facilitate capturing stereo images. In this regard, some examples of stereo cameras include a pair of imagers placed adjacent to each other that are configured to simultaneously capture side-by-side images of the environment below the UAV. The relative distance between corresponding pixels in the side-by-side images associated with a particular object (e.g., a lane, building, vegetation, fence, vehicle, power line, etc.) helps determine the distance of the object from the depth camera. This, in turn, helps determine the distance or depth of the object relative to the UAV.

[0120] Some examples of stereo cameras generate a depth mask associated with the captured images. Figure 6 An example of a portion of a depth mask 600 is shown. Some examples of depth mask 600 have the same dimensions (e.g., width and height in pixels) as depth image 500. In some examples, pixels of depth mask 600 are associated with corresponding depths of objects depicted in depth image 500 (e.g., distances of corresponding objects to a depth camera).

[0121] Some examples of the depth mask 600 include a valid pixel depth 605 and an invalid pixel depth 610. The valid pixel depth 605 corresponds to a pixel of the depth image 500 for which the stereo camera is able to determine the depth or distance to the stereo camera. The invalid pixel depth 610 corresponds to a pixel of the depth image 500 for which the stereo camera cannot determine the depth or distance to the stereo camera. In some examples, the invalid pixel depth 605 is represented as a positive integer value (e.g., n≥0 representing a relative distance), and the invalid pixel depth 610 is represented as a negative value (e.g., n=-1). In some instances, whether the depth of a particular pixel associated with a particular object can be determined depends on the corresponding size of the object (e.g., in pixels). For example, in some examples, the depth of an object that is larger than, for example, 10 pixels wide and / or 10 pixels high (e.g., a building, a vehicle, etc.) can be determined, and the depth of an object that is smaller than 10 pixels wide and / or 10 pixels high (e.g., a power line, a tree branch, etc.) cannot be determined.

[0122] The operation at block 415 involves generating an invalid pixel mask. Figure 7 Shown with Figure 5 500 of the depth image 500. Some examples of the invalid pixel mask 700 represent invalid pixels 610 from the depth mask 600. In this regard, some examples of the invalid pixel mask 700 have the same size (e.g., width and height in pixels) as the depth mask 600. In some examples, the invalid pixels 610 are represented by a particular value (e.g., 1) and the valid pixels 605 are represented by a different value (e.g., 0). Figure 7 The invalid pixel mask 700 shows a pair of lines 705A and 705B. The first line 705A and the second line 705B correspond to the first power line 505F and the second power line 505G, respectively, depicted in the depth image 500. This indicates that the valid pixel depths of the first power line 505F and the second power line 505G cannot be determined.

[0123] Some examples of the invalid pixel mask 700 are generated by a processor or other hardware of the UAV. Some examples of the invalid pixel mask 700 are generated by a ground control station (e.g., deployment system 314, local dispatch system 312, central dispatch system 310). For example, in some examples, the UAV transmits the depth image 500 to the ground control station, and the ground control station generates the invalid pixel mask 700 and transmits it to the UAV.

[0124] The operations at block 420 involve detecting obstacles that may interfere with the delivery of the payload based on the distribution of invalid pixels 610 represented by the invalid pixel mask 700 . Figure 8An example of obstacle detection logic 805 that facilitates making this determination is illustrated. Some examples of obstacle detection logic 805 are configured to detect objects associated with pixels defining a line in invalid pixel mask 700, and output an indication 810 as to whether such an object corresponds to an obstacle that would interfere with the delivery of a payload. In this regard, some examples of obstacle detection logic 805 implement a Hough transform. Invalid pixel mask 700 is input to the Hough transform, and the Hough transform outputs an indication of whether a line has been detected in invalid pixel mask 700.

[0125] Some examples of the obstacle detection logic 805 determine that an obstacle is present when a line is detected in any area of ​​the invalid pixel mask 700. For example, some examples of the obstacle detection logic 805 determine that an obstacle is present when either the first line 705A or the second line 705B in the invalid pixel mask 700 is detected.

[0126] Some examples of the obstacle detection logic 805 determine that an obstacle exists only when a line is detected in the area of ​​the invalid pixel mask 700 corresponding to the delivery zone. For example, some examples of the obstacle detection logic 805 determine that an obstacle exists when a first line 705A in the invalid pixel mask 700, but not a second line 705B in the invalid pixel mask 700, is detected, and the first line 705A passes through the cone corresponding to the delivery zone below the UAV. In this regard, some examples of the obstacle detection logic 805 determine that an obstacle exists when the line exceeds a first threshold percentage (e.g., ≥85%) of the diameter of the cone below the UAV corresponding to the delivery zone. In another example, determining that an object exists can also depend on determining that a second threshold percentage of pixels greater than a pixel in the cone in the depth image is a valid pixel (e.g., ≥85%). This second threshold can ensure that the depth image is reliable enough to confidently identify potential obstacles through invalid pixels. The cone can be a relatively narrow cone below the UAV (e.g., 200 pixel radius).

[0127] Some examples of the obstacle detection logic 805 determine that an obstacle exists when a line extends at least from a first edge of a cone under the UAV corresponding to the delivery zone to a second edge of a cone under the UAV. For example, a line that extends only halfway within the delivery zone may not be determined to be an obstacle.

[0128] In some examples, if an obstacle is determined at a particular delivery zone, an indication that there is an obstacle at the delivery zone is stored in a database. In this regard, some examples of the UAV store this information locally and / or transmit the indication to a ground control system in communication with the UAV.

[0129] Some examples of the UAV implement obstacle detection logic 805. Additionally or alternatively, some examples of the ground control station are configured to implement obstacle detection logic 805 and transmit detection results to the UAV.

[0130] The operation at block 424 involves determining whether an obstacle is detected. If no obstacle is detected, the operation is repeated from block 405. If an obstacle is detected, the operation at block 430 is performed.

[0131] The operation at block 430 involves controlling the UAV to avoid obstacles. In some examples, the UAV is controlled to move in a lateral direction during descent to avoid obstacles. For example, Fig. 9 As shown, the delivery zone is adjusted to avoid obstacles (e.g., power lines 505F). Some examples of the UAV and / or ground control station determine the amount of lateral movement required to avoid obstacles based in part on how far the obstacles extend within the delivery zone. Once the appropriate delivery zone is determined, the operation is repeated from box 405. In some examples, the position of the UAV is controlled to avoid contact between the UAV's tethered payload and the obstacle. After changing the UAV position, the UAV's cable is controlled to deliver the tethered payload to the delivery zone.

[0132] In some examples, Figure 4 Operation 400 does not begin until the UAV has descended to a predetermined height (e.g., 500 feet) above ground level. In some examples, Figure 5 Operation 400 does not begin until the UAV has descended to a first predetermined altitude above ground level (e.g., 500 feet), and operation 400 ends when the UAV has descended to a second / lower predetermined altitude above ground level (e.g., 50 feet).

[0133] Fig.10 An example of operations 1000 performed by some examples of the above devices (e.g., UAV, deployment system 314, local dispatch system 312, central dispatch system 310, etc.) when the UAV is traveling (e.g., traveling to deliver a package) is shown. In some examples, one or more of these operations are implemented via instruction codes stored in corresponding data memories of these devices. Execution of the instruction codes by corresponding processors of the devices causes these devices, alone or in combination with other devices, to perform these operations 400.

[0134] The operation at block 1005 involves receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV). One or more pixels of the depth image are associated with corresponding depth values ​​indicating the distance of one or more objects to the stereo camera. The operation at block 1010 involves determining, by the image processing system, that one or more pixels of the depth image are associated with invalid depth values. The operation at block 1015 involves inferring the presence of a potential obstacle in the environment of the UAV based on the distribution of one or more pixels of the depth image associated with invalid depth values. The operation at block 1020 involves controlling the UAV based on the inferred presence of the potential obstacle.

[0135] Some examples of operations further involve generating, by the image processing system, an invalid pixel mask associated with the depth image. Pixels in the invalid pixel mask associated with pixels in the depth image that have corresponding depth values ​​are set to a first value, and pixels in the invalid pixel mask associated with pixels in the depth image that do not have corresponding depth values ​​are set to a second value different from the first value.

[0136] Some examples of operations also involve detecting an elongated object in the environment that defines a line in the invalid pixel mask based on the invalid pixel mask. Some examples of operations further detect an elongated object in the environment that defines a line in the invalid pixel mask based on the invalid pixel mask and via a Hough transform. Some examples of the operations also involve determining that the object will interfere with a deployment operation of the UAV when the line exceeds a first threshold percentage of a diameter of a cone below the UAV.

[0137] Some examples of operations also involve detecting, by the image processing system and based on one or more pixels of the image that do not have corresponding depth values, objects corresponding to the power lines.

[0138] Some examples of operations also involve determining a delivery zone portion of the image. The delivery zone portion of the image is associated with a delivery zone of the UAV. The operations involve detecting an object that will interfere with a deployment operation of the UAV based on one or more pixels of the delivery zone portion of the image that do not have a corresponding depth value. Some examples of operations also involve determining that the detected object will interfere with the deployment operation when the detected object extends from a first edge of the delivery zone to a second edge of the delivery zone.

[0139] Some examples of operations also involve moving the UAV in a lateral direction during a subsequent descent of the UAV to avoid an inferred obstacle. Some examples of operations also involve controlling the position of the UAV to avoid contact between a tethered payload of the UAV and the detected object, and subsequently causing the cable of the UAV to deliver the tethered payload to a delivery zone.

[0140] Some examples of operations also involve causing the UAV to descend to a predetermined height above ground level before capturing the image. Some examples of operations also involve causing the UAV to capture one or more depth images using a stereo camera to infer potential obstacles below the UAV as the UAV descends from a first height to a second height.

[0141] Some examples of operations also involve storing to a database an indication that an object that will interfere with deployment operations of the UAV is present in a geographic location associated with the UAV.

[0142] VI. Conclusion

[0143] The present disclosure is not limited to the specific embodiments described in this application, which are intended to be illustrations of various aspects. Without departing from its spirit and scope, many modifications and variations may be made, which will be apparent to those skilled in the art. In addition to those listed herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0144] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. The example embodiments described herein and in the accompanying drawings are not meant to be limiting. Other embodiments may be utilized, and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0145] The box representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or additionally, the box representing the processing of information may correspond to a module, a segment, or a portion of a program code (including associated data). The program code may include one or more instructions executable by a processor to implement a specific logical function or action in a method or technology. The program code or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk or hard drive or other storage medium.

[0146] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code or data for a longer period of time, such as secondary or persistent long-term storage, such as read-only memory (ROM), optical disks or magnetic disks, compact-disc read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, as a computer-readable storage medium or a tangible storage device.

[0147] In addition, the blocks representing one or more information transfers may correspond to information transfers between software or hardware modules in the same physical device. However, other information transfers may be between software modules or hardware modules in different physical devices.

[0148] The specific arrangements shown in the figures should not be considered restrictive. It should be understood that other embodiments may include more or less of each component shown in a given figure. In addition, some of the components shown may be combined or omitted. In addition, example embodiments may include components not shown in the figures.

[0149] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented method comprising: Receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV), wherein one or more pixels of the depth image are associated with corresponding depth values ​​indicating a distance of one or more objects from the stereo camera; determining, by the image processing system, that one or more pixels of the depth image are associated with an invalid depth value; inferring the presence of a potential obstacle in the environment of the UAV based on a distribution of the one or more pixels of the depth image associated with invalid depth values; and The UAV is controlled based on the inferred presence of the potential obstacle.

2. The method according to claim 1, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: An invalid pixel mask associated with the depth image is generated by the image processing system, wherein pixels in the invalid pixel mask associated with pixels in the depth image having corresponding depth values ​​are set to a first value, and pixels in the invalid pixel mask associated with pixels in the depth image not having corresponding depth values ​​are set to a second value different from the first value.

3. The method according to claim 2, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: Elongated objects in the environment that define lines in the invalid pixel mask are detected based on the invalid pixel mask.

4. The method according to claim 3, wherein: Detecting an elongated object in the environment that defines a line in the invalid pixel mask comprises: Elongated objects in the environment that define lines in the invalid pixel mask are detected based on the invalid pixel mask and via a Hough transform.

5. The method according to claim 3, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: When the line exceeds a first threshold percentage of the diameter of the cone beneath the UAV, it is determined that the object will interfere with the deployment operation of the UAV.

6. The method according to claim 1, further comprising: Objects corresponding to the power lines are detected by the image processing system and based on the one or more pixels of the image that do not have corresponding depth values.

7. The method according to claim 1, wherein: Controlling the position of the UAV to facilitate the deployment operation includes: The UAV is moved in a lateral direction during a subsequent descent of the UAV to avoid the inferred obstacle.

8. The method according to claim 1, further comprising: Prior to capturing the image, the UAV is descended to a predetermined height above ground level.

9. The method according to claim 8, further comprising: When the UAV descends from the predetermined height to a second height, the UAV is enabled to capture one or more depth images using the stereo camera to infer potential obstacles below the UAV.

10. The method according to claim 1, wherein: Controlling the position of the UAV to facilitate the deployment operation includes: controlling a position of the UAV to avoid contact between a tethered payload of the UAV and a detected object; and The UAV's cable is then caused to deliver the tethered payload to a delivery area.

11. The method according to claim 1, further comprising: An indication of the presence of an object in a geographic location associated with the UAV that will interfere with the deployment operation of the UAV is stored in a database.

12. An unmanned aerial vehicle (UAV), comprising: camera; and A control system configured to perform operations comprising: Receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV), wherein one or more pixels of the depth image are associated with corresponding depth values ​​indicating a distance of one or more objects from the stereo camera; determining, by the image processing system, that one or more pixels of the depth image are associated with an invalid depth value; inferring the presence of a potential obstacle in the environment of the UAV based on a distribution of the one or more pixels of the depth image associated with invalid depth values; and The UAV is controlled based on the inferred presence of the potential obstacle.

13. The UAV of claim 12, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: An invalid pixel mask associated with the depth image is generated by the image processing system, wherein pixels in the invalid pixel mask associated with pixels in the depth image having corresponding depth values ​​are set to a first value, and pixels in the invalid pixel mask associated with pixels in the depth image not having corresponding depth values ​​are set to a second value different from the first value.

14. The UAV of claim 13, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: Elongated objects in the environment that define lines in the invalid pixel mask are detected based on the invalid pixel mask.

15. The UAV of claim 14, wherein: Detecting an elongated object in the environment that defines a line in the invalid pixel mask comprises: Elongated objects in the environment that define lines in the invalid pixel mask are detected based on the invalid pixel mask and via a Hough transform.

16. The UAV of claim 14, wherein: Inferring the presence of potential obstacles in the UAV's environment includes: It is determined that the object will interfere with the deployment operation of the UAV when (i) the line exceeds a first threshold percentage of the diameter of the cone below the UAV and (ii) the cone includes more than a second threshold percentage of valid pixels.

17. The UAV of claim 12, the operations further comprising: Objects corresponding to the power lines are detected by the image processing system and based on the one or more pixels of the image that do not have corresponding depth values.

18. The UAV of claim 12, wherein: Controlling the position of the UAV to facilitate the deployment operation includes: The UAV is moved in a lateral direction during a subsequent descent of the UAV to avoid the inferred obstacle.

19. The UAV of claim 12, the operations further comprising: Prior to capturing the image, the UAV is descended to a predetermined height above ground level.

20. A non-transitory computer readable medium comprising program instructions executable by one or more processors to perform operations comprising: Receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV), wherein one or more pixels of the depth image are associated with corresponding depth values ​​indicating a distance of one or more objects from the stereo camera; determining, by the image processing system, that one or more pixels of the depth image are associated with an invalid depth value; inferring the presence of a potential obstacle in the environment of the UAV based on a distribution of the one or more pixels of the depth image associated with invalid depth values; and The UAV is controlled based on the inferred presence of the potential obstacle.

Images