Automatic safe landing site selection for unmanned aerial systems

Abstract

An unmanned aerial vehicle (UAV) navigation system is disclosed that is configured to automatically identify a suitable UAV landing site, e.g., in a forced landing (e.g., emergency) scenario. In some examples, the system receives two or more overlapping images depicting a landscape below an unmanned aerial vehicle (UAV) in flight; generates a depth map of the landscape based on the two or more overlapping images; identifies, based on the depth map, regions of the landscape having a depth variance below a threshold value; determines, for each of the regions of the landscape having a depth variance below the threshold value, a landing zone quality score indicative of a depth variance and a semantic type of the region of the landscape; identifies, based on the landing zone quality scores, a suitable location for landing the UAV; and causes the UAV to automatically navigate toward and land on the suitable location.

Description

Technical Field

[0001] This disclosure relates to unmanned aerial vehicles (UAVs). Background Technology

[0002] Human pilots can remotely control unmanned aerial vehicles (UAVs). In some examples, a UAV may include one or more onboard navigation systems to automatically perform semi-autonomous or fully autonomous segments. For example, a UAV may include a GPS-based navigation system or a vision-based (e.g., image processing-based) navigation system. Summary of the Invention

[0003] Generally, this disclosure relates to systems and techniques for automatically identifying suitable landing sites for an unmanned aerial vehicle (UAV) and automatically navigating the UAV toward suitable landing sites. The techniques disclosed herein can be used as a primary UAV navigation system or a backup navigation system, such as for use when the UAV's conventional navigation system is compromised. In some examples, the navigation system may be configured to capture two or more images of a landscape below the UAV and generate a "depth" mapping that indicates the relative distance between pixels and the UAV for each pixel in at least one of the images. The navigation system may then identify one or more generally flat or planar areas of the landscape based on the depth mapping. The navigation system may then generate a landing area "quality" score for each of the generally flat or planar areas of the landscape. For example, the navigation system may include one or more machine learning-based, neural network-based, or artificial intelligence-based models or algorithms trained to rate or rank each area of ​​the landscape based on its suitability for landing (e.g., based on ground type, nearby traffic flow, etc.). When a suitable area is identified (e.g., the area with the highest landing quality score, or the nearest area with a landing quality score above the threshold), the navigation system can enable the UAV to automatically navigate toward and land in the identified area.

[0004] In one example, this disclosure describes a method comprising: receiving two or more overlapping images depicting a landscape beneath an unmanned aerial vehicle (UAV) in flight; generating a depth map of the landscape based on the two or more overlapping images; identifying regions of the landscape having a depth variance below a threshold based on the depth map; determining, for each region of the landscape having a depth variance below the threshold, a landing zone quality score indicating the depth variance and semantic type of the region; identifying a suitable location for landing the UAV based on the landing zone quality score; and enabling the UAV to automatically navigate toward and land at the suitable location.

[0005] In another example, this disclosure describes an unmanned aerial vehicle (UAV) navigation system, comprising: an unmanned aerial vehicle (UAV); an image capture device coupled to the UAV; and processing circuitry coupled to the UAV and configured to: receive from the image capture device two or more overlapping images depicting a landscape below the UAV; generate a depth map of the landscape based on the two or more overlapping images; identify regions of the landscape having a depth variance below a threshold based on the depth map; for each region of the landscape having a depth variance below the threshold, determine a landing zone quality score indicating the depth variance and semantic type of the region; identify a suitable location for landing the UAV based on the landing zone quality score; and enable the UAV to automatically navigate toward and land at the suitable location.

[0006] In another example, this disclosure describes a non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to: receive from an image capture device two or more overlapping images depicting a landscape below a UAV; generate a depth mapping of the landscape based on the two or more overlapping images; identify regions of the landscape having a depth variance below a threshold based on the depth mapping; for each region of the landscape having a depth variance below the threshold, determine a landing zone quality score indicating the depth variance and semantic type of the region; identify a suitable location for landing the UAV based on the landing zone quality score; and cause the UAV to automatically navigate toward and land at the suitable location.

[0007] Details of one or more examples of this disclosure are set forth in the following drawings and description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0008] Figure 1 This is a conceptual diagram depicting an exemplary UAV system based on one or more technologies according to this disclosure.

[0009] Figure 2 It is shown Figure 1 A block diagram of some exemplary components of a UAV system.

[0010] Figure 3 It shows that it can be generated by Figure 1 Conceptual diagrams of some examples of images generated by a UAV system.

[0011] Figure 4 This is a flowchart illustrating exemplary operation of a UAV system according to the technology disclosed herein. Detailed Implementation

[0012] Generally, this disclosure relates to systems and techniques for automatically identifying suitable landing sites for an unmanned aerial vehicle (UAV) and automatically navigating the UAV toward the identified sites. The techniques disclosed herein can be used as a primary UAV navigation system or a backup navigation system, such as for use when the UAV's conventional navigation system is compromised. In some examples, the navigation system may be configured to capture images (e.g., two or more images) of the landscape below the UAV and generate a depth mapping map based on changes in the relative positions of corresponding (e.g., matching) pixels between the images. The system may then determine or identify one or more generally flat or planar areas of the landscape based on the depth mapping map. The navigation system may then generate a landing area "quality" score for one or more flat or planar areas of the landscape, indicating the relative suitability for landing the UAV in that area. For example, the navigation system may include one or more machine learning-based, neural network-based, or artificial intelligence-based models or algorithms trained to rate or rank each area of ​​the landscape based on its suitability for landing (e.g., based on the semantic "type" (e.g., ground type) of each area of ​​the landscape). When a suitable area is identified (e.g., the area with the highest landing quality score, or the nearest location with a landing quality score above the threshold), the navigation system can enable the UAV to automatically navigate toward and land in the identified area.

[0013] Figure 1 An exemplary unmanned aerial vehicle (UAV) navigation system 100 according to the technology of this disclosure is depicted. System 100 may be included as part of the primary navigation system of UAV 102, or in other examples, the system may include a backup navigation system for use when one or more other UAV navigation systems become unavailable. Figure 1 As shown, the UAV navigation system 100 includes a UAV 102 having at least one image capture device (ICD) 104 and an integrated computing device 106.

[0014] UAV 102 may include relatively small unmanned devices capable of flight, such as via one or more rotors. While UAV 102... Figure 1While described as a hexacopter, UAV 102 can include any type of UAV, including but not limited to quadcopters, rotorcraft, fixed-wing aircraft, compound aircraft (such as tiltrotor aircraft, X2 and X3), aerostats, or any other UAV of this type, including all vertical takeoff and landing (VTOL) aircraft, vertical takeoff and landing aircraft, etc. UAV 102 can be configured to fly with varying degrees of autonomy. In some examples, UAV 102 may use light detection and ranging (LIDAR) to avoid collisions. For example, UAV 102 could be a relatively small, low-altitude, and low-speed UAV, where, in this context, small corresponds to less than 100 pounds (e.g., about 50 pounds), low altitude corresponds to an operating altitude of less than 3,000 feet above the ground (e.g., 400 feet or less), and low airspeed corresponds to an airspeed of less than 250 knots (e.g., 10 knots) or less than 50 miles per hour (mph). Furthermore, it is envisioned that UAV 102 could have hovering capabilities, meaning that UAV 102 could have the ability to maintain a roughly constant position in the air.

[0015] In some examples, detectors or other sensing devices may be mounted on UAV 102 to enable it to rapidly collect high-quality data in hard-to-reach locations. For example, large industrial companies such as utilities, oil and gas companies may own or otherwise manage complex asset infrastructures such as power line transmission towers, pipelines, data (fiber optic) cables, factories, plant buildings, solar panels, wind turbines, or other such structures or installations. Such infrastructure may require regular inspections to maintain high productivity. Oversight companies (or other controlling entities, such as government agencies) may employ any of a variety of sensing or detection devices to collect various types of data describing different aspects of the infrastructure. Some non-limiting examples of current sensing devices include x-ray systems, infrared systems, sonar systems, radar detectors, optical systems, and hybrid multispectral cameras. In some examples, the sensing device may include image capture device 104. In other examples, the sensing device may differ from image capture device 104.

[0016] In some examples, a human or other pilot may manually control the flight path of UAV 102 (e.g., speed, heading or direction, altitude, etc.). For example, a pilot may use a controller (not shown) to wirelessly transmit signals to control the flight path of UAV 102, such as during a check segment. Alternatively or additionally, UAV 102 may include one or more internal navigation systems configured to enable UAV 102 to autonomously or semi-autonomously complete one or more segments. For example, UAV 102 may include a satellite-based navigation system, such as GPS-based navigation. In some examples, UAV 102 may include a vision-based navigation system, where the UAV utilizes image capture device 104 and computing device 106 to identify different landmarks within captured images and determine a flight path based on the landmarks. In some examples, UAV 102 may include a passive surveillance-based navigation system, such as an Automatic Dependent Surveillance-Broadcast (ADS-B) system.

[0017] In some scenarios, one or more of the navigation systems in UAV 102 may malfunction or otherwise become unavailable. For example, due to severe weather, equipment failure, terrain, or other factors, UAV 102 may lose data communication with the pilot and / or other signals from auxiliary navigation systems (such as GPS signals). In one scenario, a set of infrastructure undergoing UAV-based detection may naturally emit relatively strong electromagnetic signature signals that could interfere with the vehicle's detection equipment and / or guidance systems (such as GPS-based, radio frequency, or other electromagnetic guidance systems).

[0018] In some examples according to this disclosure, upon the occurrence of a triggering condition, UAV 102 may automatically identify a suitable landing site that meets certain minimum criteria (e.g., a relatively solid, flat surface) and navigate toward it. For example, the triggering condition may include UAV 102's computing device 106 automatically detecting a loss of data communication with the UAV pilot or other navigation signal sources. In another example, the triggering condition may include computing device 106 receiving specific input indicating the triggering condition. For example, a user (such as the pilot of UAV 102) may activate a user input device configured to transmit a signal to trigger the automatic landing mode of UAV 102.

[0019] In such events or in response to detected conditions, UAV 102 can be configured to automatically identify a suitable landing location and autonomously navigate toward it (e.g., with minimal or no further human intervention). For example, according to the technology of this disclosure, UAV 102 is configured to allow ICD 104 to capture two or more overlapping images depicting the landscape 108 below the flying UAV 102. That is, ICD 104 may capture two or more images of substantially the same landscape, wherein some of the two or more images are captured from slightly different locations (e.g., orientation and / or orientation). These locations may be different, for example, because the images are captured from different lenses at different locations mounted on the UAV 102, or due to the movement of the UAV 102 between capturing different images.

[0020] Based on two or more overlapping images, UAV 102 generates a depth mapping map 304 for at least one of the two or more images of the landscape (referred to in this document as the "keyframe image"). Figure 3 For example, UAV 102 can determine the changes in the relative positions of matching or corresponding pixel pairs between two or more images, and then estimate the relative distance between UAV 102 and each pixel of the keyframe image based on the changes in relative positions. UAV 102 can store this set of estimated relative distances within a matrix and output a "patch map" 304, where the color or shading of each pixel indicates the relative distance of the pixel from the UAV.

[0021] In some, but not all, examples, UAV 102 may identify one or more relatively flat or planar regions of a landscape based on a depth mapping. For example, UAV 102 may identify regions where the relative depth (e.g., distance from UAV 102) remains relatively constant or consistent (e.g., defined as a depth variance below a threshold). In some examples, UAV 102 divides the depth mapping into multiple smaller regions and then determines the depth variance of each region by determining the average variance of the height of each region of the landscape. In other examples, UAV 102 may determine the depth variance of each region of the landscape by determining the depth difference between the “highest” point and the “lowest” point in the region.

[0022] UAV 102 can feed raw images, depth maps (e.g., a matrix of relative distances represented by the depth maps), and / or depth variances (e.g., indicators of flat or planar areas) into a landing zone quality module trained to assign relative “quality” rankings to each region in the depth maps. For example, the quality module can be trained to identify (e.g., distinguish between) different semantic “types” of landscapes, such as sites, roads, bodies of water, etc., and to rank the landing zone quality of each region of the landscape based on its depth variance (e.g., relative flatness) and its semantic type. UAV 102 can then select a single suitable location for landing the UAV, such as a location with the highest quality ranking or the nearest location with a quality ranking above a threshold, based at least in part on the quality ranking, and autonomously navigate toward and land at that suitable location.

[0023] Figure 2 It is shown Figure 1 A block diagram of some exemplary components of UAV 102 is provided. UAV 102 includes at least an image capture device 104 and an integrated computing device 106. The computing device 106 may include at least processing circuitry and a computer-readable medium storing instructions that, when executed by the processing circuitry, cause the processing circuitry to perform the techniques of this disclosure. Specifically, the memory of the computing device 106 stores a navigation module 110 configured to identify a suitable landing site and to navigate the UAV 102 toward and land at the identified landing site. More specifically, the navigation module 110 includes a depth module 112, a mass module 114, and a flight control module 116. The depth module 112 includes at least a tracking model 120 and a mapping model 122.

[0024] According to the technology disclosed herein, image capture device 104 captures two or more overlapping images 302 depicting the landscape below UAV 102 during its flight. Figure 3 Image capture device 104 can be configured to capture any suitable type of image 302, such as standard photographic images, radar images, or lidar images. In some examples, image capture device 104 can be configured to automatically capture image 302 at a predetermined frequency (e.g., 1 Hz), or in other examples, to capture images in response to user input received from the pilot of UAV 102 or a data scientist configured to control the data capture process of UAV 102. At least two images 302 are considered "overlapping" because at least one geographic area (of any arbitrary size) appears (e.g., is represented) in both images. Image capture device 104 then transmits image 302 to depth module 112 and quality module 114.

[0025] Depth module 112 receives two or more overlapping images 302 from image capture device 104. Based on the two or more overlapping images 302, depth module 112 generates a depth mapping map 304 of the landscape. Figure 3 This is used to identify generally “flat” or “planar” areas of the landscape, such as areas of the landscape with local depth variance below a threshold. For example, depth module 112 includes tracking model 120 and mapping model 122, which are configured to generate depth mapping map 304 via a feedback loop, in which tracking model 120 and mapping model 122 exchange UAV pose data 124 and landscape depth data 126.

[0026] For example, tracking model 120 is configured to receive two or more overlapping images 302 and initial system pose data 118. Initial system pose data 118 includes data indicating the initial physical location (e.g., geographic location) and initial orientation of UAV 102. Initial system pose data 118 may include data generated by UAV 102's GPS unit, inertial measurement unit (IMU), or other spatial orientation sensors. When image capture device 104 captures "keyframe" images from two or more images, initial system pose data 118 indicates the "first" pose of UAV 102, such as the initial position and orientation of UAV 102.

[0027] In some examples, the initial system attitude data 118 may be set to "empty" values ​​by default (e.g., x=0, y=0, z=0, roll=0, pitch=0, yaw=0), and the UAV 102 may subsequently monitor changes in position and orientation deviating from these initial empty values. In other examples, the position coordinates of the initial system data 118 may correspond to GPS data of the UAV 102, and the orientation coordinates of the initial system data 118 may correspond to IMU (e.g., microelectromechanical (MEM) gyroscope) data of the UAV 102. In some such examples, any position and / or orientation data subsequently communicated between the depth module 112, mass module 114, and flight control module 116 are presented in a common coordinate system, thereby enabling simpler communication between the flight control module 116 and the selected landing area location, allowing the flight control module 116 to subsequently navigate the UAV 102 toward the selected landing location.

[0028] Based on the “first” pose, the first image, and the second image from the initial pose data 118, the tracking model 120 is configured to determine or estimate the “second” pose of the UAV 102, wherein the “second” pose indicates the position and orientation (or change in position and orientation) of the UAV 102 when the UAV 102 captures the second image in two or more overlapping images. For example, the tracking model 120 may include a convolutional neural network (CNN), such as a machine learning model including one or more artificial intelligence-based algorithms trained to perform triangulation to determine the corresponding or corresponding change in the position and orientation of the UAV 102 between the times the image capture device 104 captures the two images, based on changes in the position and orientation of the landscape in the overlapping region between the two images. As a non-limiting example, the tracking model 120 may first be trained to identify corresponding or matching features or pixels between the two overlapping images, and then the relative change in the position and / or orientation of the matching features or pixels between the two images may be determined. The tracking model 120 may be further trained to estimate the corresponding change in the position and / or orientation of the image capture device 104 of the captured images based on the positional changes of the identified matching features. Based on the first posture data and estimated changes in position and / or orientation, the tracking model 120 can determine the predicted "second" posture data 124.

[0029] Tracking model 120 may then transmit first pose data 118 and second pose data 124 to mapping model 122. Mapping model 122 receives first pose data 118, a first image, predicted second pose data 124, and a second image, and for each region of the landscape depicted in the first image (e.g., for each pixel) (or at least each region or pixel of overlapping regions within the first image), determines the relative depth of the corresponding region of the landscape in the first image, where the depth indicates the distance between the corresponding region and UAV 102. For example, mapping model 122 may be trained to estimate the relative depth of each pair of matching features or pixels to determine (e.g., identify or measure) the relative changes in position between two or more pairs of matching features or pixels between the first and second images. As an illustrative example, relatively tall objects (e.g., building roofs closer to the flight altitude of UAV 102) may exhibit a relatively large change in position from the first image to the second image compared to relatively distant areas (such as manhole covers level with the street), while relatively distant areas may appear to remain in a relatively constant position between two or more images. In this way, the mapping model 122 may perform these types of geometric parallax calculations for each region or pixel in the first image to determine the relative "depth" of that region or pixel, such as the relative distance of each pixel from UAV 102. The mapping model 122 may store these estimated depth values ​​within a depth matrix. In some, but not all, examples, the mapping model 122 may visually represent the relative depth values ​​as a depth mapping map 304. Figure 3 In the depth mapping 304, the color of each pixel indicates the relative distance of that pixel to the UAV 102. For example, a generally flat or planar area of ​​the landscape may be visually represented as a region in the depth mapping 304, where the color of consecutive pixels within that region does not vary much.

[0030] Navigation module 110 (e.g., depth module 112, quality module 114, or both) identifies one or more such flat or planar regions of the landscape (e.g., regions with depth variance below a threshold) based on depth mapping 304 (e.g., a depth matrix). In some examples, navigation module 110 divides depth mapping 304 into multiple smaller regions and then determines the depth variance of each region by determining the average variance of the height of all pixels in each region of the landscape. In other examples, navigation module 110 determines the depth variance of each region of the landscape by determining the depth difference between the “highest” point in the region and the “lowest” point in the region. Alternatively, navigation module 110 may include a deep neural network (DNN) (e.g., depth module 112, quality module 114, or both) trained to recognize patterns within relative depth values ​​of the depth matrix that indicate generally flat or planar regions between adjacent groups of pixels.

[0031] As Figure 2 As part of the feedback loop shown, in some examples, the mapping model 122 is configured to feed the depth mapping map 304 (e.g., a depth matrix) back into the tracking model 120 as training data to improve subsequent estimations of the UAV 102's pose. After generating the depth mapping map 304, the depth module 112 is configured to transmit the depth mapping map 304 (e.g., the depth matrix represented by the depth mapping map 304) and, in some examples, indications of any identified flat or planar regions to the quality module 114.

[0032] The quality module 114 receives a landscape image 302 from the image acquisition device 104, a depth mapping map 304 from the depth module 112, and in some examples, indications of identified planar regions. The quality module 114 is configured to generate a landing area quality mapping map 306 based at least on the original landscape image 302 and the depth mapping map 304. Figure 3For example, based on depth mapping 304 and / or the identified flat or planar areas, quality module 114 also determines and assigns relative “landing zone quality values” for each region of the landscape, and generates a graphical representation of the landing zone quality values ​​as landing zone quality mapping 306. For example, landing zone quality mapping 306 depicts a visual representation of all landing zone quality values, which are functions of their geographic locations relative to each other. In some, but not all, examples, quality module 114 includes a neural network (e.g., one or more artificial intelligence or machine learning algorithms or models) trained to identify “suitable” regions of landscape 108 for landing UAV 102. For example, quality module 114 may be trained via semantic segmentation to identify (e.g., distinguish between) multiple different semantic meanings, types, or categories of the landscape based on any or all visual patterns, textures, colors, shapes, or any other identifiable content within landscape image 302.

[0033] As a non-limiting illustrative example, both grasslands and bodies of water can have similar depth variances; for example, both types of landscape features can include relatively flat surfaces. However, bodies of water are generally not considered "suitable" locations for landing UAV 102 because UAV 102 may sink and become unretrievable while attempting to land on the water, and / or the water may damage the electronic components of UAV 102. Therefore, although navigation module 110 may have already identified both the field area and the water area as relatively flat or planar areas, quality module 114 can subsequently identify the different semantic meanings of the two areas and, based on semantic meaning, assign a relatively high landing area quality value to the field area and a significantly lower score or value to the water area within landing area quality mapping map 306. Similarly, building roofs can be another example of locations from which UAV 102 may be difficult to retrieve. Therefore, quality module 114 can be trained to assign relatively low quality scores to roofs, roads, and other similar undesirable obstacles.

[0034] In some, but not all, examples, navigation module 110 (e.g., either or both of depth module 112 and / or mass module 114) may be configured to generate uncertainty mapping 308. Figure 3 The uncertainty mapping indicates the relative confidence in each region of the landing area quality mapping 306, for example, the relative certainty that any identified “suitable” landing site (e.g., a landscape area with a landing area quality score above a threshold) is relatively flat (e.g., with a depth variance below a threshold), free of obstructions, and composed of a suitable “type” of landscape for landing (e.g., a substantially solid, legally permissible, and low-traffic surface).

[0035] Navigation module 110 then identifies or selects landscape 108 based on depth mapping map 304, landing area quality mapping map 306, and / or uncertainty mapping map 308. Figure 1 The navigation module 110 may select at least one suitable location for landing UAV 102. For example, a suitable location may include an area with the highest landing area quality score, as determined by the quality module 114. In another example, the navigation module 110 may determine a suitable location for landing UAV 102 as a weighted average of a high landing area quality score and proximity to the UAV 102's current position. In another example, the navigation module 110 may be configured to identify one or more predetermined, predefined, or "designated" suitable landing sites within the landing area quality mapping map 306. In another example, the selected suitable location may include the nearest suitable location, for example, the nearest location with a landing quality score above a threshold. In another example, the navigation module 110 may be configured to select suitable locations at predetermined intervals along the UAV 102's latest flight path, thereby allowing the pilot or other user of the UAV 102 to more easily retrieve the landed equipment.

[0036] Upon determining a suitable location for landing UAV 102, flight control module 116 causes UAV 102 to automatically navigate toward and land at a suitable location 108. For example, flight control module 116 may determine (e.g., estimate) the attitude changes (e.g., changes in position and orientation) between the second attitude and the identified suitable landing site based on second attitude data (e.g., the “current” or latest position and orientation of UAV 102) and a depth mapping map. UAV 102 may be configured to periodically update its estimates of desired attitude changes as it approaches the landing site.

[0037] Figure 4 This is a flowchart illustrating exemplary operation of a UAV navigation system according to the technology of this disclosure. The steps of operation 400 are referred to... Figure 1 and Figure 2The UAV system 100 described herein may employ these techniques, but such techniques can be performed by any capable system. UAV 102 is configured to capture or receive two or more overlapping images (402) depicting a landscape below UAV 102 while UAV 102 is in flight. Based at least in part on the two or more images, UAV 102 generates a depth mapping indicating the relative depth (404) of each region (e.g., each pixel) of the landscape depicted in the keyframe images (e.g., a first image, a second image, or both). For example, UAV 102 may include a computing device configured to run one or more machine learning algorithms or neural networks trained to estimate changes in position of UAV 102 as UAV 102 moves, while capturing two or more images. Based on the position between the two images and any changes in matching features or pixels, the neural network may be trained to estimate the relative depth or distance between each pixel and UAV 102.

[0038] UAV 102 may subsequently generate a landing area quality mapping map (406) of keyframe images (e.g., a first image, a second image, or both). For example, UAV 102 may be trained to recognize different semantic segments or meanings to indicate different types of landscapes, or, in some examples, to identify simple binary names of “good” or “bad” type landscapes relative to their suitability for use by the landing UAV. As an illustrative example, UAV 102 may include one or more machine learning algorithms trained to identify and designate water bodies as “bad” type landscapes and open sites as “good” type landscapes for use by the landing UAV 102.

[0039] Based on the depth mapping and landing area quality mapping, UAV 102 can select a suitable landing location (408), such as by identifying a location with the highest landing area quality score as indicated by the landing area quality mapping, or by identifying the nearest location with a landing area quality score above a threshold. After identifying a single suitable landing location, UAV 102 is configured to automatically navigate and land at the identified landing location (410).

[0040] In one or more examples, the described functionality can be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that includes, for example, any medium facilitating the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium can generally correspond to: (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0041] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and that is accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0042] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be implemented entirely within one or more circuit or logic elements.

[0043] The techniques disclosed herein can be implemented in various devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented through different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) incorporating suitable software and / or firmware.

[0044] Cloud technologies used to automatically save images to web servers are not limited to local or global internet clouds. They can be private clouds and / or public clouds protected by user IDs and passwords. Passwords are not limited to one or two.

[0045] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for automatic safe landing site selection for unmanned aerial systems, comprising: Receive two or more overlapping images depicting the landscape beneath an unmanned aerial vehicle (UAV) in flight; A depth mapping map of the landscape is generated based on the two or more overlapping images; Based on depth mapping, landscape areas with depth variance below a threshold are identified by feeding the depth mapping into a deep neural network (DNN). The DNN is trained to identify one or more patterns within the depth values ​​of the depth mapping that indicate one or more planar regions in the depth mapping. For each region of the landscape with a depth variance below the threshold, determine the depth variance and the landing zone quality score indicating the semantic type of the region of the landscape; For each region of the landscape, the determined landing zone quality score is adjusted based on the determined semantic type of the region; The appropriate location for landing the UAV is identified based on the adjusted landing area mass score. as well as The UAV is automatically navigated to and lands at the appropriate location. The generation of the depth mapping map includes: Based on the first pose of the UAV, predict a first image captured by the UAV in the first pose of the two or more overlapping images, and a second image of the two or more overlapping images and the second pose of the UAV; as well as The depth of the landscape depicted in the first image is determined based on the first pose, the first image, the second pose, and the second image.

2. The method according to claim 1, The method of predicting the second pose of the UAV includes feeding the first pose, the first image, and the second image into a neural network trained to estimate the second pose, and the method further includes feeding the depth map back into the neural network as training data to improve subsequent pose estimation.

3. The method of claim 1, wherein the first posture includes a first translation of the UAV and a first orientation of the UAV, wherein the second posture of the UAV includes a second translation of the UAV and a second orientation of the UAV, and wherein navigating the UAV toward the suitable location includes determining translational changes, orientation changes, and altitude changes between the second posture and a suitable area of ​​the landscape based on the second translation and the second orientation and the depth mapping.

4. The method according to any one of claims 1 to 3, wherein the suitable location for landing the UAV comprises: Planned landing location; The region with the highest landing area quality score; The nearest location with a landing area quality score above the threshold; or A suitable location identified within a predetermined interval from the latest flight path of the UAV.

5. A system for automatic safe landing site selection for unmanned aerial systems, comprising: Unmanned aerial vehicles (UAVs); An image capture device is connected to the UAV; and Processing circuitry, connected to the UAV and configured to: Receive two or more overlapping images depicting the landscape below the UAV from the image capture device; A depth mapping map of the landscape is generated based on the two or more overlapping images; Based on depth mapping, landscape areas with depth variance below a threshold are identified by feeding the depth mapping into a deep neural network (DNN). The DNN is trained to identify one or more patterns within the depth values ​​of the depth mapping that indicate one or more planar regions in the depth mapping. For each region of the landscape having a depth variance below the threshold, determine a landing zone quality score indicating the depth variance and semantic type of the region of the landscape; For each region of the landscape, the landing zone quality score is adjusted based on the determined semantic type of the region of the landscape; The appropriate location for landing the UAV is identified based on the adjusted landing area mass score. as well as The UAV is automatically navigated to and lands at the appropriate location. The generation of the depth mapping map includes: Based on the first pose of the UAV, predict a first image captured by the UAV in the first pose of the two or more overlapping images, and a second image of the two or more overlapping images and the second pose of the UAV; as well as The depth of the landscape depicted in the first image is determined based on the first pose, the first image, the second pose, and the second image.

6. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to: Receive two or more overlapping images from an image capture device that depict the landscape below the unmanned aerial vehicle (UAV); A depth mapping map of the landscape is generated based on the two or more overlapping images; Based on depth mapping, landscape areas with depth variance below a threshold are identified by feeding the depth mapping into a deep neural network (DNN). The DNN is trained to identify one or more patterns within the depth values ​​of the depth mapping that indicate one or more planar regions in the depth mapping. For each region of the landscape having a depth variance below the threshold, determine a landing zone quality score indicating the depth variance and semantic type of the region of the landscape; For each region of the landscape, the determined landing zone quality score is adjusted based on the determined semantic type of the region; The appropriate location for landing the UAV is identified based on the landing area mass score; as well as The UAV is automatically navigated to and lands at the appropriate location. The generation of the depth mapping map includes: Based on the first pose of the UAV, predict a first image captured by the UAV in the first pose of the two or more overlapping images, and a second image of the two or more overlapping images and the second pose of the UAV; as well as The depth of the landscape depicted in the first image is determined based on the first pose, the first image, the second pose, and the second image.

Images