Neural network-equipped unmanned aircraft for enhanced mission performance

Abstract

The present invention relates to the safe navigation of unmanned aerial vehicles (UAVs) and mission execution thereby.SOLUTION: Unmanned aerial vehicles or drones execute neural networks to support surveillance, reconnaissance, reporting, and other missions. The drone survey neural network monitors the data streams from the plurality of on-board sensors in real-time during navigation to the asset along a pre-programmed flight path and / or during its mission (e.g., when scanning and surveying the asset), and the HAPS platform executes the neural network while monitoring the air traffic, enabling the neural network to classify, predict, and analyze events in that airspace of coverage in real-time, and to learn from new events that were not seen or detected before.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to and benefit of U.S. Ser. No. 63 / 068,660, filed August 21, 2020.

[0002] The present invention relates generally to the safe navigation of unmanned aerial vehicles (UAVs) and the performance of missions therewith. [Background technology]

[0003] UAVs, commonly known as drones or unmanned aircraft systems (UAS), also referred to as remotely piloted aircraft, are air vehicles without a human pilot on board. Their path is controlled either autonomously by an onboard computer or by remote control from a pilot on the ground or in another vehicle. Drones are proliferating in number as awareness of their wide and diverse commercial potential increases.

[0004] Drones may utilize Global Positioning System (GPS) navigation capabilities, such as using GPS waypoints for navigation, and tend to follow pre-programmed flight paths. They may use onboard sensors to guide the drones to or around assets that will be scanned and surveyed. Drone systems may utilize various onboard sensors (including one or more cameras, radio frequency (RF) sensors, etc.) to monitor the operating environment, pre-calculate and follow a path to the asset to be surveyed, and conduct the survey.

[0005] Despite the presence of these sensors, drones may not be equipped to use the data they provide to accommodate unplanned changes in flight path (e.g., to avoid unexpected obstacles or perform collision-avoidance maneuvers) or to accommodate GPS drift, which can affect waypoint accuracy. GPS drift occurs when a preprogrammed flight path fails to account for calibration and correction of GPS drift vectors. For example, when an operator defines a flight plan with waypoints in the absence of such corrections, the navigation system may cause the drone to veer off course. Even a slight deviation may place all or part of the asset of interest outside the preprogrammed flight path and, consequently, outside the field of view of the onboard sensors. Conventional drones may also fail to respond to abnormal or unexpected conditions in the operating environment. For example, a drone may stop collecting information if sensor readings fall below a preset trigger threshold, or it may collect too much data if it changes direction off course and begins recording sensor data before actually reaching the target.

[0006] More generally, when inspecting structures such as antennas, drones are typically controlled by an operator within the drone's line of sight. This not only requires personnel to be present at the facility for each inspection, but also demands close and sustained attention to the drone's flight. The drone must approach closely to each area of ​​the structure requiring inspection, and these areas cannot be fully understood until data is first collected and the inspection begins. Furthermore, the drone must also maintain a safe distance from the structure and maneuver around obstacles during its approach, regardless of wind or weather conditions. The task of operating a drone safely and efficiently is particularly challenging when inspecting large facilities containing many assets (such as power plants).

[0007] Drone operations can be enhanced using high altitude pseudosatellite ("HAPS") platforms, also known as high altitude, long endurance ("HALE") platforms. These are unmanned aerial vehicles that operate persistently at high altitudes (e.g., at least 70,000 feet) and can be recharged by solar radiation during the day, thus remaining aloft for extended periods and providing satellite-like airspace coverage. HAPS drones equipped with RF communications payloads can provide large areas of RF coverage and provide connectivity and real-time communications and reconnaissance services to air traffic, including drones, either alone or in conjunction with existing communications satellite constellations, or ground-based telecommunications networks, airspace reconnaissance infrastructure, airspace navigation aids, or individual air traffic communications and reconnaissance systems.

[0008] HAPS platforms can be operated at less cost and with greater flexibility than satellite constellations, which cannot be easily recovered or last-minute expanded for upgrades to meet changing bandwidth demands. Additionally, satellites are not easily integrated with existing global air traffic reconnaissance systems, making them less well-suited than HAPS platforms for monitoring drone operations and maintaining safe separation between drones and manned air traffic operating in the same airspace. Land-based alternatives, such as telecommunications facilities, generally have short-range and small-area coverage, and again, extending coverage or capability can be costly and may not even be feasible due to terrain or man-made features.

[0009] Currently, HAPS platforms are limited in their ability to provide mission-level support to drones. While they can improve safety and potentially assist with navigation, they are not designed to help drones recover from path deviations, record all necessary measurements of a target asset, and so on, without over-collecting data. Summary of the Invention [Means for solving the problem]

[0010] Neural networks are computer algorithms loosely modeled on the human brain and excel at recognizing patterns, learning nonlinear rules, and defining complex relationships between data. They can help drones navigate, provide mission support, and ensure proper asset inspections without over-collecting data. HAPS platforms can run neural networks ("HAPSNN") as they monitor air traffic. The neural network enables them to classify, predict, and analyze events within their airspace in real time and learn from new events not previously seen or detected. A HAPSNN-equipped HAPS platform can provide near-100% reconnaissance of air traffic within its airspace, and the HAPSNN can process data received from drones to facilitate safe and efficient drone operation within the airspace. The HAPSNN also enables bidirectional connectivity and real-time monitoring, thus allowing drones to better perform their intended missions.

[0011] Alternatively, or in addition, drones may run neural networks to assist in surveying, reconnaissance, reporting, and other missions. In various embodiments, the present invention utilizes unsupervised “deep learning” neural networks running onboard low-altitude survey drones. Such drone survey neural networks (“DINNs”) may monitor data streams from multiple onboard sensors in real time during navigation to an asset along a preprogrammed flight path and / or during its mission (e.g., scanning and surveying the asset). The neural networks may communicate with unmanned traffic management systems directly or through the HAPSNN and with manned air traffic to enable safe and efficient drone operation within the airspace. Using land- and / or satellite-based communications networks or a bidirectional connection to the HAPSNN, the DINN may request or receive real-time airspace change authorizations and thus adapt the drone's flight path to account for airspace conflicts with other air traffic, terrain or obstacle collisions, or to optimize the drone's flight path for more efficient mission execution. Importantly, DINN can enable target acquisition and allow the drone to compensate for GPS drift or other course deviations or unexpected target anomalies by locating all assets to be surveyed.

[0012] One application that benefits from collaboration between HAPSNN and DINN is the precise location of passive intermodulation (PIM) on active telecommunications structures. PIM results from two or more strong RF signals originating from transmitters sharing antenna wiring, transmitters using adjacent antennas, or nearby towers with competing antenna patterns. PIM appears as a set of unwanted signals generated by the mixing of two or more strong RF signals in nonlinear devices such as loose or corroded connectors, cables, duplexers, circulators, damaged antennas, or nearby rusted members such as fences, barn roofs, bolts, etc. Other sources include poorly terminated or damaged cables with splices in the shield and aging lightning arresters. PIM can be time-consuming and difficult to detect using traditional search methods. The combination of DINN and HAPSNN, providing connectivity to unmanned traffic management, may enable drones to fly autonomously and unsupervised around telecommunications assets to detect, classify, and precisely locate PIM sources. If the drone detects a new or unexpected reading, it may be able to analyze and classify the nature of the reading based on its training. When the drone's survey is complete, it may use DINN to fly to the next pre-programmed asset location, adapting its flight path in real time along the way to optimize its operation within the airspace.

[0013] Thus, in a first aspect, the present invention relates to a platform for high-altitude flight and communication with UAVs. In various embodiments, the platform comprises a wireless transceiver, computer memory, communications equipment for communicating with UAVs operating within a defined airspace via the transceiver, and a computer including a processor and electronically stored instructions executable by the processor to use data received from the UAV as input to a computationally trained predictor to identify actions to be taken based on the state of the UAV and conditions affecting its flight.

[0014] In various embodiments, the predictor is a neural network. The computer may be further configured for communication with an air traffic control infrastructure. For example, the state of the UAV includes a spatial location and a velocity, and the computer may be further configured to determine land features and their likelihood of interfering with current or future communications between the UAV and a land communication source, and to establish a communications link bridging the UAV and the land communication source if the likelihood exceeds a threshold. The land features may be determined, for example, based on a locally or remotely stored map. In some embodiments, the computer is further configured to (i) predictively determine the need for a modified flight plan for the UAV, and (ii) cause the transceiver to wirelessly obtain approval for the modified flight plan and, in response, communicate the approved modified flight plan to the UAV.

[0015] The modified flight plan may avoid possible hazards, such as weather conditions or a possible collision with another UAV. The modified flight plan may allow for unplanned surveys of land assets.

[0016] In another aspect, the invention relates to a method for controlling a defined airspace using a high-altitude pseudolite vehicle. In various embodiments, the method includes wirelessly monitoring the status of at least one unmanned aerial vehicle (UAV) operating within the defined airspace, detecting conditions affecting flight within the defined airspace, using data received from the at least one UAV as input to a computationally trained predictor to identify an action to be taken based on the status of the UAV and the detected conditions affecting flight, and causing the high-altitude pseudolite vehicle to take the action in response.

[0017] In various embodiments, the predictor is a neural network. The action may include communication with air traffic control infrastructure. The UAV state may include a spatial location and a velocity, and in various embodiments, the method further includes determining a land feature and its likelihood of interfering with current or future communication between the UAV and a land communication source, determining that the likelihood exceeds a threshold, and responsively establishing a communication link bridging the UAV and the land communication source. The land feature may be determined based on a locally or remotely stored map.

[0018] In some embodiments, the state of the UAV includes a spatial position and a velocity, and the method further includes computationally determining a predicted need for a modified flight plan for the UAV and, accordingly, obtaining approval for the modified flight plan and communicating the approved modified flight plan to the UAV. The modified flight plan of the method may avoid potential hazards, such as weather conditions or a potential collision with another UAV. Alternatively or additionally, the modified flight plan may allow unplanned surveying of land assets.

[0019] More generally, embodiments of the present invention enable a drone to autonomously navigate to a survey facility, execute a preliminary flight plan, calculate a survey path that will bring it close to the area of ​​the asset requiring survey, collect relevant survey data, and activate sensors (including one or more cameras) that indicate the specific area requiring proximity and imaging. The drone may recognize obstacles in its path, monitor relevant weather conditions, and alter both its flight path and approach pattern to avoid collisions. These operations may occur onboard the drone with or without the assistance of a HAPS. In particular, the drone may include a DINN, as discussed above, that analyzes image frames captured in real time by an onboard camera as the drone proceeds. Furthermore, without communicating with a HAPS or land communications infrastructure, the DINN may monitor data streams from multiple onboard sensors in real time during navigation to the asset along a pre-programmed flight path and / or during its mission (e.g., as it scans and surveys the asset). More typically, a DINN is configured for communication with unmanned traffic management systems and manned air traffic to enable safe and efficient drone operation within the airspace. Using bidirectional connections to land and / or satellite-based communication networks, the DINN can request or receive real-time airspace change authorizations and thus adapt drone flight paths to account for airspace conflicts with other air traffic, terrain or obstacle collisions, or to optimize the drone's flight path for more efficient mission execution. Importantly, a DINN can enable target acquisition and enable drones to compensate for GPS drift or other course deviations or unexpected target anomalies by locating all assets to be surveyed.

[0020] Thus, in a first aspect, the present invention relates to a UAV comprising, in various embodiments, a flight package, a navigation system, an image acquisition device, communications equipment, computer memory, and a computer including a processor and electronically stored instructions executable by the processor to use data received from the image acquisition device as input to a computationally trained predictor to identify and classify objects appearing in images acquired by the image acquisition device during flight.

[0021] In various embodiments, the predictor is a neural network. The UAV includes a database of actions, and the computer is configured to select and cause execution of an action from the database in response to a detected object that is classified by the predictor.

[0022] The communications equipment may be configured to interact with land and air traffic control systems. In some embodiments, the UAV also includes a weather reconnaissance module for monitoring weather conditions during drone flight, and the computer includes data from the weather reconnaissance module when selecting an action. The computer may be configured to cause the UAV to execute a preliminary flight plan around the asset to be surveyed and to calculate and execute a modified flight plan around the asset based on the object classification made by the predictor during the preliminary flight plan. The computer may be further configured to communicate with the HAPS vehicle, e.g., to execute flight commands received from the HAPS vehicle, to communicate modified flight to the HAPS vehicle to obtain approval from the air traffic control infrastructure, and / or to communicate detected but unclassified objects to the HAPS vehicle and receive classifications and associated actions to be taken from the HAPS vehicle.

[0023] In another aspect, the invention relates to a method for surveying assets using a UAV. In various embodiments, the method includes acquiring digital images in real time during flight of the UAV, computationally analyzing the acquired digital images using a computationally trained predictor to identify and classify objects appearing in the images, and taking an action based on at least one classified object. The predictor may be a neural network, and the action may be determined based on a database lookup in response to a detected object classified by the predictor. For example, the action may be altering the flight path of the drone.

[0024] In various embodiments, the method may further include monitoring weather conditions during flight of the drone, and the action may be further based on the monitored weather conditions. The method may further include obtaining a signal from the asset to be surveyed, and the action may be based on the obtained signal. The method may further include acquiring an image of the asset to be surveyed, and the action may be based on the obtained image. The method may further include causing the UAV to execute a preliminary flight plan around the asset to be surveyed based on object classification performed by the predictor during the preliminary flight plan, and to calculate and execute a revised flight plan around the asset.

[0025] In some embodiments, the method includes communicating with the HAPS vehicle, e.g., communicating the modified flight to the HAPS vehicle to obtain approval from the air traffic control infrastructure, and / or communicating the detected, but unclassified, object to the HAPS vehicle and receiving a classification and associated action to be taken from the HAPS vehicle. [Brief explanation of the drawings]

[0026] The foregoing and the following detailed description will be more readily understood when taken in conjunction with the drawings.

[0027] [Figure 1] Figure 1 conceptually illustrates a HAPS providing reconnaissance and connectivity within monitored airspace.

[0028] [Figure 2] FIG. 2 conceptually illustrates the role of the HAPSNN in anticipating and providing the need for a communications link within monitored airspace when obstacles or terrain block line-of-sight communications.

[0029] [Figure 3] Figure 3 conceptually illustrates the role of HAPSNN in anticipating the need for and proactively acquiring authorized airspace expansion for drones migrating through the monitored area.

[0030] [Figure 4] FIG. 4 conceptually illustrates HAPSNN prediction of the need for and approval of a flight plan deviation.

[0031] [Figure 5] FIG. 5 conceptually illustrates a HAPSNN providing in-airspace reconnaissance and connectivity to relay signals from multiple navigation aids to air traffic that are out of range or blocked from receiving those signals.

[0032] [Figure 6] FIG. 6 conceptually illustrates a HAPSNN, which provides reconnaissance and connectivity within airspace to monitor and avoid airspace conflicts.

[0033] [Figure 7] FIG. 7 conceptually illustrates HAPSNN prediction of the need for, and authorization for, flight plan deviations to enable efficient execution of a reconnaissance mission.

[0034] [Figure 8A]FIG. 8A is a block diagram of an exemplary HAPSNN and DINN architecture, in accordance with an embodiment of the present invention.

[0035] [Figure 8B] FIG. 8B illustrates schematically how a HAPSNN may support the operation of a DINN.

[0036] [Figure 9A] 9A and 9B show a schematic representation of the generation of a flight plan for executing a survey mission, which may be modified in real time as the drone conducts the survey. [Figure 9B] 9A and 9B show a schematic representation of the generation of a flight plan for executing a survey mission, which may be modified in real time as the drone conducts the survey.

[0037] [Figure 10] FIG. 10 is an elevation view illustrating various anomalies associated with antennas that a drone according to the present disclosure may identify.

[0038] [Figure 11] FIG. 11 conceptually illustrates drone detection of anomalous electromagnetic (EM) signatures from a transducer.

[0039] [Figure 12] FIG. 12 conceptually illustrates a drone in flight reading stored data indicating an anomaly.

[0040] [Figure 13A] 13A-13E illustrate a schematic process for establishing an optimal flight path through a complex survey environment. [Figure 13B] 13A-13E illustrate a schematic process for establishing an optimal flight path through a complex survey environment. [Figure 13C] 13A-13E illustrate a schematic process for establishing an optimal flight path through a complex survey environment. [Figure 13D] 13A-13E illustrate a schematic process for establishing an optimal flight path through a complex survey environment. [Figure 13E] 13A-13E illustrate a schematic process for establishing an optimal flight path through a complex survey environment.

[0041] [Figure 14] FIG. 14 illustrates schematically the detection and classification of centers of radiation of multiple antennas. DETAILED DESCRIPTION OF THE INVENTION

[0042] Referring first to Figures 1-7, which illustrate the functions performed by a conventional HAPS platform 105 and the manner in which these functions can be enhanced through the operation of a HAPSNN. In Figure 1, the HAPS platform 105 communicates with multiple cell phone towers, representatively shown at 108, multiple air traffic control systems, representatively shown at 110, and a fleet of aircraft, representatively shown at 113, all of which reside within a defined airspace 115 monitored by the HAPS platform 105. The HAPS platform 105 may also communicate directly and / or via cell phone towers with multiple drones 118 within the airspace 115. Finally, the HAPS platform 105 may communicate with a satellite, representatively shown at 120, for example, for geolocation to maintain a substantially fixed position. In this manner, the HAPS platform 105 obtains and updates a complete snapshot of the airspace 115, monitors air traffic therein, and serves as a communications hub between the various intercommunicating entities within the airspace 115.

[0043] As seen in FIG. 2 , HAPS 105 monitors the location, status, and speed of drones 118 and manned air traffic 133 operating within monitored airspace 115. The onboard HAPSNN recognizes that, given the drone's 118 operating altitude, terrain or other obstacles 205 will block line-of-sight communications between the drone 118 and the air traffic control system 110, and further, the aircraft 113. While the obstacles 205 are recognized and mapped in real time by HAPS 105, they may more likely be stored in a map accessible locally to HAPS 105 or via a wireless link. More specifically, HAPSNN may calculate the likelihood that the obstacles 205 will interfere with current or future communications between the drone 118 and the air traffic control system 110, and if the likelihood exceeds a threshold, register the need to establish a communications link bridging the drone 118 and the air traffic control system 110. In that case, HAPS105 responsively obtains the state (position, including altitude, and trajectory) of drone 118 by any suitable modality or combination thereof, such as observation, telemetry, signal monitoring, and / or direct communication with a global air traffic control system that monitors drone 118 and / or its flight.

[0044] As a result of this recognized need, HAPS 105 may enter the communications network as an intermediate node, relaying messages (i.e., acting as a transmission link) between drone 118 and air traffic control system 110 (e.g., UTM and LAANC) or other ground-based air traffic reconnaissance infrastructure. In the absence of HAPSNN, HAPS 105 would operate accordingly; however, for example, if drone 118 was previously communicating with HAPS 105 and air traffic control system 110, HAPS 105 may serve as a backup communications channel when direct communications between drone 118 and air traffic control system 110 are lost as drone 118 approaches obstacle 205. HAPSNN facilitates proactive and predictive intervention by HAPS 105 even in the absence of prior communications between drone 118 and air traffic control system 110. Based on stored or acquired knowledge of the locations of terrain and fixed communication features within airspace 115 and the calculated trajectory of drone 118 (which may have just entered airspace 115), HAPSNN recognizes the need for communication between drone 118 and control system 110 and causes HAPS 105 to establish a wireless link with itself as a hub. Similarly, based on knowledge of the terrain and monitored altitudes of drone 118 and manned aircraft 113, HAPSNN may cause HAPS 105 to establish a wireless link with itself as a hub between drone 118 and aircraft 113.

[0045] FIG. 3 illustrates a situation in which a drone 118, transitioning through altitude-restricted, authorized airspace 305, will encounter obstacles, shown collectively at 310, around which it must fly to remain within airspace 305. This deviation from the preprogrammed flight path can be quite large, depending on the extent of the obstacles, requiring a course correction that the drone is not equipped to handle with great accuracy, potentially leading to it missing all or part of the target when it reaches it. With HAPS 105 in communication with drone 118, HAPSNN predicts the need for additional airspace authorization and requests an extension to region 315. Once authorization is obtained, HAPS 105 communicates a relaxation of the altitude restriction to drone 118. In some embodiments, HAPS 105 can calculate a revised flight path 320 at a higher altitude for drone 118, enabling it to stay on course for the target without lateral deviations that could affect navigation accuracy.

[0046] 4, the planned and authorized flight path 405 for the drone 118 will take it through a hazardous local weather pattern that the HAPS 105 detects using onboard radar or from real-time weather updates. The HAPSNN, aware of the drone's flight path and weather conditions, recognizes the need for, and it, or another onboard computation module, computes, an alternative flight segment 410. The HAPS 105 obtains authorization for the new flight path 410 and communicates it to the drone's 118 navigation system.

[0047] 5 shows a drone 118 traveling through an obstacle that blocks line-of-sight communication between a cell phone tower 108 and a source 505 of a National Air System navigation aid (VOR, VOR / DME, TACAN, etc.). The HAPSNN infers this condition from knowledge of the terrain features in the airspace 115 and the state of the drone 118. As a corollary, the HAPSNN causes the HAPS 105 to at least relay signals from the blocked source 108, 505, or to establish a wireless link between itself and the communicating entity 108, 118, 505 as a hub.

[0048] In FIG. 6 , drone 605 is flying within its authorized airspace 610. A second drone 615 enters the airspace 610, and the HAPSNN detects that drones 605, 615 are not or cannot communicate. For example, one or both of drones 605, 615 may be “non-cooperative,” i.e., not equipped to communicate with other drones and avoid collisions along their flight paths. The HAPSNN may infer this based on communication, or the absence thereof, between the drones and / or with ground-based air traffic reconnaissance and surveillance systems. To avoid the collision, the HAPSNN determines that drone 605 should follow an alternate flight path 620, which may be temporary until drone 605 passes drone 615. The HAPSNN or another onboard computation module calculates the new flight path 620. In response, HAPS 105 obtains authorization for the new flight path 620 and communicates it to the drone's 605 navigation system.

[0049] 7 , drone 118 follows a flight path 705 within an authorized airspace corridor 707, using, for example, an RFID reader, surveying or capturing data readings from a series of assets (e.g., a series of transducers 7101...7105 along a power line 712). HAPSNN monitors the data stream received by drone 118, and if the drone detects an anomaly, HAPSNN may infer that drone 118 will request a change to its airspace authorization. In particular, if the anomaly is associated with transducer 7103, the predicted authorization need may cover area 720. HAPSNN may cause HAPS 105 to take action, for example, relay the information to the appropriate UTM / LAANC system, or request a change to airspace authorization on behalf of drone 118, which will then freely survey transducer 7103 in closer proximity and record its image for real-time or later evaluation.

[0050] FIG. 8A illustrates a representative HAPSNN architecture 800, which includes a HAPS vehicle and various hardware and software elements. Generally, multiple software subsystems, implemented as instructions stored in computer memory, are executed by a conventional central processing unit (CPU) 802. The CPU 802 may control the flight and operation of the HAPS vehicle and the functions described below, or these functions may be distributed among separate processors 802. Additionally, for efficient execution of neural network functionality, the system may include a dedicated graphics processing unit. An operating system (e.g., MICROSOFT WINDOWS®, UNIX®, LINUX®, iOS, or ANDROID®, etc.) provides low-level system functions such as file management, resource allocation, and message routing to and from hardware devices and software subsystems (including at least one non-volatile storage element 803), which executes within computer memory 804. More generally, the HAPSNN 800 may include modules implemented in hardware, software, or a combination of both. With respect to the functionality provided in software, the programs may be written in any of several high-level languages, such as PYTHON, FORTRAN, PASCAL, JAVA, C, C++, C#, BASIC, various scripting languages, and / or HTML. The software modules and mechanical and flight features involved in the operation of the HAPS vehicle are conventional and are not shown. See, e.g., U.S. Patent No. 10,618,654.

[0051] HAPSNN 800 includes a neural network module 805, a transceiver module 808, and a field-programmable gate array (FPGA) 810. The transceiver module 810 and FPGA 810 may comprise or be part of a communications facility configured to support airborne communications between vehicles and with land and satellite-based control infrastructure. HAPSNN 800 may, but need not, operate in conjunction with a drone equipped with a DINN 812. Cloud neural network module 805 may be local to the HAPS vehicle but more typically operates in the cloud, i.e., on a remote (e.g., land) server in wireless communication with the HAPS vehicle, as described below. Modules 808, 810 are typically located on the HAPS vehicle itself.

[0052] Cloud neural network module 805 includes a classification neural network 815 that processes images and data received from drones in real time via agile transceiver 808, which may be passed to cloud neural network 805. Classification neural network 815 is trained using a database 817 of training images related to the missions undertaken by the monitored drones. Classification neural network 815 processes and classifies the received images and data and detects, i.e., calculates the probability of, anomalies associated therewith. That is, anomalies may be detected based on something in the received images or when considered in parallel with other drone telemetry. For example, an otherwise unexceptional image may trigger anomaly detection when taken in conjunction with weather conditions reported by the drone. When an anomaly is detected, classification neural network 815 may consult a classification database 819 to determine an appropriate response; that is, database 819 includes records each defining an anomaly and one or more associated actions that may be taken in turn. If an anomaly is detected that does not have a database record, the image may be transmitted for human review and classification. The new classification is then added to the training database 817 and used to retrain the neural network 815. The resulting adjusted weights may be back-propagated by a cloud server associated with the neural network 805 to the DINN 812 (if present), which transmits similar mission profiles to the drone and other drones in the field. This procedure is described further below.

[0053] The agile transceiver package 808 includes Automatic Dependent Reconnaissance-Broadcast (ADS-B), Traffic Collision Avoidance System (TCAS), Secondary Reconnaissance Radar (SSR), and Automatic Dependent Reconnaissance-Rebroadcast (ADS-R) subsystems operating at 978 MHz, 1,090 MHz, and 1,030 MHz for query, response, and rebroadcast. These enable the HAPSNN 800 to "listen" to the location of manned air traffic, so the neural network 815 can computationally represent nearby traffic in 3D or 2D space and analyze any conflicts between drones and manned air traffic. This can be achieved by broadcasting the drone's location to manned air traffic or the manned air traffic's location to the drone. Urgent warnings are issued to manned and / or unmanned traffic with instructions on which direction to move to avoid collisions in the airspace.

[0054] The agile transceiver 808 may include a cellular network package, including 3G, 4G, LTE, 5G, or any future telecommunications protocol and bandwidth, to support communications links between drones operating within the HAPSNN 800's airspace, using land-based telecommunications networks utilized by some UTM systems, or using backhaul communications channels to transmit data from the HAPS to a cloud-based neural network. The VHF and UHF transceiver (TX / RX) modules may be used to monitor navigation aids such as VOR, VOR / DME, or TACAN, allowing the neural network 805 to analyze the drone and HAPS's positions using signal time-of-flight if the GPS signal is lost. This also enables leveraging satellite communications constellations to transmit or receive data, should the need arise. The drone virtual radar (DVR) data link facilitates communication with drone platforms implementing the present technology (e.g., as described in U.S. Pat. No. 10,586,462) to send and receive air traffic position information to aid in analyzing collisions or tracking drones. The neural network (NN) data link is a dedicated high-bandwidth backhaul channel that allows the HAPSNN 800 to communicate with the DINN neural network computation engine 825, which transmits real-time data received from multiple drones operating within the monitored airspace and receives predictions and action instructions obtained from the classification database 819. The FPGA 810 is employed as a hardware accelerator to run software that tunes the transceiver 808 and filters out noise.

[0055] A representative DINN 812 implemented within a drone 118 includes a neural network computation engine 825, a classification database 825, and "back-end" code for performing various data handling and processing functions as described below. In addition, the drone 118 includes communications equipment comprising or consisting of a set of agile transceivers 808 and FPGAs 810, as detailed above. The drone 118 may also include a CPU 802, storage 803, and computer memory 804.

[0056] As mentioned, DINN 812 may interact with HAPSNN 800, but either can be present and operate on its own; i.e., HAPSNN is unnecessary for successful deployment and use of DINN, but HAPSNN may fulfill a reconnaissance and safety role for an air vehicle lacking a DINN. DINN 812's role is to enable drone 118 to classify objects of interest on the asset it is surveying and obstacles that may need to be avoided during flight (e.g., recognizing cell phone towers to be surveyed and cracked antennas on such towers). Neural network 825 is configured to process and classify images received from image acquisition device 827, e.g., a video camera on drone 118. Thus, neural network 825 may be a convolutional neural network (CNN) programmed to detect and recognize objects in incoming images. These may be classified based on neural network training, and DINN 812 (e.g., back-end code) may consult classification database 830 to determine an appropriate response to the detected image. In this case, database 819 includes records defining objects, each associated with some semantic meaning or action. For example, if neural network 825 detects a tree in an incoming image, the corresponding database entry may identify the tree as an obstacle and trigger an evasive maneuver that drone navigation system 832 performs by controlling the drone's steering and propulsion systems. These are part of the drone's flight package 835, which is conventional and therefore not shown in detail, but includes a power source, a communications platform, a propulsion and steering system, an autopilot system, etc.

[0057] DINN 812 may also receive data from one or more reconnaissance systems 837, 839, which may include one or more of a DVR, UTM, LAANC, ADS-B, and TCAS systems. While these may be implemented as part of the drone's communications platform, they are depicted conceptually within DINN 812 because neural network 825 may use this data in classifying images. Similarly, weather reconnaissance system 842 would be implemented in a more conventional manner within the drone's communications platform, but is again depicted as part of DINN 812 because weather conditions may be relevant to image classification or database lookup; as shown in FIG. 4 , the same visual scene may prompt different actions depending on the weather; for example, drone 118 may give more mooring room to a building in high wind conditions.

[0058] In embodiments in which drone 118 interacts cooperatively with HAPSNN 800, the latter may provide further support and stronger classification capabilities. For example, images with detected objects that are unclassifiable by neural network 825 may be uploaded to HAPSNN 800 for review, and real-time instructions issued in return by HAPSNN may be executed by the drone's navigation system 832. Furthermore, HAPSNN 800 may update or supply different weighting files for neural network 825 in real time to better adapt the drone's mission based on the classifications being made by the drone (and communicated to HAPSNN 800 in real time). Neural network 825 responsively loads these new weighting files as they are received.

[0059] This process is illustrated in Figure 8B. Referring also to Figure 8A, the DINN 812 processes incoming image frames (e.g., at approximately 30 frames per second (FPS)) in real time, enabling the drone 118 to react quickly enough to avoid collisions and fly around the tower 850. Thus, the drone 118 may include a graphics processing unit (GPU) to support CNN operations. The real-time frame analysis allows the GPU to process the images, classify items of interest on the asset to be inspected, and inform back-end code of the classification, allowing the back-end code to execute logic that reacts to the classification, typically by performing a lookup in the classification database 830 to obtain an action corresponding to the classification.

[0060] The drone 118 transmits image data to the HAPSNN, which includes a high-precision CNN (within the compute engine 815, or even within the HAPS itself, if desired) capable of processing, for example, 60 megapixel (MP) photographic images per second. The CNN architecture is designed for speed and accuracy of classification by leveraging back-end logic running on the compute engine 825. This back-end logic can modify the CNN's weights and configuration files based on the asset being classified based on the first few images of the asset captured by the drone. These preliminary images are collected as part of a "preliminary" flight path around the asset at a safe distance and may be 60MP or greater in resolution. These preliminary images are reduced to the CNN's input image size (e.g., 224x224, or larger, depending on the asset to be investigated) and passed through successive convolutional layers (e.g., 20), followed by an average pooling layer and a fully connected layer that is pre-trained to classify different assets (e.g., 100 types of assets). Once the asset type is identified, weights and configuration files can be modified and more (e.g., four) convolutional layers can be added, followed by two fully connected layers to output probabilities and bounding boxes of objects of interest or areas that may be present on the asset. Images uploaded from the drone can be increased in size (e.g., to 448x448) because this type of classification requires more granular detail to be present. The degree of size increase can be dynamically controlled, for example, scaled up if enough detail is not detected for reliable classification.

[0061] The fully connected layers predict class probabilities and bounding boxes (i.e., cracked antenna, rust, corrosion, etc.). As an example, the final layers may use linear activation, while the convolutional layers may use leaky ReLu activation.

[0062] Once the back-end logic of the calculation engine 825 detects the presence of a class and bounding box coordinates, it can switch to and trigger the centroid tracker function to move that particular classification to the center of the field of view of the drone's image acquisition device. The back-end logic works with the ranging calculation engine to analyze the safest flight path for the drone to approach and position the image acquisition device for high-resolution scanning.

[0063] Thus, the preliminary flight path establishes the type of asset in view and registers the asset's position in 3D space relative to the drone to account for any GPS drift vectors. If any obstacles or hazards are present within the operating area, they are classified and their position in 3D space is registered. The center of gravity tracker is activated once a classification within the area of ​​interest is detected, keeping the object of interest centered within the field of view. The range calculation engine controls the drone's forward and reverse movement. Before any of these commands are executed, the drone's position in 3D space relative to the asset and any obstacles present within the operating area is obtained. This data is run through back-end logic that analyzes a safe GPS waypoint flight path that will move the drone to the area of ​​interest. In GPS-denied areas, this flight path can still be analyzed and executed using Kalman filtering of the inertial data in conjunction with center of gravity and range functionality. The flight path is fine-tuned in real time via the center of gravity tracker and range calculation engine. Note that the centroid tracker can be implemented by HAPSNN 800 rather than DINN 812.

[0064] In step 855, the HAPSNN CNN (“HP-CNN”) processes each image to detect objects therein using standard object detection routines (e.g., YOLO) and attempts to identify (i.e., classify) all detected objects based on its prior training (discussed further below). Detected objects that cannot be identified are stored in a database and made available to personnel for identification and labeling (step 857). The HP-CNN is then retrained on the augmented dataset including the newly identified and labeled object data (step 859), resulting in the generation of new CNN weights (step 862). If the real-time neural network 825 resident on the drone 118 is also a CNN (“RT-CNN”), the HAPSNN 800 may push these weights to the RT-CNN, which receives and loads them. That is, the HP-CNN and RT-CNN may be the same or substantially similar so that the CNN weights generated for the HP-CNN can be propagated across the drone fleet.

[0065] The HP-CNN (or, in some embodiments, the RT-CNN on itself) can be trained in a conventional manner. In particular, the CNN is trained on labeled images of objects likely to be encountered by drones as they perform their missions, resulting in a CNN capable of analyzing and classifying objects most likely to be encountered by the drones within their typical flight paths. Because the training set may not be exhaustive and drones will inevitably encounter unknown objects during use, the above-described process of locating unrecognized objects, storing them for manual labeling, and then retraining the CNN on an expanded dataset helps minimize the risk of accidents by constantly enriching the drones' visual vocabulary and action repertoire. This may be most efficiently accomplished using a HAPSNN as a central training hub that receives unclassifiable objects from many drones and keeps all of its neural networks updated to reflect the latest classification capabilities, although it is nonetheless conceivable that this training and retraining functionality could be implemented on individual DINNs.

[0066] 9A-13E illustrate various drone applications and how a DINN can be used to control and simplify drone operation. In FIG. 9A, the DINN 812 guides the drone 118 around an antenna 900 to be investigated according to a flight pattern 910 that can change in real time when the drone 118 detects an anomaly or structure that requires closer inspection. For example, as shown in FIG. 9B, the flight path 910 can be altered to keep the drone 118 away from a power line 915 that would be recognized as an obstruction.

[0067] Referring to FIG. 10 , antenna 1000 has a crack 1005 that can be identified by a DINN-equipped drone. The backend can execute openCV functions to fix the center of gravity on antenna 1000 and move the drone to center the antenna in the field of view to facilitate image acquisition. This can be combined with a depth map generated by a stereo RGB camera of known sensor size, lens properties, and camera focal separation to position the drone 118 close enough to the antenna for the camera to resolve sufficient detail to successfully pick up the crack. Other obstacles that can be recognized by the drone 118 include structural corrosion 1010 and paint coating anomalies 1015. The drone 118 payload can include additional sensors, such as EM sensors and infrared cameras, and the DINN 812 can include additional modules, such as spectrum analysis and / or electromagnetic computation engines. These enable the drone 118 to detect PIM signatures at the coaxial connector 1025 and other EM anomalies, such as magnetic fields emanating from the lightning arrestor 1030 (indicating possible arcing), and abnormal thermal signatures 1035. These conditions can be diagnosed and observed by the drone 118 or using an onboard computation engine and / or in cooperation with the HAPSNN 800, as described above. The DINN 812 can further include a ranging computation engine for performing data fusion between depth maps acquired as discussed above and laser rangefinder / radar / ultrasonic or other ranging sensors the drone payload may contain to derive the most accurate distance measurements, a computer vision computation engine for performing image processing and analytical functions (such as center of gravity location, as described above), and / or infrared and multispectral computation engines for analyzing infrared and other imagery acquired at wavelengths outside the visible spectrum.

[0068] As shown in FIG. 11 , under normal operation, if drone 118 detects an unusual EM signature from asset 1100, such as an anomalous, non-vertical magnetic field line 1105, while magnetic field lines are vertical, as shown at 1110, it may record the anomaly or, in embodiments in which drone 118 is in communication with HAPSNN, report it to HAPSNN, which may autonomously call in a drone with a more specialized survey payload to better diagnose the condition. Drone 118 may also communicate with or otherwise interact with the asset to be surveyed. In FIG. 12 , drone 118 reads data stored within transducer 1200 that indicates the anomaly. That is, transducer 1200 has self-diagnostic capabilities, and when an operational anomaly is previously detected, data indicating the detection will have been stored internally. If transducer 1200 is surveyed periodically by drone 118 and the abnormal condition does not require urgent attention, it may not be necessary for the transducer to communicate its presence to supervisory personnel in response to a detection. Rather, when drone 118 performs a reconnaissance flight, it can query the memory of all transducers, thereby obtaining previously stored data indicating transducer anomaly detection. This can prompt drone 118 to conduct a closer reconnaissance (as shown in the figure) in response to a condition classification and database lookup. Again, the HAPSNN recognizes the condition and sends new weights to drone 118's DINN, enabling it to make condition-specific classifications as it probes transducers 1200.

[0069] 13A-13E illustrate how the DINN 812 can establish an optimal flight path through a complex survey environment. Figures 13A and 13B illustrate a distribution substation 1300 that includes a series of outgoing transmission lines, shown collectively at 1310, a voltage regulator 1315, a step-down converter 1320, a series of switches 1325, one or more lightning arrestors 1330, and incoming transmission lines, shown collectively at 1335. Any of these components may be experiencing an anomaly or malfunction, and all may be investigated by the drone 118. As shown in Figures 13C and 13D, the drone 118, alone or in conjunction with the HAPSNN, may perform a preliminary flight pattern 1350 to gather imagery for analysis by the DINN. The DINN 812 analyzes images acquired by the drone's onboard camera and classifies the various components 1310-1335. Based on these classifications, the navigation module 832 (see FIG. 8 ) or back-end code calculates an optimized flight plan 1355 that enables all components 1310-1335 to be properly and efficiently surveyed by the drone 118. More specifically, each component present within the substation 1300 is classified during a preliminary survey flight path 1350, which are far from the substation 1300 to accommodate GPS drift vectors and other unknowns related to obstacles or inaccuracies around the initial flight path. Once a component is classified, its location in the image and the drone's location are registered. This process is repeated multiple times throughout the preliminary survey, enabling the back-end code to triangulate and locate each classified asset in 3D space, so that a more accurate survey flight path can be calculated that will move the drone and payload closer to the asset. This new flight plan 1355 is then executed, and again, the DINN 812 classifies the assets as the flight path is executed. Preliminary studies have shown that because drones are far away, DINN812 can only classify large items when the resolution of the image acquisition device is fixed.Once the closer flight path 1355 is executed, more asset details will be detected, enabling DINN 812 to classify the new item and readjust the drone's path as needed. The ranging calculation engine calculates the nearest allowable approach distance between the drone and the asset, consistent with acceptable and safe limits.

[0070] Yet another application in which a complex survey operation can be optimized by a DINN is illustrated in Figure 14. Here, the objective is to find the "radiation centers" i.e., the radiation centers 1410 of each of antennas 1410, 1415, 1420 mounted on a telecommunications structure 1400 at different elevations. R , 1415 R , 1420 R The DINN classifies the radiation centers and then, for each antenna, identifies the center of gravity and moves the radiation center to the center of the drone camera's field of view so that a barometer reading (altitude) can be recorded. The navigation module 832 can then use this altitude to generate a trajectory flight path that enables the drone 118 to acquire 3D model data for reconstruction of the antenna structure.

[0071] The terms and expressions employed herein are used as terms and expressions of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features shown and described, or portions thereof. Additionally, while certain embodiments of the present invention have been described, it will be apparent to those skilled in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the present invention. Therefore, the described embodiments are to be considered in all respects merely as illustrative and not restrictive.

[0072] The claims are as follows:

Claims

1. An unmanned aerial vehicle (UAV), Flight packages and Navigation systems, an image acquisition device; Communication equipment and Computer memory; a computer including a processor and electronically stored instructions executable by the processor to use data received from the image acquisition device as input to a predictor that is computationally trained to identify and classify objects that appear in images acquired by the image acquisition device during flight; An unmanned aerial vehicle (UAV) comprising:

2. The UAV of claim 1 , wherein the predictor is a neural network.

3. The UAV of claim 1 , wherein the communications equipment is configured to interact with land and air traffic control systems.

4. The UAV of claim 1 , further comprising a database of actions, the computer configured to select an action from the database and cause its execution in response to a detected object classified by the predictor.

5. The UAV of claim 4 , further comprising a weather reconnaissance module for monitoring weather conditions during drone flight, the computer including data from the weather reconnaissance module when selecting an action.

6. 5. The UAV of claim 4, wherein the computer is configured to cause the UAV to execute the preliminary flight plan around an asset to be surveyed based on the classification of objects performed by the predictor during the preliminary flight plan, and to calculate and execute a revised flight plan around the asset.

7. The platform of claim 6 , wherein the computer is further configured to communicate with a High Altitude Pseudolite (HAPS) vehicle.

8. The UAV of claim 7 , wherein the computer is further configured to execute flight commands received from the HAPS vehicle.

9. The UAV of claim 7 , wherein the computer is further configured to communicate the modified flight to the HAPS vehicle for approval from an air traffic control infrastructure.

10. 8. The UAV of claim 7, wherein the computer is further configured to communicate detected but unclassified objects to the HAPS vehicle and receive classifications and associated actions to be taken from the HAPS vehicle.

11. 1. A method for surveying an asset using an unmanned aerial vehicle (UAV), the method comprising: acquiring digital images in real time during flight of the UAV; computationally analyzing the obtained digital images using a computationally trained predictor to identify and classify objects appearing in the images; taking an action based on the at least one classified object; A method comprising:

12. The method of claim 11 , wherein the predictor is a neural network.

13. The method of claim 11 , wherein the action is determined based on a database lookup in response to a detected object being classified by the predictor.

14. The method of claim 11 , wherein the action is altering a flight path of the drone.

15. 15. The method of claim 14, further comprising monitoring weather conditions during flight of the drone, and wherein the action is further based on the monitored weather conditions.

16. The method of claim 14 , further comprising obtaining a signal from the asset to be investigated, and wherein the action is further based on the obtained signal.

17. The method of claim 14 further comprising the step of obtaining an image of the asset to be inspected, and wherein the action is further based on the obtained image.

18. 12. The method of claim 11, further comprising: causing the UAV to execute the preliminary flight plan around an asset to be surveyed based on object classifications made by the predictor during the preliminary flight plan, and calculating and executing a revised flight plan around the asset.

19. The method of claim 11 further comprising communicating with a High Altitude Pseudolite (HAPS) vehicle.

20. 20. The method of claim 19, further comprising communicating the modified flight to the HAPS vehicle to obtain approval from an air traffic control infrastructure.

21. 20. The method of claim 19, further comprising communicating detected but unclassified objects to the HAPS vehicle and receiving a classification and associated action to be taken from the HAPS vehicle.

23. 1. A platform for high altitude flight and communication using unmanned aerial vehicles (UAVs), comprising: A wireless transmitter / receiver; Computer memory; a communication facility for communicating with UAVs operating within the defined airspace via the transceiver; a computer including a processor and electronically stored instructions executable by the processor to use data received from the UAV as input to a computationally trained predictor to identify actions to be taken based on the state of the UAV and conditions affecting its flight; A platform that includes:

24. The platform of claim 23 , wherein the predictor is a neural network.

25. 24. The platform of claim 23, wherein the computer is further configured for communication with an air traffic control infrastructure.

26. 26. The platform of claim 25, wherein the state of the UAV includes spatial position and velocity, and the computer is further configured to determine land features and their likelihood of interfering with current or future communications between the UAV and a land communication source, and to establish a communications link bridging the UAV and the land communication source if the likelihood exceeds a threshold.

27. 27. The platform of claim 26, wherein the land features are determined based on a locally or remotely stored map.

28. 26. The platform of claim 25, wherein the state of the UAV includes a spatial position and a velocity, and wherein the computer is further configured to (i) predictively determine a need for a modified flight plan for the UAV, and (ii) cause the transceiver to wirelessly obtain approval for the modified flight plan and, in response, communicate the approved modified flight plan to the UAV.

29. 30. The platform of claim 28, wherein the modified flight plan avoids possible hazards.

30. 30. The platform of claim 29, wherein the hazard is weather conditions.

31. 30. The platform of claim 29, wherein the danger is a possible collision with another UAV.

32. 30. The platform of claim 28, wherein the modified flight plan allows for unplanned surveying of land assets.

33. 1. A method of controlling a defined airspace using a high altitude pseudolite vehicle, the method comprising: wirelessly monitoring the status of at least one unmanned aerial vehicle (UAV) operating within a defined airspace; detecting conditions affecting flight within the defined airspace; using data received from the at least one UAV as input to a computationally trained predictor to identify actions to be taken based on the state of the UAV and detected flight-affecting conditions; causing the high altitude pseudolite vehicle to take action in response thereto; A method comprising:

34. 34. The method of claim 33, wherein the predictor is a neural network.

35. The method of claim 33 , wherein the action includes communicating with an air traffic control infrastructure.

36. The state of the UAV includes a spatial position and a velocity; determining land features and their likelihood of interfering with current or future communications between the UAV and a land communications source; determining that the likelihood exceeds a threshold; In response, establishing a communications link bridging the UAV and the land communications source; 36. The method of claim 35, further comprising:

37. 37. The method of claim 36, wherein the land features are determined based on a locally or remotely stored map.

38. The state of the UAV includes a spatial position and a velocity; computationally determining a predicted need for a modified flight plan for the UAV; In response, obtaining authorization for the modified flight plan and communicating the authorized modified flight plan to the UAV.

36. The method of claim 35, further comprising:

39. 39. The method of claim 38, wherein the modified flight plan avoids a possible hazard.

40. 40. The method of claim 39, wherein the hazard is a weather condition.

41. 40. The method of claim 39, wherein the danger is a possible collision with another UAV.

42. 39. The method of claim 38, wherein the modified flight plan allows for unplanned surveying of land assets.

Images