High-altitude pseudo-satellite neural network for unmanned traffic management

Abstract

Unmanned aerial vehicles (UAVs) or "drones" run neural networks to support reconnaissance, reconnaissance, reporting, and other missions. The drone reconnaissance neural network may monitor data streams from multiple onboard sensors in real time while navigating to an asset along a pre-programmed flight path and / or during its mission (e.g., when scanning and surveying an asset). A HAPS platform may run a neural network ("HAPSNN") while monitoring air traffic. The neural network enables it to classify, predict, and analyze events within its airspace of coverage in real time, and to learn from new events not previously seen or detected.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the priority and benefit of U.S. Patent Application No. 63 / 068,660, filed on August 21, 2020.

[0002] The present invention generally relates to the safe navigation of unmanned aerial vehicles (UAVs) and the consequent mission execution.

Background Art

[0003] UAVs are generally known as drones or unmanned aircraft systems (UAS), also referred to as remotely piloted aircraft, and are aircraft without the presence of a human pilot. Their route is controlled either autonomously by an on - board computer or by remote control of a pilot on the ground or in another vehicle. The number of drones has been increasing rapidly as the recognition of their extensive and diverse commercial potential grows.

[0004] Drones can utilize global positioning system (GPS) navigation functions, such as using GPS waypoints for navigation, and tend to follow pre - programmed flight paths. These can guide the drone to assets or their surroundings that it will scan and survey using on - board sensors. Drone systems can utilize various on - board sensors (including one or more cameras, radio frequency (RF) sensors, etc.) to monitor the operating environment, pre - calculate the path to the assets to be surveyed, follow it, and conduct the survey.

[0005] Despite the presence of these sensors, drones may not be equipped to use the data they provide to respond to unplanned changes in the flight path (e.g., to avoid unexpected obstacles or perform collision avoidance maneuvers) or to compensate for GPS drift, which can affect the accuracy of howitzers. GPS drift occurs when a pre-programmed flight path fails to account for the calibration and correction of GPS drift vectors. When an operator defines a flight plan with howitzers where such corrections are absent, for example, the navigation system may cause the drone to deviate from its course. Even a slight deviation can cause all or part of the target asset to fall outside the pre-programmed flight path, and consequently, outside the field of view of the onboard sensors. Conventional drones may also fail to respond to abnormal or unexpected conditions in their operating environment. For example, a drone may stop collecting information if sensor readings fall below a pre-set trigger threshold, or it may overcollect data if it deviates from its course, changes direction, and begins recording sensor data before actually reaching the target.

[0006] More generally, when surveying structures such as antennas, drones are typically controlled by an operator within the drone's line of sight. This requires not only the presence of personnel at each survey site but also meticulous and continuous attention to the drone's flight. The drone must approach in close proximity to each area of ​​the structure being surveyed, areas that cannot be fully understood until data is first collected and the survey actually begins. Furthermore, the drone must also maintain a safe distance from the structure and maneuver to avoid obstacles during its approach, regardless of wind or weather conditions. The task of operating a drone safely and efficiently is particularly challenging when surveying large facilities (such as power plants) that contain many assets.

[0007] Drone operations can be enhanced using High Altitude Pseudo-Satellite ("HAPS") platforms, also known as High Altitude Long Endurance ("HALE") platforms. These are unmanned aerial vehicles that operate sustainably at high altitudes (e.g., at least 70,000 feet), can be recharged by solar radiation during the day, and thus remain in flight for extended periods, providing airspace with a satellite-like effective range. HAPS drones equipped with RF communications payloads can, either independently or in conjunction with existing communications satellite constellations, or ground-based telecommunications networks, airspace reconnaissance infrastructure, airspace navigation aids, or individual air traffic communications and reconnaissance systems, provide vast areas of RF effective range and enable connectivity and real-time communications and reconnaissance services for air traffic, including drones.

[0008] HAPS platforms can operate with less cost and greater flexibility than satellite constellations, which are not easily redeployed for upgrades to meet changing bandwidth demands or whose numbers cannot be increased at short notice. In addition, satellites are not easily integrated with existing Earth air traffic reconnaissance systems and do not make them as well-suited as HAPS platforms for monitoring drone movements and maintaining a secure separation of drones and manned air traffic operating in the same airspace. Land-based alternatives such as telecommunications facilities generally have a short range and small area of ​​effectiveness, and again, extending their range or capabilities is costly and may not even be feasible due to the characteristics of the terrain or artificial structures.

[0009] Currently, HAPS platforms are limited in terms of their ability to provide mission-level support to drones. While they can improve safety and potentially assist with navigation, they are not designed to help drones recover from path deviations in order to record all necessary measurements of target assets without excessive data collection. [Overview of the project] [Means for solving the problem]

[0010] Neural networks are computer algorithms that roughly model the human brain and excel at recognizing patterns, learning nonlinear rules, and defining complex relationships between data. They can help drones navigate, provide mission support, and ensure proper asset surveys without data over-collection. A HAPS platform can run a neural network ("HAPSNN") when monitoring air traffic. The neural network enables real-time classification, prediction, and analysis of events within its airspace within its effective range, and learning from new events that have not been previously seen or detected. A HAPSNN-equipped HAPS platform can provide reconnaissance of nearly 100% of air traffic within its airspace within its effective range, and HAPSNN can process data received from drones to facilitate safe and efficient drone operation within the airspace. HAPSNN also enables bidirectional connectivity and real-time monitoring, thus enabling drones to better perform their intended missions.

[0011] Alternatively, or in addition, drones can run neural networks to assist with survey, reconnaissance, reporting, and other tasks. In various embodiments, the present invention utilizes an unsupervised “deep learning” neural network that is mounted and run on a low-altitude survey drone. Such a drone survey neural network (“DINN”) can monitor data streams from multiple onboard sensors in real time while navigating to an asset along a pre-programmed flight path and / or during its task (e.g., when scanning and surveying an asset). The neural network can communicate with unmanned traffic management systems directly or through HAPSNN and with manned air traffic to enable safe and efficient drone operation within the airspace. Using land and / or satellite-based communication networks, or a bidirectional connection to HAPSNN, the DINN can request or receive real-time airspace change authorizations and thus adapt the drone’s flight path to account for airspace collisions with other air traffic, terrain or obstacle collisions, or to optimize the drone’s flight path for more efficient task execution. Importantly, DINN can enable target acquisition and locate all assets to be surveyed, allowing the drone to compensate for GPS drift or other course deviations, or unexpected target anomalies.

[0012] One application that benefits from the collaboration between HAPSNN and DINN is the precise localization of passive intermodulation (PIM) on active telecommunications structures. PIM results from two or more strong RF signals originating from transmitters sharing antenna wiring, transmitters using adjacent antennas, or nearby towers with competing antenna patterns. PIM appears as a set of undesirable signals generated by the mixing of two or more strong RF signals in nonlinear devices such as loose or corroded connectors, cables, transmit / receive unequipers, circulators, damaged antennas, or nearby rusted components such as fences, barn roofs, bolts, etc. Other sources include poorly terminated or damaged cables with splices in the shielding, and aging surge arresters. PIM can be time-consuming and difficult to detect using conventional search methods. The combination of DINN and HAPSNN, providing connectivity to unmanned traffic management, could enable drones to autonomously and unsupervisedly fly around telecommunications assets to detect, classify, and precisely locate PIM sources. If a drone detects a new or unexpected reading, it may be able to analyze and classify the nature of the reading based on its training. Once the drone's survey is complete, it can use DINN to fly to the next pre-programmed asset location, adapting its flight path in real time along the way to optimize its operation within the airspace.

[0013] Therefore, in a first aspect, the present invention relates to a platform for high-altitude flight and communication with a UAV. In various embodiments, the platform comprises a wireless transceiver, computer memory, communication equipment for communicating with a UAV operating within a defined airspace via the transceiver, and a computer including a processor and electronically stored instructions executable by the processor for using data received from the UAV as input to a computationally trained predictor to identify actions to be taken based on the state of the UAV and conditions affecting its flight.

[0014] In various embodiments, the predictor is a neural network. The computer may further be configured for communication with air traffic control infrastructure. For example, the state of the UAV includes spatial position and velocity, and the computer may further be configured to determine the likelihood of interference with current or future communications between the UAV and the land communication source, and if the likelihood exceeds a threshold, to establish a communication link that bridges the UAV and the land communication source. The land features may be determined, for example, based on a locally or remotely stored map. In some embodiments, the computer may further be configured to (i) predictively determine the need for a modified flight plan for the UAV, and (ii) cause the transceiver to wirelessly obtain authorization for the modified flight plan and, accordingly, communicate the authorized modified flight plan to the UAV.

[0015] The modified flight plan may avoid possible hazards, such as weather conditions or a possible collision with another UAV. The modified flight plan may permit unplanned surveys of land assets.

[0016] In another respect, the present invention relates to high-altitude pseudo-satellites flying object This relates to a method for controlling a defined airspace using a high-altitude pseudo-satellite flying object This includes the step of taking action in response to that.

[0017] In various embodiments, the predictor is a neural network. Actions may include communication with air traffic control infrastructure. The UAV state may include spatial position and velocity, and in various embodiments, the method further includes the steps of determining the likelihood of a land feature interfering with current or future communication between the UAV and the land communication source, determining that the likelihood exceeds a threshold, and accordingly establishing a communication link to bridge the UAV and the land communication source. The land feature may be determined based on a locally or remotely stored map.

[0018] In some embodiments, the state of the UAV includes spatial position and velocity, and the method further includes the steps of computationally determining the anticipated need for a modified flight plan for the UAV, obtaining authorization for the modified flight plan accordingly, and communicating the authorized modified flight plan to the UAV. The modified flight plan of the method can avoid potential hazards such as weather conditions or possible collisions with other UAVs. Alternatively or additionally, the modified flight plan may permit unplanned surveys of land assets.

[0019] More generally, embodiments of the present invention enable a drone to autonomously navigate to a survey facility, execute a preliminary flight plan, calculate a survey path that will bring it close to the area of ​​the asset requesting the survey, collect relevant survey data, and activate sensors (including one or more cameras) that indicate specific areas requesting close-up and imaging. The drone may recognize obstacles in its path, monitor relevant weather conditions, and modify both its flight path and approach pattern to avoid collisions. These actions may occur onboard the drone with or without HAPS assistance. In particular, the drone may include a DINN that analyzes image frames captured in real time by onboard cameras as the drone progresses, as discussed above. Furthermore, without communicating with HAPS or land communication infrastructure, the DINN may monitor data streams from multiple onboard sensors in real time during navigation to the asset along a pre-programmed flight path and / or during its mission (e.g., when scanning and surveying the asset). More typically, a DINN is configured for communication with manned air traffic management systems and manned air traffic to enable safe and efficient drone operation within the airspace. Using bidirectional connectivity to land and / or satellite-based communication networks, a DINN can request or receive real-time airspace change authorizations and thus adapt the drone's flight path to account for airspace collisions with other air traffic, terrain or obstacle collisions, or to optimize the drone's flight path for more efficient mission execution. Importantly, a DINN can enable target acquisition and locate all assets to be surveyed, allowing the drone to compensate for GPS drift or other course deviations or unexpected target anomalies.

[0020] Therefore, in the first aspect, the present invention relates to a UAV comprising, in various embodiments, a flight package, a navigation system, an image acquisition device, communication equipment, computer memory, and a computer which includes a processor and electronically stored instructions executable by the processor for using data received from the image acquisition device as input to a computationally trained predictor for identifying and classifying objects appearing in images acquired by the image acquisition device during flight.

[0021] In various embodiments, the predictor is a neural network. The UAV includes a database of actions, and the computer is configured to select an action from the database and trigger its execution in response to a detected object classified by the predictor.

[0022] The communication equipment may be configured to interact with land and air traffic control systems. In some embodiments, the UAV also includes a weather reconnaissance module for monitoring weather conditions during the drone's flight, and the computer incorporates data from the weather reconnaissance module when selecting an action. The computer may be configured to have the UAV execute a preliminary flight plan around the asset to be surveyed, and calculate and execute a modified flight plan around the asset, based on object classification performed by a predictor during the preliminary flight planning. The computer may further include HAPS flying object For example, HAPS flying object HAPS (Hyperactive Access Points) are used to obtain authorization from the air traffic control infrastructure to execute flight commands received from HAPS. flying object To communicate with and / or to detect but unclassified objects, HAPS flying object It communicates with HAPS flying object It can be configured to receive classifications and associated actions to be taken.

[0023] In another aspect, the present invention relates to a method of surveying assets using a UAV. In various embodiments, the method includes obtaining digital images in real time during flight of the UAV, computationally analyzing the obtained digital images using a predictor trained computationally to identify and classify objects appearing in the images, and taking an action based on at least one classified object. The predictor may be a neural network, and the action may be determined based on a database search in response to the detected objects classified by the predictor. For example, the action may be to modify the flight path of the drone.

[0024] In various embodiments, the method further includes monitoring weather conditions during flight of the drone, and the action may further be based on the monitored weather conditions. The method includes obtaining a signal from the asset to be surveyed, and the action may be based on the obtained signal. The method further includes obtaining an image of the asset to be surveyed, and the action may be based on the obtained image. The method may further include causing the UAV to execute a preliminary flight plan around the asset to be surveyed and calculating and executing a modified flight plan around the asset based on the classification of objects performed by the predictor during the pre-flight planning.

[0025] In some embodiments, the method includes communicating with a HAPS flying object e.g., communicating a modified flight to the HAPS to obtain approval from an air traffic control infrastructure, and / or communicating an object that has been detected but not classified to the HAPS flying object and receiving from the HAPS the classified and associated actions to be taken. flying object to the HAPS flying object BRIEF DESCRIPTION OF THE DRAWINGS

[0026] ​The above and the following detailed explanations will be easier to understand when interpreted in conjunction with the drawings.

[0027] [Figure 1] Figure 1 conceptually illustrates HAPS, which provides reconnaissance and connectivity within the monitored airspace.

[0028] [Figure 2] Figure 2 conceptually illustrates the role of HAPSNN in predicting and providing the need for communication links within monitored airspace when obstacles or terrain are blocking line-of-sight communications.

[0029] [Figure 3] Figure 3 conceptually illustrates the role of HAPSNN in predicting the need for expansion of authorized airspace for drones moving through monitored areas and in proactively acquiring it.

[0030] [Figure 4] Figure 4 conceptually illustrates the HAPSNN prediction of the need for flight plan deviation and the authorization for it.

[0031] [Figure 5] Figure 5 conceptually illustrates HAPSNN, which provides in-air reconnaissance and connectivity to relay signals from multiple navigation aids to air traffic that is out of range or where reception of those signals is blocked.

[0032] [Figure 6] Figure 6 conceptually illustrates HAPSNN, which provides reconnaissance and connectivity within the airspace to monitor and avoid airspace collisions.

[0033] [Figure 7] Figure 7 conceptually illustrates the HAPSNN prediction of the need for flight plan deviations and the authorization thereof, in order to enable the efficient execution of the survey mission.

[0034] [Figure 8A] Figure 8A is a block diagram of typical HAPSNN and DINN architectures according to embodiments of the present invention.

[0035] [Figure 8B] Figure 8B schematically illustrates the modes in which HAPSNN can support DINN operation.

[0036] [Figure 9A] Figures 9A and 9B schematically illustrate the generation of a flight plan for carrying out a survey mission, which may be modified in real time as the drone conducts the survey. [Figure 9B] Figures 9A and 9B schematically illustrate the generation of a flight plan for carrying out a survey mission, which may be modified in real time as the drone conducts the survey.

[0037] [Figure 10] Figure 10 is an elevation view illustrating various anomalies associated with the antenna that the drone according to this specification may identify.

[0038] [Figure 11] Figure 11 conceptually illustrates the detection of an abnormal electromagnetic (EM) signature from a transducer by drone.

[0039] [Figure 12] Figure 12 conceptually illustrates a drone in flight reading stored data that indicates an anomaly.

[0040] [Figure 13A] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13B] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13C]Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13D] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13E] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment.

[0041] [Figure 14] Figure 14 schematically illustrates the detection and classification of radiation centers from multiple antennas. [Modes for carrying out the invention]

[0042] First, referring to Figure 1-7, this illustrates the functions performed by a conventional HAPS platform 105 and how these functions can be improved through the operation of HAPSNN. In Figure 1, the HAPS platform 105 communicates with multiple cell phone towers, typically shown at 108, multiple air traffic control systems, typically shown at 110, and a series of aircraft, typically shown at 113, and these systems and flying object All of these reside within a defined airspace 115, which is monitored by the HAPS platform 105. The HAPS platform 105 can also communicate directly and / or via cell phone towers with multiple drones 118 within the airspace 115. Finally, the HAPS platform 105 can communicate with satellites, as exemplified in 120, for geolocation, for example, to maintain a substantially fixed position. In this way, the HAPS platform 105 takes and updates a complete snapshot of the airspace 115, monitors air traffic within it, and acts as a communication hub between various intercommunication entities within the airspace 115.

[0043] As shown in Figure 2, HAPS105 monitors the position, status, and speed of the drone 118 and manned air traffic 133 operating within the monitored airspace 115. The onboard HAPSNN, assuming the operating altitude of the drone 118, recognizes that terrain or other obstacles 205 would likely disrupt line-of-sight communication between the drone 118 and the air traffic control system 110, and furthermore, between the aircraft 113. Obstacles 205 are recognized and mapped in real time by HAPS105, and are likely to be stored in maps accessible locally to HAPS105 or via a wireless link. More specifically, HAPSNN may calculate the likelihood that obstacles 205 will interfere with current or future communication between the drone 118 and the air traffic control system 110, and if that likelihood exceeds a threshold, it registers the need to establish a communication link bridging the drone 118 and the air traffic control system 110. In that case, HAPS105 will respond by obtaining the status of the drone 118 (including altitude, position, and orbit) through any appropriate modality or combination thereof, such as observation, telemetry, signal monitoring, and / or direct communication with the Earth Air Traffic Control System that monitors the drone 118 and / or its flight.

[0044] As a result of this recognized need, HAPS105 can enter the communication network as an intermediate node and relay messages (i.e., act as a transmission link) between the drone 118 and the air traffic control system 110 (e.g., UTM and LAANC) or other ground-based air traffic reconnaissance infrastructure. In the absence of HAPSNN, HAPS105 would operate accordingly, but if, for example, the drone 118 had previously communicated with HAPS105 and the control system 110, HAPS105 could act as a backup communication channel when direct communication between the drone 118 and the control system 110 is lost as the drone 118 approaches the obstacle 205. HAPSNN facilitates proactive and predictive intervention by HAPS105 even when no prior communication has taken place between the drone 118 and the control system 110. Based on stored or acquired knowledge of the terrain and fixed communication features within airspace 115, and the calculated trajectory of drone 118 (which may have just entered airspace 115), HAPSNN recognizes the need for communication between drone 118 and control system 110 and causes HAPS 105 to establish a wireless link with itself as a hub. Similarly, based on knowledge of the terrain and monitored altitude of drone 118 and manned aircraft 113, HAPSNN may cause HAPS 105 to establish a wireless link with itself as a hub between drone 118 and aircraft 113.

[0045] Figure 3 illustrates a situation in which a drone 118, navigating through an altitude-restricted and authorized airspace 305, would encounter obstacles, collectively shown in 310, which it would have to fly around in order to remain within the airspace 305. This deviation from the pre-programmed flight path can be very large, depending on the extent of the obstacles, and may require course corrections that the drone is not equipped to handle with high accuracy, potentially leading to it missing all or part of the target when it reaches it. When using a HAPS 105 communicating with the drone 118, the HAPS NN anticipates the need for additional airspace authorization and requests an extension to area 315. Once authorization is obtained, the HAPS 105 communicates the relaxation of the altitude restriction to the drone 118. In some embodiments, the HAPS 105 may calculate a modified flight path 320 at a higher altitude for the drone 118, enabling it to remain on course for the target without lateral deviations that may affect navigation accuracy.

[0046] Similarly, in Figure 4, the planned and approved flight path 405 for drone 118 will be incorporated by HAPS105 through hazardous local weather patterns detected using its onboard radar or from real-time weather updates. Knowing the drone's flight path and weather conditions, HAPSNN recognizes the need for an alternative flight section 410, which it, or another onboard calculation module, calculates. HAPS105 obtains approval for the new flight path 410 and communicates it to the drone 118's navigation system.

[0047] Figure 5 shows a drone 118 advancing through an obstacle that obstructs line-of-sight communication between a cell phone relay tower 108 and a source 505 of National Aeronautical System navigation aid signals (VOR, VOR / DME, TACAN, etc.). HAPSNN infers this condition from its knowledge of the land features within the airspace 115 and the state of the drone 118. As a consequence, HAPSNN causes HAPS 105 to relay signals from at least the obstructed sources 108, 505, or to act as a hub to establish wireless links between itself and the communication entities 108, 118, 505.

[0048] In Figure 6, drone 605 is flying within its authorized airspace 610. A second drone 615 enters airspace 610, and HAPSNN detects that drones 605, 615 are not communicating or are unable to communicate. For example, one or both of drones 605, 615 may be “non-cooperative,” i.e., not equipped to communicate with other drones and avoid collisions in their flight paths. HAPSNN may infer this based on communication between drones and / or with ground-based air traffic reconnaissance and surveillance systems, or the absence thereof. HAPSNN determines that, in order to avoid a collision, drone 605 should follow an alternative flight path 620, which may be temporary until drone 605 passes drone 615. HAPSNN or another onboard calculation module calculates the new flight path 620. HAPS105, accordingly, obtains authorization for the new flight path 620 and communicates it to the navigation system of drone 605.

[0049] Referring to Figure 7, the drone 118 follows a flight path 705 within an authorized airspace corridor 707, for example using an RFID reader, and investigates or captures data readings from a set of assets (e.g., a set of transducers 7101...7105 along a power line 712). The HAPSNN monitors the data stream received by the drone 118, and if the drone detects an anomaly, the HAPSNN may infer that the drone 118 will request a change to its airspace authorization. In particular, if the anomaly is associated with transducer 7103, the anticipated authorization need may cover area 720. The HAPSNN may cause the HAPS 105 to take action, for example, relay the information to the appropriate UTM / LAANC system, or request a change to the airspace authorization on behalf of the drone 118, which would then be free to investigate transducer 7103 more closely and record its image for real-time or later evaluation.

[0050] Figure 8A illustrates a typical HAPSNN architecture 800, which includes a HAPS aircraft and various hardware and software elements. Generally, multiple software subsystems, implemented as instructions stored in computer memory, are executed by a conventional central processing unit (CPU) 802. The CPU 802 handles HAPS flying objectThe flight and operation of the system, and the functions described below, can be controlled, or these functions can be distributed among separate processors 802. In addition, for efficient execution of neural network functionality, the system may include a dedicated graphics processing unit. An operating system (e.g., MICROSOFT WINDOWS®, UNIX®, LINUX®, iOS, or ANDROID®, etc.) provides low-level system functions such as file management, resource allocation, and routing of messages to and from hardware devices and software subsystems (including at least one non-volatile memory element 803), which are executed in computer memory 804. More generally, the HAPSNN800 may include modules implemented in hardware, software, or a combination of both. Regarding functions provided within the software, programs can be written in any of several high-level languages ​​such as PYTHON, FORTRAN, PASCAL, JAVA®, C, C++, C#, BASIC, various scripting languages, and / or HTML. flying object The software modules and mechanical and flight features involved in its operation are conventional and not illustrated. See, for example, U.S. Patent No. 10,618,654.

[0051] The HAPSNN800 includes a neural network module 805, a transceiver module 808, and a field-programmable gate array (FPGA) 810. The transceiver module 810 and FPGA 810 may constitute, or be part of, a communications system configured to support airborne communications between aircraft and with land and satellite-based control infrastructure. The HAPSNN800 may (but may not) operate in conjunction with a drone equipped with DINN812. The cloud neural network module 805 is HAPS flying object While local to the cloud, more typically, as explained below, i.e., HAPS flying object Modules 808 and 810 can operate on a remote (e.g., on land) server in wireless communication. flying object It is located on top of itself.

[0052] The cloud neural network module 805 includes a classification neural network 815 that processes images and data received in real time from the drone via the agile transceiver 808, which may pass through to the cloud neural network 805. The classification neural network 815 is trained using a database of training images 817 related to the tasks undertaken by the monitored drone. The classification neural network 815 processes and classifies the received images and data and detects anomalies associated with them, i.e., calculates their probability. That is, anomalies can be detected when they are considered based on something in the received images or in parallel with other drone telemetry. For example, anomaly detection may be triggered when other non-exceptional images are taken in conjunction with weather conditions reported by the drone. When an anomaly is detected, the classification neural network 815 may look up the classification database 819 to determine an appropriate response, i.e., the database 819 contains records that define the anomaly and, in turn, one or more associated actions that may be taken. If an anomaly not recorded in the database is detected, the image may be transmitted for human investigation and classification. The new classification is then added to the training database 817 and used to retrain the neural network 815. The resulting adjusted weights may be backpropagated by the cloud server associated with the neural network 805 to DINN 812 (if present) to transmit similar mission profiles to drones in the field and other drones. This procedure is described further below.

[0053] The agile transceiver package 808 includes an Automated Dependent Reconnaissance Broadcast (ADS-B), a Traffic Collision Avoidance System (TCAS), a Secondary Reconnaissance Radar (SSR), and an Automated Dependent Reconnaissance Rebroadcast (ADS-R) subsystem, operating at 978 MHz, 1,090 MHz, and 1,030 MHz for querying, responding, and rebroadcasting. These enable the HAPSNN800 to "listen" to the position of manned air traffic, and thus the neural network 815 can computationally represent nearby traffic in 3D or 2D space and analyze any collisions between the drone and manned air traffic. This can be achieved by broadcasting the drone's position to manned air traffic, or the position of manned air traffic to the drone. Emergency warnings are issued to manned and / or unmanned traffic with commands regarding the direction in which they should move to avoid collisions in the airspace.

[0054] The agile transceiver 808 may include a cellular network package, including 3G, 4G, LTE, 5G, or any future telecommunications protocol and bandwidth, to support communication links between drones operating within the HAPSNN800 airspace, using land telecommunications networks utilized by several UTM systems, or using backhaul communication channels for transmitting data from HAPS to a cloud-based neural network. VHF and UHF transceiver (TX / RX) modules may be used to monitor navigation aids such as VOR, VOR / DME, or TACAN, enabling the neural network 805 to analyze the position of the drone and HAPS using the flight time of the signal in the event of GPS signal loss. This also enables the utilization of satellite communication constellations to transmit or receive data if necessary. The drone virtual radar (DVR) data link facilitates communication with drone platforms implementing the technology (as described, for example, in U.S. Patent No. 10,586,462) to transmit and receive air traffic position information for use in collision analysis or drone tracking. The neural network (NN) data link is a dedicated high-bandwidth backhaul channel that enables HAPSNN800 to communicate with the DINN neural network computation engine 825, transmitting real-time data received from multiple drones operating within the monitored airspace and receiving prediction and action commands retrieved from the classification database 819. The FPGA 810 is employed as a hardware accelerator to launch software that tunes the transceiver 808 and filters out noise.

[0055] A typical DINN812 implemented within the drone 118 includes a neural network computation engine 825, a classification database 825, and "backend" code for performing various data handling and processing functions as described below. In addition, the drone 118 includes communication equipment comprising, or consisting of, a set of agile transceiver 808 and FPGA 810, as detailed above. The drone 118 may also include a CPU 802, storage 803, and computer memory 804.

[0056] As mentioned, DINN812 can interact with HAPSNN800, but either can exist and operate on its own; that is, HAPSNN is unnecessary for the successful deployment and use of DINN, but HAPSNN can play a reconnaissance and safety role for an aircraft lacking DINN. The role of DINN812 is to enable the drone 118 to classify objects of interest on the asset being investigated and obstacles that will need to be avoided during flight (e.g., recognizing cell phone towers to be investigated and cracked antennas on such towers). The neural network 825 is configured to process and classify images received from an image acquisition device 827, e.g., a video camera on the drone 118. Thus, the neural network 825 may be a convolutional neural network (CNN) programmed to detect and recognize objects in incoming images. These can be classified based on the training of a neural network, and DINN812 (e.g., backend code) can look up the classification database 830 to determine the appropriate response to the detected image. In this case, the database 819 contains records defining objects, each associated with several semantic meanings or actions. For example, if the neural network 825 detects a tree in the incoming image, the corresponding database entry may identify the tree as an obstacle, triggering an evasive maneuver performed by the drone's navigation system 832 by controlling the drone's steering and propulsion systems. These are part of the drone's flight package 835, which is conventional and therefore not shown in detail, but includes a power supply, communication platform, propulsion and steering systems, autopilot system, etc.

[0057] DINN812 may also receive data from one or more reconnaissance systems 837, 839, which may include one or more of the following: DVR, UTM, LAANC, ADS-B, and TCAS systems. While these may be implemented as part of the drone's communication platform, they are illustrated conceptually within DINN812 because the neural network 825 may use this data when classifying images. Similarly, the weather reconnaissance system 842 would conventionally be implemented within the drone's communication platform, but again, it is shown as part of DINN812 because weather conditions may be relevant to image classification or database retrieval, as shown in Figure 4. The same visual scene may prompt different actions depending on the weather; for example, the drone 118 may give a wider mooring area to a building under strong wind conditions.

[0058] In embodiments where the drone 118 interacts collaboratively with the HAPSNN 800, the latter may provide further support and stronger classification capabilities. For example, images with detected objects that are unclassifiable by the neural network 825 may be uploaded to the HAPSNN 800 for examination, and real-time commands issued by the HAPSNN as a response may be executed by the drone's navigation system 832. Furthermore, the HAPSNN 800 may update or supply different weighting files for the neural network 825 in real time to better suit the drone's mission based on the classification performed by the drone (and communicated to the HAPSNN 800 in real time). The neural network 825 reads these new weighting files in response when they are received.

[0059] This process is shown in Figure 8B. Referring also to Figure 8A, DINN 812 processes incoming image frames (e.g., approximately 30 frames per second (FPS)) in real time, enabling the drone 118 to react fast enough to avoid collisions and fly around the tower 850. Therefore, the drone 118 may include a graphics processing unit (GPU) to support CNN operation. Real-time frame analysis allows the GPU to process the images, classify the items of interest on the assets to be investigated, notify the backend code of the classification, and enable the backend code to execute logic to react to the classification by performing a search in the classification database 830 to obtain an action corresponding to the classification.

[0060] The drone 118 transmits image data to the HAPSNN, which includes a high-precision CNN (within the computation engine 815, or optionally, further within HAPS itself) capable of processing, for example, 60 megapixels (MP) of photographic images per second. The CNN architecture is designed for speed and accuracy of classification by leveraging backend logic, which runs on the computation engine 825. This backend logic can modify the CNN's weights and configuration files based on the assets to be classified based on the first few images of the assets captured by the drone. These preliminary images are collected as part of a “backup” flight path around the assets at a safe distance and may have a resolution of 60MP or more. These preliminary images are reduced to the input image size of the CNN (e.g., 224x224, or larger depending on the assets to be investigated) and passed through a series of convolutional layers (e.g., 20), followed by an average pooling layer and fully connected layers pre-trained to classify different assets (e.g., 100 types of assets). Once the asset type is identified, the weighting and configuration files may be modified, more (e.g., four) convolutional layers may be added, followed by two fully connected layers to output the probabilities and bounding boxes of any objects or areas of interest that may exist on the asset. Images uploaded from the drone may be increased in size (e.g., up to 448x448) because this type of classification requires the presence of finer-grained detail. The degree of size increase is dynamically controlled and can be scaled up, for example, if sufficient detail is not detected for reliable classification.

[0061] The fully coupled layer predicts class probabilities and bounding boxes (i.e., cracked antennas, rust, and corrosion). For example, the final layer may use linear activation, while the convolutional layer may use leaky ReLU activation.

[0062] Once the calculation engine 825's backend logic detects the presence of class and bounding box coordinates, it can trigger a switch to the center of gravity tracker function to move its specific classification to the center of the drone's image acquisition device's field of view. The backend logic works with the ranging calculation engine to analyze the safest flight path for the drone in order to approach and position the image acquisition device for high-resolution scanning.

[0063] Therefore, the preliminary flight path establishes the type of asset within the field of view and registers the position of the asset in 3D space relative to the drone in order to account for any GPS drift vectors. If any obstacles or hazards exist within the operating area, they are classified and their positions in 3D space are registered. The center of gravity tracker is activated once classifications within the area of ​​interest are detected, keeping the object of interest centered within the field of view. The ranging calculation engine controls the forward and reverse movement of the drone. Before any of these commands are executed, the position of the drone in 3D space relative to assets and any obstacles present within the operating area is acquired. This data is invoked through backend logic to analyze a safe GPS waypoint flight path that will move the drone into the area of ​​interest, and in GPS rejection areas, this flight path can still be analyzed and executed using Kalman filtering of inertial data in conjunction with the center of gravity and ranging functionality. The flight path is fine-tuned in real time via the center of gravity tracker and ranging calculation engine. Please note that the center of gravity tracking can be performed by the HAPSNN800, not the DINN812.

[0064] In step 855, the HAPSNN CNN ("HP-CNN") processes each image to detect objects within it using a standard object detection routine (e.g., YOLO), and attempts to identify (i.e., classify) all detected objects based on its prior training (further discussed below). Detected objects that cannot be identified are stored in a database and made available to personnel for identification and labeling (step 857). The HP-CNN is then retrained on an augmented dataset containing the newly identified and labeled object data (step 859), resulting in the generation of new CNN weights (step 862). If the real-time neural network 825 residing on the drone 118 is also a CNN ("RT-CNN"), the HAPSNN 800 may push these weights to the RT-CNN, which receives and loads them. That is, the HP-CNN and RT-CNN may be the same or substantially similar so that the CNN weights generated for the HP-CNN can propagate across the drone formation.

[0065] HP-CNNs (or, in some embodiments, RT-CNNs on top of themselves) can be trained in conventional ways. In particular, the CNN is trained on labeled images of objects that the drone is likely to encounter as it performs its mission, resulting in a CNN that can analyze and classify the objects most likely to be encountered by the drone within their typical flight paths. Since the training set cannot be exhaustive and the drone will inevitably encounter unknown objects during use, the process described above—locating the whereabouts of unrecognized objects, memorizing them for manual labeling, and then retraining the CNN on an expanded dataset—helps minimize the risk of accidents by constantly enriching the drone's visual vocabulary and action repertoire. This is most efficiently performed using HAPSNNs as a central training hub receiving unclassifiable objects from many drones, and all of their neural networks can be kept updated to reflect the latest classification capabilities; however, it is also possible to implement this training and retraining functionality on individual DINNs.

[0066] Figures 9A-13E illustrate various applications of drones and the ways in which DINNs can be used to control and simplify the operation of drones. In Figure 9A, the DINN 812 guides the drone 118 around the antenna 900 so that when the drone 118 detects anomalies or structures requiring a closer inspection, it will conduct the inspection according to a flight pattern 910 that may change in real time. For example, as shown in Figure 9B, the flight path 910 may be modified so that the drone 118 does not approach a power line 915 that would be perceived as an obstacle.

[0067] Referring to Figure 10, the antenna 1000 has a crack 1005, which can be identified by a DINN-equipped drone. The backend may run an OpenCV function to fix the center of gravity on the antenna 1000 and move the drone to position the antenna in the center of the field of view to facilitate image acquisition. This can be combined with a depth map generated by a stereo RGB camera of known sensor size, lens properties, and camera focus separation to position the drone 118 close enough to the antenna for the camera to analyze enough detail to good pick up the crack. Other defects that can be recognized by the drone 118 include structural corrosion 1010 and paint coating anomalies 1015. The drone 118 payload may include additional sensors such as an EM sensor and an infrared camera, and the DINN 812 may include additional modules such as a spectral analysis and / or electromagnetic calculation engine. These enable the drone 118 to detect PIM signatures at the coaxial connector 1025, other EM anomalies such as magnetic fields emitted from the surge arrester 1030 (indicating the possibility of arc discharge), and an anomalous thermal signature 1035. These conditions can be diagnosed and observed by the drone 118, or using the onboard calculation engine and / or in cooperation with the HAPSNN800 as described above. The DINN812 may further include a ranging calculation engine to initiate data fusion between depth maps acquired as discussed above and laser rangefinders / radar / ultrasonic or other ranging sensors that may be included in the drone payload to derive the most accurate distance measurements, a computer vision calculation engine to perform image processing and analytical functions (such as center of gravity placement as described above), and / or infrared and multispectral calculation engines to analyze infrared and other images acquired at wavelengths outside the visible spectrum.

[0068] As shown in Figure 11, under normal operation, the magnetic field lines are vertical, as shown in 1110. If the drone 118 detects an unusual EM signature from asset 1100, such as an abnormal non-perpendicular magnetic field line 1105, it may record the anomaly or, in embodiments where the drone 118 communicates with HAPSNN, report it to HAPSNN, which may autonomously call in a drone with a more specialized survey payload to better diagnose the condition. The drone 118 may also communicate with or otherwise interact with the asset to be surveyed. In Figure 12, the drone 118 reads data stored in the transducer 1200 indicating an anomaly. That is, the transducer 1200 has self-diagnostic capabilities, and when an operational anomaly is detected in advance, data indicating the detection is stored internally. If the transducer 1200 is periodically surveyed by the drone 118 and the abnormal condition does not require immediate attention, it may not be necessary for the transducer to communicate its presence to supervisory personnel in response to the detection. Rather, when the drone 118 conducts a survey flight, it queries the memory of all transducers, thereby obtaining previously stored data indicating the detection of anomalies in the transducers. This can prompt the drone 118 to conduct closer surveys (as shown in the figure) in response to the classification of conditions and database searches. Again, HAPSNN recognizes the conditions and transmits new weights to the DINN of the drone 118, enabling it to perform condition-specific classification as it surveys the transducers 1200.

[0069] Figures 13A-13E illustrate how DINN812 can establish an optimal flight path through a complex survey environment. Figures 13A and 13B illustrate a power distribution substation 1300, which includes a series of outgoing transmission lines, collectively shown in 1310, a voltage regulator 1315, a drop converter 1320, a series of switches 1325, one or more surge arresters 1330, and a receiving transmission line, collectively shown in 1335. Any of these components may experience anomalies or malfunctions, and all can be surveyed by the drone 118. As shown in Figures 13C and 13D, the drone 118 may perform a preliminary flight pattern 1350, either alone or in conjunction with HAPSNN, to collect images for analysis by DINN. DINN812 analyzes the images obtained by the drone's onboard camera and classifies the various components 1310-1335. Based on these classifications, the navigation module 832 (see Figure 8) or backend code calculates an optimized flight plan 1355 that enables all components 1310-1335 to be surveyed appropriately and efficiently by the drone 118. More specifically, each component located within the substation 1300 is classified during the preliminary survey flight path 1350, and these are located far from the substation 1300 to adapt to GPS drift vectors and other unknowns related to obstacles or inaccuracies centered on the initial flight path. Once a component is classified, its position in the image and the drone's position are registered. This process is repeated multiple times throughout the preliminary survey, enabling the backend code to triangulate and position each classified asset in 3D space, thus allowing for the calculation of a more accurate survey flight path that will move the drone and payload closer to the assets. This new flight plan 1355 is then executed, and again, DINN 812 classifies the assets as the flight path is executed. Preliminary investigations suggest that because the drone is far away, the DINN812 can only classify large items if the resolution of the image acquisition device is fixed.Once the closer flight path 1355 is executed, more asset details will be detected, and DINN812 will be enabled to classify the new items and, if necessary, readjust the drone's path. The ranging engine calculates the nearest acceptable approach distance between the drone and the asset, which matches acceptable and safe limits.

[0070] Another application where complex survey operations can be optimized by DINN is illustrated in Figure 14. Here, the objective is to determine the "radiation center," that is, the radiation center 1410 of each of the antennas 1410, 1415, and 1420 mounted on the telecommunications structure 1400 at different elevations. R , 1415 R , 1420 R The task is to detect and classify the radiation centers. DINN classifies the radiation centers, then identifies the center of gravity for each antenna, and moves the radiation center to the center of the drone camera's field of view, thereby allowing the barometer reading (altitude) to be recorded. Navigation module 832 can then use this altitude to generate an orbital flight path, enabling drone 118 to acquire 3D model data for reconstructing the antenna structure.

[0071] The terms and expressions used herein are for illustrative purposes only and are not limiting, and in the use of such terms and expressions, there is no intention to exclude any equivalent or part of any feature shown or described. In addition, although one embodiment of the invention has been described, it will be apparent to those skilled in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the invention. Therefore, the embodiments described should be considered in all respects to be merely illustrative and not limiting.

[0072] The charges will be as follows:

Claims

1. A platform for high-altitude flight and communication using an unmanned aerial vehicle (UAV) (118), wherein the platform is Wireless transceiver (808), Computer memory (804) and, Communication equipment (810) for communicating with a UAV operating within the airspace (115) defined via the aforementioned transceiver, A computer (802), the computer comprising a processor and electronically stored instructions, the instructions being executable by the processor, and the computer for using survey data acquired in real time from the UAV in a complex survey environment and along a pre-programmed flight path, Equipped with, The aforementioned survey data is used as input to the predictor (815), and the predictor is Identify and classify objects in real time, When an abnormality in an object in the aforementioned complex investigation environment is identified (7103), The UAV is computationally trained to identify the actions to be taken based on the state of the UAV and the conditions affecting its flight, to calculate and execute a modified flight path (320), and to allow the UAV to deviate from the pre-programmed flight path to the modified flight path to investigate the object. The state of the UAV includes spatial position and velocity, and the computer further: Before communication between the UAV and the land communication source, based on the identified and to be taken actions provided by the predictor, While the UAV follows the modified flight path, the characteristics of the land (310) and the likelihood of them interfering with current or future communications between the UAV and the land communication source (108) are determined. In response to the determination that the likelihood exceeds the threshold, A communication link is established to bridge the UAV and the land communication source. A platform characterized by being configured in such a way.

2. The platform according to claim 1, wherein the predictor is a neural network.

3. The platform according to claim 1 or 2, wherein the computer is further configured for communication with air traffic control infrastructure.

4. The platform according to claim 1, wherein the characteristics of the land are determined based on a locally stored map.

5. The platform according to claim 1 or 2, wherein the characteristics of the land are determined based on a remotely stored map.

6. The aforementioned computer further, (i) predictively determine the need for a modified flight path for the UAV based on the object and the identified actions to be taken provided by the predictor, (ii) Based on the identified and to be taken actions provided by the predictor, upon determination that a modified flight path is required, the transceiver radioly obtains authorization for the airspace of the modified flight path from the air traffic control infrastructure and transmits the authorization for the airspace to the UAV. The platform according to claim 3, configured as described above.

7. The platform according to claim 6, wherein the modified flight plan avoids possible hazards.

8. The platform according to claim 7, wherein the aforementioned hazard is weather conditions.

9. The platform according to claim 7, wherein the aforementioned risk is a possible collision with another UAV.

10. The platform according to claim 6, wherein the modified flight plan allows for an unplanned survey of the object which is a land asset.

11. The platform according to claim 10, wherein the action to be taken is an unplanned survey of the object which is a land asset.

Images