Unmanned aerial vehicles (UAVs) with resistance to hijacking, jamming, and impersonation attacks.

Abstract

Unmanned aerial vehicles (UAVs) or "drones" run neural networks to help detect and respond to attacks. The neural networks can monitor data streams from multiple onboard sensors in real time during flight and communicate with high altitude pseudosatellite ("HAPS") platforms. For example, if the neural network detects a cyberattack but determines not to interfere with external communications, it can transfer navigation control of the drone to the HAPS.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the priority and benefit of U.S. Patent Application No. 63 / 068,660, filed on August 21, 2020.

[0002] The present invention generally relates to the navigation of unmanned aerial vehicles (UAVs) and the performance of tasks thereby, and more particularly to UAVs with resistance to electronic attacks that attempt to hijack or otherwise affect in - air operations.

Background Art

[0003] UAVs are generally known as drones or unmanned aircraft systems (UASs), also referred to as remotely piloted aircraft, and are aircraft that do not involve the boarding of a human pilot. Their path is controlled either autonomously by an on - board computer or by remote control of a pilot on the ground or in another vehicle. The number of drones has increased rapidly as the recognition of their extensive and diverse commercial potential has grown.

[0004] Drones can utilize global positioning system (GPS) navigation functions, such as using GPS waypoints for navigation, and tend to follow pre - programmed flight paths. These can guide the drone to assets that it will scan and survey using on - board sensors, or to areas around them. Drone systems can utilize various on - board sensors (including one or more cameras, radio frequency (RF) sensors, etc.) to monitor the operating environment, pre - calculate the path to the assets to be surveyed, follow it, and conduct the survey.

[0005] Despite the presence of these sensors, drones may not be equipped to use the data they provide to respond to unplanned changes in the flight path (e.g., to avoid unexpected obstacles or perform collision avoidance maneuvers) or to compensate for GPS drift, which can affect the accuracy of howitzers. GPS drift occurs when a pre-programmed flight path fails to account for the calibration and correction of GPS drift vectors. When an operator defines a flight plan with howitzers where such corrections are absent, for example, the navigation system may cause the drone to deviate from its course. Even a slight deviation can cause all or part of the asset of interest to fall outside the pre-programmed flight path, and consequently outside the field of view of the onboard sensors. More dangerous than GPS drift are electronic "spoofing" attacks that mimic GPS signals and can hijack a drone, guiding it away from its intended flight path to damage or steal it, or cause collateral damage to people or property. Other forms of cyberattacks that can be used against drones include RF saturation attacks, which are designed to disrupt connections between drones and satellite or land-based communications. As electronic attacks and their targets proliferate, the need to protect drones from malicious attempts to hijack or interfere with their operation is urgent. [Overview of the project] [Means for solving the problem]

[0006] Embodiments of the present invention detect and neutralize attempts to hijack, alter, or otherwise interfere with the navigation and control of a drone. These efforts, broadly referred to herein as “attacks,” can take numerous forms. If the drone’s navigation package is compromised, it is inconceivable that proper operation can be restored during flight, and in fact, the illegal occupation itself can be extremely difficult to detect. If the connection between the drone and a satellite or land-based control entity is severed or hijacked, it would be impossible to transfer navigation control to such entity in order to avoid the consequences that inevitably follow from the attack. Laser attacks, designed to cause disorientation of onboard optical sensors or to block light, can cause a drone to go off course, even if the navigation package itself is unaffected.

[0007] In various embodiments, the drone includes a neural network that analyzes image frames captured in real time by onboard cameras as the drone moves. Neural networks are computer algorithms that roughly model the human brain and excel at recognizing patterns, learning nonlinear rules, and defining complex relationships between data. They can help the drone navigate, provide mission support, and ensure proper asset surveys without data over-collection. The drones according to this specification can run neural networks to assist in surveys, reconnaissance, reporting, and other missions. The present invention utilizes an unsupervised “deep learning” neural network that is mounted and run on a low-altitude survey drone. Such a drone survey neural network (“DINN”) can monitor data streams from multiple onboard sensors in real time during navigation to an asset along a pre-programmed flight path and / or during its mission (e.g., when scanning and surveying an asset). The neural network can communicate with an unmanned traffic management system and manned air traffic to enable safe and efficient drone operation within the airspace. Using bidirectional connectivity to land and / or satellite-based communication networks, DINN can request or receive real-time airspace change authorizations and thus adapt drone flight paths to account for airspace collisions with other air traffic, terrain or obstacle collisions, or to optimize drone flight paths for more efficient mission execution. Importantly, DINN can enable drones to act as an independent layer of surveillance and defense to monitor the location of drone data, detect attacks, and determine appropriate actions.In the case of GPS spoofing attacks designed to hijack a drone and guide it away from its intended flight path, or RF blockade attacks designed to sever the connection between the drone and HAPSNN or land communications, DINN can track the drone's position relative to assets or other landmarks, enabling the drone to resist being forced to fly in another direction or to maintain a safe flight path until communications are reliably re-established. In the case of laser attacks intended to disorient optical sensors or block light, DINN can recognize and classify the patterns produced by such attacks, and the drone can trigger pre-programmed responses to counter the attack or alter its flight path, moving itself away from the source of the attack.

[0008] The aerial infrastructure includes High Altitude Pseudo-Satellite ("HAPS") platforms, also known as High Altitude Long Endurance ("HALE") platforms. These are unmanned aerial vehicles that operate continuously at high altitudes (e.g., at least 70,000 feet), can be recharged by solar radiation during the day, and thus remain in flight for extended periods, providing a wide effective range of airspace similar to that of satellites. HAPS drones equipped with RF communications payloads can provide a vast area of ​​RF effective range, either independently or in conjunction with existing communications satellite constellations, or ground-based telecommunications networks, airspace reconnaissance infrastructure, airspace navigation aids, or individual air traffic communications and reconnaissance systems, providing connectivity and real-time communications and reconnaissance services to air traffic, including drones.

[0009] HAPS platforms can operate with less cost and greater flexibility than satellite constellations, which are not easily redeployed for upgrades to meet changing bandwidth demands or whose numbers cannot be increased at short notice. In addition, satellites are not easily integrated with existing Earth air traffic reconnaissance systems and do not make them as well-suited as HAPS platforms for monitoring drone movements and maintaining a secure separation of drones and manned air traffic operating in the same airspace. Land-based alternatives such as telecommunications facilities generally have a short range and small area of ​​effectiveness, and again, extending their range or capabilities is costly and may not even be feasible due to the characteristics of the terrain or artificial structures.

[0010] The HAPS platform may run a neural network ("HAPSNN") when monitoring air traffic. The neural network enables it to classify, predict, and analyze events within its effective range of airspace in real time, and to learn from new events that were not previously seen or detected. A HAPSNN-equipped HAPS platform may provide reconnaissance of nearly 100% of air traffic within its effective range of airspace, and HAPSNN may process data received from drones to facilitate safe and efficient drone operation within the airspace. HAPSNN also enables two-way connectivity and real-time monitoring, thus enabling drones to perform their intended missions more effectively.

[0011] In various embodiments, the DINN collaborates with the HAPSNN. For example, if the DINN detects an attack but determines that it does not interfere with external communications, it may transfer the drone's flight control to the HAPSNN.

[0012] Therefore, in a first aspect, the present invention relates to a UAV, which in various embodiments includes a neural computation engine, a flight package, a navigation system, an image acquisition device, a communication system including a plurality of agile transceivers, a computer memory including a plurality of pre-stored waypoints and flight images, and a computer, which includes a processor for using data received from one or more of the image acquisition device, a communication array, or a navigation system as input to a predictor trained to (a) detect an attack on the UAV, (b) identify mitigation actions to be taken in response, and (c) cause the actions to be taken, and instructions stored in the computer memory and executable by the processor.

[0013] If the attack is GPS spoofing, the action may be to have the drone navigate without GPS and to coordinate the agile transceiver to multiple National Aeronautical Navigation Aids (HAPS) according to a pre-stored waypoint flight plan. The drone may navigate without GPS navigation using inertial navigation with optical flow and / or Kalman filtering. If the attack is GPS spoofing, the action may be to trigger the transfer of navigation control to HAPS. If the attack is RF blockade, the action is to execute a pre-programmed flight plan that does not require external communication. In some embodiments, the pre-programmed flight plan relies on (i) at least one of inertial navigation with Kalman filtering or optical flow to have the drone navigate according to a pre-programmed waypoint flight plan, and (ii) using image matching to classify the time the drone reached each waypoint. If the attack is a laser attack, the action may be to plot a flight path away from the laser and have the drone follow it.

[0014] In some embodiments, the computer is further configured to communicate with the HAPS vehicle to computationally identify mitigation actions. The predictor may be, for example, a neural network. In various embodiments, the communication equipment is configured to interact with land and air traffic control systems. The UAV may further have a database of predetermined actions, and the computer is configured to select an action from the database in response to a detected attack and trigger its execution.

[0015] In another aspect, the present invention relates to a method for operating a UAV. In various embodiments, the method includes the steps of: obtaining digital images in real time during the flight of the UAV; obtaining GPS signals for navigating the UAV; communicating with land or air infrastructure via radio signals; computationally analyzing the obtained digital images, GPS signals, and / or radio signals based thereon using a predictor computationally trained to detect cyberattacks on the UAV; computationally identifying mitigation actions to be taken in response to detected cyberattacks; and taking those actions. [Brief explanation of the drawing]

[0016] The above and the following detailed explanations will be easier to understand when interpreted in conjunction with the drawings.

[0017] [Figure 1] Figure 1 conceptually illustrates HAPS, which provides reconnaissance and connectivity within the monitored airspace.

[0018] [Figure 2] Figure 2 conceptually illustrates the role of HAPSNN in predicting and providing the need for communication links within monitored airspace when obstacles or terrain are blocking line-of-sight communications.

[0019] [Figure 3]Figure 3 conceptually illustrates the role of the HAPSNN in predicting the need for an extended authorized airspace for a drone moving through a monitored area and actively acquiring it.

[0020] [Figure 4] Figure 4 conceptually illustrates the HAPSNN prediction of the need for and authorization for a flight plan deviation.

[0021] [Figure 5] Figure 5 conceptually illustrates the HAPSNN that provides surveillance and connectivity within an airspace to relay signals from multiple navigation aid facilities to air traffic that is out of range or whose signal reception is blocked.

[0022] [Figure 6] Figure 6 conceptually illustrates the HAPSNN that provides surveillance and connectivity within an airspace to monitor and avoid airspace conflicts.

[0023] [Figure 7] Figure 7 conceptually illustrates the HAPSNN prediction of the need for and authorization for a flight plan deviation to enable the efficient execution of an investigation mission.

[0024] [Figure 8A] Figure 8A is a block diagram of a representative HAPSNN and DINN architecture according to an embodiment of the present invention.

[0025] [Figure 8B] Figure 8B schematically illustrates a manner in which the HAPSNN can support the operation of the DINN.

[0026] [Figure 9A] Figures 9A and 9B schematically show the generation of a flight plan for performing an investigation mission, which can be changed in real time as the drone conducts the investigation. [Figure 9B]Figures 9A and 9B schematically illustrate the generation of a flight plan for carrying out a survey mission, which may be modified in real time as the drone conducts the survey.

[0027] [Figure 10] Figure 10 is an elevation view illustrating various anomalies associated with the antenna that the drone according to this specification may identify.

[0028] [Figure 11] Figure 11 conceptually illustrates the detection of an abnormal electromagnetic (EM) signature from a transducer by drone.

[0029] [Figure 12] Figure 12 conceptually illustrates a drone in flight reading stored data that indicates an anomaly.

[0030] [Figure 13A] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13B] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13C] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13D] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment. [Figure 13E] Figures 13A-13E schematically illustrate the process for establishing the optimal flight path through a complex survey environment.

[0031] [Figure 14] Figure 14 schematically illustrates the detection and classification of radiation centers from multiple antennas.

[0032] [Figure 15A]Figures 15A-15C schematically illustrate the response to an RF blocking attack in various embodiments. [Figure 15B] Figures 15A-15C schematically illustrate the response to an RF blocking attack in various embodiments. [Figure 15C] Figures 15A-15C schematically illustrate the response to an RF blocking attack in various embodiments.

[0033] [Figure 16A] Figures 16A-16C schematically illustrate the response to laser attack in various embodiments. [Figure 16B] Figures 16A-16C schematically illustrate the response to laser attack in various embodiments. [Figure 16C] Figures 16A-16C schematically illustrate the response to laser attack in various embodiments. [Modes for carrying out the invention]

[0034] First, referring to Figure 1-7, this illustrates the functions performed by a conventional HAPS platform 105 and how these functions can be enhanced through the operation of HAPSNN. In Figure 1, the HAPS platform 105 communicates with multiple cell phone towers, representatively shown at 108, multiple air traffic control systems, representatively shown at 110, and a series of aircraft, representatively shown at 113, all of which are located within a defined airspace 115 monitored by the HAPS platform 105. The HAPS platform 105 can also communicate with multiple drones 118 within the airspace 115, directly and / or via cell phone towers. Finally, the HAPS platform 105 can communicate with satellites, representatively shown at 120, for example, for geolocation to maintain a substantially fixed position. Thus, the HAPS platform 105 acquires and updates a complete snapshot of the airspace 115, monitors air traffic within it, and acts as a communication hub between various intercommunication entities within the airspace 115.

[0035] As shown in Figure 2, HAPS105 monitors the position, status, and speed of the drone 118 and manned air traffic 133 operating within the monitored airspace 115. The onboard HAPSNN, assuming the operating altitude of the drone 118, recognizes that terrain or other obstacles 205 would likely disrupt line-of-sight communication between the drone 118 and the air traffic control system 110, and furthermore, between the aircraft 113. Obstacles 205 are recognized and mapped in real time by HAPS105, and are likely to be stored in maps accessible locally to HAPS105 or via a wireless link. More specifically, HAPSNN may calculate the likelihood that obstacles 205 will interfere with current or future communication between the drone 118 and the air traffic control system 110, and if that likelihood exceeds a threshold, it registers the need to establish a communication link bridging the drone 118 and the air traffic control system 110. In that case, HAPS105 will respond by obtaining the status of the drone 118 (including altitude, position, and orbit) through any appropriate modality or combination thereof, such as observation, telemetry, signal monitoring, and / or direct communication with the Earth Air Traffic Control System that monitors the drone 118 and / or its flight.

[0036] As a result of this recognized need, HAPS105 can enter the communication network as an intermediate node and relay messages (i.e., act as a transmission link) between the drone 118 and the air traffic control system 110 (e.g., UTM and LAANC) or other ground-based air traffic reconnaissance infrastructure. In the absence of HAPSNN, HAPS105 would operate accordingly, but if, for example, the drone 118 had previously communicated with HAPS105 and the control system 110, HAPS105 could act as a backup communication channel when direct communication between the drone 118 and the control system 110 is lost as the drone 118 approaches the obstacle 205. HAPSNN facilitates proactive and predictive intervention by HAPS105 even when no prior communication has taken place between the drone 118 and the control system 110. Based on stored or acquired knowledge of the terrain and fixed communication features within airspace 115, and the calculated trajectory of drone 118 (which may have just entered airspace 115), HAPSNN recognizes the need for communication between drone 118 and control system 110 and causes HAPS 105 to establish a wireless link with itself as a hub. Similarly, based on knowledge of the terrain and monitored altitude of drone 118 and manned aircraft 113, HAPSNN may cause HAPS 105 to establish a wireless link with itself as a hub between drone 118 and aircraft 113.

[0037] Figure 3 illustrates a situation in which a drone 118, navigating through an altitude-restricted and authorized airspace 305, would encounter obstacles, collectively shown in 310, which it would have to fly around in order to remain within the airspace 305. This deviation from the pre-programmed flight path can be very large, depending on the extent of the obstacles, and may require course corrections that the drone is not equipped to handle with high accuracy, potentially leading to it missing all or part of the target when it reaches it. When using a HAPS 105 communicating with the drone 118, the HAPS NN anticipates the need for additional airspace authorization and requests an extension to area 315. Once authorization is obtained, the HAPS 105 communicates the relaxation of the altitude restriction to the drone 118. In some embodiments, the HAPS 105 may calculate a modified flight path 320 at a higher altitude for the drone 118, enabling it to remain on course for the target without lateral deviations that may affect navigation accuracy.

[0038] Similarly, in Figure 4, the planned and approved flight path 405 for drone 118 will be incorporated by HAPS105 through hazardous local weather patterns detected using its onboard radar or from real-time weather updates. Knowing the drone's flight path and weather conditions, HAPSNN recognizes the need for an alternative flight section 410, which it, or another onboard calculation module, calculates. HAPS105 obtains approval for the new flight path 410 and communicates it to the drone 118's navigation system.

[0039] Figure 5 shows a drone 118 advancing through an obstacle that obstructs line-of-sight communication between a cell phone relay tower 108 and a source 505 of National Aeronautical System navigation aid signals (VOR, VOR / DME, TACAN, etc.). HAPSNN infers this condition from its knowledge of the land features within the airspace 115 and the state of the drone 118. As a consequence, HAPSNN causes HAPS 105 to relay signals from at least the obstructed sources 108, 505, or to act as a hub to establish wireless links between itself and the communication entities 108, 118, 505.

[0040] In Figure 6, drone 605 is flying within its authorized airspace 610. A second drone 615 enters airspace 610, and HAPSNN detects that drones 605, 615 are not communicating or are unable to communicate. For example, one or both of drones 605, 615 may be “non-cooperative,” i.e., not equipped to communicate with other drones and avoid collisions in their flight paths. HAPSNN may infer this based on communication between drones and / or with ground-based air traffic reconnaissance and surveillance systems, or the absence thereof. HAPSNN determines that, in order to avoid a collision, drone 605 should follow an alternative flight path 620, which may be temporary until drone 605 passes drone 615. HAPSNN or another onboard calculation module calculates the new flight path 620. HAPS105, accordingly, obtains authorization for the new flight path 620 and communicates it to the navigation system of drone 605.

[0041] Referring to Figure 7, the drone 118 follows a flight path 705 within an authorized airspace corridor 707, for example using an RFID reader, and investigates or captures data readings from a set of assets (e.g., a set of transducers 7101...7105 along a power line 712). The HAPSNN monitors the data stream received by the drone 118, and if the drone detects an anomaly, the HAPSNN may infer that the drone 118 will request a change to its airspace authorization. In particular, if the anomaly is associated with transducer 7103, the anticipated authorization need may cover area 720. The HAPSNN may cause the HAPS 105 to take action, for example, relay the information to the appropriate UTM / LAANC system, or request a change to the airspace authorization on behalf of the drone 118, which would then be free to investigate transducer 7103 more closely and record its image for real-time or later evaluation.

[0042] Figure 8A illustrates a typical HAPSNN architecture 800, which includes a HAPS aircraft and various hardware and software elements. Generally, multiple software subsystems, implemented as instructions stored in computer memory, are executed by a conventional central processing unit (CPU) 802. The CPU 802 can control the flight and operation of the HAPS vehicle and the functions described below, or these functions may be distributed among separate processors 802. In addition, for efficient execution of neural network functionality, the system may include a dedicated graphics processing unit. An operating system (e.g., MICROSOFT WINDOWS®, UNIX®, LINUX®, iOS, or ANDROID®) provides low-level system functions such as file management, resource allocation, and routing of messages to and from hardware devices and software subsystems (including at least one non-volatile memory element 803), which are executed in computer memory 804. More generally, the HAPSNN 800 may include modules implemented in hardware, software, or a combination of both. Regarding the functions provided within the software, the program may be written in any of the following: Python, Fortran, Pascal, Java®, C, C++, C#, BASIC, various scripting languages, and / or several high-level languages ​​such as HTML. The software modules and mechanical and flight features involved in the operation of the HAPS vehicle are conventional and not illustrated. See, for example, U.S. Patent No. 10,618,654.

[0043] The HAPSNN800 includes a neural network module 805, a transceiver module 808, and a field-programmable gate array (FPGA) 810. The transceiver module 810 and FPGA 810 may constitute, or be part of, a communications system configured to support airborne communications between aircraft and with land and satellite-based control infrastructure. The HAPSNN800 may (but may not) operate in conjunction with a drone equipped with the DINN812. The cloud neural network module 805 is local to the HAPS vehicle, but more typically, it may operate in the cloud, i.e., on a remote (e.g., land) server for wireless communications with the HAPS vehicle, as described below. Modules 808 and 810 are typically located on the HAPS vehicle itself.

[0044] The cloud neural network module 805 includes a classification neural network 815 that processes images and data received in real time from the drone via the agile transceiver 808, which may pass through to the cloud neural network 805. The classification neural network 815 is trained using a database of training images 817 related to the tasks undertaken by the monitored drone. The classification neural network 815 processes and classifies the received images and data and detects anomalies associated with them, i.e., calculates their probability. That is, anomalies can be detected when they are considered based on something in the received images or in parallel with other drone telemetry. For example, anomaly detection may be triggered when other non-exceptional images are taken in conjunction with weather conditions reported by the drone. When an anomaly is detected, the classification neural network 815 may look up the classification database 819 to determine an appropriate response, i.e., the database 819 contains records that define the anomaly and, in turn, one or more associated actions that may be taken. If an anomaly not recorded in the database is detected, the image may be transmitted for human investigation and classification. The new classification is then added to the training database 817 and used to retrain the neural network 815. The resulting adjusted weights may be backpropagated by the cloud server associated with the neural network 805 to DINN 812 (if present) to transmit similar mission profiles to drones in the field and other drones. This procedure is described further below.

[0045] The agile transceiver package 808 includes an Automated Dependent Reconnaissance Broadcast (ADS-B), a Traffic Collision Avoidance System (TCAS), a Secondary Reconnaissance Radar (SSR), and an Automated Dependent Reconnaissance Rebroadcast (ADS-R) subsystem, operating at 978 MHz, 1,090 MHz, and 1,030 MHz for querying, responding, and rebroadcasting. These enable the HAPSNN800 to "listen" to the position of manned air traffic, and thus the neural network 815 can computationally represent nearby traffic in 3D or 2D space and analyze any collisions between the drone and manned air traffic. This can be achieved by broadcasting the drone's position to manned air traffic, or the position of manned air traffic to the drone. Emergency warnings are issued to manned and / or unmanned traffic with commands regarding the direction in which they should move to avoid collisions in the airspace.

[0046] The agile transceiver 808 may include a cellular network package, including 3G, 4G, LTE, 5G, or any future telecommunications protocol and bandwidth, to support communication links between drones operating within the HAPSNN800 airspace, using land telecommunications networks utilized by several UTM systems, or using backhaul communication channels for transmitting data from HAPS to a cloud-based neural network. VHF and UHF transceiver (TX / RX) modules may be used to monitor navigation aids such as VOR, VOR / DME, or TACAN, enabling the neural network 805 to analyze the position of the drone and HAPS using the flight time of the signal in the event of GPS signal loss. This also enables the utilization of satellite communication constellations to transmit or receive data if necessary. The drone virtual radar (DVR) data link facilitates communication with drone platforms implementing the technology (as described, for example, in U.S. Patent No. 10,586,462) to transmit and receive air traffic position information for use in collision analysis or drone tracking. The neural network (NN) data link is a dedicated high-bandwidth backhaul channel that enables HAPSNN800 to communicate with the DINN neural network computation engine 825, transmitting real-time data received from multiple drones operating within the monitored airspace and receiving prediction and action commands retrieved from the classification database 819. The FPGA 810 is employed as a hardware accelerator to launch software that tunes the transceiver 808 and filters out noise.

[0047] A typical DINN812 implemented within the drone 118 includes a neural network computation engine 825, a classification database 825, and "backend" code for performing various data handling and processing functions as described below. In addition, the drone 118 includes communication equipment comprising, or consisting of, a set of agile transceiver 808 and FPGA 810, as detailed above. The drone 118 may also include a CPU 802, storage 803, and computer memory 804.

[0048] As mentioned, DINN812 can interact with HAPSNN800, but either can exist and operate on its own; that is, HAPSNN is unnecessary for the successful deployment and use of DINN, but HAPSNN can play a reconnaissance and safety role for an aircraft lacking DINN. The role of DINN812 is to enable the drone 118 to classify objects of interest on the asset being investigated and obstacles that will need to be avoided during flight (e.g., recognizing cell phone towers to be investigated and cracked antennas on such towers). The neural network 825 is configured to process and classify images received from an image acquisition device 827, e.g., a video camera on the drone 118. Thus, the neural network 825 may be a convolutional neural network (CNN) programmed to detect and recognize objects in incoming images. These can be classified based on the training of a neural network, and DINN812 (e.g., backend code) can look up the classification database 830 to determine the appropriate response to the detected image. In this case, the database 819 contains records defining objects, each associated with several semantic meanings or actions. For example, if the neural network 825 detects a tree in the incoming image, the corresponding database entry may identify the tree as an obstacle, triggering an evasive maneuver performed by the drone's navigation system 832 by controlling the drone's steering and propulsion systems. These are part of the drone's flight package 835, which is conventional and therefore not shown in detail, but includes a power supply, communication platform, propulsion and steering systems, autopilot system, etc.

[0049] DINN812 may also receive data from one or more reconnaissance systems 837, 839, which may include one or more of the following: DVR, UTM, LAANC, ADS-B, and TCAS systems. While these may be implemented as part of the drone's communication platform, they are illustrated conceptually within DINN812 because the neural network 825 may use this data when classifying images. Similarly, the weather reconnaissance system 842 would conventionally be implemented within the drone's communication platform, but again, it is shown as part of DINN812 because weather conditions may be relevant to image classification or database retrieval, as shown in Figure 4. The same visual scene may prompt different actions depending on the weather; for example, the drone 118 may give a wider mooring area to a building under strong wind conditions.

[0050] In embodiments where the drone 118 interacts collaboratively with the HAPSNN 800, the latter may provide further support and stronger classification capabilities. For example, images with detected objects that are unclassifiable by the neural network 825 may be uploaded to the HAPSNN 800 for examination, and real-time commands issued by the HAPSNN as a response may be executed by the drone's navigation system 832. Furthermore, the HAPSNN 800 may update or supply different weighting files for the neural network 825 in real time to better suit the drone's mission based on the classification performed by the drone (and communicated to the HAPSNN 800 in real time). The neural network 825 reads these new weighting files in response when they are received.

[0051] This process is shown in Figure 8B. Referring also to Figure 8A, DINN 812 processes incoming image frames (e.g., approximately 30 frames per second (FPS)) in real time, enabling the drone 118 to react fast enough to avoid collisions and fly around the tower 850. Therefore, the drone 118 may include a graphics processing unit (GPU) to support CNN operation. Real-time frame analysis allows the GPU to process the images, classify the items of interest on the assets to be investigated, notify the backend code of the classification, and enable the backend code to execute logic to react to the classification by performing a search in the classification database 830 to obtain an action corresponding to the classification.

[0052] The drone 118 transmits image data to the HAPSNN, which includes a high-precision CNN (within the computation engine 815, or optionally, further within HAPS itself) capable of processing, for example, 60 megapixels (MP) of photographic images per second. The CNN architecture is designed for speed and accuracy of classification by leveraging backend logic, which runs on the computation engine 825. This backend logic can modify the CNN's weights and configuration files based on the assets to be classified based on the first few images of the assets captured by the drone. These preliminary images are collected as part of a “backup” flight path around the assets at a safe distance and may have a resolution of 60MP or more. These preliminary images are reduced to the input image size of the CNN (e.g., 224x224, or larger depending on the assets to be investigated) and passed through a series of convolutional layers (e.g., 20), followed by an average pooling layer and fully connected layers pre-trained to classify different assets (e.g., 100 types of assets). Once the asset type is identified, the weighting and configuration files may be modified, more (e.g., four) convolutional layers may be added, followed by two fully connected layers to output the probabilities and bounding boxes of any objects or areas of interest that may exist on the asset. Images uploaded from the drone may be increased in size (e.g., up to 448x448) because this type of classification requires the presence of finer-grained detail. The degree of size increase is dynamically controlled and can be scaled up, for example, if sufficient detail is not detected for reliable classification.

[0053] The fully coupled layer predicts class probabilities and bounding boxes (i.e., cracked antennas, rust, and corrosion). For example, the final layer may use linear activation, while the convolutional layer may use leaky ReLU activation.

[0054] Once the calculation engine 825's backend logic detects the presence of class and bounding box coordinates, it can trigger a switch to the center of gravity tracker function to move its specific classification to the center of the drone's image acquisition device's field of view. The backend logic works with the ranging calculation engine to analyze the safest flight path for the drone in order to approach and position the image acquisition device for high-resolution scanning.

[0055] Therefore, the preliminary flight path establishes the type of asset within the field of view and registers the position of the asset in 3D space relative to the drone in order to account for any GPS drift vectors. If any obstacles or hazards exist within the operating area, they are classified and their positions in 3D space are registered. The center of gravity tracker is activated once classifications within the area of ​​interest are detected, keeping the object of interest centered within the field of view. The ranging calculation engine controls the forward and reverse movement of the drone. Before any of these commands are executed, the position of the drone in 3D space relative to assets and any obstacles present within the operating area is acquired. This data is invoked through backend logic to analyze a safe GPS waypoint flight path that will move the drone into the area of ​​interest, and in GPS rejection areas, this flight path can still be analyzed and executed using Kalman filtering of inertial data in conjunction with the center of gravity and ranging functionality. The flight path is fine-tuned in real time via the center of gravity tracker and ranging calculation engine. Please note that the center of gravity tracking can be performed by the HAPSNN800, not the DINN812.

[0056] In step 855, the HAPSNN CNN ("HP-CNN") processes each image to detect objects within it using a standard object detection routine (e.g., YOLO), and attempts to identify (i.e., classify) all detected objects based on its prior training (further discussed below). Detected objects that cannot be identified are stored in a database and made available to personnel for identification and labeling (step 857). The HP-CNN is then retrained on an augmented dataset containing the newly identified and labeled object data (step 859), resulting in the generation of new CNN weights (step 862). If the real-time neural network 825 residing on the drone 118 is also a CNN ("RT-CNN"), the HAPSNN 800 may push these weights to the RT-CNN, which receives and loads them. That is, the HP-CNN and RT-CNN may be the same or substantially similar so that the CNN weights generated for the HP-CNN can propagate across the drone formation.

[0057] HP-CNNs (or, in some embodiments, RT-CNNs on top of themselves) can be trained in conventional ways. In particular, the CNN is trained on labeled images of objects that the drone is likely to encounter as it performs its mission, resulting in a CNN that can analyze and classify the objects most likely to be encountered by the drone within their typical flight paths. Since the training set cannot be exhaustive and the drone will inevitably encounter unknown objects during use, the process described above—locating the whereabouts of unrecognized objects, memorizing them for manual labeling, and then retraining the CNN on an expanded dataset—helps minimize the risk of accidents by constantly enriching the drone's visual vocabulary and action repertoire. This is most efficiently performed using HAPSNNs as a central training hub receiving unclassifiable objects from many drones, and all of their neural networks can be kept updated to reflect the latest classification capabilities; however, it is also possible to implement this training and retraining functionality on individual DINNs.

[0058] Figures 9A-13E illustrate various applications of drones and the ways in which DINNs can be used to control and simplify the operation of drones. In Figure 9A, the DINN 812 guides the drone 118 around the antenna 900 so that when the drone 118 detects anomalies or structures requiring a closer inspection, it will conduct the inspection according to a flight pattern 910 that may change in real time. For example, as shown in Figure 9B, the flight path 910 may be modified so that the drone 118 does not approach a power line 915 that would be perceived as an obstacle.

[0059] Referring to Figure 10, the antenna 1000 has a crack 1005, which can be identified by a DINN-equipped drone. The backend may run an OpenCV function to fix the center of gravity on the antenna 1000 and move the drone to position the antenna in the center of the field of view to facilitate image acquisition. This can be combined with a depth map generated by a stereo RGB camera of known sensor size, lens properties, and camera focus separation to position the drone 118 close enough to the antenna for the camera to analyze enough detail to good pick up the crack. Other defects that can be recognized by the drone 118 include structural corrosion 1010 and paint coating anomalies 1015. The drone 118 payload may include additional sensors such as an EM sensor and an infrared camera, and the DINN 812 may include additional modules such as a spectral analysis and / or electromagnetic calculation engine. These enable the drone 118 to detect PIM signatures at the coaxial connector 1025, other EM anomalies such as magnetic fields emitted from the surge arrester 1030 (indicating the possibility of arc discharge), and an anomalous thermal signature 1035. These conditions can be diagnosed and observed by the drone 118, or using the onboard calculation engine and / or in cooperation with the HAPSNN800 as described above. The DINN812 may further include a ranging calculation engine to initiate data fusion between depth maps acquired as discussed above and laser rangefinders / radar / ultrasonic or other ranging sensors that may be included in the drone payload to derive the most accurate distance measurements, a computer vision calculation engine to perform image processing and analytical functions (such as center of gravity placement as described above), and / or infrared and multispectral calculation engines to analyze infrared and other images acquired at wavelengths outside the visible spectrum.

[0060] As shown in Figure 11, under normal operation, the magnetic field lines are vertical, as shown in 1110. If the drone 118 detects an unusual EM signature from asset 1100, such as an abnormal non-perpendicular magnetic field line 1105, it may record the anomaly or, in embodiments where the drone 118 communicates with HAPSNN, report it to HAPSNN, which may autonomously call in a drone with a more specialized survey payload to better diagnose the condition. The drone 118 may also communicate with or otherwise interact with the asset to be surveyed. In Figure 12, the drone 118 reads data stored in the transducer 1200 indicating an anomaly. That is, the transducer 1200 has self-diagnostic capabilities, and when an operational anomaly is detected in advance, data indicating the detection is stored internally. If the transducer 1200 is periodically surveyed by the drone 118 and the abnormal condition does not require immediate attention, it may not be necessary for the transducer to communicate its presence to supervisory personnel in response to the detection. Rather, when the drone 118 conducts a survey flight, it queries the memory of all transducers, thereby obtaining previously stored data indicating the detection of anomalies in the transducers. This can prompt the drone 118 to conduct closer surveys (as shown in the figure) in response to the classification of conditions and database searches. Again, HAPSNN recognizes the conditions and transmits new weights to the DINN of the drone 118, enabling it to perform condition-specific classification as it surveys the transducers 1200.

[0061] Figures 13A-13E illustrate how DINN812 can establish an optimal flight path through a complex survey environment. Figures 13A and 13B illustrate a power distribution substation 1300, which includes a series of outgoing transmission lines, collectively shown in 1310, a voltage regulator 1315, a drop converter 1320, a series of switches 1325, one or more surge arresters 1330, and a receiving transmission line, collectively shown in 1335. Any of these components may experience anomalies or malfunctions, and all can be surveyed by the drone 118. As shown in Figures 13C and 13D, the drone 118 may perform a preliminary flight pattern 1350, either alone or in conjunction with HAPSNN, to collect images for analysis by DINN. DINN812 analyzes the images obtained by the drone's onboard camera and classifies the various components 1310-1335. Based on these classifications, the navigation module 832 (see Figure 8) or backend code calculates an optimized flight plan 1355 that enables all components 1310-1335 to be surveyed appropriately and efficiently by the drone 118. More specifically, each component located within the substation 1300 is classified during the preliminary survey flight path 1350, and these are located far from the substation 1300 to adapt to GPS drift vectors and other unknowns related to obstacles or inaccuracies centered on the initial flight path. Once a component is classified, its position in the image and the drone's position are registered. This process is repeated multiple times throughout the preliminary survey, enabling the backend code to triangulate and position each classified asset in 3D space, thus allowing for the calculation of a more accurate survey flight path that will move the drone and payload closer to the assets. This new flight plan 1355 is then executed, and again, DINN 812 classifies the assets as the flight path is executed. Preliminary investigations suggest that because the drone is far away, the DINN812 can only classify large items if the resolution of the image acquisition device is fixed.Once the closer flight path 1355 is executed, more asset details will be detected, and DINN812 will be enabled to classify the new items and, if necessary, readjust the drone's path. The ranging engine calculates the nearest acceptable approach distance between the drone and the asset, which matches acceptable and safe limits.

[0062] Another application where complex survey operations can be optimized by DINN is illustrated in Figure 14. Here, the objective is to determine the "radiation center," that is, the radiation center 1410 of each of the antennas 1410, 1415, and 1420 mounted on the telecommunications structure 1400 at different elevations. R , 1415 R , 1420 R The task is to detect and classify the radiation centers. DINN classifies the radiation centers, then identifies the center of gravity for each antenna, and moves the radiation center to the center of the drone camera's field of view, thereby allowing the barometer reading (altitude) to be recorded. Navigation module 832 can then use this altitude to generate an orbital flight path, enabling drone 118 to acquire 3D model data for reconstructing the antenna structure.

[0063] DINN812 can be configured to detect a cyberattack, determine one or more actions necessary to mitigate or neutralize the attack, and carry out those actions. In some cases, this may involve HAPSNN800. For example, DINN812 may detect that the navigation package 832 has been compromised by an attack and transfer control of the navigation to HAPSNN800, which can monitor the location of the drone 118 and the images recorded by the image acquisition device 827 in real time, and navigate the drone 118 around obstacles, for example, returning it to its starting point or the operator's location. In other cases, DINN812 may recognize the presence of a cyberattack but may not be able to recognize its characteristics, in which case it would be impossible to identify the actions to be taken in response. In this situation, it may communicate data revealing the presence of the attack to HAPSNN800. The HAPSNN800 may, either independently or after communication with land resources, infer the nature of the attack and mitigation strategies, which are communicated to the drone 118, and again, the mitigation strategies may involve the HAPSNN800 assuming control of the drone 118's movements.

[0064] In the case of a GPS spoofing attack designed to hijack a drone and guide it away from its intended flight path, or an RF saturation attack designed to sever the connection between the drone and HAPSNN or land communications, DINN812 can track the drone's position relative to assets or other landmarks, enabling the drone 118 to resist being forced to navigate in another direction or to maintain a safe flight path until communications are reliably re-established. In the case of a laser attack intended to disorient optical sensors or block light, DINN812 can recognize and classify the patterns produced by such an attack, trigger a response by the drone to counterattack or alter the drone's flight path to evade the source of the attack.

[0065] In its simplest form, a GPS spoofing attack consists of RF signals that obscure the RF signals from the actual GPS constellation. The spoofing attacker replaces the actual location information with any information necessary to confuse the GPS into believing it is moving away from its intended location (essentially, the GPS spoofer guides the GPS to make corrections to move it back to where it should be, in the direction the spoofer wants it to move). The flight path of the drone 118 to the asset is typically pre-planned using GPS waypoints that the drone will execute, but the DINN 812 uses optical flow sensors to correct and adjust the flight path in real time as it classifies the asset and any obstacles or hazards within the operating environment. For example, Figure 15A shows the drone 118 on its flight path to a target of interest while being guided by the HAPS 105, which is not necessarily the case, but is possible. The position of the drone 118 is monitored by HAPS 105 and the drone itself, for example, based on GPS and / or the land cellular network 1515 and / or navigation aids. The onboard RF source 1505 generates an RF field 1510 that suppresses RF communications within a spherical effective area around the source 1505. The drone's flight path 1420 passes through the RF field 1510, and as shown in Figure 14B, when the drone 118 enters the RF field 1510, communication with HAPS 105 and the ground cellular network 1515 is severed. Recognizing the nature of the attack, the drone 118's DINN 812 triggers the execution of a pre-programmed avoidance flight path 1525 until communication is no longer affected by the RF field 1510 and communication is re-established. DINN812 calculates a new flight plan 1530 that keeps the drone 118 outside the range of the interfering RF field 1510 (based on the destination and, in some cases, the estimated shape of the RF field 1510), either on its own or more typically with assistance from HAPS105.

[0066] In particular, DINN812 can plot a new flight path for drone 118 via a conventional algorithm that generates a 3D map of the backend logic and operating environment and plots the positions of drone 118, assets, and / or obstacles within this environment. This data is stored in memory associated with the neural computation engine 825 and can be optimized to enable real-time position calculation and analysis. This allows the drone to easily track its actual position relative to where it needs to be and relative to landmarks or obstacles. Kalman filtering can be applied to track the movement of drone 118 using the inertial navigation capabilities of system 832, which is data fused with neural network classification that may utilize pre-stored how-to point image matching, compass data, optical flow tracking of assets or landmarks using RGB sensors, location information of National Aeronautical Aids, and / or GPS location information to track the drone's movement. This enables DINN812 to plot the drone's position in the 3D environment in real time and detect whether one or more of the sensors is giving an abnormal reading that places the drone 118 at a different location in the 3D environment relative to where it should be, and where the other sensors indicate it is. For example, if during a spoofing attack the GPS indicates that the drone 119 is moving away from its intended waypoint, but inertial navigation aids, a landmark-tracking optical flow sensor, and National Aeronautical Navigation Aids indicate that the drone 118 is moving toward the waypoint, DINN812 would recognize that a GPS spoofing attack is taking place. Once a scenario is classified, algorithms designed to steer the drone to counter the attack, or other pre-programmed operations to move the drone away from the attack, can be triggered.

[0067] The data link between DINN812 and HAPSNN800 is typically encrypted and may utilize a handshake to ensure that each DINN812 is identified by the HAPSNN800 before the communication channel is opened. This serves to prevent unauthorized communication attempts or impersonation by unauthorized users or DINNs.

[0068] In its simplest form, a laser or optical attack is designed to damage the imaging sensor of an optical flow device, such as an RGB or IR camera. If the sensor is not damaged, the shutter speed or aperture may be affected, blurring the image and thus rendering the sensor unusable, as patterns or colors can no longer be distinguished. This effectively prevents the optical flow sensor from being used for navigation purposes, as the neural network cannot classify the patterns in the image. Such a scenario is relatively easy to detect, for example, whether a predetermined threshold number of blurred images are detected within a given period when the sensor is pointed in a specific direction. An algorithm can be activated via backend logic to change the orientation of the optical flow sensor and detect whether the blurred images cease. If the blurred images resume when the sensor is pointed back to its original orientation, DINN812 will recognize and classify this scenario as a laser attack and plot the area / orientation in the 3D environment as a no-fly zone. Assuming that the laser or other optical attack is directional, this is a simple solution to be implemented. For example, Figure 16A shows a drone 118 on a flight path 1605 to a target of interest while being guided by HAPS 105, although this is not necessarily the case, but is possible. The position of the drone 118 is monitored by HAPS 105 and the drone itself, for example, based on GPS and / or land cellular network 1615 and / or navigation aids. The laser attacker 1605 targets the drone 118 (Figure 16B), DINN 812 detects the pattern of the laser attack, and backend logic enables evasive action to be taken. As shown in Figure 16C, DINN 812 and / or the HAPSNNN of HAPS 105 may generate a new flight path 1620 around or away from the source of the laser attack 1605.

[0069] If GPS signals, navigation aid signals, and data link signals to HAPSNN or the land communications network are all lost simultaneously, DINN812 classifies the scenario as an RF saturation attack, at which point the backend logic may invoke a conventional algorithm that leverages a 3D environment stored in memory, associated with the neural computation engine 825, to plot a path outside and away from the current operating environment. The actual flight path of the drone 118 relative to the 3D plot is monitored and adjusted using optical flow and inertial navigation Kalman filtering until the drone 118 can re-establish RF communication.

[0070] The terms and expressions used herein are for illustrative purposes only and are not limiting, and in the use of such terms and expressions, there is no intention to exclude any equivalent or part of any feature shown or described. In addition, although one embodiment of the invention has been described, it will be apparent to those skilled in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the invention. Therefore, the embodiments described should be considered in all respects to be merely illustrative and not limiting.

[0071] The charges will be as follows:

Claims

1. It is an unmanned aerial vehicle (UAV), Flight package and UAV navigation system and UAV image acquisition device, UAV communication equipment including multiple agile transceivers, A computer memory containing multiple pre-stored waypoints and flight images of a pre-programmed flight path, It includes a drone survey neural network (DINN) that includes a neural computation engine with neural network-based predictors, The neural computation engine includes a processor and instructions stored in computer memory and executable by the processor, in order to reconstruct a real-time 3D map of the UAV operating environment using real-time data received from one or more of the UAV image acquisition device, the UAV communication equipment, or the UAV navigation system, and to position the UAV in real time on the 3D map, The aforementioned neural network-based predictor, The real-time 3D map and real-time UAV data are received, (a) Identify and classify that an attack has occurred on the UAV, (b) identify the characteristics and type of the attack, and (c) identify whether the attack characteristics are unclassifiable by DINN. The neural computation engine is computationally trained in such a manner, The aforementioned real-time data is sent to a High Altitude Pseudo-Satellite System (HAPS) or a cloud-based neural computing engine. Based on the HAPS or cloud neural computation engine inspection of the aforementioned real-time data, updates are received in real time with different weighted files for neural network-based predictors in line with the relaxation strategy. An unmanned aerial vehicle (UAV) configured in such a way.

2. The UAV according to claim 1, wherein the neural computation engine is configured to identify mitigation actions to be taken in response to classified attack types and to take such actions.

3. The UAV according to claim 1, wherein the UAV communication equipment is configured to communicate with land and high-altitude pseudo-satellite (HAPS) based communication networks to establish a data link with a HAPS or cloud-based neural computation engine.

4. The UAV according to claim 1, wherein a cloud-based or HAPS-based neural computation engine enables dynamic scaling of neural computation power in real time and assists in the classification of data uploaded from the UAV.

5. The UAV according to claim 1, further comprising a database of actions, wherein the neural computation engine is configured to select a mitigation action from the database of actions in response to a detected attack classified by the predictor and to trigger its execution.

6. The UAV according to claim 1, wherein the attack is GPS spoofing, and the mitigation action is to cause the UAV to navigate without GPS.

7. The UAV according to claim 1, wherein the UAV communication equipment is tuned to a plurality of National Aeronautical Navigation Aids and tracks the position of the UAV in accordance with a pre-stored waypoint flight plan.

8. The UAV according to claim 1, wherein the navigation of the UAV is performed using optical flow or inertial navigation, and the position of the UAV along a pre-stored waypoint flight plan is tracked using a classification of real-time flight images from the UAV compared to pre-stored images of waypoints along the pre-stored waypoint flight plan, the pre-stored images of waypoints being separate and distinct from the real-time flight images.

9. The UAV according to claim 1, wherein the attack is GPS spoofing, and the action is to cause the transfer of navigation control to a HAPS vehicle-based neural computation engine, the HAPS vehicle-based neural computation engine being separate from and distinct from the neural computation engine of the UAV.

10. The UAV according to claim 1, wherein the attack is RF blocking, and the action is the execution of a pre-stored flight plan that does not require external communication, and the pre-stored flight plan is separate from and distinct from a pre-programmed UAV reconnaissance mission flight plan.

11. The UAV according to claim 10, wherein the pre-stored flight plan relies on (i) at least one of inertial navigation or optical flow for navigating the UAV along a pre-stored waypoint flight plan, and (ii) classifying when the UAV reaches each waypoint using image matching with pre-stored waypoint images.

12. The UAV according to claim 1, wherein the attack is a laser attack, and the action is to plot a flight path away from the source of the laser and cause the UAV to follow it.

13. The UAV according to claim 1, wherein the neural computation engine is further configured to communicate with the HAPS vehicle to computationally identify the mitigation action.

Images