Threat Assessment of Unmanned Aerial Systems Using Machine Learning

A machine learning-based system integrates diverse data sources to assess UAS threats, addressing the challenge of low-altitude UAS detection and discrimination, enhancing protection of critical targets by differentiating benign from threatening UASs in urban areas.

JP7586614B2Active Publication Date: 2024-11-19RAYTHEON CO
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2022575380
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-08
Filing Date
2021-04-23
Publication Date
2024-11-19
Estimated Expiration
2041-04-23

AI Technical Summary

Technical Problem

The proliferation of inexpensive small unmanned aerial systems (UASs) poses a significant threat to various targets due to their low-altitude operation and limited line-of-sight, making detection, discrimination, and threat assessment difficult, especially in urban areas, and existing countermeasures are costly and ineffective without prior identification.

Method used

A system utilizing machine learning to combine disparate data from multiple sources, including crowd-sourced information and sensor data, to assess the threat level of UASs in real-time through anomaly detection and threat assessment, employing Object Based Production (OBP) and Activity Based Intelligence (ABI) techniques.

Benefits of technology

Enables effective and efficient threat assessment of UASs, providing actionable intelligence for protecting critical infrastructure and non-traditional targets by distinguishing benign from potentially threatening UASs, even in complex urban environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586614000001
    Figure 0007586614000001
  • Figure 0007586614000002
    Figure 0007586614000002
  • Figure 0007586614000003
    Figure 0007586614000003
Patent Text Reader

Abstract

The method includes obtaining (704) a plurality of data items (309) from a plurality of disparate data sources (308), the plurality of data items relating to an unmanned aerial system (UAS) (306, 502) flying within an area. The method also includes processing (710) the plurality of data items using a machine learning algorithm (314) to determine a threat level of the UAS to one or more targets (506, 508) within the area. The method further includes outputting (712) the threat level of the UAS for review by a user.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure is directed generally to unmanned aerial systems. More specifically, the present disclosure is directed to threat assessment of unmanned aerial systems (or other manned or unmanned vehicles) using machine learning. [Background technology]

[0002] Inexpensive small unmanned aerial systems (UASs), such as drones, are proliferating rapidly. Numerous uses for UAS have been proposed and implemented, such as infrastructure and utility inspection, surveillance, courier delivery, urban mobility, etc. While most use cases involving UAS are beneficial, some UAS can be used for malicious purposes, and these UAS can pose a significant threat to a wide range of targets. Unfortunately, detection, discrimination, and threat assessment of UAS can be difficult, especially since UAS often fly at low altitudes. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure relates to threat assessment of unmanned aerial systems using machine learning. [Means for solving the problem]

[0004] In a first embodiment, the method includes obtaining a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to an unmanned aerial system (UAS) flying within an area. In some embodiments, at least one data source includes crowd-sourced information of UAS sightings using a specially designed application. The method also includes processing the plurality of data items with a machine learning algorithm to determine a threat level of the UAS to one or more targets within the area. The method further includes outputting the threat level of the UAS for review by a user.

[0005] In a second embodiment, a device includes at least one memory configured to store instructions and at least one processor coupled to the at least one memory. The at least one processor, upon execution of the instructions, is configured to obtain a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to an unmanned aerial system (UAS) flying within an area. The at least one processor, upon execution of the instructions, is also configured to process the plurality of data items using a machine learning algorithm to determine a threat level of the UAS to one or more targets within the area. The at least one processor, upon execution of the instructions, is further configured to output the threat level of the UAS for review by a user.

[0006] In a third embodiment, a non-transitory computer readable medium includes instructions that, when executed, cause at least one processor to obtain a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to an unmanned aerial system (UAS) flying within an area. The non-transitory computer readable medium also includes instructions that, when executed, cause the at least one processor to process the plurality of data items with a machine learning algorithm to determine a threat level of the UAS to one or more targets within the area. The non-transitory computer readable medium further includes instructions that, when executed, cause the at least one processor to output the threat level of the UAS for review by a user.

[0007] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] FIG. 1 illustrates an example system for assessing the degree of anomaly and potential threat of an unmanned aerial system (UAS) using machine learning, in accordance with the present disclosure.

[0010] [Diagram 2] FIG. 1 illustrates an exemplary device for assessing the degree of anomaly and potential threat of a UAS using machine learning, in accordance with the present disclosure.

[0011] [Diagram 3] FIG. 1 illustrates an exemplary process for assessing the degree of anomaly and potential threat of a UAS using machine learning, in accordance with the present disclosure.

[0012] [Figure 4] FIG. 1 illustrates an exemplary profile map for assessing the degree of anomaly and potential threat of a UAS in accordance with the present disclosure.

[0013] [Diagram 5] FIG. 1 illustrates an example heatmap displaying a UAS threat, according to the present disclosure.

[0014] [Figure 6] FIG. 1 illustrates an example geospatial representation of a counter-UAS solution capable of performing UAS detection, tracking, and assessment in accordance with the present disclosure.

[0015] [Figure 7] FIG. 1 illustrates an exemplary method for assessing the degree of anomaly and potential threat of a UAS using machine learning, in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] 1 to 7 described below and the various embodiments used to illustrate the principles of the present invention in this patent document are for illustrative purposes only and should not be construed as limiting the scope of the invention in any way. Those skilled in the art will understand that the principles of the present invention may be implemented in any type of suitably arranged device or system.

[0017] The ubiquitous availability of small unmanned aerial systems (UAS) poses an increasing number of security threats to public safety that are difficult to prevent using traditional means. For example, in urban areas where legitimate UAS are present, where all UAS are operating at low altitudes or within limited lines of sight, it can be difficult to assess the threat level of small UAS. Also, most traditional solutions provide only limited information available for distinguishing small UAS. As a result, protecting potential targets (such as critical infrastructure resources, major events with large crowds, etc.) against attacks by small UAS is difficult.

[0018] The U.S. Federal Aviation Administration (FAA) continues to develop and evolve rules and regulations for the operation of UAS by defining restricted and prohibited airspace for UAS. Protected airspace generally includes major airports and national security locations, but does not encompass critical infrastructure, key resources, or non-traditional targets such as sporting events. Additionally, compliance with FAA rules and regulations requires cooperation from users, but operators of hostile UAS are unlikely to cooperate prior to an attack. It should be noted that the current rules for commercial use of UAS and the differences between hobbyists and commercial actors are an area of ​​intense research and development, innovation, and rapid evolution of technology, regulations, and laws.

[0019] A variety of counter UAS approaches have been developed to neutralize or destroy UASs identified as hostile. These approaches can be used to accomplish one or more of several goals, such as (1) grounding the UAS or initiating a "return to home" maneuver by disrupting its operation through RF, GPS, or other signal jamming, (2) taking control of the target UAS by spoofing, (3) destroying the target UAS using lasers, projectiles, or other mechanisms, or (4) entangling the blades of the UAS using nets. However, these approaches are expensive and are only useful after the identification of a hostile UAS. There is no practical, low-cost system to assess the potential threat level to military or civilian targets by detected entities, in this particular example, particularly UASs operating at low altitudes or within limited line of sight.

[0020] The present disclosure provides a system and method for assessing the degree of anomaly and potential threat of a UAS using machine learning. As described in more detail below, the disclosed system and method provides a centralized UAS tracking entity that combines disparate data from multiple sources and uses machine learning to assess the threat level of one or more UAS in real time. This can include two related functions: 1. anomaly detection, and 2. threat assessment. Generally, if a UAS is not determined to be anomalous, it is not a threat. In practice, there may be many anomaly detections (e.g., nuisance alarms), and adjudication of anomaly detections is where threat assessment is important. In some embodiments, the entity performs anomaly and threat assessment of each UAS (or other entity) for protection of conventional and non-conventional targets. The assessment can be based on a contextual technique called Object Based Production (OBP). OBP is an intelligence community approach that organizes data by detected objects or entities. Analytics are then performed on that combined data set to develop a context. A variety of analysis techniques can be used, including Activity Based Intelligence (ABI).

[0021] Although the present disclosure is described with respect to UAS, it will be understood that the principles disclosed herein are applicable to other types of vehicles or detected entities. For example, the disclosed embodiments can also be used with manned air vehicles (e.g., low flying aircraft) and vehicles in other domains (e.g., maritime or land traffic). The disclosed embodiments can also be used in conjunction with other detection events or entities, such as chemical, biological, radiological, nuclear, or explosive detection.

[0022] 1 illustrates an example system 100 for assessing the degree of anomaly and potential threat of a UAS using machine learning, according to the present disclosure. As shown in FIG 1, the system 100 includes a plurality of user devices 102a-102d, at least one network 104, at least one server 106, and at least one database 108. However, it should be noted that other combinations and arrangements of components may be used herein.

[0023] In this example, each user device 102a-102d is coupled to or communicates through a network 104. Communication between each user device 102a-102d and the network 104 may occur in any suitable manner, such as via a wired or wireless connection. Each user device 102a-102d represents any suitable device or system used by at least one user to provide information to or receive information from a server 106 or database 108. Example types of information may include detector readings, UAS vehicle information, temporal data, geospatial activity data, etc.

[0024] Any suitable number and type of user devices 102a-102d may be used in system 100. In this particular example, user device 102a represents a desktop computer, user device 102b represents a laptop computer, user device 102c represents a smartphone, and user device 102d represents a tablet computer. However, any other or additional types of user devices may be used in system 100. Each user device 102a-102d includes any suitable structure configured to transmit and / or receive information.

[0025] Network 104 facilitates communication between various components of system 100. For example, network 104 may communicate Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, or other suitable information between network addresses. Network 104 may include one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems in one or more locations. Network 104 may also operate according to any suitable communication protocol or protocols.

[0026] Server 106 is coupled to network 104 and is coupled to or otherwise in communication with database 108. Server 106 supports retrieval of information from database 108 and processing of that information. Of course, database 108 may also be used within server 106 to store information, in which case server 106 may store the information itself.

[0027] Among other things, server 106 processes information used in assessing the degree of anomaly and potential threats of a UAS using machine learning. Server 106 includes any suitable structure configured to assess the degree of anomaly and potential threats of a UAS using machine learning. In some embodiments, server 106 includes one or more processors, one or more memories, and one or more communication interfaces. However, it should be noted that server 106 may be implemented in any suitable manner to perform the described functions. It should also be noted that, although described herein as a server, the device actually implementing server 106 may represent one or more desktop computers, laptop computers, server computers, or other computing or data processing devices or systems.

[0028] The database 108 stores various information used, generated, or collected by the server 106 and the user devices 102a-102d. For example, the database 108 may store detector readings, UAS vehicle information, temporal data, geospatial activity data, etc.

[0029] There are multiple possible ways to implement system 100 to provide the described functionality for assessing the degree of anomaly and potential threats of a UAS using machine learning. For example, in some embodiments, server 106 and database 108 are owned, operated, or managed by a common entity. In other embodiments, server 106 and database 108 are owned, operated, or managed by different entities. However, it should be noted that the present disclosure is not limited to any particular organizational implementation.

[0030] Although FIG. 1 illustrates an example of a system 100 for assessing the degree of anomaly and potential threat of a UAS using machine learning, various modifications may be made to FIG. 1. For example, the system 100 may include any number of user devices 102a-102d, a network 104, a server 106, and a database 108. Also, while FIG. 1 illustrates one database 108 coupled to the network 104, any number of databases 108 may be present in any location or locations accessible by the server 106, and each database 108 may be directly or indirectly coupled to the server 106. Additionally, while FIG. 1 illustrates one example operating environment in which machine learning may be used to assess the degree of anomaly and potential threat of a UAS, this functionality may be used in any other suitable system.

[0031] FIG. 2 illustrates an example device 200 for assessing the degree of anomaly and potential threats of a UAS using machine learning, according to the present disclosure. One or more instances of the device 200 may be used to at least partially implement the functionality of the server 106 of FIG. 1, for example. However, the functionality of the server 106 may be implemented in any other suitable manner. Also, the same or similar arrangement of components may be used to at least partially implement the functionality of one or more of the user devices 102a-102d of FIG. 1. However, the functionality of each user device 102a-102d may be implemented in any other suitable manner.

[0032] 2, device 200 represents a computing device or system including at least one processing device 202, at least one storage device 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing device 202 may execute instructions loadable into memory 210. Processing device 202 may include any suitable number and type of processors or other devices in any suitable arrangement. Examples of processing device 202 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0033] Memory 210 and persistent storage 212 are examples of storage device 204, which represents any structure capable of storing and facilitating retrieval of information (such as, temporarily or permanently, data, program code, and / or other suitable information). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 212 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0034] The communications unit 206 supports communication with other systems or devices. For example, the communications unit 206 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network, such as the network 104. The communications unit 206 may support communication over any suitable physical or wireless communication link.

[0035] The I / O unit 208 allows for the input and output of data. For example, the I / O unit 208 may provide a connection for user input through a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 208 may also provide a connection for any "add-on" sensing devices 214, such as sensors or cameras, that may be added to the device 200 as an accessory. The I / O unit 208 may also send output to a display, printer, or other suitable output device. However, it should be noted that if the device 200 does not require local I / O, such as when the device 200 can be accessed remotely, the I / O unit 208 may be omitted.

[0036] In some embodiments, the instructions executed by the processing device 202 may include instructions to implement the functionality of the server 106 described above. For example, the instructions executed by the processing device 202 may include instructions for assessing the degree of anomaly and potential threat of a UAS using machine learning.

[0037] Although Figure 2 illustrates an example of a device 200 for assessing the degree of anomalies and potential threats of a UAS using machine learning, various modifications may be made to Figure 2. For example, computing devices and systems come in a wide variety of configurations, and Figure 2 does not limit the disclosure to any particular computing device or system.

[0038] 3 illustrates an example process 300 for assessing the degree of anomaly and potential threats of a UAS using machine learning in accordance with the present disclosure. For ease of explanation, process 300 is described as being performed using system 100 of FIG 1, which may be implemented using one or more devices 200 of FIG 2. However, process 300 may involve the use of any suitable device in any suitable system.

[0039] As shown in Figure 3, process 300 may be performed by a server 302, which is owned or controlled by an entity 304. Server 302 represents a computing device or system capable of processing information to assess the degree of anomaly and potential threats of a UAS, such as UAS 306. In some embodiments, server 302 represents (or is represented by) server 106 of Figure 1 or device 200 of Figure 2 described above. In some embodiments, server 302 may be a cloud-based server or group of servers or other computing device.

[0040] UAS 306 represents a small UAS that typically operates at low altitude and may operate within one or more areas with limited line of sight, such as urban or suburban areas. In process 300, UAS 306 is in flight and proceeds along a flight path. Because information about UAS 306 may not be readily or immediately known to entity 304, process 300 enables entity 304 to gather information about UAS 306 at server 302. Although FIG. 3 illustrates only one UAS 306, it will be understood that process 300 may be used with multiple UAS 306.

[0041] After receiving information regarding the presence of UASs 306 within the area of ​​interest, the server 302 evaluates the degree of anomaly and potential threat of each UAS 306 (such as whether the UAS 306 is expected, anomalous, harmless, a threat, etc.). For example, the server 302 may receive information from a data source 308 that a UAS 306 has been detected within the area of ​​interest. The detection information may be obtained from one or more of a number of different data sources 308. The data sources 308 represent disparate information sources that may be provided to the server 302 for use in evaluating the threat of the UAS 306. The data sources 308 represent any suitable structure, device, or system that may provide data or information to the server 302 via any suitable communication technology, such as over a network. Some example data sources 308 may include, but are not limited to, sensors, commercial (third party) information sources, government or military information sources, public crowds of individuals with mobile devices (public crowds), etc.

[0042] To assess the degree of anomalies and potential threats of each UAS 306, the server 302 obtains a number of data items 309 about the UAS 306 from the data source 308. The data items 309 can be categorized into a number of disparate data types 310. FIG. 3 shows the following examples of data types 310 representing the data items 309 that can be used by the server 302 in the process 300. Of course, additional or alternative data types are possible and are within the scope of this disclosure. Type T1 represents detector readings, which can include, for example, readings from a detector or sensor such as a radar, camera, etc. Type T2 represents vehicle information of the UAS 306, which can include, for example, the make and model of the UAS 306, the registered owner of the UAS 306, performance characteristics exhibited by a UAS of that type, etc. Type T3 represents temporal activity data of the UAS 306, which can include, for example, the time the UAS 306 is in flight, the day or date of flight, how long the UAS 306 has been flying, etc. Type T4 represents vehicle speed data of the UAS 306, which may include, for example, speed and direction information of the UAS 306 during an entire time period or at one or more specific points in time.

[0043] Type T5 represents geospatial activity data associated with the UAS 306 or surrounding area, which may include the type of geospatial context information available in online mapping applications, such as traffic information, weather information, proximity and direction to potential targets (such as sports stadiums or military bases). This type of information is useful because it allows the server 302 to assess threats to many targets within the area of ​​each UAS 306. This is in contrast to existing systems that typically assess risk to only one or a few targets based on sensors placed on or around those targets.

[0044] Type T6 represents the time of occurrence of other events not specifically related to the UAS 306 or its movements, but which may have an impact on the threat assessment of the UAS 306. This may include, for example, timing information for a sporting event occurring at a nearby stadium at a particular time, the delivery of critical supplies to a nearby location, activity scheduled at a nuclear facility, etc.

[0045] Type T7 represents business entity data such as business records related to the entity associated with UAS 306, such as corporate data, criminal records of employees or owners, etc. This type of data can associate a particular actor with UAS 306 based on flight plan registration in an unmanned traffic management (UTM) system. Based on that information, server 302 can determine, for example, that UAS 306 is a drone operated by a package carrier and is making a delivery.

[0046] Type T8 represents business license information. Data of this type informs server 302 regarding any specific authorizations of the operator of the UAS, such as whether the entity has a commercial license to deliver products, take aerial photography, gather news, etc. Type T9 represents information from a government-issued national watch list. Data of this type indicates whether the UAS 306 (or that type of UAS) or an actor associated with the UAS 306 is on a watch list.

[0047] Some data items 309 may be categorized into multiple data types 310. For example, data items 309 may include UAS track data in the form of Electro-Optical / Infrared (EO / IR) images and video, radar data, radio frequency (RF) signals, or sequences of GPS coordinates of the UAS 306. These data items 309 may be categorized, for example, as both detector readings (type T1) and vehicle speed data (type T4). In some embodiments, track information of the UAS 306 may be obtained via crowdsourcing techniques. An example of this type of technique is described in applicant's co-pending patent application (Attorney Docket No. 18-11470-US-NP (rayn01-81470)), which is incorporated herein by reference in its entirety. Crowdsourced tracking information may fill gaps in other track data that may be limited due to line of sight or other issues. Other data items 309 may include contextual information from government and / or commercial data sources, and these data items 309 may be classified as one or more of data types T1-T9 or as another data type 310.

[0048] After obtaining data items 309 having data types 310, the server 302 can extract one or more features 312 from the data items 309 of each data type 310. Typically, the features 312 are normalized scores, vectors, or other values ​​that represent how anomalous the values ​​of the data items 309 are. In some embodiments, the server 302 can extract one feature 312 (specifically identified as V1-V9 in FIG. 3 ) for each data item 309 of data types T1-T9. That is, the server 302 can extract feature V1 for data items 309 of data type T1, feature V2 for data items 309 of data type T2, etc. The specific mechanism for extracting features V1-V9 from data types T1-T9 is data dependent. Because some data patterns are not intuitive or obvious, the determination of features 312 depends on subject matter expertise across multiple data sets and multiple domains.

[0049] Using the features 312, the server 302 executes machine learning algorithms 314 to assess the degree of anomaly and potential threat of each UAS 306. The threat assessment of the UAS 306 is performed using the machine learning algorithms 314 based on the analysis and understanding of contextual Activity Based Intelligence (ABI). Using the machine learning algorithms 314, the server 302 fuses disparate data items 309 including track data and contextual information for the UAS 306 and examines relationships between the data to assess the threat of the UAS 306. The extraction of features 312 from the disparate data items 309 helps to convert both the physical data and the non-physical contextual ABI into a format suitable for use by the machine learning algorithms 314. In some embodiments, the machine learning algorithms 314 can generate a response using predictive analytics.

[0050] The following examples illustrate relationships between disparate data items 309 of different data types 310 from different sources 308. The server 302 may obtain tracks of the UAS 306 from one or more of the data sources 308. In some embodiments, the server 302 may combine track data from the different sources 308 to generate a probable track of the UAS 306. The server 302 may extract one or more track information features 312 from the track information. However, in the absence of other information, the track information by itself may not be enough information to assess the threat of the UAS 306. Thus, the server 302 may execute a machine learning algorithm 314 by combining the track information features 312 with other features 312 related to the UAS 306 to establish a pattern of life. A pattern of life includes behaviors and movements associated with a particular entity over a given period of time. In this embodiment, server 302 may obtain and examine historical sets of life patterns of different UAS for different missions (crop inspection, power line inspection, bombing runs, etc.) or other actors or objects (e.g., law enforcement / medical / news helicopter operations, birds, etc.) and compare them to the track information of UAS 306.

[0051] As a particular example, if the server 302 determines that the UAS 306 moves in a zigzag pattern, this may be consistent with a crop inspection mission. If the server 302 determines that the UAS 306 moves in a figure-eight pattern, this may be consistent with a surveillance mission. Using machine learning algorithms 314, the server 302 can detect one or more anomalies in the track data by analyzing any differences between the current flight path of the UAS 306 and the historical patterns of track data for that type of UAS 306 or multiple UAS. Anomaly detection can also include determining in real time whether the flight path violates any FAA airspace restrictions, such as Temporary Flight Restrictions (TFRs) or Notices to Airmen (NOTAMs), or whether the UAS 306 is operating outside of an FAA-designated UAS safe flight area.

[0052] The life pattern may represent one feature 312 that the server 302 uses in machine learning algorithms 314 to perform a threat assessment, such as determining a threat level. Another feature 312 may be whether the UAS 306 has a transponder and is broadcasting information, such as its identifier (ID). This feature 312 may reference registered flight plans to see if there is a correspondence. Other features 312 may include geospatial and temporal information, such as the time the UAS 306 is flying, events occurring near that time, etc.

[0053] If the server 302 detects a track anomaly, the server 302 can use the contextual data in conjunction with machine learning algorithms 314 to further inform the threat assessment. For example, the server 302 can cross-reference the owner information of the UAS 306 obtained from a UAS registration source with one or more databases maintained by state or federal agencies or from commercial sources. The server 302 can analyze the contextual data from such databases to identify any records of concern or anomalous information within the records. The server 302 can combine the results of such contextual ABIs with the physical track data of the UAS 306 as input features 312 to the machine learning algorithms 314 for threat assessment of the UAS 306.

[0054] In some embodiments, using machine learning algorithms 314, the server 302 can generate a profile of the UAS 306. The profile of the UAS 306 can be cross-referenced with a profile map to enable a determination of the threat of the UAS 306. FIG. 4 illustrates an example profile map 400 for assessing the degree of UAS anomaly and potential threat, according to the present disclosure. As shown in FIG. 4, the map 400 is a two-dimensional profile map with one feature V1 mapped on the x-axis and another feature V2 mapped on the y-axis. For ease of illustration, the map 400 shows only two dimensions. In an actual implementation, each of the features V1-V9 may be mapped to a different axis / dimension. Over time, a curve 402 may be determined using the machine learning algorithm 314. Data points on one side of the curve 402 may represent UAS activity that is anomalous and potentially threatening, and data points on the other side of the curve 402 may represent expected or non-anomalous benign UAS activity.

[0055] In some embodiments, the implementation of algorithms 314 is based on Privacy by Design principles and best practices. This can be reflected in different algorithm layers that operate on different types of data (e.g., public vs. private, unclassified vs. classified, etc.). Different layers can be triggered by different scenarios, business rules, or different agency trigger thresholds. For example, intelligence agencies may have different access than local law enforcement agencies.

[0056] In some embodiments, the generated profile of the UAS 306 can be reduced or converted to a score 316. The score 316 indicates the severity of the threat of the UAS 306. For example, a higher score 316 may indicate a higher threat level and a lower score 316 may indicate a lower threat level, or a higher score 316 may indicate a lower threat level and a lower score 316 may indicate a higher threat level. The score 316 can be output by the server 302 in a manner that is understandable by a human user. In some embodiments, the score 316 can be communicated to a human user through one or more graphical user interfaces (GUIs) 318. The use of a score 316 that indicates the threat of the UAS 306 improves the user experience because the user can easily understand the threat level.

[0057] In some embodiments, the score 316 is presented on the GUI 318 along with analytical information that explains or justifies how the machine learning algorithm 314 arrived at its decision (such as how the score 316 was determined or the reason for the designation of a threat level). This supporting information can help generate user confidence that the threat assessment is accurate. In some embodiments, the GUI 318 is organized such that all relevant information is made readily available in one place for human adjudication, for taking further action if warranted, and securely archived for possible future legal proceedings or to preserve the chain of evidence after action review.

[0058] In some embodiments of the process 300 shown in FIG. 3, the information available for distinguishing small UAS, such as UAS 306, may be limited. Often, small UAS may be confused with birds or even ground clutter by radar and other sensing or detection mechanisms. In addition, the actual occurrence of a truly threatening UAS is (currently) a rare event. Thus, data representative of true UAS threats may be limited. To mitigate the limited data problem, synthetic high-fidelity data may be generated to train the machine learning algorithm 314 for threat assessment. For example, the training data may be generated by the server 302 or by another computing device or system associated with the process 300. In some embodiments, the training data may be generated by combining actual track data of previously observed UAS activity with threat modeling. Data provided by government agencies or commercial data sources may be used to augment the track data with contextual data.

[0059] One exemplary purpose of process 300 is to enable protection of non-traditional soft targets, such as critical infrastructure (such as military installations, major government buildings or transportation hubs) or major national events (such as major sporting events, concerts, or holidays or celebrations) from attacks by small hostile UAS. Protection of such targets within dense urban areas can be challenging, especially when the targets are spread across a large geographic area, as it may be difficult or impractical to establish a safe and sufficiently large protection radius.

[0060] To address this issue, the process 300 can use location-based geospatial risk analysis and contextual information within the protected area in which the UAS 306 is operating as part of the threat assessment. In this approach, the process 300 (such as the server 302) can classify targets within a geographic area into a fixed number of groups. Targets within each group are associated with a weighting factor that reflects the importance of the targets and the likelihood that they may be selected as potential targets by the operator of the UAS 306. In some embodiments, for example, a heat map can be used to display the concentration of targets within a geographic area.

[0061] FIG. 5 illustrates an example heat map 500 for displaying UAS threats according to the present disclosure. As shown in FIG. 5, the heat map 500 represents a geographic area. A UAS 502 (such as UAS 306) is observed flying within the area, and the flight path 504 of the UAS 502 is shown on the heat map 500. The heat map 500 also illustrates a number of targets within the area, including one or more primary targets 506 (such as military bases) and one or more soft targets 508. As described above, the targets 506-508 may be categorized into one or more groups, where the targets 506-508 within each group are associated with a weighting factor that reflects the importance of the target 506-508 and the likelihood that the target 506-508 may be selected as a potential target by the UAS 502. In some embodiments, a weighting score may be displayed and viewed for each target 506-508. When there are a large number of targets 506-508, as shown in the heat map 500, the heat map 500 can be displayed at different zoom levels to make it easier to distinguish between the individual targets 506-508 and to view the weighted scores. At the zoomed in levels, the heat map 500 provides an overall view of where the targets 506-508 are distributed, including where the targets 506-508 are most concentrated. The heat map 500 can change over time depending on many factors, such as events (such as sporting events, political rallies, etc.) that may be occurring in the area at a given time.

[0062] For each location shown in the heat map 500, a threat assessment parameter called a target concentration factor (TCF) can be calculated by aggregating the weighting factors and based on the concentration (density) and geospatial distribution characteristics of the targets 506-508 near the location. Thus, the TCF is a location-based quantification of risk and threat to the targets 506-508 of the UAS 502 based on geospatial location and area, threat type, target geospatial distribution, etc. In some embodiments, the TCF is normalized to be within a particular range of values, such as the range [0,1]. A higher TCF may indicate the presence of more important targets 506-508 near the location, while a lower TCF may indicate fewer targets 506-508 or less important targets 506-508. Of course, this is merely an example and other embodiments may include other ranges, values, and meanings for the TCF.

[0063] Each TCF may be calculated by the server 302 or by another computing device associated with the process 300. The server 302 may use each TCF as a data item 309 for extraction of features 312 in the threat assessment process 300. Each TCF may change over time as events (such as sporting events, political rallies, local or national threats by political opponents, etc.) occur over time. Each TCF may also change based on the stakeholder. For example, a TCF for a private industry stakeholder may be different from a TCF for a government agency involved in a different asset. As a specific example, a military police force protecting a military base may have a different threat threshold than a local law enforcement agency protecting a soccer field located near the military base. Thus, the TCF within an area may be different for each party.

[0064] FIG. 3 illustrates an example of a process 300 for assessing the degree of anomaly and potential threat of a UAS using machine learning, although various modifications may be made to FIG. 3. For example, the number of servers, data sources, data items, data types, or features may differ from those illustrated in FIG. 3. FIG. 4 illustrates an example of a profile map 400 for assessing the degree of anomaly and potential threat of a UAS, and FIG. 5 illustrates an example of a heat map 500 for displaying a threat of a UAS, although various modifications may be made to FIG. 4 and FIG. 5. For example, the profile map 400 and heat map 500 illustrated herein are merely examples, and any other suitable profile map and heat map may be generated for a specified area or environment.

[0065] Figure 6 illustrates an example geospatial representation of a counter-UAS solution 600 capable of performing UAS detection, tracking, and threat assessment in accordance with the present disclosure. For ease of explanation, the counter-UAS solution 600 will be described as being used with the process 300 of Figure 3 and may include the system 100 of Figure 1. However, the counter-UAS solution 600 may involve the use of any suitable devices, processes, and systems.

[0066] As shown in Figure 6, counter-UAS solution 600 includes multiple groups of actors 601-604. Actors 601 include traditional air defense actors (such as military forces) for protection from planes, missiles, etc. Actors 602 include newer air defense actors, such as military police. Actors 603 include local non-military agencies (such as civilian police forces) such as local law enforcement. Actors 604 include private actors such as businesses, industries, utilities, concerned individuals (e.g., celebrities or individuals who are very privacy conscious), etc.

[0067] Each group of actors 601-604 has one or more counter actions that the actors 601-604 can take in the event of a UAS sighting. For example, both actors 601 and 602 have the authority for interdiction when a UAS is observed, such as destroying the UAS or disrupting the operation of the UAS. The type of interdiction performed can depend on how close the UAS is to a particular protected asset 606. For example, the type of interdiction can be more significant depending on whether the UAS enters a surveillance zone 607, an engagement zone 608, or a restricted zone 609 associated with the asset 606. Each zone 607-609 can be defined by a radius from the asset 606. In some embodiments, actors 603 and 604 do not have the authority to perform any interdiction of the UAS. However, actors 603 and 604 can issue directives (such as directives to evacuate) to people and companies.

[0068] Counter-UAS solution 600 is also defined by altitude zones 611-613, where altitude zone 611 is low altitude (e.g., below 400 feet), altitude zone 612 is medium altitude (e.g., 400 feet to 2,000 feet), and altitude zone 613 is high altitude (e.g., above 2,000 feet). Of course, these values ​​are merely examples and other embodiments may include a different number of altitude zones with different thresholds. In some cases, the threat assessment process disclosed herein may be performed in low altitude zone 611 and surveillance zone 607, as represented by the dashed line indicated at 615. However, other use cases of the threat assessment process may also be possible.

[0069] Although Figure 6 illustrates an example of a counter-UAS solution 600 capable of performing UAS detection, tracking, and threat assessment, various modifications may be made to Figure 6. For example, solution 600 may include any suitable number of actors, surveillance zones, altitude zones, and counter actions.

[0070] Figure 7 illustrates an example method 700 for assessing the degree of anomaly and potential threats of a UAS using machine learning in accordance with the present disclosure. For ease of explanation, the method 700 of Figure 7 may be described as being performed using the server 302 of Figure 3, which may be implemented using one or more devices 200 of Figure 2. However, the method 700 may involve the use of any suitable device in any suitable system.

[0071] 7, in step 702, an event is detected. The detection event is associated with the detection of a UAS flying within an area. The detection event can be notified to a server. This can include, for example, notifying the server 302 of the detection event from a sensor network, one of the data sources 308, etc.

[0072] In step 704, a plurality of data items are obtained from a plurality of disparate data sources. The plurality of data items are associated with a UAS flying within the area. This may include, for example, the server 302 obtaining a plurality of data items 309 associated with the UAS 306 from a plurality of data sources 308. In step 706, a TCF for the UAS is calculated. The TCF indicates the concentration and importance of one or more targets within the area. This may include, for example, the server 302 calculating a TCF for one or more targets 506-508. The TCF can be used as one or more data items 309 for the machine learning algorithm 314.

[0073] At step 708, a number of features are extracted from the data items. Each feature indicates how anomalous the data item is. In some embodiments, the number of features is input to a machine learning algorithm 314. This may include, for example, the server 302 extracting a number of features 312 from the data items 309 for input to the machine learning algorithm 314. At step 710, the number of data items is processed with the machine learning algorithm to assess a UAS threat level for one or more targets in the area. This may include, for example, the server 302 using the machine learning algorithm 314 to assess a threat level of the UAS 306.

[0074] At step 712, the UAS threat level is output for review by the user. In some embodiments, the threat level is represented as a score indicating the severity of the threat posed by the UAS to one or more targets in the area. In some embodiments, the score is also presented on a graphical user interface along with analysis information indicating how the machine learning algorithm assessed the threat level. This may include, for example, the server 302 presenting the score 316 on the GUI 318.

[0075] Although Figure 7 illustrates one example of a method 700 for assessing the degree of anomaly and potential threat of a UAS using machine learning, various modifications may be made to Figure 7. For example, while illustrated as a series of steps, various steps in Figure 7 may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0076] Among other things, the above-described embodiments provide for the extension of air defense to non-traditional actors, including civilian, commercial, and law enforcement actors. In some cases, threat assessment information determined by the disclosed embodiments can fill gaps in the military kill chain, Intelligence Community (IC) intelligence cycle, and Law Enforcement Suspicious Activity Report investigation cycle. In some cases, threat assessment information can also be provided to traditional or institutional kill chain actors or other law enforcement actors with additional active mitigation authority.

[0077] In some embodiments, various functions described in this patent document are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium accessible by a computer, such as a read only memory (ROM), a random access memory (RAM), a hard disk drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable media include media that can store data permanently and media that can store data and later be overwritten, such as rewritable optical disks or erasable storage devices.

[0078] It may be advantageous to provide definitions of certain words and phrases used throughout this patent document. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof, adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean inclusion rather than limitation. The term "or" is inclusive and / or. "Associated with" and its derivatives may mean including, contained within, interconnected with, containing, contained within, connected to or with, coupled to or with, communicable with, cooperate with, interleave with, juxtaposed to, adjacent to, associated with or with, having the property of, relating to or with, and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A, B, and C.

[0079] The description herein should not be construed as implying that any particular element, step, or function is an essential or critical element required for inclusion in the scope of the claims. The scope of the claimed subject matter is defined solely by the allowed claims. Moreover, none of the claims invoke 35 U.S.C. 112(f) with respect to any of the appended claims or claim elements unless the precise words "means for" or "step for" followed by a specific phrase identifying the function are expressly used in a particular claim. The use of terms such as "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller" in the claims is understood and intended to refer to structures known to those skilled in the art, as further modified or enhanced by features of the claims themselves, and is not intended to invoke 35 U.S.C. 112(f).

[0080] While this disclosure has described certain embodiments and generally associated methods, modifications and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. A method for detecting and responding to threats comprising: generating training data by combining actual track data of one or more past unmanned aerial system (UAS) activities with threat modeling; training a machine learning algorithm using the training data; obtaining a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to UASs flying within an area; processing the plurality of data items with the trained machine learning algorithm to determine a threat level of the UAS to one or more targets within the area; outputting the threat level of the UAS for review by a user; A method comprising:

2. the threat level is expressed as a score indicating the severity of a threat posed by the UAS to the one or more targets in the area; The method of claim 1.

3. The score is presented on a graphical user interface along with analytical information showing how the machine learning algorithm determined the threat level. The method of claim 2.

4. extracting a plurality of features from the data items, each feature indicating how anomalous the data items are; The plurality of features are input to the machine learning algorithm. The method of claim 1.

5. calculating a target concentration factor (TCF) indicative of the concentration and importance of the one or more targets within the area; The TCF includes one of the plurality of data items. The method of claim 1.

6. the plurality of data items are classified into a plurality of disparate data types; The method of claim 1.

7. the machine learning algorithm is further trained using contextual data from one or more government or commercial data sources; The method of claim 1.

8. At least one of the plurality of heterogeneous data sources includes a public cloud source. The method of claim 1.

9. at least one memory configured to store instructions; at least one processor coupled to the at least one memory, which, when executing the instructions, generating training data by combining actual track data of one or more historical unmanned aerial system (UAS) activities with threat modeling; training a machine learning algorithm using the training data; obtaining a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to UASs flying within the area; processing the plurality of data items with the trained machine learning algorithm to determine a threat level of the UAS to one or more targets within the area; outputting the threat level of the UAS for review by a user; at least one processor configured to Including, the device.

10. the threat level is expressed as a score indicating the severity of a threat posed by the UAS to the one or more targets in the area; The device of claim 9.

11. a display configured to present the score on a graphical user interface along with analytical information indicating how the machine learning algorithm determined the threat level; The device of claim 10 further comprising:

12. the at least one processor is further configured to extract a plurality of features from the data items, each feature indicative of how anomalous the data items are; The plurality of features are input to the machine learning algorithm. The device of claim 9.

13. The at least one processor is further configured to calculate a Target Concentration Factor (TCF), the TCF being indicative of a concentration and importance of the one or more targets within the area; The TCF includes one of the plurality of data items. The device of claim 9.

14. the plurality of data items are classified into a plurality of disparate data types; The device of claim 9.

15. the machine learning algorithm is further trained using contextual data from one or more government or commercial data sources; The device of claim 9.

16. At least one of the plurality of heterogeneous data sources includes a public cloud source. The device of claim 9.

17. When executed, the method includes the steps of: generating training data by combining actual track data of one or more historical unmanned aerial system (UAS) activities with threat modeling; training a machine learning algorithm using the training data; acquiring a plurality of data items from a plurality of disparate data sources, the plurality of data items relating to UASs flying within the area; processing the plurality of data items with the trained machine learning algorithm to determine a threat level of the UAS to one or more targets within the area; outputting the threat level of the UAS for review by a user; A non-transitory computer-readable medium containing instructions.

18. the threat level is expressed as a score indicating the severity of a threat posed by the UAS to the one or more targets in the area; 20. The non-transitory computer readable medium of claim 17.

19. The score is presented on a graphical user interface along with analytical information showing how the machine learning algorithm determined the threat level.

20. The non-transitory computer readable medium of claim 18.

20. The instructions, when executed, further cause the at least one processor to extract a plurality of features from the data items, each feature indicative of how anomalous the data items are; The plurality of features are input to the machine learning algorithm.

20. The non-transitory computer readable medium of claim 17.

Citation Information

Patent Citations

  • An urban safety overall risk analysis method and system

    CN109934465A

  • Rearview monitoring device of vehicle and method thereof

    JP1994227318A

  • Image processing system, image processing device and image processing method

    JP2016021649A

  • Model creation device, terminal device, hazardous place discovery system, and program

    JP2018081549A

  • Flying body and information processing system

    JP2019086902A