Crowdsourced Detection and Tracking of Unmanned Aerial Systems

A crowdsourced system using mobile devices for UAS detection and tracking addresses the challenges of conventional methods by leveraging user networks for efficient and cost-effective threat assessment and mitigation.

JP7582741B2Active Publication Date: 2024-11-13RAYTHEON CO
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022575377
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-08
Filing Date
2021-04-23
Publication Date
2024-11-13
Estimated Expiration
2041-04-23

AI Technical Summary

Technical Problem

Detection, differentiation, and threat assessment of unmanned aerial systems (UAS) are difficult, especially in high clutter environments with limited lines of sight, and conventional detection methods are costly and manually intensive, limiting their effectiveness against non-compliant or hostile actors.

Method used

A crowdsourced detection and tracking system that utilizes mobile devices of multiple users to collect information about UAS, processes it through a centralized entity, and provides evaluations and notifications to users and law enforcement, leveraging machine learning for threat assessment and mitigation.

Benefits of technology

Enables efficient, scalable, and cost-effective detection and tracking of UAS by harnessing a network of users, filling gaps in traditional sensor coverage and improving the efficiency of threat assessment and mitigation processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582741000001
    Figure 0007582741000001
  • Figure 0007582741000002
    Figure 0007582741000002
  • Figure 0007582741000003
    Figure 0007582741000003
Patent Text Reader

Abstract

The method includes receiving (502) a first notification of an unmanned aerial system (UAS) (306) flying within an area. The method also includes sending (502) a second notification to a plurality of mobile devices (102a-102d, 200, 304) of a plurality of users (302) located within the area, the second notification requesting the plurality of users to locate the UAS. The method further includes receiving (508) information about the UAS from one or more of the mobile devices of one or more of the users, the information being acquired by the one or more mobile devices. The method also includes determining (510) a reputation for the UAS based on the information about the UAS received from the one or more mobile devices. Additionally, the method includes sending (512) a third notification to at least some of the mobile devices, the third notification including information about the reputation of the UAS.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure is directed generally to unmanned aerial systems and, more specifically, to crowd-sourced detection and tracking of unmanned aerial systems or other manned or unmanned vehicles. [Background technology]

[0002] Inexpensive small unmanned aerial systems (UASs), such as drones, are proliferating rapidly. Numerous uses for UAS have been proposed and implemented, such as infrastructure and utility inspection, surveillance, courier delivery, urban mobility, etc. While most use cases involving UAS are beneficial, some UAS can be used for malicious purposes, and these UAS can pose a significant threat to a wide range of targets. Unfortunately, detection, discrimination, and threat assessment of UAS can be difficult, especially since UAS often fly at low altitudes. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure relates to crowdsourced detection and tracking of unmanned aerial systems. [Means for solving the problem]

[0004] In a first implementation, the method includes receiving a first notification of an unmanned aerial system (UAS) flying within an area. The method also includes sending a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the plurality of users to locate the UAS. The method further includes receiving information about the UAS from one or more of the mobile devices of one or more of the users, the information being acquired by the one or more mobile devices. The method also includes determining an evaluation of the UAS based on the information about the UAS received from the one or more mobile devices. Additionally, the method includes sending a third notification to at least some of the mobile devices, the third notification including information about the evaluation of the UAS.

[0005] In a second implementation, the system includes at least one memory configured to store instructions and at least one processor coupled to the at least one memory. The at least one processor, upon executing the instructions, is configured to receive a first notification of a UAS flying within the area. The at least one processor, upon executing the instructions, is also configured to send a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the plurality of users to locate the UAS. The at least one processor, upon executing the instructions, is further configured to receive information regarding the UAS from one or more of the mobile devices of the one or more users of the users, the information being obtained by the one or more mobile devices. The at least one processor, upon executing the instructions, is also configured to determine an assessment of the UAS based on the information regarding the UAS received from the one or more mobile devices. Additionally, the at least one processor, upon executing the instructions, is configured to send a third notification to at least some of the mobile devices, the third notification including information regarding the assessment of the UAS.

[0006] In a third implementation, the non-transitory computer-readable medium includes instructions that, when executed, cause the at least one processor to receive a first notification of a UAS flying within the area. The non-transitory computer-readable medium also includes instructions that, when executed, cause the at least one processor to send a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the plurality of users to search for the UAS. The non-transitory computer-readable medium further includes instructions that, when executed, cause the at least one processor to receive information about the UAS from one or more of the mobile devices of one or more of the users, the information being obtained by the one or more mobile devices. The non-transitory computer-readable medium also includes instructions that, when executed, cause the at least one processor to determine an assessment of the UAS based on the information about the UAS received from the one or more mobile devices. Additionally, the non-transitory computer-readable medium includes instructions that, when executed, cause the at least one processor to send a third notification to at least some of the mobile devices, the third notification including information about the assessment of the UAS.

[0007] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] FIG. 1 illustrates an example system for crowdsourced detection and tracking of unmanned aerial systems (UAS) in accordance with the present disclosure.

[0010] [Diagram 2] FIG. 1 illustrates an exemplary device for crowdsourced detection and tracking of UAS in accordance with the present disclosure.

[0011] [Diagram 3]FIG. 1 illustrates an example environment in which crowdsourced detection and tracking of UAS according to the present disclosure can be performed.

[0012] [Figure 4] FIG. 1 illustrates an example geospatial representation of a counter-UAS solution in which crowdsourced detection and tracking of UAS in accordance with the present disclosure can be performed.

[0013] [Diagram 5] FIG. 1 illustrates an exemplary method for crowdsourced detection and tracking of UAS in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] 1 through 5 described below, and the various implementations used to illustrate the principles of the present invention in this patent document, are for illustrative purposes only and should not be construed as limiting the scope of the invention in any way. Those skilled in the art will understand that the principles of the present invention may be implemented in any type of suitably arranged device or system.

[0015] As noted above, unmanned aerial systems or "UAS" (such as drones or other unmanned aerial vehicles) are proliferating rapidly. Numerous beneficial uses for UAS have been proposed and implemented. However, some UAS may be used for malicious purposes, and these UAS may pose a significant threat to a wide range of targets. Unfortunately, detection, differentiation, and threat assessment of UAS can be difficult, especially since UAS often fly at low altitudes. Furthermore, UAS often fly in high clutter environments with limited lines of sight, and small UAS are easily transported to different locations, which can launch them as pop-up threats with short attack timelines. In addition, conventional equipment for detection, differentiation, assessment, and mitigation is expensive and manually intensive, which limits their use.

[0016] Some existing approaches for UAS detection rely on cooperative communication, such as when the UAS or its operator provides information about the UAS (such as flight plan, reserved airspace, and vehicle location) to stakeholders who observe the UAS. These approaches are primarily focused on enabling legal, commercial, and recreational actors. However, these approaches are not sufficient or effective in dealing with non-compliant, non-cooperative, and / or hostile actors. In such cases, it is often necessary to detect and evaluate the UAS without information voluntarily provided by the UAS or its operator. It may be possible to protect populated areas or areas with critical infrastructure with detection sensors such as radar or advanced imaging systems. However, these solutions are generally considered too costly in terms of acquisition, staffing, and maintenance for large-scale deployment in urban or suburban areas, especially considering the limited line of sight in low altitude and high clutter environments such as urban or suburban "canyons" due to man-made structures.

[0017] The present disclosure provides systems and methods for crowdsourced detection and tracking of UAS. As described in more detail below, the disclosed systems and methods provide for a centralized UAS tracking entity to receive information about one or more UAS from mobile devices of a plurality of users, such as users in a crowd. Based on the received information, the UAS tracking entity can perform an assessment of each UAS. The UAS tracking entity can also send a notice of the assessment to at least some of the users, and the UAS tracking entity can further send the assessment information to other entities, such as law enforcement.

[0018] Although the present disclosure is described with respect to a UAS, it will be understood that the principles disclosed herein are applicable to other types of vehicles in areas of limited sensor coverage. For example, the disclosed implementations can also be used with manned aerial vehicles (e.g., low-flying aircraft) and vehicles in other domains (e.g., maritime or land traffic).

[0019] Figure 1 illustrates an example system 100 for crowdsourced detection and tracking of UAS according to the present disclosure. As shown in Figure 1, the system 100 includes a plurality of user devices 102a-102d, at least one network 104, at least one server 106, and at least one database 108. However, it should be noted that other combinations and arrangements of components may be used herein.

[0020] In this example, each user device 102a-102d is coupled to or communicates through network 104. Communication between each user device 102a-102d and network 104 may occur in any suitable manner, such as via a wired or wireless connection. Each user device 102a-102d represents any suitable device or system used by at least one user to provide information to or receive information from server 106 or database 108. Example types of information may include UAS photographs, UAS position data, communications or control signals associated with a UAS, etc.

[0021] Any suitable number and type of user devices 102a-102d may be used in system 100. In this particular example, user device 102a represents a desktop computer, user device 102b represents a laptop computer, user device 102c represents a smartphone, and user device 102d represents a tablet computer. However, any other or additional types of user devices may be used in system 100. Each user device 102a-102d includes any suitable structure configured to transmit and / or receive information.

[0022] Network 104 facilitates communication between various components of system 100. For example, network 104 may communicate Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, or other suitable information between network addresses. Network 104 may include one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems in one or more locations. Network 104 may also operate according to any suitable communication protocol or protocols.

[0023] Server 106 is coupled to network 104 and is coupled to or otherwise in communication with database 108. Server 106 supports retrieval of information from database 108 and processing of that information. Of course, database 108 may be used within server 106 to store information, in which case server 106 may store the information itself. Among other things, server 106 processes information used for crowdsourced detection and tracking of UAS. Examples of server 106 processing operations are provided below.

[0024] The server 106 includes any suitable structure configured to process information for crowdsourced detection and tracking of UAS. In some implementations, the server 106 includes one or more processors, one or more memories, and one or more communication interfaces. However, it should be noted that the server 106 may be implemented in any suitable manner to perform the described functions. It should also be noted that, although described herein as a server, the device actually implementing the server 106 may represent one or more desktop computers, laptop computers, server computers, or other computing or data processing devices or systems. In some implementations, the server 106 may include or be part of a cloud-based processing environment.

[0025] The database 108 stores various information used, generated, or collected by the server 106 and the user devices 102a-102d. For example, the database 108 may store UAS photographs, UAS location and flight path (e.g., wake or trajectory) data, communication or control signals associated with the UAS, etc. In some cases, at least a portion of the information to be stored by the database 108 may be received directly from the user devices 102a-102d or indirectly, such as via the server 106. In some cases, the information may be received from a sensor, such as a radar or advanced imaging system.

[0026] There are multiple possible ways to implement system 100 to provide the described functionality for crowdsourced detection and tracking of UAS. For example, in some implementations, server 106 and database 108 are owned, operated, or managed by a common entity. In other implementations, server 106 and database 108 are owned, operated, or managed by different entities. However, it should be noted that this disclosure is not limited to any particular organizational implementation.

[0027] Although FIG. 1 illustrates an example system 100 for crowdsourced detection and tracking of UAS, various modifications may be made to FIG. 1. For example, system 100 may include any number of user devices 102a-102d, a network 104, a server 106, and a database 108. Also, while FIG. 1 illustrates one database 108 coupled to network 104, any number of databases 108 may reside in any location or locations accessible by server 106, and each database 108 may be directly or indirectly coupled to server 106. Additionally, while FIG. 1 illustrates one example operating environment in which crowdsourced detection and tracking of UAS may be used, this functionality may be used in any other suitable system.

[0028] FIG. 2 illustrates an example device 200 for crowdsourced detection and tracking of UAS according to the present disclosure. One or more instances of the device 200 may be used to at least partially implement the functionality of the server 106 of FIG. 1, for example. However, the functionality of the server 106 may be implemented in any other suitable manner. Also, the same or similar arrangement of components may be used to at least partially implement the functionality of one or more of the user devices 102a-102d of FIG. 1. However, the functionality of each user device 102a-102d may be implemented in any other suitable manner.

[0029] 2, device 200 represents a computing device or system including at least one processing device 202, at least one storage device 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing device 202 may execute instructions loadable into memory 210. Processing device 202 may include any suitable number and type of processors or other devices in any suitable arrangement. Examples of processing device 202 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0030] Memory 210 and persistent storage 212 are examples of storage device 204, which represents any structure capable of storing and facilitating retrieval of information (such as, temporarily or permanently, data, program code, and / or other suitable information). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 212 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0031] The communications unit 206 supports communication with other systems or devices. For example, the communications unit 206 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network, such as the network 104. The communications unit 206 may support communication over any suitable physical or wireless communication link.

[0032] The I / O unit 208 allows for the input and output of data. For example, the I / O unit 208 may provide a connection for user input through a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 208 may also provide a connection for any "add-on" sensing devices 214, such as sensors or cameras, that may be added to the device 200 as an accessory. The I / O unit 208 may also send output to a display, printer, or other suitable output device. However, it should be noted that if the device 200 does not require local I / O, such as when the device 200 can be accessed remotely, the I / O unit 208 may be omitted.

[0033] In some implementations, the instructions performed by the processing device 202 may include instructions to implement functionality of the server 106. For example, the instructions performed by the processing device 202 may include instructions for crowdsourced detection and tracking of UAS. In other implementations, the instructions performed by the processing device 202 may include instructions to implement functionality of the user devices 102a-102d. For example, the instructions performed by the processing device 202 may include instructions for providing UAS-related information to the server 106 and receiving UAS-related notifications from the server 106.

[0034] Although Figure 2 illustrates an example of a device 200 for crowdsourced detection and tracking of UAS, various modifications may be made to Figure 2. For example, computing devices / systems and mobile devices come in a wide variety of configurations, and Figure 2 does not limit the disclosure to any particular computing device or system, or to any particular mobile device.

[0035] 3 illustrates an example environment 300 in which crowdsourced detection and tracking of UAS may be performed in accordance with the present disclosure. For ease of explanation, the environment 300 is described as being used in conjunction with the system 100 of FIG. 1, which may be implemented using one or more devices 200 of FIG. 2. However, the environment 300 may involve the use of any suitable devices in any suitable system.

[0036] As illustrated in Figure 3, the environment 300 includes at least one user 302 located within an area in which a UAS 306 is flying. While Figure 3 illustrates only one user 302, it will be understood that the environment 300 typically includes more than one user 302 and can include many users 302 (e.g., tens, hundreds, or thousands) located within an area proximate the UAS 306 or its flight path. Also, while Figure 3 illustrates only one UAS 306, it will be understood that the environment 300 may include multiple UASs 306, with one or more users 302 located within an area proximate each UAS 306 or its flight path.

[0037] Each user 302 owns or has access to a mobile device 304. Each mobile device 304 represents a computing device or system capable of detecting and tracking a UAS, such as UAS 306. In some implementations, each mobile device 304 includes one or more sensors (such as a camera, a Wi-Fi sensor, a BLUETOOTH sensor, a microphone, etc.) that can sense or detect information about the UAS 306. Exemplary types of mobile devices 304 can include mobile phones, tablet computers, laptop computers, etc. In some implementations, the mobile device 304 represents (or is represented by) one of the user devices 102a-102d of FIG. 1 or device 200 of FIG. 2 described above.

[0038] Each mobile device 304 may include an application (or “app”) 305 installed on the mobile device 304. The app 305 provides a graphical user interface (GUI) for interaction with the user 302 and program instructions for UAS detection and tracking. The app 305 also facilitates communication between the mobile device 304 and a server 310 over a network 308 to share information about UAS 306. In some implementations, the app 305 is provided by an entity 312 that collects and processes UAS detection and tracking related information. In certain implementations, the app 305 is licensed to each user 302 and installed on each mobile device 304 as part of a collaborative “freemium” business model. In such a business model, each user 302 is considered a “prosumer” (a new compound word of “producer” and “consumer”) because the user 302 both generates UAS-related information for use by and receives UAS-related information from the server 310.

[0039] The app 305 enables recruitment of many users 302 to join a community and search for low altitude UAS such as UAS 306 using short-range sensing technology already available on the users' mobile devices 304. In some implementations, the app 305 may provide rewards to each user 302 that submits useful information about the UAS 306. For example, the rewards may be credits for in-app purchases, access to information about spotted UAS 306, etc.

[0040] In this example, UAS 306 represents a small UAS that typically operates at low altitude and may operate within one or more areas with limited line of sight, such as an urban or suburban area. In environment 300, UAS 306 is in flight and proceeds along a flight path. Because information about UAS 306 may not be readily or immediately known to entity 312, environment 300 enables users 302 to use their mobile devices 304 to gather information about UAS 306 and provide that information to server 310, which is owned or controlled by entity 312.

[0041] Server 310 represents a computing device or system that can process information for detection and tracking of UAS, such as UAS 306. In some implementations, server 310 represents (or is represented by) server 106 of FIG. 1 or device 200 of FIG. 2 described above. In some implementations, server 310 can be a cloud-based server or group of servers or other computing devices.

[0042] The server 310 performs tracking of the UAS 306 by receiving, correlating, and processing information about the UAS 306 from the mobile device 304 (such as images or videos, audio recordings, control signal detections, user observations or comments obtained via the GUI of the app 305, etc.). In some implementations, the information can be obtained from multiple sensing positions or angles. Also, in some implementations, the information can be GPS tagged or otherwise correlated with location information. The server 310 can process the collected information using a track estimation algorithm that can generate track details of the UAS 306. The track details can describe the previous and estimated future track of the UAS 306, such as its altitude, flight path, speed, etc. In response to the received information, the server 310 can determine the location of the UAS 306 and track its flight path using multiple data items in one or more triangulation algorithms. For example, image processing of multiple images or videos captured from different positions can be used in combination with triangulation algorithms to determine the location of the UAS 306. Tracking accuracy may depend on the number and spatial distribution of mobile devices 304, the quality of sensor inputs, and the availability of other information sources for a particular UAS contact.

[0043] Using the information received from the mobile device 304, the server 310 can also perform differentiation of the UAS 306, such as by identifying its unique make, model, and payload (if any). For example, the server 310 can use image and audio recognition algorithms, as well as human observations input via the GUI of the app 305. The server 310 can also perform differentiation of likely intent, adjudication of potential threats, and recommendation of appropriate mitigation actions for one or more actors in the spectrum. For example, an industrial operator may need to temporarily suspend its UAS operations, a law enforcement agency may need to locate the ground control station of the UAS 306 and ticket or arrest the UAS operator, or a federal agency may need to use kinetic or non-kinetic counters. Additionally, the server 310 can perform detection, location, and identification of the ground control station and operator of the UAS 306. These can be distinguished in a similar manner to the distinction made between UAS306 itself.

[0044] The server 310 can use one or more algorithms to differentiate and adjudicate potential threats, such as algorithms utilizing object-based production (OBP), activity-based intelligence (ABI), or both. Such algorithms can perform an initial assessment of the UAS 306 and its potential threat. Based on the initial assessment, the server 310 can escalate to another legitimate actor for further adjudication or mitigation, or de-escalate if the assessment indicates a low threat. In some cases, the server 310 can also send specific notifications to one or more users 302 to request more information or recommend an action, such as "seek shelter" or "your flight operations."

[0045] In some implementations, the information provided to the server 310 may be aggregated as a data set that may enable one or more artificial intelligence (AI) or other machine learning (ML) algorithms or routines. The accuracy of the AI / ML algorithms or routines may improve over time as the network of users 302 and mobile devices 304 grows, thus providing a larger reference database for training and evaluation. In particular, crowdsourcing with multiple observations from multiple mobile devices 304 may enable a way to ground truth sensor data and human-machine interface (HMI) data, as well as validation of discrimination and adjudication decisions. Additionally, in some implementations, the aggregated data set may be packaged as a product or service that may be sold, licensed, or otherwise provided to interested parties.

[0046] The detection and collection of information regarding the UAS 306 can be initiated by at least one of the users 302, such as when one or more of the users 302 observes the flight of the UAS 306. In another example, the trigger that starts the process can be that another third party system (e.g., an Unmanned Traffic Management (UTM) system, a radar system, an optical tracking system such as the Multi-Spectral Targeting System (MTS) by Raytheon Technologies, etc.) detects the presence of an unidentified object. The third party system can then request the server 310 to send a geospatial alert notification to the user 302 via the app 305 on the user's mobile device 304. In yet another example, the trigger that starts the process can be a social media post of a UAS flight in the area, which can automatically generate a geospatial alert notification to the user 302 via the app 305.

[0047] As one particular example, a user 302 may observe a UAS 306 and report the observation to a server 310 over the network 308 using the app 305. The server 310 may then send a notification via the app 305 to a group of users 302 in a nearby area (such as within a one kilometer radius of the observing user 302). The notification informs the group of users 302 of any known information about the UAS 306 and requests the users 302 to search for the UAS 306 and record any possible information about the UAS 306. For example, the user 302 may capture a photo or video of the UAS 306, record audio of sounds made by the UAS 306, generate text comments about the UAS 306, etc. In some implementations, the app 305 can monitor one or more internal sensors (e.g., GPS, IMU, compass, etc.) of the mobile device 304, one or more external “add-on” sensors (such as sensing device 214), or other (open source or licensed) geospatial surveying apps to extract information (direction, bearing, etc.) that can be used for track generation calculations. The user 302 or mobile device 304 then reports information about the UAS 306 to the server 310 via the app 305.

[0048] The server 310 receives the UAS 306 information, including any location information, and examines geospatial information of the space around the UAS 306 to determine if there are any vulnerable areas (such as congested events, important assets, military bases, etc.). The server 310 also notifies a group of users 302 in a larger area (e.g., within a 5-kilometer radius of the UAS 306) to locate the controller or operator of the UAS 306. For example, the users 302 may use their mobile devices 304 to electronically sense the network or transmission band (e.g., Wi-Fi, radio frequency (RF), cellular, etc.) over which the controller and the UAS 306 communicate. The users 302 may also simply walk around and visually locate the UAS controller or operator and report the information to the server 310 via the app 305.

[0049] Once the server 310 has obtained information about the UAS 306 or its controller / operator, the server 310 performs an assessment of the UAS 306 and its potential threats. In some implementations, machine learning techniques can be used to perform the assessment of the UAS 306. An example of this type of technique is described in the applicant's co-pending patent application (Attorney Docket No. 20-13800-US-NP(rayn01-13800)), which is incorporated by reference in its entirety. Based on this assessment, the server 310 can escalate to another authorized actor for further adjudication or mitigation, or can de-escalate if the assessment indicates a low threat. The server 310 can also alternatively transmit the assessment information of the UAS 306 to one or more of the users 302 or to one or more interested third parties. Interested third parties can include law enforcement, entities involved with the UAS 306 or business assets near its flight path, property owners, etc.

[0050] In some implementations, the freemium business model may include a subscription-based service offered by the entity 312. In a subscription-based service, a subscriber may receive a greater level of information about the UAS 306. The subscriber may include one or more of the users 302. Other subscribers may include subscribers who may not be interested in searching for UAS, but who would like to be aware of UAS in their vicinity. For example, a subscriber may be a celebrity who would like to know when a UAS is near their home. However, it should be noted that the approach described in this patent document is not limited to any particular business model or other implementation for encouraging users to participate.

[0051] In environment 300, server 310 can use information from mobile device 304 or other disparate sources of data (e.g., geospatial mapping, criminal records databases, etc.) to integrate and compress the intelligence cycle, law enforcement suspicious activity investigatory cycle, and military "kill chain" for pop-up UAS threats to improve the efficiency of the adjudication and action decision process. In traditional approaches, multiple inefficiencies exist in the process due to lack of education, clear roles, responsibilities, authorities, and mitigation solutions. This prevents timely and efficient notification, adjudication, and appropriate mitigation of potential threats by all actors in the community. Environment 300 integrates and compresses all three processes by defining clear roles, responsibilities, authorities, and mitigation solutions for different actors (including both non-traditional and traditional actors) and by reducing friction and time associated with communication and coordination between actors in the kill chain. For example, UAS information provided to the server 310 by the mobile device 304 can fill gaps in traditional kill chain sensor coverage.

[0052] The environment 300 shifts most of the acquisition, staffing, and sustainment costs of sensing infrastructure to a large potential network of users 302, each with access to their own independent mobile devices 304. This solves the problem of scaling and the cost required to deploy, staff, and maintain low altitude sensing infrastructure. As the network of users 302 grows, the coverage provided by the environment 300 scales up. This can be particularly advantageous for soft targets such as large public events.

[0053] Although Figure 3 illustrates an example of an environment 300 in which crowdsourced detection and tracking of UAS may be performed, various modifications may be made to Figure 3. For example, operating environments come in a wide variety of configurations, and Figure 3 does not limit the disclosure to any particular environment.

[0054] 4 illustrates an example geospatial representation of a counter-UAS solution 400 capable of performing crowdsourced detection and tracking of UAS in accordance with the present disclosure. For ease of explanation, the counter-UAS solution 400 is described as being used with the environment 300 of FIG. 3 and may include the system 100 of FIG. 1. However, the counter-UAS solution 400 may involve the use of any suitable devices, environments, and systems.

[0055] As shown in FIG. 4, counter-UAS solution 400 includes multiple groups of actors 401-404. Actors 401 include traditional air defense (such as military forces) for protection from planes, missiles, etc. Actors 402 include newer air defense actors, such as military police. Actors 403 include local non-military agencies, such as local law enforcement (such as civilian police forces). Actors 404 include private actors, such as business, industry, utilities, concerned individuals (e.g., celebrities or individuals who are very privacy conscious), etc.

[0056] Each group of actors 401-404 has one or more counter actions that the actors 401-404 can take in the event of a UAS sighting. For example, both actors 401 and 402 have the authority for interdiction when a UAS is observed, such as destroying the UAS or disrupting the operation of the UAS. The type of interdiction performed can depend on how close the UAS is to a particular protected asset 406. For example, the type of interdiction can be more significant depending on whether the UAS enters a surveillance zone 407, an engagement zone 408, or a restricted zone 409 associated with the asset 406. Each zone 407-409 can be defined by a radius from the asset 406. In some implementations, the actors 403 and 404 do not have the authority to perform any interdiction of the UAS. However, the actors 403 and 404 can issue commands (such as commands to evacuate) to people and companies.

[0057] Counter-UAS solution 400 is also defined by altitude zones 411-413, where altitude zone 411 is low altitude (e.g., below 400 feet), altitude zone 412 is medium altitude (e.g., 400 feet to 2,000 feet), and altitude zone 413 is high altitude (e.g., above 2,000 feet). Of course, these values ​​are merely examples and other implementations may include a different number of altitude zones with different thresholds. In some cases, the crowdsourced detection and tracking systems and methods disclosed herein may be implemented in low altitude zone 411 and surveillance zone 407, as represented by the dashed line indicated at 415. However, other use cases of the crowdsourced detection and tracking systems and methods may be possible.

[0058] Although Figure 4 illustrates an example of a counter-UAS solution 400 capable of performing crowdsourced detection and tracking of UAS, various modifications may be made to Figure 4. For example, solution 400 may include any suitable number of actors, surveillance zones, altitude zones, and counter actions.

[0059] FIG 5 illustrates an example method 500 for crowdsourced detection and tracking of UAS according to the present disclosure. For ease of explanation, the method 500 of FIG 5 may be described as being performed in an environment 300 using a system 100, which may be implemented using one or more devices 200 of FIG 2. However, the method 500 may involve the use of any suitable devices and systems in any suitable environment.

[0060] 5, in step 502, notification of a UAS flying within an area is received. The notification is received from at least one of the users or from a third party system that detects the presence of an unidentified object within the area. This may include, for example, server 310 receiving a notification regarding UAS 306 from at least one of users 302 or a third party system.

[0061] At step 504, another notification is sent to multiple mobile devices of multiple users located within the area. The notification requests the users to locate the UAS. In some implementations, the notification is provided to the users via an application installed on the mobile device. This may include, for example, the server 310 sending the notification to the mobile device 304, which is provided to the user 302 via the app 305.

[0062] In step 506, a further notification may be sent to the mobile devices and to other mobile devices of other users located in the larger area, requesting the users and the other users to locate the controller of the UAS or the operator of the UAS. This may include, for example, the server 310 sending a notification to the mobile devices 304 in the larger area.

[0063] At step 508, information about the UAS is received from one or more mobile devices of one or more users. The information is obtained by the one or more mobile devices. In some implementations, the information about the UAS includes one or more images or videos of the UAS captured by the mobile device, radio control signal information from the UAS or a controller of the UAS detected by the mobile device, position data of the UAS obtained by the mobile device, track data of the UAS, or a combination thereof. This may include, for example, the server 310 receiving information about the UAS 306 from the mobile device 304.

[0064] At step 510, a reputation of the UAS is determined based on information about the UAS received from one or more mobile devices and / or other disparate data sources. This may include, for example, the server 310 determining a reputation of the UAS 306 based on information received from the mobile device 304.

[0065] At step 512, a general notification is sent to at least some of the mobile devices. The notification includes information regarding the reputation of the UAS. This may include, for example, the server 310 sending a reputation notification to at least some of the mobile devices 304. The reputation information sent to the mobile devices 304 may be relatively general information regarding the UAS 306 (although it may be more detailed if desired).

[0066] At step 514, a detailed notification is sent to one or more third parties. The notification includes more detailed information regarding the assessment of the UAS, such as specific detailed identification information or a detailed assessment of the threat posed by the UAS. The one or more third parties can include at least one subscriber of a service associated with the UAS detection, law enforcement, at least one owner, at least one entity involved with the UAS or business assets near its flight path, or combinations thereof. This can include, for example, server 310 sending a detailed report notification to the third parties.

[0067] Although Figure 5 illustrates one example of a method 500 for crowdsourced detection and tracking of UAS, various modifications may be made to Figure 5. For example, while illustrated as a series of steps, various steps in Figure 5 may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0068] Among other things, the above-mentioned modes of operation provide an extension of air defense to non-conventional actors, including civilian, commercial, and law enforcement actors. In some cases, UAS information collected through the disclosed modes of operation can fill gaps in the military kill chain, the Intelligence Community (IC) intelligence cycle, and the law enforcement suspicious activity report investigation cycle. Also, in some cases, the collected information can be provided to conventional or institutional kill chain actors or other law enforcers with additional proactive mitigation authority.

[0069] In some implementations, various functions described in this patent document are implemented or supported by a computer program that is formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium accessible by a computer, such as a read only memory (ROM), a random access memory (RAM), a hard disk drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable media include media that can store data permanently and media that can store data and later be overwritten, such as rewritable optical disks or erasable storage devices.

[0070] It may be advantageous to provide definitions of certain words and phrases used throughout this patent document. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof, adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean inclusion rather than limitation. The term "or" is inclusive and / or. "Associated with" and its derivatives may mean including, contained within, interconnected with, containing, contained within, connected to or with, coupled to or with, communicable with, cooperate with, interleave with, juxtaposed to, adjacent to, associated with or with, having the property of, relating to or with, and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A, B, and C.

[0071] The description herein should not be construed as implying that any particular element, step, or function is an essential or critical element required for inclusion in the scope of the claims. The scope of the claimed subject matter is defined solely by the allowed claims. Moreover, none of the claims invoke 35 U.S.C. 112(f) with respect to any of the appended claims or claim elements unless the precise words "means for" or "step for" followed by a specific phrase identifying the function are expressly used in a particular claim. The use of terms such as "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller" in the claims is understood and intended to refer to structures known to those skilled in the art, as further modified or enhanced by features of the claims themselves, and is not intended to invoke 35 U.S.C. 112(f).

[0072] While this disclosure has described particular implementations and generally associated methods, modifications and permutations of these implementations and methods will be apparent to those skilled in the art. Thus, the above description of exemplary implementations does not define or constrain the disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. receiving a first notification of an unmanned aerial system (UAS) flying within the area; sending a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the users to locate the UAS; sending a third notification to at least some of the mobile devices and to other mobile devices of other users located within a larger area, the third notification requesting the users and the other users to locate a controller of the UAS or an operator of the UAS; receiving information about the UAS from one or more of the mobile devices of one or more of the users, the information being obtained by the one or more mobile devices; determining a reputation of the UAS based on information about the UAS received from the one or more mobile devices; sending a fourth notification to at least some of the mobile devices, the fourth notification including information regarding the evaluation of the UAS; A method comprising:

2. The information regarding the UAS is: one or more images or videos of the UAS captured by the one or more mobile devices; and wireless control signal information from the UAS or a controller of the UAS detected by the one or more mobile devices; position data of the UAS obtained by the one or more mobile devices; The method of claim 1 , comprising at least one of:

3. the second notification is provided to the user via an application installed on the mobile device. The method of claim 1.

4. the first notification is received from one of the users or from a third party system that detects the presence of an unidentified object in the area. The method of claim 1.

5. and transmitting a fifth notification to one or more third parties, the fifth notification including second information regarding the evaluation of the UAS. The method of claim 1.

6. The one or more third parties include at least one of a subscriber of a service associated with UAS detection, a law enforcement officer, at least one property owner, and at least one entity involved with a business asset near the UAS or its flight path; The method according to claim 5.

7. at least one memory configured to store instructions; a processor coupled to the at least one memory and configured to execute the instructions, Receive initial notification of an unmanned aerial system (UAS) flying within the area; sending a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the users to locate the UAS; sending a third notification to at least some of the mobile devices and to other mobile devices of other users located within a larger area, the third notification requesting the users and the other users to locate a controller of the UAS or an operator of the UAS; receiving information about the UAS from one or more of the mobile devices of one or more of the users, the information being obtained by the one or more mobile devices; determining an assessment of the UAS based on information about the UAS received from the one or more mobile devices; sending a fourth notification to at least some of the mobile devices, the fourth notification including information regarding the evaluation of the UAS; At least one processor configured to: Including, the device.

8. The information regarding the UAS is: one or more images or videos of the UAS captured by the one or more mobile devices; and wireless control signal information from the UAS or a controller of the UAS detected by the one or more mobile devices; position data of the UAS obtained by the one or more mobile devices; The device of claim 7 , comprising at least one of:

9. the second notification is provided to the user via an application installed on the mobile device. The device of claim 7.

10. the at least one processor is configured to receive the first notification from one of the users or from a third party system that detects the presence of an unidentified object in the area. The device of claim 7.

11. The at least one processor is further configured to send a fifth notification to one or more third parties, the fifth notification including second information regarding the evaluation of the UAS. The device of claim 7.

12. The one or more third parties include at least one of a subscriber of a service associated with UAS detection, a law enforcement officer, at least one property owner, and at least one entity involved with a business asset near the UAS or its flight path. The device of claim 11.

13. When executed, the method includes the steps of: Receive a first notification of an unmanned aerial system (UAS) flying within the area; transmitting a second notification to a plurality of mobile devices of a plurality of users located within the area, the second notification requesting the users to locate the UAS; sending a third notification to at least some of the mobile devices and to other mobile devices of other users located within a larger area, the third notification requesting the users and the other users to locate a controller of the UAS or an operator of the UAS; receiving information about the UAS from one or more of the mobile devices of one or more of the users, the information being obtained by the one or more mobile devices; determining an assessment of the UAS based on information about the UAS received from the one or more mobile devices; transmitting a fourth notification to at least some of the mobile devices, the fourth notification including information regarding the evaluation of the UAS; A non-transitory computer-readable medium containing instructions.

14. The information regarding the UAS is: one or more images or videos of the UAS captured by the one or more mobile devices; and wireless control signal information from the UAS or a controller of the UAS detected by the one or more mobile devices; position data of the UAS obtained by the one or more mobile devices; 14. The non-transitory computer readable medium of claim 13, comprising at least one of:

15. the second notification is provided to the user via an application installed on the mobile device.

14. The non-transitory computer readable medium of claim 13.

16. The instructions, when executed, further cause the at least one processor to receive the first notification from one of the users or from a third party system that detects the presence of an unidentified object in the area.

14. The non-transitory computer readable medium of claim 13.

17. The instructions, when executed, further cause the at least one processor to send a fifth notification to one or more third parties, the fifth notification including second information regarding the evaluation of the UAS.

14. The non-transitory computer readable medium of claim 13.

Citation Information

Patent Citations

  • Method of operating a uav, computer readable medium, and uav flight system

    JP2017536620A

  • Drone real-time interactive communications system

    US20200106818A1

  • Security event detection and threat assessment

    US20200162489A1