Event camera based tracking of rotor craft

Abstract

Systems and methods herein provide for tracking of a rotorcraft. In one embodiment, a tracking system includes an event camera. The event camera comprises an array of pixels with each pixel being operable to record a change in brightness, and to generate an event for each change in brightness. The tracking system also includes a processing system operable to process events from the event camera in real time, to identify one of the events as being new, to assign a track identification (ID) to the new event, and to update the track ID with subsequent events from the event camera by computing pixel distances between the events to track a target that causes the event camera to generate the events.

Description

EVENT CAMERA BASED TRACKING OF ROTOR CRAFTCross Reference to Related Applications

[0001] This patent application claims priority to, and thus the benefit of an earlier filing date from, U.S. Provisional Patent Application No. 63 / 538,219 (filed September 13, 2023), the contents of which are hereby incorporated by reference.Background

[0002] Current object tracking algorithms can predict a future position of a target by extrapolating current estimates of the target’s velocity. For example, a motion model and a Kalman filter may be used to provide updated and projected position estimates. However, changes in the force applied to the target that impact trajectory are not directly measured and can only be inferred once these changes in force begin to impact velocity. This limits the ability to accurately predict the future position of the target, particularly when the target is traveling through complex trajectories. Such is the case when tracking small rotorcraft that are capable of rapid changes in motion and complex acceleration (e.g., as compared to fixed wing aircraft).

[0003] Kinetic defeat systems are limited by range to consistently hit small unmanned aerial systems / vehicles (UAVs, a.k.a. “drones”, “rotorcraft”, etc.). For example, at longer distances, the delay between firing a round and the round’s impact requires an accurate position prediction to lead the target. For engagements with swarms of UAVs, range and efficient use of ammunition is important. Rotor speed monitoring has been traditionally performed with traditional frame-based cameras, but the required high-speed cameras are expensive and computationally intensive. This limits the size, weight, power and cost (SWaP-C) of kinetic defeat systems. What is needed is an improved method to predict future positions of small rotorcraft to provide targeting solutions for kinetic defeat systems.Summary

[0004] For rotorcraft, the force applied to a vehicle body by each rotor is proportional to the speed of that rotor. Thus, monitoring the speed of each rotor can allow for the estimation of the force on the vehicle body, and lead to an improved future motion prediction and classification (i.e., identification) of the vehicle body. The embodiments herein allow for thesimultaneous measurement of a target’s instantaneous position, speed, and acceleration for improved predictions via the use of event cameras. Event cameras offer analog processing at each pixel to reduce data-rates, and generally only report changes occurring in a scene for rapidly changing objects. This allows real-time rotor speed monitoring of UAVs with smaller, less expensive cameras with dramatically reduced data processing requirements.

[0005] In one embodiment, a tracking system includes an event camera comprising an array of pixels with each pixel being operable to record a change in brightness, and to generate an event for each change in brightness. The tracking system also includes a processing system operable to process the events from the event camera in real time, to identify one of the events as being new, to assign a track identification (ID) to the new event, and to update the track ID with subsequent events from the event camera by computing pixel distances between the events to track a target causing the event camera to generate the events. From there, the processing system may be operable to identify a rotorcraft based on the changes in rotor frequencies. And the processing system can communicate this information as well as tracking information to a targeting system (e.g., radar or lidar) to direct a kinetic defeat system at the rotorcraft.

[0006] The system may also include a detection module operable to reject false alarms based on the rotor frequencies of the UAV. In some embodiments, the processing system comprises a motion model (e.g., a Kalman filter) operable to predict the motion of the UAV based on the changes in the rotor frequencies. The motion model may also be operable to detect a payload weight and / or a weight distribution of the UAV based on the changes in the rotor frequencies. The motion model may also be operable to determine environmental conditions affecting the rotorcraft, which can be conveyed to a targeting system for more accurate targeting.

[0007] The various embodiments disclosed herein may be implemented in a variety of ways as a matter of design choice. For example, some embodiments herein are implemented in hardware whereas other embodiments may include processes that are operable to implement and / or operate the hardware. Other exemplary embodiments, including software and firmware, are described below.Brief Description of the Drawings

[0008] Some embodiments of the present invention arc now described, by way of example only, and with reference to the accompanying drawings. The same reference number represents the same element or the same type of element on all drawings.

[0009] FIG. 1 is a block diagram of a processing system operable with an event camera to track a target, in one exemplary embodiment.

[0010] FIG. 2 is a block diagram of an event association module of the processing system of FIG. 1, in one exemplary embodiment.

[0011] FIG. 3 is a block diagram of a tracking module of the processing system of FIG. 1 , in one exemplary embodiment.

[0012] FIG. 4 is a block diagram of bounding boxes generated by the processing system of FIG. 1, in one exemplary embodiment.

[0013] FIG. 5 is a graph of rotor frequencies of a rotorcraft, in one exemplary embodiment.

[0014] FIGS. 6A and 6B arc plots of wavelets used to determine rotor frequencies of a rotorcraft, in one exemplary embodiment.

[0015] FIGS. 7A-7D illustrate rotors of various aircraft types detected by the processing system of FIG. 1, in one exemplary embodiment.

[0016] FIGS. 8 A and 8B illustrate rotor speeds of a rotorcraft detected by the processing system of FIG. 1, in one exemplary embodiment.

[0017] FIGS. 9 A and 9B illustrate rotor speeds of a rotorcraft in changing wind conditions detected by the processing system of FIG. 1, in one exemplary embodiment.

[0018] FIG. 10 is another block diagram of the processing system of FIG. 1 operable to track a rotorcraft, in one exemplary embodiment.

[0019] FIGS. 1 1 A and 1 IB are graphs illustrating a correlation between rotorcraft motion and instantaneously derived acceleration from rotor speed, in one exemplary embodiment.

[0020] FIGS. 12A-12C are graphs illustrating predicted locations of a rotorcraft based on rotorcraft acceleration, in one exemplary embodiment.

[0021] FIG. 13 is a flowchart of a process of tracking a target, in one exemplary embodiment.

[0022] FIG. 14 is a block diagram of an exemplary computing system in which a computer readable medium provides instructions for performing methods herein.Detailed Description of the Drawings

[0023] The figures and the following description illustrate specific exemplary embodiments. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody certain principles and are included within the scope of the embodiments. Furthermore, any examples described herein are intended to aid in understanding the embodiments and arc to be construed as being without limitation to such specifically recited examples and conditions. As a result, the embodiments arc not limited to any of the examples described below.

[0024] An event camera, also known as a neuromorphic camera, silicon retina, or dynamic vision sensor, operates differently from conventional frame-based cameras. For example, instead of capturing images using a shutter, event cameras respond to local changes in brightness. In an asynchronous operation, each pixel in an event camera operates independently and asynchronously. When it detects a change in brightness that exceeds a threshold, it generates an “event” - i.e., a discrete packet containing the pixel address, a timestamp, and a polarity (e.g., an increase or a decrease in brightness). Event cameras report timestamps with microsecond resolution.

[0025] Event cameras can also operate with analog processing circuitry for each pixel in a focal plane array. Such circuitry provides in- silica processing. For the embodiments herein, the circuitry is designed to report events when the illumination on a pixel changes above acertain level. This processing allows extremely fast scene monitoring (e.g., at greater than 40,000 frames per second) with reduced data-rates when compared to traditional cameras. In this regard, event cameras can provide motion and edge detection when adapted for use in the embodiments herein.

[0026] The event cameras shown and described herein efficiently resolve high-speed motion and are used for the detection of targets with rotors against varyingly cluttered backgrounds. The event cameras can provide target detection of rotorcraft by imaging highspeed rotor motion directly. For example, few things in nature have a periodic repetitive behavior of greater than 50 Hz. UAVs, on the other hand, have rotor speeds that can vary between 5,000 and 20,000 revolutions per minute (RPM’s). An event camera can provide target detection and background segmentation in the time domain by filtering data to recover signals with strongly peaked frequency characteristics at frequencies not found in natural phenomena.

[0027] With this in mind, FIG. 1 is a block diagram of a processing system 100 for detecting and tracking a target 120 (e.g., a rotorcraft) using an event camera, in one exemplary embodiment. The processing system 100 ingests raw event camera data from an event camera 102 in the event camera data stream module 104 where the events of the event camera 102 are processed. For example, the event camera data stream module 104 may filter out slower moving events such as birds using threshold detection methods that allow faster moving events to pass through, such as those associated with rotorcraft. From there, the events undergo an association with the target 120, in the event association module 106.

[0028] The event association module 106 is operable to detect a new target and / or update an existing one. For example, the event association module 106 may detect a new target 120 and then assign that target a track identification (ID). Then, as subsequent events of that target 120 are processed by the event camera data stream module 104, the event association module 106 may associate those events with the track ID. The event association module 106 may do so by computing, from the event camera data stream, the distance between pixels in a pixel space of the event camera 102.

[0029] After event association, the tracking module 110 may determine a center of mass position and velocity of the target 120. In some embodiments, the tracking module 110 canestimate the current position of the target 120. For example, based on events generated by pixels in the event camera 102 and the scene being viewed by the event camera 102, the tracking module 110 may identify a likely position of the target 120. And as the target 120 moves through the scene, different pixels of the event camera 102 will begin generating events. Based on a change in time between those pixels generating events and the distances between those pixels, the module 118 may determine the position and velocity of the target 120 in the scene. In some embodiments, the module 118 may estimate the velocity of the target using a motion model and previous estimates of position. The module 118 may also use rotor analysis to provide estimations of acceleration, improve tracking, and improve future position estimation. Event association and target tracking are shown and described in more detail below in FIGS. 2 and 3, respectively.

[0030] The processing system 100 includes an image processor 122 that is operable to process image data of each event from the event camera 102. For example, the image processor 122 may include a bounding box module 108 that is operable to form a bounding box around the target 120 during tracking of the target. Once the target 120 has been detected in the imagery of the event camera 102 and the target 120 has been assigned a track ID to be currently tracked, a bounding box may be formed around the imagery of the target 120 such that other imagery may be filtered out. The bounding box allows the imagery of the target 120 to be processed according to its track ID.

[0031] From there, the bounding box of the target 120 may be segmented with a segmentation module 112 of the image processor 122 to detect various features of the target 120, including rotor speed of the target 120. This image processing may be performed by independently analyzing frequency characteristics of events in each segmented portion of the bounding box, via the frequency analysis module 114. From there, the processing system 100 may determine the rotor speed of the target 120 with the rotor speed determination module 116. An example of the bounding box processing is shown and described in further detail in FIG. 4.

[0032] Generally, targets with multiple rotors have strong frequency dependent signals in each segment that overlap the rotor. Masked regions without a rotor present will not have a strong periodic structure and are neglected from further analysis. Once the rotors of the target120 are identified, the number and speed of the rotors may be used for target classification, as well as predictive tracking of the target 120. Thus, the processing system 100 has the unique ability to efficiently analyze and process data of the event camera 102 to derive traits of the target 120 and to predict movement of the target 120.

[0033] In some embodiments, the processing system 100 may include an image stabilizer 124 (e.g., a gimbal, a lens shift, etc.) to compensate for target motion rates across a scene. For example, if the target 120 were moving sufficiently fast through the scene, triggers associated with the rotor rates may not be detectable with simple image processing. The image stabilizer 120 could stabilize the image so that that rotors affect the same pixels of the event camera 102 through multiple rotor rotations. And the image stabilizer 120 may be tied specifically to event camera triggering. Alternatively, or additionally, if trigger and pixel locations are processed together, rotor rate triggers might be separable from trigger rates of gross target motion. This would allow the processing system 100 to identify triggers that are likely from target motion in the scene and filter them out so that triggers associated with rotor motion would stand out in the signal.

[0034] FIG. 2 is a block diagram of the event association module 106, in one exemplary embodiment. In this embodiment, the event association module 106 detects an event 202 in the event camera data stream. For example, when the target 120 is imaged by certain pixels of the pixel array of the event camera 102, those pixels will begin generating events. The event association module 106 may then perform a distance calculation 204 on these events to determine whether they are new events or events of targets already being tracked and associated with a track ID. For example, pixels that are relatively close in proximity to one another in the pixel array may create events when imaging the target 120. Thus, when the event camera data stream module 104 records a new event, the event association module 106 may measure the distance in pixel space between the new event and the current location of each existing track using a Euclidean distance formula. If the event is within a set threshold distance from an historical track, the event association module 106 associates the event with that track, and the track is updated in the module 210. If the event is outside the distance threshold, the event association module 106 may start a new track that is then used for associating future events in the module 208.

[0035] The event association module 106 may remove tracks from memory if the tracks have not been updated for a certain time. When the event association module 106 assigns a track ID and begins updating that track, that information may be passed to the tracking module 110 such that the events caused by the motion of the target 120 can be processed together. In some embodiments, events generated from pixels that are further away in the event camera 120’ s pixel array may either generate new track IDs in the new track module 208 or be disregarded as noise. The event association module 106 may include a running memory of existing tracks with a unique ID for each track which are updated in the module 206 with subsequent events.

[0036] FIG. 3 is a block diagram of the tracking module 110, in one exemplary embodiment. In this embodiment, the tracking module 110 estimates a current position of a target 120 based on the historical record of events for the track. For smaller targets, such as UAVs, the tracking module 110 may employ a Proportional-Integral-Derivative (PID) controller comprising modules 304, 306, and 308. With this PID controller, each new event may be treated as a new measurement of the target 120’s position, and the estimated position may be determined based on a term proportional to the difference between the new event detection 202 and the previous position 302, the derivative of the changes in this value with respect to time, and the integral of this value with respect to time. For larger targets, the center of mass of events in a fixed time period may be used instead.

[0037] FIG. 4 is a block diagram of bounding boxes generated by the processing system of FIG. 1, in one exemplary embodiment. In this embodiment, once a target 120 has been detected and is currently being tracked, image data in the event camera data stream is associated with the track. And a first bounding box 402-1 is formed around the target 120 by the image processor 122 of FIG. 1. Thus, when the target 120 moves through the pixel space of the event camera 102, new event data including image data is generated at different pixels in the event camera 102. Once that event data is associated with the existing track of the target 120, the image data in the bounding box 402 moves through the scene that the event camera 102 is viewing. That is, the bounding box 402 tracks with the target 120 in the scene to filter out other event / image data from the target 120.

[0038] The bounding boxes may be determined by calculating the minimum and maximum position in both axes of events associated with a track-ID over a period of time. This method may determine the target 120’ s extent for use in determining segmentation positioning. For example, as this is a continuous process as long as the target 120 remains in the scene (i.e., as long as there is a track for the target 120), a number of bounding boxes 402-1 - 402-N (where the reference number “N” is an integer greater than ‘T’and not necessarily equal to any other “N” reference designated herein) may be retained for frequency analysis (e.g., module 114 of FIG.l). In this regard, the segmentation module 112 of the image processor 122 of FIG. 1 segments each of the bounding boxes 402 into an NxM array of bounding box segments 402-1-1 - 402-M-N (where the reference number “M” is an integer greater than ‘T’and not necessarily equal to any other “N” reference designated herein). As seen herein, the target 120 is positioned in the bounding box 402- 1. Rotors of the target 120 are found in certain segments of the bounding box 402- 1. New bounding boxes 402 are formed at a frequency of the pixels of the event camera 102 generating the events. That is, as subsequent event data is received from the pixels of the event camera 102, that event data is assigned to the track ID of the target 120. And, as mentioned, the bounding boxes 402 move with the target 120 in the scene. The frequency at which the bounding boxes 402 are generated can be used for frequency analysis of the rotor speeds of the target 120.

[0039] FIG. 5 is a graph 500 of the rotor speed analysis performed by the frequency analysis module 114 of FIG. 1. Here, the bounding boxes 402 are collected at the event rate of the event camera 102. In this embodiment, the analysis is performed on the left rotor of the target 120 and thus a portion of the bounding box segments 402-1-1 - 402-M-N where the rotor lies in the bounding box 402. To detect the presence of a rotor for this specific target / track ID each of bounding box segments 402-1-1 - 402-M-N is processed independently.

[0040] There are numerous methods for detecting periodic signals in time-series data, but not all of them are well suited to asynchronous data generated by the event camera 102. Thus, two methods to determine the rotor speed are presented. The first method takes the event data stream and calculates the event rate as a function of time. This corresponds to how much “motion” there is in that specific segment of the image of the bounding box 402. The event rate data is then analyzed with a peak- finding algorithm to detect spikes 502-1 - 502 -N in the eventrate. The spikes 502 occur when something (e.g., a rotor or otherwise) passes through the specific segment. The position of the spikes 502 is then analyzed. If a consistent spacing between the spikes is detected, this indicates periodic motion consistent with the presence of a rotor of the target 120. As illustrated herein, the peaks 502 are separated by about 6,000 s. This corresponds to a rotor frequency of about 10,000 RPMs (e.g., l / (.006s peak separation) x 60s)).

[0041] The widths 504 of the peaks 502 are attributed to the rotor rotating through different bounding box segments 402-1-1 - 402-M-N over time. That is, as the rotor spins, different pixels of the event camera 102 are illuminated and generate events. But, over time, the bounding box segments 402-1-1 - 402-M-N generating the most event data represent the peaks 502 the rotor speed of the target 120. Alternatively, the autocorrelation of the event rate may be used to identify a strong periodicity in the result. In any case, once frequency analysis has occurred, the image processor 122 can determine if a sufficiently periodic signal is present in each segmented section. And if the periodicity passes a set threshold, that periodicity may be interpenetrated as the speed of the rotor.

[0042] The second method is a wavelet-based template search as illustrated in FIGS. 6 A and 6B. For this technique, the event data for previous K events prior to a time TO may be used to calculate a correlation with a bank of wavelets. For example, the correlation (C_L) with the L’th wavelet (F_L) in the bank of wavelets may be calculated as:C_L=Sum(F_L(Ti-T0), { i, 1 ,K] ).

[0043] When the events match a characteristic frequency of a specific wavelet 602, a strong correlation exists and the magnitude of C_L is large (604). When there is no correlation between the wavelet 606’ s frequency and the timing of the events, smaller values (608) are obtained. By designing the bank of wavelets to match the frequency range of interest, this method can estimate the spectrogram of the signal, and spikes in this spectrogram indicate the presence of a rotor. This method works well for data from the event camera 102 as the sparse nature of events means that the frequency analysis module 114 needs to process O(K*L) calculations. Methods to require evenly spaced data, such as traditional fast Fourier transforms,require significant data augmentation that eliminates much of the advantages of spare event camera data.

[0044] Unique information for target classification is generally not available to previous systems. But, while being operable to determine a location of the target 120, to predict motion of the target 120, and to determine the rotor speed of the target 120, the embodiments herein are also operable to determine a number of rotors of the target 120 and determine a shape of the target 120. For example, simple fixed wing UAVs and planes will have a single rotor, and a characteristic shape that may be extracted from the track and rotor segmentation analysis. Quadcopter UAVs can be separated from other platforms because of the varying numbers of rotors.

[0045] FIGS. 7A-7D illustrate rotors of various aircraft types that can be detected and classified by the processing system of FIG. 1, in one exemplary embodiment. FIG. 7A illustrates a silhouette of a UAV target 120A with twin propellers 730A-1 and 730A-2. FIG. 7B illustrates a traditional helicopter target 120B having a main rotor 730B-1 and a tail rotor 730B-2. FIG. 7C illustrates a propeller aircraft target 120C with a single propeller 730C-1. And FIG. 7D illustrates a jet aircraft having no propellers.

[0046] When the processing system 100 generates a bounding box around the UAV 120A of FIG. 7A and segmentation of the bounding box occurs, detection of multiple propellers 730A-1 and 730A2 occurs in different segments of the bounding box. For example, if the bounding box is generated around the target 120 A and there are multiple frequency characteristics in differing segments of the bounding box, the processing system may determine that the target 120A is a UAV. The position and velocity determination as well as the rotor speed determination may also be indicators that the target 120A is a UAV and may classify the target 120A as such. The target 120B having different rotor frequency characteristics and detections may indicate that the target is indeed a helicopter such that the processing system 100 may classify it as such. And the propeller frequency characteristics of the target 120C has one region of rotor motion that may indicate that it is a propeller plane such that the processing system may also classify it as such. However, there are generally no frequency characteristics with respect to the target 120D as its silhouette represents a fighter jet. Since there is no rotor,other more traditional targeting systems like radar and lidar may be used to identify the target 120D.

[0047] The rotor speed of a target 120 may also yield unique information about a payload of the target 120. For example, a quadcopter UAV target 120 carrying munitions will be substantially heavier than an unloaded version, and the rotor speed will be higher to overcome the increased force from gravity, as illustrated by the rotor speed graphs 800-1 and 800-2 in FIGS. 8 A and 8B, respectively. This changed rotor speed can be detected and monitored continuously to determine if munitions and / or contraband are being released. For example, if the target 120 (e.g., a UAV) drops a payload, the rotor speed will decrease to conserve power as rotor speed is proportional to the weight of the target 120. The processing system 100 may detect the changes in payloads via rotor speed analysis to identify threats and prioritize responses in situations with more than one target 120. This technique offers a complimentary means to optical identification of drone payloads, which cannot determine payload mass directly.

[0048] Another advantage of the processing system 100 regards the remote estimation of wind speeds. As winds vary to maintain a constant flight path (i.e., which autonomously controlled drones are programmed to do), a target 120 must change rotor speed to account for the motion of the wind, as illustrated the changing rotor speeds 940A, 940B, 942 A, and 942B in FIGS. 9A and 9B. Thus, a rotorcraft whose trajectory remains constant, but whose rotor speeds are continually varying, is likely be flying in a region where wind is present. Wind can also impact the trajectory of munitions targeting the target 120. By identifying if high winds are present, the processing system 100 can inform an operator of a targeting system such that corrective actions can be taken. Potentially, these variations in rotor speed compared to the physical trajectory can be analyzed to extract the wind velocity and direction to the target 120. Such information can be output to a traditional ballistics calculator to improve a targeting solution.

[0049] In some embodiments, Kalman filters may be used to incorporate new measurements and previous state estimates to make predictions for the future state of a target 120. When employed in the tracking module 110 of FIG. 1, a Kalman filter may configured to function with a motion model that encodes physical properties of the target 120 usingmeasurements about the motion of the target 120. In this regard, the processing system 100 performs direct measurements of the target’s acceleration. And properties of the target 120 being tracked become especially pronounced when the target 120 is undergoing complex motion, such as the case when the target 120 is attempting to dodge incoming fire from a kinetic defeat system.

[0050] FIG. 10 is a block diagram of the processing system 100 incorporating a Kalman filter prediction algorithm. For example, acceleration estimation can be an important pail of predictive tracking. Position 1010 and velocity 1012 of a target 120 may thus be augmented by predicted instantaneous accelerations 1008 derived from rotor speed measurements 1002 from the event camera 102 and a motion model 1006. The motion model 1006 may be selected based on the identification 1004 of the target 120. And the processing system 100 may directly measure the current acceleration 1008 of the target 120 without relying on historical data.

[0051] With this collected data, the processing system 100 can also validate improvements in predictive tracking enabled by measurement (i.e., not estimation) of target acceleration. For example, the processing system 100 may directly measure target acceleration from rotor frequency analysis. The Kalman-filter 1014 may be used to determine a target 120’ s instantaneous acceleration for future position prediction enable large improvements over existing tracking and targeting systems 1016, especially for targeting solutions at longer ranges. Analysis of rotor speed, rotor position, relative size of regions with rotor motion, and total number of rotors may also provide unique information in the classification of the target type and target threat level.

[0052] The embodiments herein show that rotor speed can be extracted by analyzing event camera data in a segmented bounding box around a target track. FIGS. 11A and 1 IB illustrate one reduction to practice where the processing system 100 was used under real-world conditions with a hovering drone against a suburban backdrop. For example, in FIG. 11 A the processing system 100 determined the left rotor speed 1106 and the right rotor speed 104 as well as the UAV’s position. Rotor speed is determined in FIG. 5. For example, an updated speed may be calculated each time a spike in the event rate is detected by a rotor passing the bounding box. The UAV’s position may be determined in FIG. 3 and shows the target 120 is initially stationary,then moves down, and then move hack up to the original position. Based on subsequent rotor speed analysis of the left rotor 1116 and the right rotor 1114 (c.g., in real time), the UAV’s acceleration 1112 was estimated after the fact by taking the second derivatives of the center of mass motion 1102 of FIG. 11 A of the drone during vertical flight. The center of mass acceleration 1112 of FIG. 11B shows a strong correlation with rotor speed. Thus, rotor speed can be used for instantaneous estimation of drone acceleration for providing a unique input into existing tracking systems for improved performance.

[0053] FIGS. 12A-12C illustrate the drone’s predicted location 0.25 seconds in the future using three methods. FIG. 12A illustrates a graph 1200 of the location of the target without the inclusion of acceleration, leading to large deviations in the presence of acceleration. Plot 1202 represents the predicted location of the drone with no acceleration, whereas plot 1204 represents the actual location. FIG. 12A corresponds to a motion model that assumes constant velocity (i.e., no acceleration) and shows how neglecting acceleration can introduce errors. FIG. 12B illustrates a graph 1220 of the location of a target with the inclusion of modeled / estimated acceleration, with associated errors in acceleration prediction that may add some uncertainty. FIG. 12B shows that, while including acceleration into the motion model can reduce these errors, predictions of instantaneous acceleration based on past motion can still be “noisy”. This noise can degrade future position estimation. Plot 1222 represents the predicted location of the drone, whereas plot 1224 represents the actual location. And FIG. 12C illustrates a graph 1240 of the location of a target with the inclusion of measured acceleration from rotor monitoring and rotor frequency analysis. Plot 1244 represents the predicted location of the drone, whereas plot 1242 represents the actual location. FIG. 12Cshows that the use of measured acceleration derived from instantaneous rotor speed. This may lower errors that would be introduced by estimating acceleration from past positions. The inclusion of acceleration from rotor frequency analysis improves the accuracy and prediction as two plots 1242 and 1244 are closely aligned with one another.

[0054] FIG. 13 is a flowchart of a process 1300 of predicting rotorcraft motion, in one exemplary embodiment. In this embodiment, the processing system 100 operates an event camera 102 to capture events of a target rotorcraft 120, in the process element 1302. The event camera 102 comprises an array of pixels with each pixel being operable to record a change inbrightness when imaging a scene. For example, when any pixel within the event camera 102 detects a change in brightness that is above and / or below a certain threshold, the pixel records that change and generates an event corresponding to the change in brightness. From there, that information is conveyed to the event camera data stream module 104 (e.g., a data interface) of the processing system 100 where the events of the event camera 102 are processed, in the process element 1304. Then, the processing system 100 determines whether the events are part of an existing tracking, in the process element 1306. For example, the event association module 106 may identify the pixel in the pixel array of the event camera 102 from a particular event. The event association module 106 may then search the existing tracks to determine whether the event of that pixel is close enough in proximity to a pixel in the pixel array that has been associated with an event that is currently being tracked. If so, the event association module 106 may update the track with the new event data, in the process element 1308, and the processing system 100 returns to processing events from the event camera 102, in the process element 1304. Otherwise, the event association module 106 may assign a track ID to the new event to begin tracking subsequent events from the event camera 102 (i.e., processing events from the event camera in the process element 1304).

[0055] To limit a number of false “hits”, the processing system 100 may set event thresholds to filter out certain events generated by the event camera 102. For example, anytime there is a change in brightness in a particular pixel of the event camera 102, that pixel will generate an event. In this regard, the processing system 100 may set a threshold for the number of events being generated by a pixel. To illustrate, if a pixel generates an event once per second, the processing system 100 may deem this event to be noise or other useless information. The processing system 100 may then disregard the event and not search the tracking database or generate a new event. If, however, a pixel is generating 100 or even 1000 events per second, the processing system 100 may deem those events to be “track worthy” and search for an existing tracker start a new track. Of course, the threshold of generated events for searching tracks or starting a new track may be chosen as a matter of design choice. Similarly, a threshold for the amount of brightness change for generating an event may be established in the event camera 102 as a matter of design choice.

[0056] Again, once the track is started, the processing system 100 may search for and detect one or more rotors of a rotorcraft from the events of the event camera 102. The processing system 100 may then locate rotors of the target rotorcraft 120 and generate bounding boxes (e.g., bounding boxes 402 of FIG. 4) about the received imagery from the event camera 102. These bounding boxes may operate to block out other events (e.g., changes in pixel brightness) generated by the event camera 102, so as to reduce unnecessary processing. Once these bounding boxes are formed, the bounding boxes may be segmented (e.g., via the segmentation module 112) such that rotor frequencies of the target 120 may be extracted from the events (e.g., via the frequency analysis module 114). From there, the processing system 100 may predict motion and / or location of the target 120 based on changes in the rotor frequencies of the rotorcraft (e.g., via the position and velocity determination module 118 and the rotor speed determination module 116).

[0057] Any of the above embodiments herein may be rearranged and / or combined with other embodiments. Accordingly, the concepts herein are not to be limited to any particular embodiment disclosed herein. Additionally, the embodiments can take the form of entirely hardware or comprising both hardware and software elements. Portions of the embodiments may be implemented in software, which includes but is not limited to firmware, resident software, microcode, etc. FIG. 14 illustrates a computing system 1400 in which a computer readable medium 1406 may provide instructions for performing any of the methods disclosed herein.

[0058] Any of the various computing and / or control elements shown in the figures or described herein may be implemented as hardware, as a processor implementing software or firmware, or some combination of these. For example, an element may be implemented as dedicated hardware. Dedicated hardware elements may be referred to as “processors,” “controllers,” or some similar terminology. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be constmed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, a network processor, application specific integrated circuit (ASIC) or other circuitry, field programmable gate array (FPGA), read only memory (ROM) for storing software,random access memory (RAM), non-volatile storage, logic, or some other physical hardware component or module.

[0059] In one embodiment, instructions stored on a computer readable medium direct a computing system of any of the devices and / or servers discussed herein to perform the various operations disclosed herein. In some embodiments, all or portions of these operations may be implemented in a networked computing environment, such as a cloud computing system. Cloud computing often includes on-demand availability of computer system resources, such as data storage (cloud storage) and computing power, without direct active management by a user. Cloud computing relies on the sharing of resources, and generally includes on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service.

[0060] FIG. 14 depicts one illustrative cloud computing system 1400 operable to perform the above operations by executing programmed instructions tangibly embodied on one or more computer readable storage mediums. The cloud computing system 1400 generally includes the use of a network of remote servers hosted on the internet to store, manage, and process data, rather than a local server or a personal computer (e.g., in the computing systems 1402-1 - 1402-N). Cloud computing enables users to use infrastructure and applications via the internet, without installing and maintaining them on-premises. In this regard, the cloud computing network 1420 may include virtualized information technology (IT) infrastructure (e.g., servers 1424-1 - 1424-N, the data storage module 1422, operating system software, networking, and other infrastructure) that is abstracted so that the infrastructure can be pooled and / or divided irrespective of physical hardware boundaries. In some embodiments, the cloud computing network 1420 can provide users with services in the form of building blocks that can be used to create and deploy various types of applications in the cloud on a metered basis.

[0061] Various components of the cloud computing system 1400 may be operable to implement the above operations in their entirety or contribute to the operations in part. For example, a computing system 1402-1 may be used to perform analysis of lidar data, and then store that analysis in a data storage module 1422 (e.g., a database) of a cloud computing network 1420. Various computer servers 1424-1 - 1424-N of the cloud computing network 1420 may beused to operate on the data and / or transfer the analysis and / or the data to another computing system 1402-N.

[0062] Some embodiments disclosed herein may utilize instructions (e.g., code / software) accessible via a computer-readable storage medium for use by various components in the cloud computing system 1400 to implement all or parts of the various operations disclosed hereinabove. Examples of such components include the computing systems 1402-1 - 1402-N.

[0063] Exemplary components of the computing systems 1402-1 - 1402-N may include at least one processor 1404, a computer readable storage medium 1414, program and data memory 1406, input / output (VO) devices 1408, a display device interface 1412, and a network interface 1410. For the purposes of this description, the computer readable storage medium 1414 comprises any physical media that is capable of storing a program for use by the computing system 1402. For example, the computer-readable storage medium 1414 may be an electronic, magnetic, optical, electromagnetic, infrared, semiconductor device, or other non-transitory medium. Examples of the computer- readable storage medium 1414 include a solid-state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Some examples of optical disks include Compact Disk - Read Only Memory (CD-ROM), Compact Disk - Read / Write (CD- R / W), Digital Versatile Disc (DVD), and Blu-Ray Disc.

[0064] The processor 1404 is coupled to the program and data memory 1406 through a system bus 1416. The program and data memory 1406 include local memory employed during actual execution of the program code, bulk storage, and / or cache memories that provide temporary storage of at least some program code and / or data in order to reduce the number of times the code and / or data are retrieved from bulk storage (e.g., a hard disk drive, a solid state drive, or the like) during execution.

[0065] Input / output or VO devices 1408 (including but not limited to keyboards, displays, touchscreens, microphones, pointing devices, etc.) may be coupled either directly or through intervening VO controllers. Network adapter interfaces 1410 may also be integrated with the system to enable the computing system 1402 to become coupled to other computingsystems or storage devices through intervening private or public networks. The network adapter interfaces 1410 may be implemented as modems, cable modems, Small Computer System Interface (SCSI) devices, Fibre Channel devices, Ethernet cards, wireless adapters, etc. Display device interface 1412 may be integrated with the system to interface to one or more display devices, such as screens for presentation of data generated by the processor 1404.

Claims

AMENDED CLAIMS received by the International Bureau on 14 January 2025 (14.01.2025)Claims1. A tracking system, comprising: an event camera comprising an array of pixels with each pixel being operable to record a change in brightness, and to generate an event for each change in brightness; and a processing system operable to process events from the event camera in real time, to identify one of the events as being new, to assign a track identification (ID) to the new event, and to update the track ID with subsequent events from the event camera by computing pixel distances between the events to track a target that causes the event camera to generate the events, wherein the processing system comprises an image processor operable to process segments of image data containing at least one rotor of the target, wherein the processing system is further operable to determine a rotor speed of the at least one rotor based on the processed segments, and wherein the processing system comprises a motion model operable to predict motion of the target based on changes in the rotor speed of the target.

2. The tracking system of claim 1, wherein: the image processor is further operable to generate bounding boxes around the image data of the target, and to process segments of the bounding boxes containing the at least one rotor of the target; and the processing system is further operable to determine the rotor speed of the at least one rotor based on the processed segments of the bounding boxes.

3. (Cancelled)4. The tracking system of claim 2, wherein: the processing system is further operable to detect a payload weight of the target based on changes in the rotor speed of the target.

5. The tracking system of claim 2, wherein: the processing system is further operable to determine environmental conditions affecting the target based on changes in the rotor speed of the target.24AMENDED SHEET (ARTICLE 19)6. The tracking system of claim 5, wherein: the processing system is further operable to communicate the environmental conditions to a targeting system to direct a kinetic defeat system at the target.

7. The tracking system of claim 1, wherein: the processing system comprises an image processor operable to generate bounding boxes around image data of the target, and to process segments of the bounding boxes containing the target; and the processing system is further operable to identify the target based on the processed segments of the bounding boxes.

8. A method of tracking a target, the method comprising: operating an event camera to capture events, the event camera comprising an array of pixels with each pixel being operable to record a change in brightness, and to generate an event for each change in brightness; and processing the events from the event camera in real time; identifying one of the events as being new; assigning a track identification (ID) to the new event; updating the track ID with subsequent events from the event camera by computing pixel distances between the events to track a target causing the event camera to generate the events; with an image processor, processing segments of image data containing at least one rotor of the target, determining a rotor speed of the at least one rotor based on the processed segments; and with a motion model, predicting motion of the target based on changes in the rotor speed of the target.

9. The method of claim 8, further comprising: generating bounding boxes around the image data of the target; processing segments of the bounding boxes containing the at least one rotor of the target; and25AMENDED SHEET (ARTICLE 19)determining the rotor speed of the at least one rotor based on the processed segments of the bounding boxes.

10. (Cancelled)11. The method of claim 9, further comprising: detecting a payload weight of the target based on changes in the rotor speed of the target.

12. The method of claim 9, further comprising: determining environmental conditions affecting the target based on changes in the rotor speed of the target.

13. The method of claim 12, further comprising: communicating the environmental conditions to a targeting system to direct a kinetic defeat system at the target.

14. The method of claim 8, further comprising: generating bounding boxes around image data of the target; process segments of the bounding boxes containing the target; and identifying the target based on the processed segments of the bounding boxes.

15. A non-transitory computer readable medium comprising instructions that, when executed by a processing system, are operable to direct the processing system to track a target, the instructions directing the processing system to: operate an event camera to capture events, the event camera comprising an array of pixels with each pixel being operable to record a change in brightness, and to generate an event for each change in brightness; and process the events from the event camera in real time; identify one of the events as being new; assign a track identification (ID) to the new event; update the track ID with subsequent events from the event camera by computing pixel distances between the events to track a target causing the event camera to generate the events;26AMENDED SHEET (ARTICLE 19)via an image processor, process segments of image data containing at least one rotor of the target, determine a rotor speed of the at least one rotor based on the processed segments; and via a motion model, predicting motion of the target based on changes in the rotor speed of the target.

16. The computer readable medium of claim 15, further comprising instructions direct the processing system to: generate bounding boxes around the image data of the target; process segments of the bounding boxes containing the at least one rotor of the target; and determine, based on the processed segments of the bounding boxes, at least one of the rotor speed of the at least one rotor of the target or identify the target.

17. (Cancelled)18. The computer readable medium of claim 16, further comprising instructions direct the processing system to: detect a payload weight of the target based on changes in the rotor speed of the target.

19. The computer readable medium of claim 16, further comprising instructions direct the processing system to: determine environmental conditions affecting the target based on changes in the rotor speed of the target.

20. The computer readable medium of claim 19, further comprising instructions direct the processing system to: communicate the environmental conditions to a targeting system to direct a kinetic defeat system at the target.27AMENDED SHEET (ARTICLE 19)

Images