Method and system for target detection and classification to aid drone defence systems

Abstract

A model-based artificial intelligence (Al) system and method that applies trained decision- making on extracted features for target classification, including at least one sensor for detecting a point target, an interactive multiple model (IMM) filter for extracting features related to motion kinematics of the point target, and a classifier for applying trained decision-making on the extracted features to determine classification of the point target

Description

METHOD AND SYSTEM FOR TARGET DETECTION AND CLASSIFICATION TO AID DRONE DEFENCE SYSTEMSBACKGROUND OF THE INVENTION1. Field of the Invention

[0001] The present invention is directed to drone detection and classification, and in particular to a model-based artificial intelligence (Al) system and method that applies trained decisionmaking on extracted features for target classification.2. Description of the Related Art

[0002] The recent popularization and misuse of Uncrewed Aircraft Systems (UASs), or also known as RPAS (Remotely Piloted Aircraft Systems) and in particular small UAS (sUAS) aircraft (commonly referred to as a “drones”), has highlighted certain risks in low altitude drone flight, especially in urban areas, involving privacy constraints and collision hazards with ground-based structures and other low-flying aircraft and birds. This has given rise to a growing and critical need for anti-drone systems that use effective detection and surveillance algorithms and technologies.

[0003] Existing technologies rely on sensor data from radio sources, radar, acoustics, and / or visual sensors for detection and identification of drone targets. In particular, existing sensor technologies for drone target detection include 1) radio frequency (RF) sensors that scan for the RF broadcast from drones to their ground control stations, 2) sonic sensors that listen for the sonic signature of the drones and propellers, 3) radar sensors that transmit radio waves to an object and use the reflection signal to determine the range and other information about a target, and 4) visual sensors, including electro-optical, infra-red, and laser-based cameras (LiDAR), to provide imaging data in the form of still images and / or videos for further human assessment or post-processing by computer algorithms designed to carry out detection and identification.

[0004] Taha and Shoufan provide a review of existing literature on drone detection and classification using machine learning methods[B. Taha and A. Shoufan, "Machine Learning- Based Drone Detection and Classification: State-of-the-Art in Research," in IEEE Access, vol. 7, pp. 138669-138682, 2019, doi: 10.1109 / ACCESS.2019.2942944],

[0005] At low elevation, flying birds and drones are the principal low speed targets that require disambiguation. Flying birds have similar Radar Cross Section (RCS), same velocity range, similar signal fluctuation, and approximate signal amplitude to drones[J. Gong, J. Yan, D. Li, D.Kong, and H. Hu, “Interference of radar detection of drones by birds,” Progress In Electromagnetics Research, vol. 81 , pp. 1-11 , 2019], The similarity between small drones and birds therefore presents major challenges associated with class separation The problem of identifying birds vs. drones has been discussed in the literature [R. Kretzschmar, N.Karayiannis, and H. Richner, “A comparison of feature sets and neural network classifiers on a bird removal approach for wind profiler data,” in Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium. IEEE, 2000, vol. 2, pp. 279-284, and S. Haykin and C. Deng, “Classification of radar clutter using neural networks,” IEEE Transactions on Neural Networks, vol. 2, no. 6, pp. 589-600,1991],

[0006] Visual-based methods of detection suffer from certain drawbacks due to different weather conditions, while acoustic-based methods are very sensitive to ambient noise and therefore tend to fail in loud areas, and Radio-frequency (RF) based techniques are not suitable for autonomous flying drones [P. Molchanov, R.I.A. Harmanny, J.J.M. de Wit, K. Egiazarian, and J. Astola, “Classification of small uavs and birds by micro-doppler signatures,” International Journal of Microwave and Wireless Technologies, vol. 6, no. 3-4, pp. 435-444, 2014],

[0007] It is also known in the art to discriminate drones and flying birds using Micro-Doppler (M- D) characteristics of a target [J.J.M. de Wit, R.I.A. Harmanny, and G. Premel-Cabic, “Micro- doppler analysis of small uavs,” in 2012 9th European Radar Conference. IEEE, 2012, pp. 210— 213, and J. L. Garry and G. E. Smith, “Experimental observations of micro-doppler signatures with passive radar,” IEEE Transactions on Aerospace and Electronic Systems, vol. 55, no. 2, pp. 1045-1052, 2019], Radar operates by detecting changes in the characteristics of a transmitted electromagnetic signal reflected from a target. For example, the carrier frequency of the returned signal is shifted if the target moves. In addition to the bulk rigid-body movement of the target, micro-motions such as vibration or rotations of any structure of the target (such as a propeller) also causes frequency modulation on the returned signal, a phenomenon referred to as Micro-Doppler effect [V. C. Chen, F. Li, S. -. Ho and H. Wechsler, "Micro-Doppler effect in radar: phenomenon, model, and simulation study," in IEEE Transactions on Aerospace and Electronic Systems, vol. 42, no. 1 , pp. 2-21 , Jan. 2006, doi: 10.1109 / TAES.2006.1603402], However, micro-Doppler has limitations, such as a lower range of detection than conventional methods (typically limited to around 500 meters) and works better with radar signals of higher frequency (e.g., K-band or W-band). Moreover, radar cannot detect a hovering target.

[0008] Additional relevant prior art includes US20170261613A1 : Counter drone system (long- range LIDAR-based); US9715009: Deterrent for unmanned aerial systems (ground based radar; an optical, infrared, or laser range finder; an omnidirectional (RF) antenna; and a directional RF antenna); US9977117: Systems and methods for detecting, tracking and identifying small unmanned systems such as drones (radar-based); US2017 / 092138, which teaches multimodal detection and characterization; and US10588021 which teaches use of Al for improving sensing and identification.SUMMARY OF THE INVENTION

[0009] It is an aspect of the present invention to provide a model-based artificial intelligence (Al) system and method for target classification using features from interactive multiple model (IMM) tracking filters and flight trajectories that uniquely describe the flight of each target. The system and method set forth herein uses an Al classifier that applies trained decision-making on the extracted features to determine the classification.

[0010] The above aspects can be attained by a system for target classification, comprising: at least one sensor for detecting a point target; an interactive multiple model (IMM) filter for extracting features related to motion kinematics of the point target; and a classifier for applying trained decision-making on the extracted features to determine classification of the point target, and by a method for target classification, comprising: detecting a point target; extracting features related to motion kinematics of the point target using an interactive multiple model (IMM) filter; and applying trained decision-making on the extracted features to determine classification of the point target..

[0011] These together with other aspects and advantages which will be subsequently apparent, reside in the details of construction and operation as more fully hereinafter described and claimed, reference being had to the accompanying drawings forming a part hereof, wherein like numerals refer to like parts throughout.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 shows a real-time UAV (Uncrewed Aircraft Vehicle), also known as RPA (Remotely Piloted Aircraft) detection and classification system, according to an embodiment.

[0013] FIG. 2 is a block diagram of functional components of an Al server of the system shown in FIG. 1 for classifying targets using sensor and video data, according to an embodiment.

[0014] FIG. 3 shows exemplary flight trajectories for a bird (FIG. 3A) and a UAV (FIG. 3B),according to an embodiment.

[0015] FIG. 4 shows details of an interactive multiple models (IMM) filter of the Al server shown in FIG. 2 for extracting features for classification, according to an embodiment.

[0016] FIG. 5 shows components of a decision tree classifier for discriminating between UAV, birds and other (ground targets) targets using features extracted by the IMM filter of FIG. 4, according to an embodiment.

[0017] FIG. 6 shows an exemplary output of the decision tree in FIG. 5.DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] For the purpose of the present disclosure, a point target is a radar target that is small compared with the pulse volume, which is the cross- sectional area of the radar beam multiplied by half the length of the radar pulse. Therefore, the cross-sectional area of the point target cannot be estimated and micro-Doppler and other signatures cannot be observed. A point object or point target for optical sensors such as infrared cameras, thermal cameras and RGB camera is characterized by the energy of the signal component being concentrated on one or several pixels making it impossible to distinguish between the objects.

[0019] FIG. 1 shows a real-time UAV detection and classification system 100, according to an embodiment. Raw data from at least one sensor 105 and, optionally camera 110, is transmitted as messages to an Al server 130 via a switch 140 over a TCP connection, wherein the message format comprises a header followed by a data payload. Sensor 105 can be a radar sensor, electro-optical sensor (EG sensor), LIDAR sensor, thermal or RGB camera, or other suitable sensor for detecting an object as a point target and providing its spatiotemporal coordinates as raw data measurements. If sensor 105 is a camera, such as an infrared camera, the coordinates of the detected object are extracted from the frame and only its estimated coordinates are used in signal processing algorithm. Optional video camera 110 can be a conventional PTZ (pan-tilt-zoom) camera capable of directional and zoom control.

[0020] A command and control system 150 synchronizes operation of sensor 105, video camera 110, and Al server 130 via switch 140 during data collection and classification, and allows an operator to adjust specified parameters such as radar detection threshold, zone masking, sidelobe masking, and camera zoom, which can be sent to radar sensor 105 and video camera 110. Radar track data along with metadata of the header file can be saved for further processing along with video data synchronized with the radar data for ground truth.

[0021] FIG. 2 is a block diagram of functional components of Al server 130 for classifying targets using raw sensor measurements 200 and video data 220 from sensor 105 and optional video camera 110, respectively. The functional components of Al server 130 can be divided into four categories: raw data acquisition, target tracking and feature extraction using interactive multiple model (IMM) filter 210, and classification by an Al classifier 280. Since the maneuverability of a flying bird target is usually higher than a small UAV, variations in speed, acceleration, and curvature of the flight trajectories during relatively straight and turning segments can be used to differentiate targets. Also, since one single model cannot represent the behavior of a target at all times, multiple motion models can be used by IMM filter 210 to represent target maneuvers. According to an exemplary embodiment, Constant Velocity (CV), Constant Acceleration (CA), Horizontal Coordinated Turn (HCT) and 3D Coordinated Turn (3DCT) motion models are used by IMM filter 210 for small UAV and bird classification. The CV and CA models describe uniform motion with constant speed and constant acceleration, respectively. The HCT model describes uniform motion in a horizontal plane with minimal vertical maneuvering, while the 3DCT model is used by IMM filter 210 when there are target maneuvers in 3D space.

[0022] The state space model of a given trajectory can be defined as follows:where X(k) = [x, vx, ax, y yy, ay, z, vz, az, a>]Tis the taget state at time step k, and contains the position, velocity, acceleration in X-Y-Z plane and the turn rate a>. F is the transition matrix and >(k) is the noise at time k: For the CV, CA, HCT, and 3DCT motion models, the discrete state space equations are as follows:where where T denotes the sampling time and >, Q denotes the turning rate for the HCT and 3DCT models, respectively.

[0023] A UAS is normally used for area surveillance or package delivery and therefore typically follows a predetermined course and maintains a steady flight with few maneuvers, for safety reasons. The motion model for a UAV therefore oscillates between CV and CA, although the HCT model can also be selected to adapt taget turning movements.

[0024] In contrast, bird flight is more varied than UAV flight since a bird can fly from very low altitude to very high altitude and perform certain unique movements, such as vertical and horizontal turns. The motoin model for a bird therefore switches between CV, CA, HCT, and 3DCT.

[0025] TABLE A shows the maneuvering state mnk of target n at time kTABLE A

[0026] The initial state xno of a target n is a sample from the respective distribution of target kinematic characteristics such as speed, acceleration, and turn rate. TABLE B summarises the target kinematic feature ranges utilised to construct trajectories according to an embodiment.TABLE B

[0027] FIG. 3 shows exemplary flight trajectories for a bird (FIG. 3A) and a UAV (FIG. 3B), according to an embodiment. The simulation of classification of a flying target is carried out according to the described embodiments assuming that the target under investigation is one of two categories of targets: bird or UAV.

[0028] Due to the significant difference among the distribution of speed, acceleration, and curvature parameters resulting from the inherent differences in the kinematics of bird and UAV (drone) flight, IMM filter 210 switches between motion models based upon probability, as discussed below (e.g. the tracking filter for birds can cause frequent changes in motion models compared to the filter for UAV flight tracking).

[0029] TABLE C lists the features used by Al classifier 280 for differentiating targets, according to an exemplary embodiment. The features are grouped in two sets: static trajectory (geometry) based features 260 and dynamic model features 270 used by Al classifier 280 for outputting a classified target 290.TABLE C

[0030] Al classifier 280 can, optionally, also use optical imagery of the target captured by video camera 110. In an exemplary embodiment, a pan-tilt-zoom camera can be used to visualize a target at long distances, and an optical classifier detector 230 and optical classifier tracker 240 can run in parallel and independently from radar classification based on feature extraction to augment the accuracy of the radar classification with visual data and signature of a target. For example, Al classifier 280 can provide coordinates to the camera 110 based on classification using radar data in order to focus the camera for optical detection and tracking via detector 230 and tracker 240.

[0031] System 100 uses the dynamics and geometric characteristics of the target trajectories for physics-based feature extraction by Al classifier 280, which applies a trained decision on the extracted features to determine the classification. As discussed below with reference to FIGs. 5 and 6, Al classifier 280 trains a decision tree for target classification.

[0032] FIG. 4 shows details of interactive multiple models (IMM) filter 210 for extracting features for classification using the set of motion models for capturing the complex behavior of a flying target and switching between models based upon probability. IMM filter 210 comprises aplurality of filters (FILTER 1 , FILTER 2... FILTER N) that are matched with respective motion models CV, CA, HCT, and 3DCT. During every sampling instant, the plurality of filters communicate information between each other due to switching. Each filter estimates the state at every sampling instance and the final state estimate is the combination of all state estimates of all the filters.

[0033] If M1, M2, M3...Mnare n models of I MM filter 210 and M<(k) is the model in use at time k, then for each model M1, state and observation equations can be expressed as follows:( fe) JF ,.p a - F a 1 * , 3, 4 where cu is the process noise, v is the measurement noise, F is the linear or nonlinear motion model and H is the measurement model. The overall system state vector is X = [x; vx; ax; y; vy; ay; z; vz; az; >]T; where the state vector consists of position, velocity, acceleration and turn rate.

[0034] At each time step, IMM filter 210 performs functions of mixing, filtering and combination, as follows.

[0035] For the mixing function, IMM filter 210 includes a mixer 212 for mixing the state estimates X^k-^k-1), X2(k-1\k-1) ... XN(k-1\k-1) provided by all filters FILTER 1 , FILTER 2... FILTER N from the previous time step, in order to set the initial conditions for the model- matched filter that is most suitable model at the current time step.

[0036] For the mixing function, IMM filter 210 filters each model using the mixed initial state X(k) and observation Z(k) as input and a linear or non-linear filter according to the motion models used to capture the complex flight behavior of the target. In embodiments, a Kalman filter (KF) can be used for CV and CA linear models, and an Extended Kalman filter (EKF) can be used for HCT and 3DCT nonlinear models. Each FILTER 1 , FILTER 2... FILTER N produces an estimate of state mean m^k) and covariance P(k) for each model M1as output and also calculates the likelihood of observation for each model to update the model probabilities iJ(k).

[0037] For the mixing function, a combiner 218 combines the state mean m(k) and covariance P(k) and outputs the extracted features 250, as follows:where X(k) = [x, vx, ax, y, yy, ay, z, vz, az, >]Tis the taget state at time step k, and contains the position, velocity, acceleration in X-Y-Z plane and the turn rate a>.

[0038] As discussed above, low-flying bird targets have better maneuverability than small UAVs, which influences the IMM model switching probabilities (.^(k)) such that that the IMM filter tracking for birds changes motion models frequently in comparison with the UAV flight.Therefore, as discussed above, IMM model probabilities play an important role in identifying flying targets. Based on the filtered results, the average model conversion frequency f is calculated for a target track as follows:where / jJ(k) is the model transition probability 215 of the jthmodel at a kthtime instance and L is the length of the target track. As a result of better maneuverability of flying bird targets, f is higher for birds than for drones.

[0039] Directional bearings, which is another significant feature of a target’s flight, can be measured using curvature, as follows:where prime denotes the differentiation with respect to time.

[0040] Al classifier 280 trains a decision tree 500, as shown in FIG. 5, for discriminating between UAVs, birds and other (ground targets) targets using features extracted by the IMM filter 210. The decision tree comprises a first stage tree 510 which is based on simple fuzzyrules that utilize kinematics features of the radar output tracks, and a second stage tree 520 derived by training the classifier 280 with training data comprising kinematics and IMM filter features, as discussed above.

[0041] The first stage decision tree 510 uses the height of the target to distinguish between inflight and ground targets. However, since drone take-off and landing have similar height profiles to ground features, Al classifier 280 can use data of the target captured by optical classifier detector 230 and optical classifier tracker 240 from camera 110 (with or without additional data such as target velocity, RCS, etc.) to identify a ground target 505.

[0042] The second stage decision tree 520 is trained using the extracted features 260 and 270 of each target class. The pairand Y(i>) represent sample data of the training set where,represents a 21 dimension feature vector extracted from Ithradar track and Y(i> e {1 ; 2; 3} is the class label associated with / Yhtarget track.

[0043] Consequently, the entire training dataset with N total sample data points corresponding to all target tracks, is given by:

[0044] The second stage decision tree 520 splits each node / leaf in such a way that each subbranch models specific maneuvers (curvilinear flight 530, hovering 540, straight flight 550) of the target based on the extracted features, namely velocity, acceleration and turn angles.Curvilinear flight 530 can be further modelled as loops 560, 3D turn 570, 2D turn 580 while straight flight 550 can be further modelled as elevation 590, speed 600, curvature 610.

[0045] The output of Al classifier 280 is a matrix with each row corresponding to a node and columns corresponding to node number, positive child node number, negative child node number, function used, split value, data size, and majority class, as shown in FIG. 6. For any input test track, classification is performed by checking the condition at each node and followingthe tree until the leaf node is reached, resulting in the label (i.e. bird or UAV).

[0046] Experimental testing confirms that the Al classifier 280 provides greater accuracy than Baysian inferencing methods (naive Baysian-NB) and SVM (Support Vector Machine) methods, as shown in TABLE D.TABLE D

[0047] A person of skill in the art will understand that the embodiments described can be used to enhance safety and security of drone operations and counter unauthorized drone traffic.

[0048] The many features and advantages of the invention are apparent from the detailed specification and, thus, it is intended by the appended claims to cover all such features and advantages of the invention that fall within the true spirit and scope of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope of the invention.

Claims

CLAIMS1 . A system for target classification, comprising: at least one sensor for detecting a point target; an interactive multiple model (IMM) filter for extracting features related to motion kinematics of the point target; and a classifier for applying trained decision-making on the extracted features to determine classification of the point target.

2. The system of claim 1 , further including a camera configured to be focused on the point target by the classifier for visual verification of the classification and in response updating classification accuracy of the classifier.

3. The system of claim 2, wherein the camera is a pan-tilt-zoom camera.

4. The system of claim 1 , wherein the extract features include velocity, acceleration and turn angle.

5. The system of claim 1 , wherein the sensor is one of a radar sensor, electro-optical sensor, LIDAR sensor or camera.

6. The system of claim 1 , wherein the interactive multiple model (IMM) filter switches between multiple motion models representing the features related to motion kinematics of the point target based upon probability.

7. The system of claim 6, wherein the multiple motion models include Constant Velocity (CV), Constant Acceleration (CA), Horizontal Coordinated Turn (HCT) and 3D Coordinated Turn (3DCT).

8. The system of claim 7, wherein the interactive multiple model (IMM) filter includes a plurality of filters (FILTER 1 , FILTER 2... FILTER N) that are matched with respective motion models CV, CA, HCT, and 3DCT for estimating the maneuvering state of the point target at successive sampling instances, a mixer for mixing state estimates provided by plurality of filters (FILTER 1 , FILTER 2... FILTER N) from a previous sampling instance to set the initial conditions for each of the plurality of filters that is a most suitable model at a current sampling instance, and a combiner that combines state mean m(k) and covariance P(k) and outputs the extracted features.

9. The system of claim 4, wherein the classifier trains a decision tree for discriminating between UAVs, birds and ground targets using the features extracted by the interactive multiple model (IMM) filter to determine classification of the point target.

10. The system of claim 8, wherein the decision tree comprises a first stage tree that uses the motion kinematics of the point target to distinguish between airborne targets and ground, and a second stage tree that splits each node / leaf such that each sub-branch models specific maneuvers of the point target based on the extracted features.11 . The system of claim 10, wherein the maneuvers include curvilinear flight, hovering and straight flight.

12. The system of claim 11 , wherein curvilinear flight further is modelled by loops, 3D turns and 2D turns and straight flight is modelled by elevation, speed, and curvature.

13. The system of claim 9, wherein the output of the classifier is a matrix with each row corresponding to a node and columns corresponding to node number, positive child node number, negative child node number, function used, split value, data size, and majority class, such for a given trajectory of the target object, classification is performed by checking the condition at each node of the second stage tree and following the resulting branches until a leaf node is reached, wherein the leaf node provides a label for classifying the target object as oneof either a bird or UAV.

14. A method for target classification, comprising: detecting a point target; extracting features related to motion kinematics of the point target using an interactive multiple model (IMM) filter; and applying trained decision-making on the extracted features to determine classification of the point target.

15. The method of claim 14, further including focusing on the point target for visual verification of the classification and in response updating classification accuracy.

16. The method of claim 14, wherein the extract features include velocity, acceleration and turn angle.

17. The method of claim 14, wherein the interactive multiple model (IMM) filter switches between multiple motion models representing the features related to motion kinematics of the point target based upon probability.

18. The method of claim 16, wherein the multiple motion models include Constant Velocity (CV), Constant Acceleration (CA), Horizontal Coordinated Turn (HCT) and 3D Coordinated Turn (3DCT).

19. The method of claim 17, wherein the interactive multiple model (IMM) filter includes a plurality of filters (FILTER 1 , FILTER 2... FILTER N) that are matched with respective motion models CV, CA, HCT, and 3DCT for estimating the maneuvering state of the point target at successive sampling instances, a mixer for mixing state estimates provided by plurality of filters (FILTER 1 , FILTER 2... FILTER N) from a previous sampling instance to set the initial conditionsfor each of the plurality of filters that is a most suitable model at a current sampling instance, and a combiner that combines state mean m(k) and covariance P(k) and outputs the extracted features.

20. The method of claim 14, wherein the trained decision-making includes training a decision tree for discriminating between UAV, birds and ground targets using the features extracted by the interactive multiple model (IMM) filter to determine classification of the point target.21 . The method of claim 20, wherein the decision tree comprises a first stage tree that uses the motion kinematics of the point target to distinguish between airborne targets and ground, and a second stage tree that splits each node / leaf such that each sub-branch models specific maneuvers of the point target based on the extracted features.

Images