Drone detection and early warning system

Abstract

The invention relates to the drone detection and early warning system providing real¬ time and automatic drone detection and warning to users, using the camera images obtained by users through the security system for perimeter surveillance.

Description

[0001] DRONE DETECTION AND EARLY WARNING SYSTEM

[0002] Technical Field of the Invention

[0003] The invention relates to the drone detection and early warning system providing realtime and automatic drone detection and warning to users, using the camera streams obtained by users through the security system for perimeter surveillance.

[0004] State of the Art

[0005] Today, there are various drone (unmanned aerial vehicle) detection systems. These systems can use one or more of the radar, RF signal scanner, LIDAR, camera, acoustic (over sound) systems to detect drone or drones in the sky. These systems monitor the presence of drones by collecting signals from a certain area in the sky or by sending and receiving signals. The monitorable area and the correct detection quality may vary according to conditions such as the monitoring range capability of the system, weather, light, electronic and magnetic wave presence and its intensity. When the relevant systems detect at least one drone, they provide an audible and / or visual warning to the relevant security units. In this way, a possible drone presence in the protected area is automatically detected and a warning is created by means of a system.

[0006] Application number "US2015302858" in the state of the art involves receiving a sound signal through a microphone and recording a digital sound sample of the sound signal through a sound card. The invention involves the implementation of broad spectrum matching via the processor to compare the feature frequency spectrum with at least one drone sound signature stored in a database. The invention operates using sound and operates according to the method of matching the sounds it receives with a database of drone sounds. It is understood that it is made with conventional methods and does not use artificial intelligence. On the other hand, it is understood that the proposed system is within the database constraint and will be limited and insufficient to operate against new generation or voice-manipulated drones with a sound spectrum outside of predefined range of frequencies and in cases where noise as an external sound is present. Application number "WO2018044231" in the state of the art is has a centralized management software to manage the detection, interception and seizure of control of the drones using a scanner, a GPS spoofer, and an RF jammer. The invention relates to a signal scanner with a multi-antenna array supporting 433 MHz, 2.4 GHz and 5.0-5.8 GHz scanning frequencies, a GPS, a radio frequency (RF) jammer and a host system on a Linux-based operating system. In this invention, the drone is also required to emit radio frequency. In an example of an autonomous task, a drone performs its missions automatically and that minimizes emitted radio frequency during the mission. As it is not remotely controlled in an automatic mission, therefore it can be difficult to detect by RF sensors. Conversely, urban areas or locations with high levels of electromagnetic interference can also present challenges by increasing the complexity of detection.

[0007] In the current state of the art, existing studies have developed either single detection systems or drone detection and alarm systems by combining multiple systems. Depending on the decision-making logic in compact systems, the error of one sensor in detection may overshadow the correct detection of other sensors if other systems depend on initial system’s decision. On the other hand, with such compact and complex systems, there can be barriers to deployment with existing installation structures. A system designed to scan large areas should be easy to deploy. For this reason, it is crucial to develop a drone detection system that analyzes data directly from imagery, such as from a camera, and remains unaffected by signal interference.

[0008] Summary and Objects of the Invention

[0009] The invention relates to the drone detection and early warning system that offers realtime and automated drone detection and alerts to users. This system utilizes camera images obtained through the security system for perimeter surveillance.

[0010] An object of the invention is to enhance energy efficiency. The system processes images from the cameras to detect drones and issuewarnings via connected modules. It consumes less energy than high-energy solutions like radar, RF, and LIDAR, thus making it possible to work with limited energy resources, whether at fixed locations and on mobile platforms such as land vehicles, yachts, and ships. In addition, by designated the multimedia computer as a remote server, energy consumption at the detection site can be further minimized.

[0011] Another objective of the invention is to incorporate customizable alarm mechanisms. Unlike radar and RF-based systems that operate within the frequency range of drones, this system directly analyses the image of the object. By classifying the image, it allows for the creation of additional early warning and alarm rules. These rules can be extended to other objects such as birds, airplanes and helicopters, and can be adjusted by the user as needed.

[0012] Another object of the invention is to ensure that it is easily scalable. Since the system does not depend on any other hardware other than the multimedia computer for the execution of the main task, it can scale linearly depending on the camera density over the area (e.g.: campus, premises, site, infrastructure) where the task needs to be performed as it receives video streams over the network from the multimedia computer (this does not occur when it is positioned as a remote server) and the cameras already installed. This makes it easier to deploy compared to system approaches that offer monolithic and multiple hardware dependencies. The same ease of scaling applies to relatively small task areas. (E.g.: small and medium-sized boats, private properties)

[0013] Another object of the invention is to enable automatic data labeling. The system automatically records the detections and classifications during task in order to improve the artificial intelligence-supported object detection and classification systems it uses and to provide sustainable service by adapting to new environment and object conditions more easily, and provides direct operational efficiency to system improvement studies by minimizing the high time and labor costs of data labeling.

[0014] Another object of the invention is to enable effective execution of tasks at close range and without emitting signals. Since the system makes detection based on the analysis of the direct incoming video streams , it is not affected by the disadvantages of systems that work on electromagnetic signal analysis (complex electromagnetic signal analysis, sensor fusion applications, signal database and range, difficulty in detection at close range and low altitude, difficulty in working in dense electromagnetic signal areas (e.g.: cities, public areas, metal-based mining sites)). Another object of the invention is to enable cooperation with present camera systems and to eliminate the need for a physical driver for camera positioning.

[0015] Another object of the invention is to log both visual and metadata for each detection and classification, utilizing a data recording module to enhance system performance and enable continuous improvement through future system updates.

[0016] Another object of the invention is to ensure cost efficiency. The system offers a significant cost advantage by avoiding high-tech signal propagation and detection-based systems, instead utilizing existing camera hardware as a primary data collection tool.

[0017] Description of the Drawings

[0018] Figure 1. The drawing showing the schematic view of the system of the invention.

[0019] Description of the References in Figures

[0020] 1 . Multimedia computer

[0021] 2. Data coordination module

[0022] 3. Artificial intelligence object detection module

[0023] 4. Visual template matching module

[0024] 5. Object tracking module

[0025] 6. Object classification module

[0026] 7. Camera positioning module

[0027] 8. Multiple warning module

[0028] 9. Data recording module

[0029] 10. System monitoring module

[0030] 20. Camera

[0031] 30. User monitor

[0032] Detailed Description of the Invention

[0033] The invention relates to the drone detection and early warning system that provides real-time and automatic detection and alerts to users by utilizing camera images from security systems used for perimeter surveillance. The invention provides drone detection (as well as birds, helicopters, and aircrafts) along with early warnings for individuals and legal entities seeking to monitor their airspace using standard cameras.lt enables early detection and delivers audible and visual alerts to prevent potential illegal activities caused by a drone (UAVs-unmanned aerial vehicles) or multiple drones.

[0034] The multimedia computer (1 ) is equipped with a processor that instantly processes the images from the cameras (20) and provides audible and visual warnings to the user. All modules operate on the multimedia computer (1 ).

[0035] The data coordination module (2) ensures that the live streams from the cameras (20), designated by the user for surveillance and drone detection and configured through the interface, are routed to the subsequent modules individually and without disrupting their sequence.

[0036] The artificial intelligence object detection module (3) analyzes each image frame transmitted from the data coordination module to classify and detect within a restrictive visual bounding box and its coordinates on the given frame, provided the object belongs to a class defined during model training. The artificial intelligence object detection module (3), not being limited thereto, can operating on individual frames for object detection, as well as on sequentially fed video frames, detecting flying objects and visually marking them in bounding boxes.

[0037] The visual template matching module (4) ensures continuity in detection by conducting a new visual search over a broad area around the last detected location when the artificial intelligence object detection module (3) fails to detect in the subsequent frames due to varying lighting, background, weather conditions, or other visual variables of the object to be detected. This process works on the successive images processed by the artificial intelligence object detection module (3), analyzing a specific number of previously detected object frames each time. The method is triggered when the total number of previously detected objects in new image frames decreases; and the detected objects from the last set of frames are enclosed within restrictive coordinates, cut from the image, and overlaid on the frame that triggered the module. These cut objects are scanned for similarity within the visual area where they were last detected, and the most matching similar visual region and object depiction are identified. This process is repeated in the remaining cut objects, and the region of the rediscovered objects are marked within restrictive coordinates, as done by the artificial intelligence object detection module (3).

[0038] The object tracking module (5) enables determining whether the object detections provided by the artificial intelligence object detection module (3) and the visual template matching module (4) pertain to the same object. It tracks the object’s movement by inferring its next location based on its velocity, position, and acceleration, ensuring stable tracking throughout sequences of video frames.

[0039] The object classification module (6) estimates the probability of object’s class using advanced artificial intelligence that accounts for temporal dependencies withing the image sequence, which may consist of one or more frames from the artificial intelligence object detection module (3). To determine the object’s class, sequential image frames - cropped from the original image frame and contain only relevant sequences- are fed into a deep learning artificial intelligence architecture trained to handle temporal visual information. This deep learning model has been previously trained with similar data. Unlike a single object image, the data in this structure transmitted to the trained model provides more and more efficient information for classification as it also includes the distinctive characteristic temporal movements for more accurate classification. By using continuous temporal visual object data sequences, the object classification module (6) enhances its ability to classify non-drone flying objects such as airplanes, helicopters, and birds.

[0040] The camera positioning module (7) transmits the orientation and zoom-in or zoom-out commands to the camera (20) which is capable of moving up, down, left, right, and adjusting zoom, as specified by the user for drone surveillance. After the object classification module (6) identifies the class as a drone, the camera positioning module (7) aims to maintain the detected drone in the center of the image frame. The camera positioning module (7) adjusts the camera based on the drone’s position by issuing commands to move the frame up or down, or left to right, and adjusts the zoom according to the measured area. If multiple drones are detected in the frame simultaneously, the module retrieves orientation commands from the camera (20) and issues a zoom-out command to the camera (20) if a new drone is detected while it is already being oriented toward another drone. The module’s operations can be customized to fit various scenarios and are not limited to the examples provided. The multiple warning module (8), upon detecting and classifying at least one drone, triggers both an audible and visual alarms on the user monitor (30) through the visual interface of the current user monitoring and listening unit over the network. It notifies the user with information such as the real-time detection feed, the identified target, and the specific name of the camera (20) unit, among other possible details.

[0041] The data recording module (9) logs the detection coordinates and estimated object classes for each image source. The data is used to enhance the system's advanced artificial intelligence modules for detection and classification, while also minimizing the need for manual class labeling of image frames and manually identified object coordinates. This streamlines the process of classifying objects as either drone or nondrone.

[0042] The system monitoring module (10) tracks the temperature, pressure, humidity levels of the hardware components on the multimedia computer (1 ) running the system. It conducts consistency, quality, and reliability tests using unit tests during the live operation of the aforementioned modules. Additionally it monitors the power and network data, as well as multimedia data transmitted from the system, performing periodic checks to ensure overall system reliability. It also provides a configuration interface for the cameras (20) supplying video streams to the system, and its functionality extends beyond these operations.

[0043] The multimedia computer (1 ) interfaces with the user monitor (30) and the camera (20), enabling the transmission of the camera (20) streams to its internal modules. The multimedia computer (1 ) sends orientation commands to the camera (20) and system setting and monitoring tools, and delivers audible and visual alerts related to detections and classifications to the user monitor (30).

[0044] In the system, the multimedia computer (1 ) is installed at the location where the airspace monitoring against drones is required (e.g., facility, structure, mobile vehicle, or region) or can be placed remotely, accessible via a network. The connection details of existing cameras (20) associated with the system or new cameras to be added, are configured through the user monitor (30) with the assistance of the system monitoring module (10) which includes an interface for the multimedia computer (1 ) over the network. Once configured, the system begins drone detection and monitoring for the defined image sources. The images received from the cameras (20) are processed individually by the data coordination module (2). For each image source, the workflow proceeds as follows: images from one or more camera (20) sources are coordinated by the data coordination module (2) and transmitted separately to the subsequent modules for further processing. The image frames from the live video stream are processed by the artificial intelligence object detection module (3), where flying objects are detected. During detection, objects are enclosed within bounding rectangles based on their coordinate. For objects that can evade detection by artificial intelligence object detection module (3), the visual template matching module (4) applies template matching to ensure continuity in detection. To maintain the trace of detected objects and confirm they are the same across frames, the object tracking module (5) links discrete detections between image frames. The object classification module (6) then predicts the class of the detected and tracked object to identify its type. If the object or objects that are detected, tracked and classified is a drone, the camera (20) receives directional commands from the camera positioning module (7) for physical tracking, while audible alert and a visual feed of detection are simultaneously sent to the monitoring interface via the multiple warning module (8). In this way, the system provides instant drone detection and warning task. At the same time, the system records detection and classification estimations in the background, gathering data for future system improvements through the data recording module (9). In parallel to all these, the system monitoring module (10) continuously verifies the reliability of the system, ensuring that all the operational units are functioning correctly and in compliance with standards.

Claims

CLAIMS1 . Drone detection and early warning system, characterized in that it comprises;- at least one data coordination module (2) processing the live streams coming from the user-provided cameras (20) for surveillance and detection against drones and whose connection information is defined to the multimedia computer (1 ) via an interface, ensuring uninterrupted transmission to the artificial intelligence object detection module (3); at least one artificial intelligence object detection module (3) separating image frames from the live video stream provided by the data coordination module (2) and detects flying objects; a visual template matching module (4) ensuring continuity in detections by performing a new visual template matching over a wide search area at the last known detection location in case the artificial intelligence object detection module (3) fails to detect in sequential frames due to variations in lighting, background, weather conditions and visual variables of the object to be detected; an object tracking module (5) enables determining whether the object detections provided by the artificial intelligence object detection module (3) and the visual template matching module (4) belong to the same object, tracking with positional inference and verification regarding the next location of the object based on its velocity, position, and acceleration in the image for stable tracking of the object throughout video sequence; at least one object classification module (6) estimating the probability of object class using artificial intelligence by analysing the sequence and, taking into account the temporal dependency, in order to determine the identity of the detected object that coming from the artificial intelligence object detection module (3); at least one camera positioning module (7) that transmits orientation, zoom-in and / or zoom-out commands to the camera to track the object classified as drones by the object classification module (6); a multiple warning module (8) issuing audible and visual alarm through a visual interface to alert users when objects are classified as drones by the object classification module (6); at least one data recording module (9) logging the detection coordinates and estimated object classes separately by image source, in order to minimize the need for manual class labeling of image frames, to be used for improving theartificial intelligence object detection module (3) and object classification module (6) for drone or non-drone classes; at least one multimedia computer (1 ) periodically checking the system reliability, comprising at least one system monitoring module (10) that controls the temperature, pressure, and humidity values of the multimedia computer’s (1) hardware components, communicating with the camera (20), managing the operation of the system’s modules with the processor thereon which enables the images coming from the camera (20) to be transferred to other modules and processed, and processing the camera images in real time, providing audible and visual warnings to the users.- at least one camera capturing images of the area where it is located (20)- a user monitor (30) providing the end user with audible and visual alert s based on the detections and classifications from the multiple warning module (8), and allowing the user to define at least one camera (20) within the system using the configuration tools provided by the system monitoring module (10).

2. A drone detection and early warning system according to claim 1 , characterized in that it comprises multiple warning modules (8) with an interface allowing the user to view the stream of instant detection, the target and the specific name of the camera unit.

3. A drone detection and early warning system according to claim 1 , characterized in that it comprises a camera positioning module (7) that halts orientation commands to the camera in case multiple drones are detected simultaneously, and issues a zoom-out command in case a new drone is detected while already sending orientation commands for another drone to the camera.

4. A drone detection and early warning system according to claim 1 , characterized in that it comprises an object classification module (6) that is capable of identifying the class of non-drone flying objects.

5. A drone detection and early warning system according to claim 1 , characterized in that it comprises an artificial intelligence object detection module (3) thatvisually marks detected flying objects by enclosing them within a bounding box areas.

Images