Illegal unmanned aerial vehicle source tracing method and system based on fusion of visible light, thermal imaging and radio frequency fingerprints

Abstract

The invention discloses an illegal unmanned aerial vehicle source tracing method and system based on visible light, thermal imaging and radio frequency fingerprint fusion. The method comprises the following steps: synchronously acquiring target visible light, a thermal infrared image and a radio frequency signal; extracting appearance, thermal radiation and radio frequency fingerprint features in parallel; multi-modal features are dynamically weighted and fused through an adaptive cross-modal attention fusion network (ACMAF-Net), and unmanned aerial vehicle model identification and individual association are completed; reconstructing a three-dimensional track of the unmanned aerial vehicle by using multi-sensor observation data; the possible positions of a take-off point and an operator are deduced by combining track features and geographic information. The system comprises a multi-mode sensing subsystem, a data processing and fusion subsystem, a traceability analysis subsystem, a display control storage subsystem and the like. According to the invention, through three-mode complementary fusion, the limitation of a single technology is overcome, all-weather and high-precision detection identification and effective traceability of the illegal unmanned aerial vehicle are realized, and the low-altitude security and protection capability is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of low-altitude security and target monitoring technology, specifically relating to a system and method for detecting, identifying and tracing illegal drones by integrating multimodal sensor information. Background Technology

[0002] With the rapid popularization of drone technology, its applications in logistics, surveying, entertainment and other fields are becoming increasingly widespread. However, this has also brought serious problems such as illegal intrusion, privacy violations, and threats to public safety. Sensitive areas such as airports, military bases, and nuclear power plants are frequently subjected to illegal drone intrusions, urgently requiring efficient and accurate monitoring and countermeasures.

[0003] Currently, the main monitoring technologies for drones include:

[0004] 1. Radar detection: It has limited detection capabilities for low-altitude, slow-moving, and small targets ("low, slow, and small"), has a high false alarm rate, and cannot identify the target type and identity.

[0005] 2. Radio Spectrum Detection: This method detects and identifies drones by monitoring the communication signals between the drone and its remote controller. Its drawbacks include ineffectiveness against drones flying along preset routes or operating in silent conditions, and its accuracy is susceptible to environmental interference.

[0006] 3. Acoustic detection: Identification is achieved using the unique acoustic signature of the drone's rotor. However, its effective range is short, and its performance deteriorates sharply in complex urban noise environments.

[0007] 4. Photoelectric detection:

[0008] Visible light imaging: depends on ambient light and fails at night, in foggy or hazy conditions, or under strong backlight conditions.

[0009] Infrared thermal imaging: It can work at night, but its detection range for low-temperature targets is limited, and its image resolution is low, making it difficult to perform fine-grained identification.

[0010] Individual monitoring technologies, limited by their physical principles, all have insurmountable shortcomings and cannot meet the comprehensive monitoring needs of illegal drones in complex environments, requiring all-weather, high-precision, and traceable monitoring. Therefore, effectively integrating multi-source heterogeneous information to achieve complementary advantages has become the key to improving the overall performance of the system. Summary of the Invention

[0011] (a) Purpose of the invention

[0012] This invention aims to overcome the shortcomings of existing technologies and provide a method and system for tracing illegal drones based on multimodal fusion of visible light, thermal imaging, and radio frequency fingerprinting. This method achieves reliable detection, accurate identification, and trajectory tracking of illegal drones through deep fusion and intelligent processing of heterogeneous sensor information, ultimately tracing their origin to their takeoff point and the possible location of the operator, forming a complete chain of electronic evidence.

[0013] The core algorithm processing flow of this invention can be abstractly represented by the following functional relationship: ,in, These represent the simultaneously acquired visible light image, thermal infrared image, and radio frequency signal, respectively. These are the feature extraction functions for each modality; An adaptive cross-modal feature fusion function; This framework provides functions for trajectory reconstruction and source tracing inference. It constitutes a complete technological chain from multi-source perception to advanced decision-making.

[0014] (II) Technical Solution

[0015] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0016] A method for tracing the origin of illegal drones based on the fusion of visible light, thermal imaging, and radio frequency fingerprinting, characterized by the following steps:

[0017] S1: Synchronous acquisition of multimodal data

[0018] Using sensing devices deployed at monitoring points, visible light images, thermal infrared images, and radio frequency signals of the target are collected simultaneously.

[0019] S2: Parallel Extraction of Multimodal Features

[0020] S2.1: Preprocess and enhance the visible light image, use a deep learning-based target detection network to locate the UAV, and extract its appearance and structural feature vectors. The target detection network adopts an improved YOLOv7-tiny architecture, and its loss function... Defined as:

[0021]

[0022] in, Instruction No. Does each grid contain the target? To predict the center coordinates and width and height of the bounding box, For confidence level, This represents the category probability.

[0023] S2.2: Non-uniformity correction and enhancement are performed on the thermal infrared image. A semantic segmentation network is used to extract the UAV target region and analyze its thermal radiation distribution to extract thermal feature vectors. From the segmented thermal regions, the following key features are calculated. :

[0024] 1. Temperature statistical characteristics: ,in For pixel temperature, This represents the total number of pixels within the region.

[0025] 2. Coordinates of thermal centroid :

[0026] 3. Thermal distribution texture features: Calculate the Local Binary Pattern (LBP) histogram based on temperature gradient.

[0027] S2.3: The radio frequency (RF) signal is detected, down-converted, and pre-filtered. RF fingerprint feature vectors, representing the uniqueness of the device hardware, are extracted from the transient and steady-state signals, as well as through a deep network. The radio frequency fingerprint feature The extraction specifically includes:

[0028] 1. Transient characteristics: The rising edge of the signal amplitude envelope is subjected to exponential fitting to extract the time constant.

[0029] 2. Steady-state characteristics: Calculate the error vector magnitude (EVM) to characterize the modulation quality.

[0030] in, For actual sampling points, For the ideal constellation point, This represents the average signal power.

[0031] 3. Deep features: The preprocessed I / Q signal sequence is input into a one-dimensional convolutional neural network (1D-CNN), which automatically learns and outputs high-dimensional discriminative features.

[0032] S3: Adaptive Fusion of Multimodal Features

[0033] The feature vector extracted in step S2 , , The input is fed into an adaptive cross-modal attention fusion network (ACMAF-Net). This network first calculates the correlation between features of different modalities through a cross-attention mechanism, and then dynamically generates fusion weights by combining real-time environmental information (such as light intensity and weather) with the recognition confidence of each modality. Weighted fusion is performed to output a unified fusion feature. The adaptive weights are used for final target classification and recognition. Calculated using the following formula: ,in, For the first Real-time recognition confidence of each modality, Assess the environmental suitability for the mode (e.g., the ratio of light intensity to visible light). and These are learnable parameters that control the weights of confidence and environmental influence. The core calculation of the cross-attention mechanism is as follows: , where the query matrix Key matrix Value matrix They are obtained by linear projection of the feature vectors of different modes. is the dimension of the key vector.

[0034] S4: Unmanned Aerial Vehicle (UAV) Identification and Trajectory Reconstruction

[0035] S4.1: Based on fusion features Classify drone models and associate individual identities (matching with the radio frequency fingerprint database).

[0036] S4.2: Utilizing observational information such as azimuth, pitch, and time difference of arrival (TDOA) acquired by multiple sensors (multiple monitoring nodes), the three-dimensional spatial state (position, velocity) of the UAV is estimated and smoothed using a geometric positioning model and the Extended Kalman Filter (EKF) algorithm, reconstructing its high-precision flight trajectory. The geometric positioning model is based on the TDOA (Time Difference of Arrival) principle, and its equations are as follows: ,in, Coordinates of the drone For the first Coordinates of each monitoring station To achieve the goal of The distance between the stations At the speed of light, For the signal to arrive at the Stations and Reference Stations Time difference, The core recursive equation of the Extended Kalman Filter (EKF) algorithm is as follows:

[0037] 1. State prediction:

[0038]

[0039]

[0040] 2. Measurement Update:

[0041]

[0042]

[0043]

[0044] in, This is the predicted state value. Let be the error covariance matrix. Here is the state transition matrix. These are the covariance matrices of process noise and observation noise, respectively. For Kalman gain, For observation models, Its Jacobian matrix, These are actual observed values.

[0045] S5: Deduction of takeoff point and operator position

[0046] S5.1: Based on the reconstructed flight trajectory origin features and combined with digital elevation model (DEM) data, the most likely takeoff point is inferred under the constraint of a flat ground area.

[0047] S5.2: Establish a maximum likelihood model for operator position inference. This model comprehensively considers the UAV flight trajectory pattern (e.g., hovering, flight path), the line-of-sight constraint between the operator and the UAV (based on GIS terrain data), and the possible remote control signal strength (RSSI) attenuation model. It searches for the position that maximizes the conditional probability within the potential field of view and outputs the confidence region where the operator may exist. The maximum likelihood estimation problem is formally defined as: .in, The desired operator position. This is the reconstructed sequence of drone trajectory points. Conditional probability. Modeling was performed at a known operator position. and historical trajectory Under these conditions, the current location of the drone The model incorporates a line-of-sight function to indicate the likelihood of occurrence. With probability weights based on flight patterns.

[0048] S6: Comprehensive Source Tracing Report Generation and Early Warning

[0049] By integrating the drone model / individual ID, 3D flight trajectory, inferred takeoff point, and operator confidence area information output from the above steps, a structured electronic traceability report is generated. Simultaneously, real-time warnings are triggered based on preset rules, using methods such as audio-visual alerts and network monitoring.

[0050] An illegal drone tracing system implementing the above method, characterized in that it includes:

[0051] Multimodal sensing subsystem: includes a visible light camera module, a thermal imaging camera module, and a software-defined radio (SDR) radio frequency receiver module, used for synchronous acquisition of raw data.

[0052] Data processing and fusion subsystem: includes an edge computing unit, and is equipped with a visible light image processing unit, a thermal image processing unit, a radio frequency signal processing unit, and the core ACMAF-Net fusion algorithm, which are used to execute steps S2 and S3.

[0053] Source analysis subsystem: running trajectory reconstruction algorithm (such as EKF) and takeoff point / operator inference model, used to perform steps S4 and S5.

[0054] Data storage and display control subsystem: includes a database server and a monitoring terminal, used to store the radio frequency fingerprint database and historical trajectory data, and to realize system status monitoring, visualization of traceability results and release of early warning information.

[0055] (III) Beneficial Effects

[0056] Compared with the prior art, the present invention has the following significant advantages:

[0057] 1. Strong environmental robustness: It integrates three modes: visible light (daytime advantage), thermal imaging (nighttime / adverse weather advantage), and radio frequency (non-line-of-sight / communication advantage), realizing true all-weather, all-time domain monitoring.

[0058] 2. High accuracy in identification and tracing: By introducing radio frequency fingerprint features with unique hardware characteristics, a leap has been achieved from "target discovery" to "identifying specific individual devices," providing direct electronic evidence for judicial evidence collection. Combined with multi-source positioning and intelligent inference models, it can effectively trace back to the takeoff point and the operator's area.

[0059] 3. Intelligent adaptive fusion: The adopted ACMAF-Net network can dynamically adjust the fusion strategy according to environmental changes and the data quality of each modality, ensuring that the system always works in the optimal or near-optimal state.

[0060] 4. Good passive detection concealment: The system mainly receives signals (light, heat, radio waves) radiated or emitted by the target itself, making it difficult to be detected by UAV counter-reconnaissance equipment and highly concealed in deployment.

[0061] 5. Flexible system expansion: The modular design allows for flexible configuration of the number and type of sensor nodes according to specific scenario requirements and budget, and it is easy to integrate with existing security platforms. Attached Figure Description

[0062] Figure 1 This is a block diagram of the overall architecture of the system of the present invention.

[0063] Figure 2 This is a flowchart illustrating the working principle of the method of the present invention.

[0064] Figure 3 A schematic diagram of the U-Net network structure for segmenting hot targets.

[0065] Figure 4 This is a diagram of the structure of a one-dimensional convolutional neural network (1D-CNN).

[0066] Figure 5 This is a flowchart of the radio frequency fingerprint extraction process.

[0067] Figure 6 This is a schematic diagram of the Adaptive Cross-Modal Attention Fusion Network (ACMAF-Net) structure.

[0068] Figure 7 This is a schematic diagram illustrating the principle of multi-node collaborative localization (TDOA / AOA).

[0069] Figure 8 Example visualization of trajectory reconstruction and takeoff point inference results.

[0070] Figure 9 This is a schematic diagram of the hardware structure of an integrated sensor tower.

[0071] Figure 10 This is a system software architecture diagram. Detailed Implementation

[0072] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. This description is intended to enable those skilled in the art to implement the present invention and is not intended to limit its scope of protection.

[0073] Example: Airport Airspace Protection Zone Monitoring System

[0074] In this embodiment, the system is deployed in the airspace protection zone of an international airport.

[0075] Hardware deployment and integration:

[0076] Reference Figure 1 and Figure 8 Four monitoring points with wide field of view and coverage of key directions were selected around the airport, and an integrated sensor tower was deployed at each point. The sensor tower integrates:

[0077] Visible light module: A 4K resolution, image-stabilized camera with 30x optical zoom.

[0078] Thermal imaging module: a 640×512 resolution, NETD<50mK uncooled long-wave infrared thermal imager.

[0079] Radio frequency module: A software-defined radio device with an operating frequency band covering 70MHz-6GHz, used for detecting common frequency bands of drones (such as 2.4GHz, 5.8GHz).

[0080] Computing and Communication Unit: Built-in high-performance edge computing box (equipped with NVIDIA Jetson AGX Orin module), GPS / BeiDou timing module, gigabit fiber optic network interface and backup power supply.

[0081] Each sensor tower is connected to the airport security command center via a dedicated airport fiber optic ring network and uses the PTP protocol to achieve microsecond-level time synchronization.

[0082] Software system and algorithm flow:

[0083] Reference Figure 2 and Figure 9 The system software is built on the ROS2 framework, achieving loose coupling and modularity.

[0084] 1. Data Acquisition and Preprocessing: Data is acquired synchronously by each sensor driver module. Visible light and thermal imaging video streams undergo distortion correction and spatiotemporal alignment. Radio frequency signals undergo down-conversion, filtering, and digitization.

[0085] 2. Feature extraction:

[0086] Visual features: A lightweight YOLOv7-tiny network is used to detect drone targets in videos in real time, and a 128-dimensional appearance feature vector is extracted from the target region using a MobileNetV3 network. .

[0087] Thermal characteristics: U-Net network is used (see structure) Figure 3 Semantic segmentation is performed on the thermal image to extract the thermal region of the UAV, and then its thermal centroid, average temperature, temperature standard deviation, etc. are calculated to form a 64-dimensional feature vector. .

[0088] Radio frequency fingerprint: see process Figure 10 First, the signal is detected, then transient features (such as envelope rising edge fitting parameters), steady-state features (such as EVM, I / Q offset, spectral asymmetry), and features are simultaneously extracted using a 1D-CNN (structure see...). Figure 4 The extracted deep features together constitute a 256-dimensional radio frequency fingerprint feature vector. .

[0089] 3. Adaptive fusion and recognition: [This will] integrate and recognize... , , Input ACMAF-Net network (structure see...) Figure 5The network dynamically calculates fusion weights based on current light sensor readings (determining visible light confidence) and radio frequency (RF) environmental noise assessments (determining RF confidence). For example, the weight allocation for nighttime might be: thermal imaging (0.5), RF (0.4), and visible light (0.1). The fused features are used for the final decision, outputting target presence, model (e.g., "DJI Mavic 3"), and RF fingerprint matching results (e.g., "matching database ID: UAV_015").

[0090] 4. Cooperative localization and trajectory generation: When a target is detected simultaneously by at least two sensor towers, the system initiates cooperative localization. For example... Figure 6 As shown, the 3D coordinates of the target are calculated using the time difference of signal arrival at each tower and the TDOA algorithm. An extended Kalman filter is then used to smooth the coordinate sequence, and a high-precision 3D trajectory is plotted on the geographic information system.

[0091] 5. Source tracing and inference: Based on trajectory data over a period of time (see...) Figure 7 The system automatically analyzes:

[0092] Takeoff point: Tracing back to the starting point of the trajectory, find a flat area on the ground digital elevation model that is in a direction that is reasonable with the tangent of the trajectory, and give a confidence ellipse.

[0093] Operator Area: Combining the terrain occlusion model, assuming that the operator needs to maintain line-of-sight communication with the drone, within the intersection area of ​​the visible fields of each point on the trajectory, the range is further narrowed down according to the drone's flight pattern (such as hovering points often being close to the operator) to generate a possible activity area.

[0094] 6. Early Warning and Evidence Collection: Once a target is identified as an illegal intrusion (e.g., not on the reporting list), the system will highlight the target, trajectory, and inferred area on the command center's large screen, issue an audible and visual alarm, and automatically generate an encrypted electronic evidence package containing all feature data, images, and inference results for subsequent handling and possible legal proceedings.

[0095] Performance verification:

[0096] In simulated tests, the system achieved an average detection rate of 96.5% for five common drones under various conditions, including day, night, and fog, a model identification accuracy of 95.2%, and an individual RF fingerprint association accuracy of 89.7%. With a three-node network, the positioning accuracy was better than 5 meters (RMS), and the takeoff point inference error (CEP50) was less than 50 meters.

[0097] The above embodiments fully illustrate the technical feasibility and superiority of the present invention. Those skilled in the art can adjust the sensor configuration, number of nodes, and algorithm parameters according to different application scenarios (such as urban security, border patrol, and nuclear power plant protection), and all such adjustments should fall within the protection scope of the present invention.

Claims

1. A method for tracing the origin of illegal drones based on the fusion of visible light, thermal imaging, and radio frequency fingerprinting, characterized in that, Includes the following steps: S1: Simultaneously acquire visible light images, thermal infrared images, and radio frequency signals of the target area; S2: Extract the visual appearance feature vectors of the UAV from the visible light image, thermal infrared image, and radio frequency signal, respectively. thermal radiation distribution characteristic vector and radio frequency fingerprint feature vector ; S3: Transfer the feature vector , and The input is fused into an adaptive cross-modal attention fusion network to obtain fused features. And based on this fusion feature, the detection and identification of the drone are completed; S4: Reconstruct the three-dimensional flight trajectory of the UAV based on multi-sensor observation data; S5: Based on the three-dimensional flight trajectory and combined with geographic information system data, infer the takeoff point of the UAV and the possible location area of ​​the operator; S6: Generate a multimodal comprehensive traceability report that includes drone identity information, flight trajectory, takeoff point, and operator area.

2. The method according to claim 1, characterized in that, In step S2, the extraction of the radio frequency fingerprint feature vector includes: after preprocessing the radio frequency signal, extracting the transient features of the signal, including extracting the time constant based on exponential fitting of the envelope rising edge. Extracting steady-state features includes calculating the magnitude of the error vector. The deep features are extracted using a one-dimensional convolutional neural network; the multiple features are then fused and dimensionality reduced to form the radio frequency fingerprint feature vector. .

3. The method according to claim 1, characterized in that, In step S3, the adaptive cross-modal attention fusion network uses a cross-attention mechanism. The correlation between different modal features is calculated, and based on real-time environmental information and the recognition confidence of each modality, a weighting formula is used. Dynamically generate fusion weights and perform weighted fusion.

4. The method according to claim 1, characterized in that, Step S4 specifically includes: utilizing the Time Difference of Arrival (TDOA) observation information obtained from multiple monitoring nodes, and applying the distance difference equation... A geometric positioning model is established to calculate the position of the UAV, and the extended Kalman filter (EKF) algorithm is used to continuously estimate and smooth the position and velocity state of the UAV through state prediction and measurement to update the recursive equation, thereby reconstructing the three-dimensional flight trajectory.

5. The method according to claim 1, characterized in that, In step S5, the inference of the takeoff point specifically involves: based on the starting point of the reconstructed trajectory, finding the most likely point that satisfies the trajectory's initial motion characteristics within the flat ground area constrained by the digital elevation model (DEM); the inference of the possible location area of ​​the operator specifically involves: establishing a maximum likelihood model with the operator's position as a latent variable. This model integrates the flight trajectory pattern and terrain occlusion line-of-sight constraints, and searches for the area within the potential field of view that maximizes the conditional probability of the trajectory.

6. An illegal drone tracing system for implementing the method of any one of claims 1-5, characterized in that, include: The multimodal sensing unit includes a visible light camera, a thermal imaging camera, and a software-defined radio (SDR) device for synchronous data acquisition; An edge computing processing unit, connected to the multimodal sensing unit, is used to run image processing, radio frequency signal processing, and feature extraction algorithms; The central processing and fusion unit is used to run adaptive cross-modal attention fusion networks, trajectory reconstruction algorithms, and source inference models. The data storage and display control unit is used to store the radio frequency fingerprint database and historical data, and provides a human-machine interface and early warning information output.

7. The system according to claim 6, characterized in that, The multimodal sensing unit and edge computing processing unit are integrated into an integrated sensing tower, which includes a protective shell, a radio frequency antenna array and meteorological sensor mounted on the top, a three-axis stabilized gimbal that carries visible light and thermal imaging cameras in the middle, and a computing and communication module cabin built into the bottom.

8. The system according to claim 6, characterized in that, The system software architecture is built on the robot operating system ROS2 and adopts a distributed microservice design, including device driver nodes, perception algorithm nodes, fusion decision nodes, and human-machine interaction nodes. The nodes communicate with each other through the data distribution service DDS.

9. The system according to claim 6, characterized in that, It also includes a collaborative monitoring network consisting of multiple traceability system nodes, with each node synchronizing time and communicating data with each other via wired or wireless networks to jointly complete the positioning and trajectory tracking of UAVs in a wide area.

Images