Unmanned aerial vehicle identification early warning method and system

Abstract

The invention provides an unmanned aerial vehicle identification early warning method and system. The method comprises the following steps: acquiring RGB image data, thermal radiation data and spectral data of an unmanned aerial vehicle no-fly zone through a visible light camera, an infrared thermal imager and a multispectral imager; performing transmission preprocessing on the RGB image data, the thermal radiation data and the spectral data, and adaptively adjusting multi-level fusion of a fusion weight based on real-time environmental parameters to generate target fusion data; based on a deep learning model and a tracking prediction algorithm, performing unmanned aerial vehicle identification early warning on the target fusion data, and generating early warning data; and transmitting the target fusion data and the corresponding abnormal event log to a cloud server, and updating the deep learning model by adopting the target fusion data and the abnormal event log. Through cooperative work of a multi-mode sensor, visible light, infrared, multispectral and other wave bands are covered, all-weather and full-scene unmanned aerial vehicle detection is achieved, the fusion weight is adjusted in real time based on real-time environment parameters, and the accuracy of the recognition result under the complex air situation is ensured.

Description

Technical Field

[0001] This invention relates to the field of airspace management technology, and in particular to a method and system for identifying and issuing early warnings for unmanned aerial vehicles (UAVs). Background Technology

[0002] Traditional drone monitoring methods have significant limitations: manual observation is constrained by environmental factors such as day-night cycles and inclement weather, making it difficult to achieve 24 / 7 coverage, and is also costly and slow to respond; while early radar systems have wide-area detection capabilities, they inherently suffer from low target resolution and high false alarm rates, and are particularly difficult to accurately detect micro-drones with wingspans of less than 1 meter. In addition, single-sensor solutions (such as relying solely on visible light cameras or infrared thermal imagers) are inadequate for meeting the needs of complex air situation control due to poor environmental adaptability and limited information dimensions.

[0003] Currently, the mainstream drone identification and early warning technologies mainly include the following categories, but each has its own application limitations:

[0004] Radar detection technology utilizes radar systems to emit electromagnetic waves, and based on the principle of drone's airframe reflecting these waves, it achieves detection and position measurement, acquiring information such as target distance, altitude, azimuth, and speed. However, the "low, slow, and small" characteristics of drones—low flight altitude, slow speed, and small size—result in poor performance of traditional radar systems during detection. Even though new dedicated radars have improved performance in recent years through signal design, signal processing algorithms, and ground clutter suppression, they still face challenges such as difficulty in capturing micro-drones and a high false alarm rate in complex environments.

[0005] Radio signal monitoring technology: This technology detects drones by analyzing the radio signals emitted by their flight control and image transmission systems. Single-station direction finding can obtain the drone's azimuth and roughly estimate the distance, while multi-station time difference positioning technology can accurately calculate the position and obtain the drone's unique radio "fingerprint information" to build blacklists and whitelists for control. However, this technology relies on the drone continuously emitting signals, making it difficult to detect drones flying silently or using frequency hopping communication. In non-line-of-sight or obstructed scenarios, the effective detection range is significantly reduced.

[0006] Photoelectric identification and tracking technology is divided into visible light identification and tracking and infrared identification and tracking. Visible light identification and tracking uses visible light cameras to acquire images, which is suitable for daytime use and has a lower equipment cost, but the identification effect drops significantly at night or in rainy or foggy weather. Infrared identification and tracking uses the thermal radiation of the drone's power system and electronic components to achieve detection, and can work in all weather conditions, but it cannot obtain detailed features for model identification, and the equipment cost is high. Single sensor solutions cannot meet the detection needs in complex environments.

[0007] Sound monitoring technology: Based on the noise generated by the rotation of drone propellers, sound signals are collected through a microphone array, and voiceprint features are extracted and matched with a database to achieve monitoring and model identification. As a passive monitoring method, it has good concealment and low equipment cost, but it is easily interfered with by background noise, has a short detection range, limited positioning accuracy, and limited application scenarios.

[0008] In summary, existing technologies either rely on a single sensor or suffer from poor environmental adaptability, insufficient detection accuracy, and strong scenario limitations, making it difficult to meet the current demand for high-precision, all-weather, and low-false-alarm identification and early warning of UAVs in no-fly zones under complex air conditions. Summary of the Invention

[0009] In view of this, the present invention proposes a method and system for drone identification and early warning, which solves the technical problem that existing drone identification and early warning technologies are unable to meet the requirements for high-precision and low-false-alarm identification and early warning of drones in no-fly zones under the current complex air conditions.

[0010] The present invention provides a method for identifying and warning of unmanned aerial vehicles (UAVs), comprising the following steps:

[0011] RGB image data, thermal radiation data, and spectral data of no-fly zones for drones were collected using visible light cameras, infrared thermal imagers, and multispectral imagers.

[0012] The RGB image data, thermal radiation data, and spectral data are transmitted and preprocessed to generate target RGB image data, target thermal radiation data, and target spectral data.

[0013] Using the target RGB image data, the target thermal radiation data, and the target spectral data, the fusion weights are adaptively adjusted in conjunction with real-time environmental parameters, and multi-level data fusion is performed to generate target fused data;

[0014] Based on deep learning models and tracking prediction algorithms, drone identification and early warning are performed on the target fusion data to generate early warning data.

[0015] The target fusion data and the corresponding abnormal event logs are transmitted to the cloud server, and the deep learning model is updated using the target fusion data and the abnormal event logs.

[0016] Optionally, the step of transmitting and preprocessing the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data includes:

[0017] The RGB image data, the thermal radiation data, and the spectral data are validated to generate a validation dataset.

[0018] The data in the verification dataset is filtered for outliers to generate a filtered dataset;

[0019] The filtered dataset is compressed to generate a compressed dataset, which is then asynchronously transmitted to the edge nodes via a Kafka message queue.

[0020] Feature extraction is performed on the compressed dataset through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

[0021] Optionally, the compressed dataset includes visible light data, infrared data, and multispectral data; the step of extracting features from the compressed dataset through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data includes:

[0022] The visible light data is subjected to distortion correction and adaptive histogram equalization to generate initial RGB image data;

[0023] The infrared data is converted into physical temperature values ​​and non-uniformity correction is performed to generate initial thermal radiation data;

[0024] The multispectral data is radiometrically corrected, and the normalized vegetation index is calculated to generate initial spectral data.

[0025] The initial RGB image data, the initial thermal radiation data, and the initial thermal radiation data are spatiotemporally aligned to generate intermediate RGB image data, intermediate thermal radiation data, and intermediate thermal radiation data.

[0026] The intermediate RGB image data, the intermediate thermal radiation data, and the intermediate thermal radiation data are compressed respectively, and feature extraction is performed through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

[0027] Optionally, the step of transmitting and preprocessing the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data further includes:

[0028] When abnormal features appear in the target RGB image data, the target thermal radiation data, and the target spectral data, pattern matching of the abnormal features is performed using a preset rule base to generate abnormal type data.

[0029] When the abnormal data type is communication interruption, a preset retry processing operation is performed;

[0030] When the abnormal data type is data anomaly, a preset redundancy processing operation is performed;

[0031] When the abnormal data type is synchronous loss, a preset degradation mode strategy is executed.

[0032] Optionally, the step of using the target RGB image data, the target thermal radiation data, and the target spectral data, adaptively adjusting the fusion weights in conjunction with real-time environmental parameters, and performing multi-level data fusion to generate target fused data includes:

[0033] By combining a long short-term memory network with a preset physical rule base and real-time environmental parameters, dynamic weights corresponding to the target RGB image data, the target thermal radiation data, and the target spectral data are dynamically generated respectively.

[0034] The target RGB image data, the target thermal radiation data, and the target spectral data are weighted and fused according to the dynamic weights to generate initial fused data;

[0035] The initial fused data is subjected to multimodal feature extraction, feature filtering and fusion to generate intermediate fused data;

[0036] By combining the improved DS evidence theory with the preliminary decision data corresponding to the intermediate fused data and the environment adaptive voting weight, feature fusion is performed on the intermediate fused data to generate the target fused data.

[0037] Optionally, the step of extracting multimodal features from the initial fused data, and then performing feature filtering and fusion to generate intermediate fused data includes:

[0038] Visible light modal features were extracted from the initial fused data using a lightweight YOLOv7-Edge model.

[0039] Infrared modal features in the initial fused data are extracted by combining a residual network with a convolutional block attention module.

[0040] Multispectral modal features in the initial fused data are extracted by jointly modeling with a one-dimensional convolutional neural network and a support vector machine.

[0041] The visible light modal features, the infrared modal features, and the multispectral modal features are concatenated to generate a high-dimensional vector;

[0042] The importance score of each dimension in the high-dimensional vector is calculated using a self-attention mechanism;

[0043] Select the dimension data corresponding to the importance scores that meet the preset score threshold, and construct discriminative features.

[0044] Optionally, the step of generating warning data by performing UAV identification and early warning on the target fusion data based on a deep learning model and a tracking prediction algorithm includes:

[0045] Target detection data is generated by performing target detection on the target fusion data using a YOLOv8 detector.

[0046] The target detection data is used to perform trajectory tracking through a deep learning model to generate continuous trajectory data.

[0047] The intrusion probability corresponding to the continuous trajectory data is calculated by combining the LSTM trajectory prediction model with the environmental compensation algorithm.

[0048] When the intrusion probability is greater than a preset probability threshold and a target in the continuous trajectory data enters the electronic fence, the warning signal and countermeasure command corresponding to the intrusion probability are encapsulated into a preset format to generate warning data.

[0049] Optionally, the step of performing target detection on the target fusion data using a YOLOv8 detector to generate target detection data includes:

[0050] Multi-scale features of the target fusion data are extracted using the YOLOv8 multi-head detection architecture;

[0051] The detection head uses CIoU Loss to optimize the bounding box and combines it with the angle prediction branch to perform target detection on the multi-scale features and generate target localization boxes.

[0052] The multi-scale features are used to perform model identification and threat level classification through a classification head to generate attribute data.

[0053] Target detection data is constructed using the target bounding box and the attribute data.

[0054] Optionally, the step of using the target detection data to perform trajectory tracking through a deep learning model and generating continuous trajectory data includes:

[0055] An adaptive IOU matching mechanism is adopted, which combines the target detection data and the Hungarian algorithm to associate the target location box with the historical trajectory to generate associated data;

[0056] Extract the multimodal features corresponding to the associated data to generate initial visible light texture features and initial infrared thermal radiation features;

[0057] The initial visible light texture features and the initial infrared thermal radiation features are extracted by two independent ResNet-34 branches to generate target visible light texture features and target infrared thermal radiation features.

[0058] The target visible light texture features and the target infrared thermal radiation features are aligned across modes using a Transformer encoder, and the cosine similarity between the features is calculated.

[0059] The cosine similarity is compared with a preset similarity threshold to generate multimodal feature matching data;

[0060] The Kalman filter algorithm is used to predict the position at the next moment by using the multimodal feature matching data and the corresponding bounding box coordinate data, thereby generating position prediction data.

[0061] The position prediction data is corrected using the actual coordinate data at the current moment, and the trajectory status data is updated to generate continuous trajectory data.

[0062] The present invention also provides a drone identification and early warning system, comprising:

[0063] The data acquisition module is used to acquire RGB image data, thermal radiation data, and spectral data of no-fly zones for drones using a visible light camera, an infrared thermal imager, and a multispectral imager.

[0064] The data preprocessing module is used to transmit and preprocess the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data;

[0065] The target data fusion generation module is used to use the target RGB image data, the target thermal radiation data, and the target spectral data, and adaptively adjust the fusion weights in combination with real-time environmental parameters, and perform multi-level data fusion to generate target fused data;

[0066] The early warning data generation module is used to perform UAV identification and early warning on the target fusion data based on a deep learning model and a tracking prediction algorithm, and generate early warning data.

[0067] The cloud collaboration module is used to transmit the target fusion data and the corresponding abnormal event logs to the cloud server, and to update the deep learning model using the target fusion data and the abnormal event logs.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] This invention acquires RGB image data, thermal radiation data, and spectral data of no-fly zones for drones using a visible light camera, an infrared thermal imager, and a multispectral imager. The RGB image data, thermal radiation data, and spectral data are preprocessed for transmission to generate target RGB image data, target thermal radiation data, and target spectral data. Using the target RGB image data, target thermal radiation data, and target spectral data, and combining real-time environmental parameters, the fusion weights are adaptively adjusted, and multi-level data fusion is performed to generate target fused data. The fusion weights are adjusted in real-time based on environmental parameters, suppressing visible light weights in strong light environments to ensure system performance stability under extreme conditions. Based on a deep learning model and tracking prediction algorithm, drone identification and early warning are performed on the target fused data, generating early warning data. The target fused data and corresponding abnormal event logs are transmitted to a cloud server, and the deep learning model is updated using the target fused data and abnormal event logs. This invention achieves high precision, high robustness, and high real-time performance in the drone identification and early warning system, providing reliable technical support for proactive defense in complex scenarios, demonstrating significant technological advancement and broad application prospects. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating a drone identification and early warning method according to Embodiment 1 of the present invention;

[0072] Figure 2 This is a schematic diagram of the end-edge-cloud collaborative architecture connection process according to Embodiment 1 of the present invention;

[0073] Figure 3 This is a schematic diagram of the edge-cloud collaboration process in Embodiment 1 of the present invention;

[0074] Figure 4 This is a flowchart illustrating a drone identification and early warning method according to Embodiment 2 of the present invention;

[0075] Figure 5 This is a schematic diagram of an end-side data acquisition system architecture according to Embodiment 2 of the present invention;

[0076] Figure 6 This is a schematic diagram of the data transmission protocol stack structure according to Embodiment 2 of the present invention;

[0077] Figure 7 This is a schematic diagram of a standardized processing framework for multimodal sensor data according to Embodiment 2 of the present invention;

[0078] Figure 8 This is a flowchart illustrating the hardware synchronization mechanism for implementing the FPGA synchronization controller according to Embodiment 2 of the present invention.

[0079] Figure 9 This is a schematic diagram of a time synchronization scheme according to Embodiment 2 of the present invention;

[0080] Figure 10 This is a flowchart illustrating the self-healing mechanism on the end side of Embodiment 2 of the present invention.

[0081] Figure 11 This is a schematic diagram of a multimodal data fusion and decision-making architecture for complex scenarios according to Embodiment 2 of the present invention;

[0082] Figure 12 This is a schematic diagram of the YOLOv8 multi-head detection architecture according to Embodiment 2 of the present invention;

[0083] Figure 13 This is a flowchart illustrating the real-time intrusion detection and early warning system according to Embodiment 2 of the present invention;

[0084] Figure 14 This is a structural block diagram of a drone identification and early warning system according to Embodiment 3 of the present invention. Detailed Implementation

[0085] This invention provides a method and system for identifying and warning of unmanned aerial vehicles (UAVs), which addresses the technical problem that existing UAV identification and warning technologies are insufficient to meet the requirements for high-precision, low-false-alarm identification and warning of UAVs in no-fly zones under current complex air conditions.

[0086] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0087] Example 1

[0088] Please see Figure 1 The present invention provides a method for identifying and warning of unmanned aerial vehicles (UAVs) according to Embodiment 1, which includes the following steps:

[0089] Step 1001: Collect RGB image data, thermal radiation data, and spectral data of the no-fly zone for drones using a visible light camera, an infrared thermal imager, and a multispectral imager.

[0090] In this embodiment of the invention, the no-fly zone for drones refers to sensitive airspace such as the vicinity of airports and military restricted areas. By employing a multimodal sensor working collaboratively, covering visible light, infrared, and multispectral bands, all-weather, all-scenario drone detection is achieved, obtaining RGB image data, thermal radiation data, and spectral data of the no-fly zone for drones. This multimodal sensor includes a visible light camera, an infrared thermal imager, and a multispectral imager; the sensor selection and parameter configuration are shown in Table 1.

[0091] Table 1 Sensor Selection and Parameter Configuration

[0092]

[0093] Selection criteria: All three sensors support external trigger input and PTP (IEEE 1588v2) time synchronization protocol, and operate within a temperature range of -30℃ to +60℃, ensuring stability in extreme environments. The multi-band complementary design (visible light + thermal radiation + spectral) significantly improves recognition robustness.

[0094] Furthermore, this invention can implement a hardware-level synchronization control mechanism, specifically employing the following technical means:

[0095] The PTP Precision Time Protocol (IEEE 1588 standard) is used to provide a unified clock source for the visible light camera (1080p@30fps), the infrared thermal imager (FLIR T1020), and the multispectral imager (Micasense RedEdge-MX). The time synchronization error is ≤1ms (measured standard deviation 0.3ms) through FPGA hardware timestamps.

[0096] A multi-sensor coaxial mounting bracket was designed, and the rotation matrix R and translation vector T were calculated using the OpenCV checkerboard calibration method to achieve spatial coordinate system alignment (reprojection error <0.5 pixels). By adopting the IEEE 1588v2 Precise Time Protocol (PTP) to provide a unified clock source for the visible light camera, infrared thermal imager, and multispectral imager, and combining it with FPGA hardware timestamps, a time synchronization error ≤1ms and a spatial coordinate system alignment error <0.5 pixels were achieved, significantly outperforming traditional solutions (time deviation >100ms).

[0097] Step 1002: Transmit and preprocess the RGB image data, thermal radiation data, and spectral data to generate target RGB image data, target thermal radiation data, and target spectral data.

[0098] In this embodiment of the invention, a verification dataset is generated by verifying RGB image data, thermal radiation data, and spectral data. Outlier filtering is performed on the data in the verification dataset to generate a filtered dataset. The filtered dataset is then compressed to generate a compressed dataset, which is asynchronously transmitted to edge nodes via a Kafka message queue. Feature extraction is performed on the compressed dataset at the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data. When abnormal features appear in the sensor data of a drone in a no-fly zone, pattern matching is performed on the abnormal features using a preset rule base to generate abnormal type data. When the abnormal type data indicates a communication interruption, a preset retry processing operation is executed. When the abnormal type data indicates a data anomaly, a preset redundancy processing operation is executed. When the abnormal type data indicates synchronization loss, a preset degradation mode strategy is executed.

[0099] Step 1003: Using target RGB image data, target thermal radiation data, and target spectral data, and combining real-time environmental parameters, adaptively adjust the fusion weights and perform multi-level data fusion to generate target fused data.

[0100] In this embodiment of the invention, dynamic weights corresponding to target RGB image data, target thermal radiation data, and target spectral data are dynamically generated by combining a Long Short-Term Memory (LSTM) network with a preset physical rule base and real-time environmental parameters. The target RGB image data, target thermal radiation data, and target spectral data are then weighted and fused according to these dynamic weights to generate initial fused data. Multimodal feature extraction is performed on the initial fused data, followed by feature filtering and fusion to generate intermediate fused data. Finally, feature fusion is performed on the intermediate fused data using an improved DS evidence theory combined with preliminary decision data corresponding to the intermediate fused data and environmental adaptive voting weights to generate target fused data.

[0101] Step 1004: Based on the deep learning model and tracking prediction algorithm, perform UAV identification and early warning on the target fusion data to generate early warning data.

[0102] Deep learning models include the YOLOv8 multi-head detection architecture, LSTM (Long Short-Term Memory) network, LSTM trajectory prediction model, ResNet (Residual Network) -34 branch, and Transformer Encoder. The YOLOv8 multi-head detection architecture includes the CSPDarknet53 backbone network, SPPF (Spatial Pyramid Pooling-Fast) spatial pyramid pooling, and PANet (Path Aggregation Network) feature pyramid. Transformer is a deep learning architecture based on a self-attention mechanism, adept at modeling long-distance dependencies between elements in a sequence. The Encoder is one of the core components of Transformer, encoding the input sequence (such as text or image sequence features) through a self-attention mechanism, outputting a feature representation containing global dependency information, and is widely used in natural language processing, computer vision, and other fields.

[0103] The YOLOv8 multi-head detection architecture performs multi-scale feature extraction, target bounding box optimization, and multi-label classification on target fusion data to generate target bounding boxes, model identification results, and threat levels.

[0104] LSTM networks perform weight calculations on real-time environmental parameters to generate dynamic weights for data-level fusion, adapting to the multimodal data fusion needs under different environments.

[0105] The LSTM trajectory prediction model performs time-series dependency modeling and intrusion probability calculation on continuous trajectory data, and generates intrusion probability by combining environmental compensation algorithm.

[0106] The ResNet-34 branch and TransformerEncoder perform cross-modal feature extraction and alignment operations on the visible light texture features and infrared thermal radiation features corresponding to the target localization box, generating aligned feature vectors that provide a basis for target identity verification.

[0107] The tracking and prediction algorithm includes an improved DeepSORT algorithm, which incorporates an adaptive IOU matching mechanism, the Hungarian algorithm, and Kalman filtering. The improved DeepSORT algorithm performs association matching between the target bounding box and historical trajectory data, dynamically adjusting the IOU threshold, verifying feature similarity, and correcting position prediction to generate continuous trajectory data.

[0108] In this embodiment of the invention, target detection data is generated by using a YOLOv8 detector to analyze the fused target data. A deep learning model is then used to track the target bounding box, generating continuous trajectory data. An LSTM trajectory prediction model combined with an environmental compensation algorithm is used to calculate the intrusion probability corresponding to the continuous trajectory data. When the intrusion probability exceeds a preset probability threshold and the target in the continuous trajectory data enters the electronic fence, the warning signal and countermeasure command corresponding to the intrusion probability are encapsulated in a preset format to generate warning data.

[0109] Step 1005: Transmit the target fusion data and the corresponding abnormal event logs to the cloud server, and update the deep learning model using the target fusion data and abnormal event logs.

[0110] It is worth mentioning that this invention adopts an edge-cloud collaborative architecture, which achieves efficient collaboration between data collection, edge processing, and cloud decision-making through a three-tier architecture:

[0111] (1) Edge side (sensor array): Multi-source sensors (visible light, infrared, multispectral) achieve physical time synchronization (error ≤ 1ms) and spatial calibration through FPGA synchronization controller; data is compressed and uploaded to edge nodes, and key data (abnormal events) are reported to the cloud in real time.

[0112] (2) Edge computing nodes: perform multimodal data fusion (dynamic weighted fusion, deep learning recognition); call local models for real-time decision-making, and link with the countermeasure system through the MQTT protocol; upload feature data to the cloud for model iteration.

[0113] (3) Cloud platform: Stores historical data and manages model training clusters; dynamically distributes updated models (TensorRT engine) and policy configurations; realizes global trajectory prediction and threat level assessment.

[0114] In embodiments of the present invention, such as Figure 2 The flowchart of the edge-cloud collaborative architecture is shown in the figure. On the edge side, visible light / infrared / multispectral sensors are connected through an FPGA hardware synchronization module, using the IEEE 1588v2 protocol to achieve clock synchronization at the ≤1ms level, and a physical coaxial bracket ensures spatial coordinate alignment. The edge-cloud collaborative architecture supports flexible expansion of multimodal sensors, and the cloud model, i.e., the cloud platform iteration and edge node strategy dynamic update mechanism ensure the long-term adaptability of the system.

[0115] In terms of edge-cloud collaboration, such as Figure 3As shown, after feature extraction from the collected raw data, the edge nodes use the Zstd algorithm to efficiently compress the feature data at a compression ratio of 4:1, reducing network bandwidth consumption. This reduces transmission costs and improves cloud storage and analysis efficiency while ensuring data validity. The cloud platform trains or optimizes models (such as object detection and classification models) based on global data, and converts the models into a format that edge devices can efficiently run in (such as INT8 quantization models) through the TensorRT engine. This ensures that the edge nodes always use the latest version of the model, improving real-time inference performance and accuracy. During operation, if the edge nodes detect hardware failures, data anomalies, or business logic errors, they will upload the abnormal event logs (including timestamps, error codes, and context information) to the cloud platform in real time. This provides the cloud platform with fault diagnosis data and supports root cause analysis and system optimization. Based on historical data and real-time feedback, the cloud platform dynamically generates new business strategies (such as threshold adjustments and algorithm parameter optimizations) and distributes them to the edge nodes through configuration files. This enables dynamic adaptation of edge-side strategies, enhancing the system's responsiveness to environmental changes or business needs. Through the aforementioned bidirectional data flow, the system can reduce redundant data transmission at the edge, centrally process complex tasks in the cloud, and optimize resource allocation; models and strategies can be rapidly iterated and updated, ensuring real-time performance; at the same time, closed-loop management of abnormal events ensures stable system operation and improves stability.

[0116] Example 2

[0117] Please see Figure 4 The second embodiment of the present invention provides a method for identifying and warning of unmanned aerial vehicles (UAVs), which includes the following steps:

[0118] Step 4001: Collect RGB image data, thermal radiation data, and spectral data of the no-fly zone for drones using a visible light camera, an infrared thermal imager, and a multispectral imager.

[0119] In embodiments of the present invention, such as Figure 5 As shown, within the no-fly zone for drones, three types of equipment—visible light cameras, infrared thermal imagers, and multispectral imagers—are deployed and simultaneously begin data acquisition. The FPGA synchronization controller leads the synchronization acquisition logic: first, it establishes a unified time base for the system through PTP clock distribution; then, through trigger signal generation, it precisely distributes trigger commands to the visible light trigger, infrared trigger, and multispectral trigger branches, synchronously driving the three types of equipment to start acquisition. Specifically, the visible light camera outputs corresponding "RGB image data" through the GigEVision interface; the infrared thermal imager transmits image data containing "thermal radiation data" through the SDI 3G interface; and the multispectral imager outputs image data carrying "spectral data." Simultaneously, the FPGA synchronization controller on the left, synchronized with the system time base using the PTP clock, ensures consistent acquisition timing for the three types of equipment.

[0120] The multimodal data collected by the three types of devices (all transmitted in the form of "image data") will subsequently undergo processing in stages such as data reception, timestamp marking, and data alignment buffering. However, the core of the first step is to use the multi-source devices, dual FPGA synchronous control (including PTP clock distribution and trigger branching), and dedicated interfaces (GigE Vision / SDI 3G, etc.) shown in the diagram to complete the synchronous collection of multi-dimensional data in the no-fly zone. Then, the data is sent to the edge computing node in real time, where the edge computing node performs preliminary reception and buffering to ensure the timeliness and integrity of the collected data. Finally, the data is uploaded to the cloud analysis platform to form a preliminary data flow framework of "collection-edge reception-cloud association".

[0121] Step 4002: Perform data verification on the RGB image data, thermal radiation data, and spectral data to generate a verification dataset.

[0122] In this embodiment of the invention, the system receives RGB image data, thermal radiation data, and spectral data, and then checks the format integrity and protocol compliance through a data verification module to construct a verification dataset to ensure that the input data meets preset standards.

[0123] Step 4003: Filter outliers in the validation dataset to generate a filtered dataset.

[0124] In this embodiment of the invention, a dynamic threshold algorithm is used to remove noise, outliers, and abnormal timestamp data from the verification dataset, thereby filtering outliers in the verification dataset and obtaining a filtered dataset to ensure the data quality of subsequent processing.

[0125] Step 4004: Compress the filtered dataset to generate a compressed dataset and transmit it asynchronously to the edge node via a Kafka message queue.

[0126] In its invention, the Kafka message queue primarily serves as an "asynchronous data transmission bridge between edge sensors and edge nodes." Specifically, it supports streaming transmission of multi-source heterogeneous data (visible light, infrared, multispectral), reducing bandwidth usage with the Zstandard compression algorithm; it improves data transmission efficiency through partitioning strategies to meet the real-time data output requirements of high frame rate sensors (such as visible light cameras at 1080p@30fps); and it ensures data reliability using a replication mechanism (replica count ≥ 3), reducing the data loss rate from 5.2% in traditional solutions to 0.03%, providing stable data input for subsequent edge node fusion analysis and early warning.

[0127] It should be noted that the Zstandard (Zstd) algorithm is a core compression technology for preprocessing multi-source heterogeneous data transmission. It is used to compress visible light images, infrared thermal radiation data, and multispectral data, achieving a compression ratio of ≥4:1 and ≥90%, effectively reducing transmission bandwidth consumption. This algorithm, combined with Kafka message queues, ensures low data loss rates, while its fast compression and decompression characteristics meet the real-time processing requirements of edge nodes, supporting the end-to-end real-time performance and reliability of the system.

[0128] In this embodiment of the invention, a Kafka-based streaming compression pipeline is developed: RGB images, 16-bit thermal radiation data, and 5-band spectral data are encapsulated into JSON format, compressed using the Zstandard algorithm (compression ratio ≥ 4:1), and then transmitted to edge nodes. Specifically, the Zstandard algorithm is used to efficiently compress the data, reducing network bandwidth consumption while preserving key features to obtain a compressed dataset. The processed data is asynchronously transmitted via a Kafka message queue, whose partitioning strategy and replication mechanism (replication factor ≥ 3) ensure data reliability and high throughput. The Kafka-based streaming compression pipeline improves the multimodal data compression ratio to 4:1 and reduces the data loss rate from 5.2% in traditional solutions to 0.03%, laying the foundation for high-precision fusion.

[0129] It is worth mentioning that, such as Figure 6 The data transmission protocol stack structure diagram shows that, to meet the high throughput and low latency transmission requirements of multimodal sensor data, this system adopts a five-layer protocol stack design (as shown in the figure), combining compression acceleration, Time-Sensitive Networking (TSN), and high-frequency wireless transmission technology to achieve end-to-end optimized data transmission. Application Layer: Data volume is reduced and transmission load is lowered through the Zstd compression algorithm (compression rate 90%+, real-time processing capability). Transport Layer: Based on the UDP acceleration protocol (supporting multi-threaded concurrency), combined with Reed-Solomon forward error correction (20% redundancy), packet loss resistance is enhanced while ensuring throughput. Network Layer: TSN over IP (IEEE 802.1Qbv time-aware scheduling) is adopted, and deterministic latency (<1ms) of critical data streams is guaranteed through hard priority partitioning (priority 6). Data Link Layer: Jumbo Frames (maximum transmission unit 9000 bytes) are supported, reducing protocol overhead and improving the efficiency of large frame data transmission. Physical layer: Based on Wi-Fi 6 (802.11ax), it utilizes a 160MHz channel bandwidth and 1024-QAM modulation to achieve a high-speed wireless transmission with a theoretical peak rate of 3.6Gbps.

[0130] Step 4005: Extract features from the compressed dataset through edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

[0131] Furthermore, the compressed dataset includes visible light data, infrared data, and multispectral data. Step 2005 also includes the following sub-steps:

[0132] Step S11: Perform distortion correction and adaptive histogram equalization on the visible light data to generate initial RGB image data;

[0133] Step S12: Convert the infrared data into physical temperature values ​​and perform non-uniformity correction to generate initial thermal radiation data;

[0134] Step S13: Perform radiometric correction on the multispectral data and calculate the normalized vegetation index to generate initial spectral data;

[0135] Step S14: Spatiotemporally align the initial RGB image data, initial thermal radiation data, and initial thermal radiation data to generate intermediate RGB image data, intermediate thermal radiation data, and intermediate thermal radiation data.

[0136] Step S15: Compress the intermediate RGB image data, intermediate thermal radiation data, and intermediate thermal radiation data respectively, and extract features through edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

[0137] In embodiments of the present invention, such as Figure 7 As shown, this invention proposes a standardized processing framework for multimodal sensor data, covering the entire chain from raw data acquisition to edge analysis. The process is based on a modular design, processing visible light, infrared, and multispectral data through separate paths, ultimately achieving spatiotemporal alignment and efficient analysis of cross-modal data.

[0138] Specifically, the raw data first enters the sensor type branch module, where a dedicated processing path is selected based on the visible light / infrared / multispectral characteristics. For visible light data, distortion correction is used to eliminate lens geometric distortion, and contrast-limited adaptive histogram equalization (CLAHE) is combined to enhance local details, resulting in initial RGB image data. For infrared data, temperature calibration and non-uniformity correction algorithms are used to convert the raw thermal radiation signal into physical temperature values ​​and eliminate pixel response differences, resulting in initial thermal radiation data. For multispectral data, after radiometric correction to correct atmospheric scattering and sensor response deviations, the Normalized Difference Vegetation Index (NDVI) is further calculated to quantify vegetation health, resulting in initial spectral data.

[0139] Subsequently, all processed data enters the spatiotemporal alignment module, which achieves strict synchronization and spatial registration of multi-source data based on nanosecond-level timestamps and calibration matrices, yielding intermediate RGB image data, intermediate thermal radiation data, and intermediate thermal radiation data. To reduce transmission and storage overhead, the data is compressed using the Zstandard algorithm (compression rate ≥90%) while retaining key features. Finally, edge nodes perform edge analysis on the compressed data using the Canny operator or a deep learning model-driven approach to extract target contours and key features, generating target RGB image data, target thermal radiation data, and target spectral data.

[0140] It should be noted that temperature calibration is a crucial step in infrared data preprocessing. This involves collecting raw electrical signals (such as grayscale values ​​and voltage values) from infrared sensors at different temperatures using a blackbody radiation source with known temperatures (e.g., multiple calibration points such as -40℃, 0℃, and 50℃). A mapping model between the sensor output value and the physical temperature value is then established (usually a polynomial fitting or piecewise linear correction) to eliminate systematic errors caused by individual sensor differences (such as inconsistent responses and temperature drift). Ultimately, the raw infrared data is converted into physically meaningful temperature values ​​(unit: ℃), ensuring that it accurately reflects the actual thermal radiation temperature of the UAV target and providing reliable initial data for subsequent infrared feature extraction.

[0141] It is worth mentioning that, such as Figure 8 As shown, relying on an FPGA synchronization controller (Xilinx·Zynq·UltraScale+·MPSoC), the process uses the PTP master clock as the core time source and obtains the precise PTP clock (IEEE·1588v2) through the GPS module to establish a globally unified reference. After the timestamp engine encodes the PTP clock, it drives the trigger signal generation module to synchronously output pulses with a period of 1ms and a rise edge accuracy of ±100ns to visible light trigger, infrared trigger, and multispectral trigger, forcing the three types of sensors to start acquiring data at the same time. This eliminates the time difference in the acquisition of multimodal data from the hardware layer and realizes the underlying logic of time synchronization.

[0142] After acquisition, the data receiving module aggregates the multimodal raw data (visible light data, infrared data, and multispectral data), and then adds a 64-bit hardware timestamp (format: seconds.nanoseconds) to each frame of data. The hardware timestamp is added when the data is returned. Finally, the data alignment buffer module, relying on the DDR4 cache, sorts and calibrates the multimodal data according to the timestamp (maximum alignment delay ≤2ms). This not only corrects timing misalignments caused by transmission delays but also reserves a "time-consistent" data source foundation for subsequent spatial calibration and multi-source fusion, completing the final step in spatiotemporal synchronization.

[0143] It is worth mentioning that, such as Figure 9As shown, this solution uses a high-precision GPS clock source (1PPS signal, ±20ns accuracy) as the time reference and an FPGA synchronization controller to achieve spatiotemporal synchronization and data alignment for multiple sensors. The controller constructs a distributed clock network based on the PTP protocol (IEEE 1588v2), periodically generating 1ms synchronization frames and sending trigger signals to the visible light camera (GigE Vision interface), infrared thermal imager (Camera Link interface), and multispectral imager (SDI 3G interface). Upon receiving the trigger command, each sensor completes image acquisition and adds a precise time stamp to the raw data using the FPGA controller's hardware-level timestamp module (error <100ns). The acquired data undergoes spatiotemporal alignment processing via edge computing nodes (maximum time difference ≤1ms) and is finally uploaded to the cloud analysis platform via the TSN network. This architecture achieves a complete link from nanosecond-level time synchronization to multimodal data fusion through a layered design.

[0144] To achieve accurate fusion of multimodal sensor data, a sensor spatial registration model based on geometric transformation needs to be established. The registration matrix shown in the figure is calculated using a joint calibration algorithm (such as a combination of Zhang Zhengyou's calibration method and the iterative nearest point algorithm), the core objective of which is to eliminate sensor installation errors and unify the coordinate system.

[0145] Multi-sensor spatial registration matrix:

[0146]

[0147] The calibration parameters are shown in Table 2:

[0148] Table 2 Calibration Parameters

[0149] Sensor pair Rotation angle θ (°) <![CDATA[Translation (t x , t y )(mm)]]> Reprojection error Visible light - infrared 0.35 (2.1，-1.8) 0.42 pixels Visible light - multispectral 1.20 (-3.5，0.7) 0.58 pixels Infrared Radar 2.75 (5.2，-2.3) 0.31m

[0150] Step 4006: When abnormal features appear in the target RGB image data, target thermal radiation data and target spectral data, the abnormal features are pattern matched using a preset rule base to generate abnormal type data.

[0151] In fault diagnosis, the preset rule base consists of matching logic data between abnormal characteristics and fault types (communication interruption, data anomaly, synchronization loss), as well as corresponding fault handling strategy data (such as the number of retry attempts for communication interruption, and the activation conditions for backup units in data anomalies). In dynamic weight control, it includes trigger condition data for six typical environments (dense fog, nighttime, strong light, etc.) (such as parameter thresholds like visibility <200m and humidity >80%), and sensor weight adjustment strategy data for each environment (such as increasing the infrared weight to 0.6 in dense fog). All this data is generated based on sensor physical characteristics, actual test data, and expert experience calibration, ensuring the scientific validity and effectiveness of the rules.

[0152] In embodiments of the present invention, such as Figure 10 The diagram illustrates the self-healing mechanism on the edge side of this invention. This mechanism, based on real-time status monitoring and fault classification, constructs a closed-loop process of "abnormal triggering - type judgment - processing execution - feedback optimization." It continuously monitors the sensor's operating status. When an abnormal signal is detected (such as communication interruption, data anomaly, or synchronization loss), i.e., when abnormal features appear in the target RGB image data, target thermal radiation data, and target spectral data, pattern matching is performed on the abnormal features using a preset rule base, classifying them into three main types and triggering the corresponding processing branch.

[0153] Step 4007: When the exception type data is communication interruption, execute the preset retry processing operation.

[0154] The default retry procedure is to first attempt to retry the connection three times (applicable to transient interference). If the retry fails, a redundant communication interface is enabled to ensure channel availability.

[0155] In this embodiment of the invention, for abnormal data scenarios involving communication interruption, the system first attempts to retry the connection three times (applicable to transient interference). If the retry fails, redundant communication interfaces are enabled to ensure channel availability.

[0156] Step 4008: When the abnormal data type is data abnormal, execute the preset redundancy processing operation.

[0157] The default redundancy handling operation is to analyze the root cause through the sensor self-test program. If it is determined to be a hardware failure, the redundant backup unit is activated and a maintenance request is reported. If it is a temporary interference, the data is corrected by interpolation or filtering algorithms.

[0158] In this embodiment of the invention, for abnormal data, the root cause is analyzed through the sensor self-test program. If it is determined to be a hardware failure, the redundant backup unit is activated and a maintenance request is reported. If it is temporary interference, the data is corrected by interpolation or filtering algorithms.

[0159] Step 4009: When the abnormal data type is synchronous loss, execute the preset degradation mode strategy.

[0160] The default degradation mode strategy is to start the time / space synchronization calibration process. If the recovery is successful, the normal operation mode will be restored; otherwise, the low-precision synchronization degradation mode will be switched to retain the basic functions.

[0161] In this embodiment of the invention, for abnormal data due to synchronization loss, a time / space synchronization calibration process is initiated (such as PTP protocol realignment or calibration matrix update). If recovery is successful, normal operation mode is restored; otherwise, a low-precision synchronization degradation mode is switched to retain basic functionality. This mechanism achieves hierarchical fault tolerance through a three-order response strategy of "retry → redundancy → degradation," optimizing resource consumption while ensuring reliability. Furthermore, all processing results are fed back to the monitoring module to dynamically optimize anomaly detection thresholds and processing strategies, further enhancing the system's adaptive capabilities.

[0162] It is worth mentioning that, to ensure the continued availability of the multimodal sensor system when some modules fail, this invention proposes a hierarchical degradation strategy based on module redundancy and functional replacement, as shown in Table 3. This strategy dynamically switches between redundant modes and multimodal data fusion mechanisms, degrading to a software time synchronization scheme when hardware-level PTP synchronization fails, thus balancing system function retention with performance loss.

[0163] Table 3. Tiered Degradation Strategy

[0164]

[0165]

[0166] Specifically, when any of the visible light / infrared / multispectral sensors fails, the system prioritizes using redundant modes to maintain core functions: if the visible light camera fails due to insufficient light, the infrared thermal imager provides basic identification capabilities through thermal radiation characteristics, but due to limited thermal imaging resolution, the daytime identification distance decreases by 30% (based on point spread function (PSF) simulation and the inverse square law of light intensity); if the multispectral sensor fails, the system switches to an RGB and thermal feature fusion strategy, using visible light color information and thermal imager temperature characteristics to compensate for band loss, but sacrificing the sensitivity of multispectral sensors to material differences (accuracy decreases by 50%). For hardware-level synchronization module failure scenarios, the system degrades to software time synchronization (accuracy drops from nanoseconds to 10ms), leading to increased data alignment errors at edge nodes (maximum time difference ≤ 1ms → 10ms), and a doubling of target positioning error (assuming target speed v = 5m / s, x ≈ v·Δt = 50mm). This strategy sacrifices time accuracy for system availability, meeting the fault tolerance requirements of the IEC 61158 standard for industrial control systems.

[0167] Step 4010: Using target RGB image data, target thermal radiation data, and target spectral data, and combining real-time environmental parameters, adaptively adjust the fusion weights and perform multi-level data fusion to generate target fused data.

[0168] Existing methods rely on a single fusion layer (such as pixel-level average weighting), which leads to a sharp drop in feature discriminative power in scenarios such as sudden changes in lighting or haze interference (measured recall decreases by >40%). This invention establishes a three-level fusion framework, such as... Figure 11 The diagram shown illustrates a multimodal data fusion and decision-making architecture for complex scenarios. Through a three-tiered progressive processing approach—data-level, feature-level, and decision-level—it achieves end-to-end analysis from raw sensor data to final early warning decisions.

[0169] First, in the data-level fusion stage, the system receives raw data from visible light, infrared, and spectral sensors and performs preliminary fusion based on dynamic weight allocation to generate basic data representations. Subsequently, the feature-level fusion module extracts high-dimensional features from the three types of data: visible light data is processed using a convolutional neural network to extract 1024-dimensional visual features (such as edges and textures); infrared data is processed using a thermal radiation model to generate 2048-dimensional thermal features (such as temperature distribution and thermal patterns); and spectral data is processed using band reflectance analysis to obtain 512-dimensional material features. These features are combined through a dynamic weighting mechanism (weights are adjusted in real time according to data quality and environmental changes) to form high-dimensional feature vectors. Finally, the decision-level fusion employs a multi-strategy collaborative mechanism: confidence allocation and combination rules based on DS evidence theory quantify the consistency and conflict of multi-source evidence; the trajectory analysis module models the target's motion behavior; and a Transformer self-attention mechanism is introduced to dynamically focus key features and suppress noise interference. Finally, the fusion of DS theory and trajectory analysis results generates a comprehensive decision-making basis and outputs early warning information such as risk level or action classification, resulting in target fusion data.

[0170] This invention proposes a two-layer dynamic weight control mechanism that integrates a data-driven LSTM model (i.e., Long Short-Term Memory network) and a physical rule base (i.e., a pre-defined physical rule base). Through the synergistic effect of temporal modeling and expert knowledge, it achieves adaptive weight allocation for multimodal sensors. This mechanism comprises two core components: an LSTM weight controller and a weight decision rule base.

[0171] The LSTM weight controller optimizes the weights of visible light (α), infrared (β), and spectral (γ) sensors in real time by modeling the temporal dependency between environmental parameters and weight allocation. The input layer receives a three-channel environmental parameter sequence containing illumination intensity (Lux), ambient temperature (T), and visibility (VIS). After feature extraction by a 16-dimensional fully connected encoder, a two-layer LSTM network captures the temporal dependency of environmental changes. The decoder uses a two-layer fully connected network (ReLU activation) to generate three weight outputs, and a Softmax function constrains their sum to 1 (α+β+γ=1) to ensure the physical interpretability of the weight allocation.

[0172] The weight decision rule library addresses the response latency issue of LSTM models in sudden environmental changes by pre-setting scenario trigger conditions and weight adjustment strategies. For example, in dense fog mode (VIS < 200m & humidity > 80%), the infrared weight is increased (β↑ 0.6) and the visible light weight is decreased (α↓ 0.1), leveraging the weak sensitivity of infrared wavelengths (8-14μm) to water molecule scattering to enhance penetration capability. In electromagnetic interference scenarios (signal-to-noise ratio < -10dB), the weight of the spectral sensor is preferentially increased (γ↑ 0.7) due to its inherent anti-electromagnetic interference characteristics based on optical interference principles. The weight decision rule library covers six typical environments (dense fog, nighttime, strong light, rain / snow, dust storms, and electromagnetic interference), and each rule is calibrated based on sensor physical characteristics and actual test data to ensure the scientific validity and effectiveness of the weight adjustment strategies.

[0173] The collaborative working mechanism of the two adopts a hierarchical decision-making mechanism: the weight decision rule base prioritizes matching the current environment scenario (by detecting trigger conditions through a sliding window), and if the match is successful, the preset weight strategy is directly applied (response latency <1ms); in non-extreme scenarios, the LSTM model predicts the weight allocation for the next 3 seconds based on historical environment data (10-second window), and adapts to the gradual change of environment through dynamic learning; when the difference between the weight decision rule base and the LSTM output exceeds the threshold (Δw>0.2), a weighted fusion strategy is adopted (the rule base weight accounts for 70%) to ensure the balance between physical rationality and data-driven approach.

[0174] Furthermore, step 4010 also includes the following sub-steps:

[0175] S21. By combining a long short-term memory network with a preset physical rule base and real-time environmental parameters, dynamic weights are dynamically generated for the target RGB image data, target thermal radiation data, and target spectral data, respectively.

[0176] Long Short-Term Memory Networks (LSTM) are primarily used to process time-dependent data, enabling dynamic decision-making and prediction.

[0177] The preset physical rule base is a logical set built based on sensor physical characteristics, test data, and expert experience. It includes triggering conditions (such as visibility <200m) and weight adjustment strategies (such as increasing the infrared weight to 0.6 in dense fog) for 6 typical environments (dense fog, nighttime, etc.), supplementing the response delay of LSTM to sudden environments. It also includes matching rules for fault types (communication interruption, synchronization loss, etc.) and processing strategies (such as retrying 3 times for communication interruption), supporting the system's self-healing from faults.

[0178] In this embodiment of the invention, LSTM is responsible for temporal modeling and gradual environment adaptation, while a pre-set physical rule base handles sudden environmental events and faults, jointly ensuring the robustness and real-time performance of the system in complex scenarios. In dynamic weight control, LSTM learns the temporal characteristics of environmental parameters such as illumination, temperature, humidity, and visibility to generate dynamic weights (α, β, γ) corresponding to target RGB image data, target thermal radiation data, and target spectral data, supporting multi-source data fusion. In trajectory prediction and intrusion warning, LSTM calculates the probability of drone intrusion based on temporal data such as speed and orientation angle of historical trajectories, achieving dynamic warning.

[0179] S22. The target RGB image data, target thermal radiation data and target spectral data are weighted and fused according to dynamic weights to generate initial fused data.

[0180] In this embodiment of the invention, the traditional scheme uses static weight allocation (e.g., α = β = γ = 0.33), which leads to a sharp decrease in feature discriminativeness when the environment changes abruptly (measured recall rate attenuation >40% in dense fog scenes). In contrast, this invention, in the data-level fusion stage, receives target RGB image data, target thermal radiation data, and target spectral data from visible light, infrared, and spectral sensors, and performs preliminary fusion based on dynamic weight allocation (α, β, and γ correspond to visible light, infrared, and spectral modes, respectively), generating basic data representation, i.e., initial fused data. It supports adaptive enhancement for abnormal scenes: automatically increasing the infrared weight β to 0.6 in foggy weather, and increasing the multispectral weight γ to 0.5 in night mode. The data-level dynamic weighted fusion formula is:

[0181] I fusion =α(t)·I RGB +β(t)·I Thermal +γ(t)·I Spectral ;

[0182] Among them, I fusion The output of the multi-source sensor data fusion is the initial fused data; α(t) is the visible light dynamic weighting coefficient, dynamically generated by the LSTM network, and its value changes with time t; I RGB The visible light modal raw data, i.e., the target RGB image data, reflects the visual details of the scene; β(t) is the infrared dynamic weight coefficient, dynamically generated by the LSTM network, and its value changes with time t; I Thermal The raw infrared thermal radiation mode data, i.e., the target thermal radiation data, highlights temperature difference information; γ(t) is the spectral dynamic weighting coefficient, dynamically generated by the LSTM network, and its value changes with time t; I Spectral The raw multispectral modal data, i.e. the target spectral data, contains scene features within a specific spectral range.

[0183] S23. Extract multimodal features from the initial fused data, and perform feature filtering and fusion to generate intermediate fused data.

[0184] Furthermore, step S23 also includes the following sub-steps:

[0185] S231. Extract visible light modal features from the initial fused data using a lightweight YOLOv7-Edge model;

[0186] S232. Extract infrared modal features from the initial fused data by combining a residual network with a convolutional block attention module;

[0187] S233. Extract multispectral modal features from the initial fused data by jointly modeling with a one-dimensional convolutional neural network and a support vector machine;

[0188] S234. The visible light modal features, infrared modal features, and multispectral modal features are concatenated to generate a high-dimensional vector;

[0189] S235. Calculate the importance scores of each dimension in a high-dimensional vector using a self-attention mechanism;

[0190] S236. Filter the dimension data corresponding to the importance scores that meet the preset score threshold, and construct discriminative features.

[0191] The lightweight YOLOv7-Edge model is used for feature extraction of visible light modes. With its high frame rate and high resolution, it can extract 1024-dimensional feature vectors in well-lit environments, accurately capturing details such as the drone's outline and markings, supporting model recognition, and adapting to the real-time processing needs of edge nodes.

[0192] The Residual Network (ResNet-50) is applied to infrared modal feature extraction. Combined with the Convolutional Block Attention Module (CBAM), it generates 2048-dimensional heat source distribution features. The residual connection solves the gradient degradation problem of deep networks, enhances the characterization ability of heat source areas such as UAV power systems and electronic components, and assists in target localization.

[0193] The Convolutional Block Attention (CBAM) module works in conjunction with ResNet-50 to dynamically focus on key heat source areas of the UAV in the infrared mode through channel attention and spatial attention mechanisms, suppressing background clutter interference and improving the target discrimination in complex environments (such as fog and night).

[0194] One-dimensional convolutional neural networks (1D-CNN) are used for multispectral modal processing and are jointly modeled with support vector machines (SVM). By analyzing the electromagnetic radiation characteristics of different bands, 512-dimensional material reflectivity features are extracted, which can effectively distinguish drones from background clutter such as smoke and dust, and enhance the detection capability in extreme environments.

[0195] Support vector machines and 1D-CNN work together to process multispectral data, achieve target classification based on extracted spectral features, enhance the recognition accuracy of UAVs in complex backgrounds by using pattern matching, and provide spectral dimension discrimination basis for multi-source fusion.

[0196] The preset score threshold refers to the critical value set in advance for filtering features after multimodal fusion in the Transformer feature selection stage of feature-level attention fusion.

[0197] In this embodiment of the invention, a feature-level fusion strategy based on multimodal feature extraction and Transformer self-attention mechanism is proposed. By dynamically evaluating the feature contribution of different modalities, redundant dimensionality compression and discriminative feature enhancement are achieved. To address the differences in characteristics between visible light, infrared, and multispectral sensors, customized models are used to extract high-dimensional features: For the visible light modality, a 1024-dimensional feature vector is extracted based on a lightweight YOLOv7-Edge model, resulting in visible light modal features that focus on target contour and color information, suitable for contour recognition under complex lighting conditions (such as edge detection and texture analysis). For the infrared modality, a ResNet-50 network combined with a convolutional block attention module (CBAM) is used to generate 2048-dimensional heat source distribution features, i.e., infrared modal features, enhancing the representation ability of key thermal regions through channel and spatial attention mechanisms. For the multispectral modality, a 512-dimensional material reflectance feature is extracted through joint modeling using 1D-CNN and support vector machine (SVM), resulting in multispectral modal features. The correlation between spectral bands is used to distinguish the reflectance characteristics of different materials.

[0198] To eliminate redundancy in cross-modal features and improve fusion performance, a feature optimization module based on a self-attention mechanism was designed. The specific process is as follows:

[0199] Feature concatenation: The Transformer feature optimization module fuses the three types of feature vectors, concatenating them into a unified high-dimensional representation. Among them, F concat This is a high-dimensional feature vector resulting from the concatenation of multimodal features, integrating visible light, infrared, and multispectral feature information, and serves as the fundamental input for subsequent feature processing and fusion. RGB This is a feature vector extracted from the visible light (RGB) modality, with a dimension of 1024, containing detailed features such as visual texture and color of the scene. Thermal The feature vector extracted for the infrared thermal radiation mode has a dimension of 2048 and focuses on features such as the target's heat source distribution and temperature differences. SpectralIt is a feature vector extracted from the multispectral mode, with a dimension of 512, containing features such as spectral reflectance in specific bands, and is suitable for target differentiation in complex lighting environments. The concatenated feature vector is located in a 3584-dimensional real space, where 3584 is calculated from 1024 (visible light) + 2048 (infrared) + 512 (multispectral), representing the total dimension of the feature vector.

[0200] Self-attention computation: The importance scores of each feature dimension are dynamically calculated through the self-attention layer of the Transformer encoder. Where Q and K are the query and key matrices, respectively, and d is the scaling factor; k This is the scaling factor.

[0201] Redundancy Removal: A subset of discriminative features is selected based on a preset score threshold (Top-50% retained). Experiments show that this method can reduce redundant dimensions by up to 43% while maintaining 98.7% of the original information fidelity.

[0202] The data from the multimodal feature extraction are shown in Table 4:

[0203] Table 4 Data from multimodal feature extraction

[0204]

[0205]

[0206] S24. By combining the improved DS evidence theory with the preliminary decision data corresponding to the intermediate fusion data and the environmental adaptive voting weight, feature fusion is performed on the intermediate fusion data to generate the target fusion data.

[0207] DS evidence theory is an uncertainty reasoning method for multi-source information fusion. Its core lies in assigning basic probabilities to evidence from different sources (such as decision results from visible light, infrared, and multispectral modes) and combining them with evidence combination rules to arrive at a final conclusion in order to deal with the uncertainty and conflict in multi-source information.

[0208] It should be noted that when performing multi-level fusion using target RGB image data, target thermal radiation data, and target spectral data to adaptively adjust fusion weights based on real-time environmental parameters, support is first provided by "constructing a spatiotemporal correlation metadata database": The time-series database InfluxDB is used to record environmental parameters (illuminance Lux, temperature / humidity T / H) and equipment status (such as focal length f, distortion coefficients k1, k2). This provides prior knowledge for "adaptively adjusting fusion weights based on real-time environmental parameters," ensuring that weight adjustments align with the current environment and actual equipment status. Simultaneously, the application of this metadata database directly impacts the multi-level fusion process, resolving the fusion distortion problem caused by acquisition delays in multimodal data. This reduces the data loss rate from 5.2% in traditional solutions to 0.03%, laying the foundation for generating high-precision target fusion data in this step.

[0209] In this embodiment of the invention, during the decision-level fusion stage, the system treats the decision outputs of multimodal sensors as "confidence votes" on the target state, integrating the confidence levels of each sensor through the combination rules of DS theory. For example, when a visible light sensor mistakenly identifies sunlight reflection as a drone, while an infrared sensor determines it is not a drone, the system calculates the degree of conflict between the two decisions (conflict factor K) and eliminates the misjudgment by dynamically adjusting the weights. The core of this method lies in quantifying the degree of contradiction between sensors and redistributing confidence levels using environmental information, performing feature fusion on intermediate fusion data, and generating target fusion data, thereby avoiding the error of a single sensor dominating the final decision.

[0210] It should be noted that this invention employs an improved Dempster-Shafer (DS) evidence theory framework, achieving dynamic fusion of heterogeneous information by constructing a multi-source sensor confidence matrix. Addressing the shortcomings of traditional evidence theory's combination rules in high-conflict scenarios, an adaptive conflict factor adjustment mechanism based on entropy weighting is introduced: when the inter-sensor conflict factor K > 0.9, a Yager weighted synthesis strategy is adopted; when K ∈ [0.6, 0.9], an evidence discount factor λ = 1 - 0.5k is introduced for correction. This algorithm effectively solves the misjudgment problem caused by environmental interference such as sunlight reflection (e.g., misidentifying mirror reflection as a metallic drone), and can improve feature matching accuracy by 32% in complex lighting scenarios.

[0211] The optimized confidence aggregation formula for the DS evidence theory is as follows:

[0212]

[0213] Where, m final (A) represents the final fused Basic Probability Assignment (BPA), indicating the confidence level of proposition A (e.g., "identified as a drone") after the fusion of multimodal (visible light, infrared, multispectral) evidence. Conflict Factor Resolve misjudgments caused by sensor conflicts (such as misidentifying sunlight reflection as a drone). RGB (B) represents the original basic probability assignment for the visible light modality, indicating the confidence level of the single-modal evidence for proposition B. Thermal (C) represents the original basic probability assignment for the infrared modality, indicating the confidence level of the single-modal evidence for proposition C. Spectral (D) represents the original basic probability assignment of the multispectral modalities, indicating the confidence level of single-modal evidence for proposition D. A, B, C, and D are propositions in the recognition framework (such as target type, scene state, etc., which need to be defined in conjunction with the specific task, for example, A is "is a drone", B is "not a drone", etc.). B∩C∩A is the intersection matching condition of the multimodal propositions, requiring the recognition results of visible light (B), infrared (C), and multispectral (implicit association D) to be logically compatible with the final proposition A (the intersection is not empty), and is used to screen "consistent evidence". or or The conflict condition represents a logical contradiction between multimodal evidence (with no intersection), and is used to calculate the conflict factor K.

[0214] Notably, the system constructs an LSTM-based environmental perception network. By analyzing 12-dimensional environmental parameters such as temperature, humidity, light intensity, and visibility in real time, it dynamically generates sensor weight allocation strategies. In high-temperature scenarios, infrared sensors are more reliable due to the significant temperature difference between the target and the background, and their weight is dynamically increased to 0.7. Thermodynamic models verify the penetrating advantage of infrared imaging in the 8-14μm band, further increasing the infrared channel weight to 0.7. In dense fog environments, a particle scattering model enhances the weight ratio of millimeter-wave radar. This mechanism employs an online learning architecture, updating the weight matrix every 30 minutes to ensure rapid system response to environmental changes (response latency <200ms). In strong light environments, the weight of visible light sensors is suppressed to reduce overexposure interference. This weight adjustment is based on real-time environmental parameters (such as temperature, visibility, and light intensity) and is automatically completed through preset physical characteristic functions without manual intervention.

[0215] Step 4011: Based on the deep learning model and tracking prediction algorithm, perform UAV identification and early warning on the target fusion data to generate early warning data.

[0216] Furthermore, step 4011 also includes the following sub-steps:

[0217] S31. Target detection is performed on the target fusion data using the YOLOv8 detector to generate target detection data.

[0218] Furthermore, step S31 also includes the following sub-steps:

[0219] S311. Extract multi-scale features of target fusion data using the YOLOv8 multi-head detection architecture;

[0220] S312. Using the detection head, CIoU Loss is used to optimize the bounding box and combine it with the angle prediction branch to perform target detection on multi-scale features and generate target localization boxes.

[0221] S313. Perform model identification and threat level classification on multi-scale features using a classification head to generate attribute data;

[0222] S314. Update the initial bounding box using attribute data to generate the target bounding box.

[0223] The YOLOv8 multi-head detection architecture refers to the multi-branch detection head structure designed by YOLOv8 for multi-scale target detection. It contains 3 to 4 feature map output branches, each corresponding to a different downsampling factor (4×, 8×, 16×, 32×), covering the detection needs of targets from small to large.

[0224] The Detection Head specifically refers to the decoupled head module in YOLOv8 that is responsible for bounding box regression and class prediction, as opposed to the "coupled head" in YOLOv5 (which outputs objectivity, class, and bounding box simultaneously).

[0225] CIoU Loss optimization of bounding boxes refers to YOLOv8 using the CompleteIoU loss function to optimize bounding box regression, solving the problem that traditional IoU Loss is insensitive to "center point offset, aspect ratio, and aspect ratio".

[0226] The Class Head refers to the class prediction branch in YOLOv8 that is independent of the regression head. It adopts a task decoupling + soft label design and outputs the class probability distribution instead of the hard classification result.

[0227] In embodiments of the present invention, such as Figure 12 As shown, in the edge-cloud collaborative architecture, the YOLOv8 multi-head detection engine is deployed on edge computing nodes as a core component of the multi-source heterogeneous data processing pipeline. This module directly receives spatiotemporally aligned multimodal data (visible light / infrared / multispectral), and through lightweight design and multi-scale feature fusion, achieves real-time high-precision detection of UAV targets, providing structured input for subsequent threat assessment. The specific generation process of the target localization box is as follows: input target fusion data (the fusion result of spatiotemporally aligned multimodal data, corresponding to the "input image" step in the figure), relying on the built-in process of the YOLOv8 detector:

[0228] (1) Deep feature extraction is performed on the fused data through the CSPDarknet53 backbone network to mine multi-dimensional texture and contour information (such as UAV shape and thermal radiation features) of RGB images, thermal radiation and spectral data.

[0229] (2) SPPF Spatial Pyramid Pooling: Multi-scale aggregation of the output features of the backbone network enhances the feature representation of small targets (such as low-altitude micro UAVs) and solves the problem of "feature loss" for small targets in traditional detection.

[0230] (3) PANet Feature Pyramid: Through the feature fusion of "top-down + bottom-up", shallow detailed features (such as the outline edge of the drone) and deep semantic features (such as the abstract features of the "drone" category) complement each other in both directions, thereby improving the robustness of localization;

[0231] (4) Detection head branch: Based on the fused multi-scale features, the target localization box is directly output (corresponding to "Detection head branch → target localization box" in the figure), realizing the accurate spatial positioning of UAV in multi-modal fused data.

[0232] (5) Classification head branch: Perform model identification and threat level classification on multi-scale features to obtain model identification and threat level classification data, and construct target detection data.

[0233] It is worth mentioning that the multi-task joint recognition engine used in this invention includes:

[0234] Detection Model: A multi-head detection architecture based on YOLOv8 is used. This architecture includes multiple branches (such as different downsampling layers) and can simultaneously detect targets at multiple scales (from "micro-drones" to "large drones"). It extracts multi-scale features from target fusion data, adapting to the feature diversity of multimodal fusion data (RGB images, infrared thermal radiation, spectral data) (such as local features of infrared hotspots and global contours of RGB images), thus solving the problem of "missed detections due to large differences in target scale" in complex scenarios. The YOLOv8-based multi-head detection architecture achieves collaborative optimization of target detection, model recognition, and threat level classification, supporting continuous tracking of drone targets with speeds ≥30m / s, with an end-to-end latency ≤200ms (measured with Jetson AGX).

[0235] The backbone network consists of CSPDarknet53 and PANet feature pyramids. CSPDarknet53 employs cross-stage local connectivity (CSP) to optimize gradient propagation, reducing computational cost while enhancing the extraction of complex multimodal features (such as the "thermal radiation distribution" of a drone in infrared and the "body texture" in RGB). The PANet feature pyramids construct a bidirectional feature propagation mechanism (shallow details → deep semantics, deep semantics → shallow details), allowing the model to simultaneously capture both "drone edge contours" (shallow layer) and "category abstract features (such as the semantics of 'drone')" (deep layer).

[0236] Loss functions: Focal Loss (α′=0.8, γ′=2); FL(p t )=-α t (1-p t ) γ′ log(p t ), where α′ is the positive and negative sample weight balance coefficient; γ′ is the modulation factor for hard-to-classify samples; p t α refers to the probability that a predicted sample is of the positive class. t The sample class weight coefficients; log(p) t The logarithmic loss term quantifies the difference between prediction and reality, addressing the problem of negative sample dominance.

[0237] Data augmentation: CycleGAN is used to generate rain / fog / nighttime samples, expanding the training set by 3.8 times. CycleGAN is a style transfer algorithm without paired data, which can generate drone samples in harsh environments such as "rain / fog" and "nighttime" (without real-world sampling), making up for the lack of "complex environment samples" in real-world scenarios.

[0238] The identification model is an EfficientNet-B4 + Transformer classifier. EfficientNet-B4 is a lightweight network based on a composite scaling strategy (simultaneously scaling depth, width, and resolution), which can efficiently extract high-dimensional features from multimodal fusion data, balancing computational cost and feature representation capability, and adapting to the computing power limitations of edge computing nodes. The Transformer classifier uses a self-attention mechanism to capture long-range dependencies in feature vectors (such as the correlation between infrared thermal radiation features and RGB texture features), improving the accuracy of drone model recognition (10 categories) and threat level classification (low / medium / high).

[0239] Input: 2048-dimensional feature vector;

[0240] Output: Drone model (10 categories) + threat level (low / medium / high), construct attribute data, and combine with target location bounding box to construct target detection data.

[0241] It should be noted that this invention proposes a multi-task joint training mechanism based on YOLOv8, aiming to achieve collaborative optimization of target detection, model recognition, and threat level classification through a unified model architecture. The core innovations of this mechanism are reflected in three aspects: detection head optimization, classification head enhancement, and adversarial training enhancement, which significantly improves model performance and robustness in complex scenarios.

[0242] Detector head optimization: CIoU Loss and angle prediction branch

[0243] To address the limitations of traditional IoU loss functions for rotating targets (such as UAVs), this study introduces CIoU Loss as an alternative to standard IoU, the mathematical expression of which is:

[0244]

[0245] Among them, L CIoU CIoU loss is used to measure the overall difference between the predicted bounding box and the ground truth bounding box. The smaller the value, the closer the predicted box is to the actual drone location. IoU is the intersection-over-union ratio, which refers to the difference between the predicted box b and the ground truth box b. gt The degree of overlap; ρ(b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal length of the minimum bounding rectangle; α” is the balance coefficient, dynamically adjusting the aspect ratio loss weight; v is the aspect ratio consistency index; αν is the penalty term, used to compensate for orientation deviation. In addition, by adding an angle prediction branch, the model can directly output the target's rotation angle, effectively mitigating the bounding box offset problem caused by UAV attitude changes.

[0246] Category Head Enhancement: Multi-Label Category Output Design.

[0247] To achieve joint modeling of model identification and threat level classification, a multi-label classification output layer was designed. This module contains two fully connected layers:

[0248] Model Recognition Branch: Outputs classification results for 10 drone models.

[0249] Threat Level Classification Branch: Outputs three-classification results: low / medium / high threat.

[0250] By sharing the feature extraction layer and separating the task heads, the model improves the efficiency of collaborative training among multiple tasks while ensuring feature consistency.

[0251] Adversarial training enhancement: Focal Loss optimization and dynamic data augmentation.

[0252] To address class imbalance and interference from complex environments, this invention proposes the following enhancement strategies:

[0253] Adaptive Focal Loss: This method dynamically balances the weights of easy and difficult samples by parameterizing α1 and γ1. The formula is:

[0254]

[0255] Where L is the total Focal Loss value, a measure of the overall difference between the model's predictions and the true labels; the smaller the value, the closer the prediction is to the truth. t α1 is the positive and negative sample balance coefficient; γ1 is the modulation factor for hard-to-classify samples, which enhances the model's sensitivity to small targets and fuzzy samples. This represents the cross-entropy loss.

[0256] Dynamic data augmentation strategy: Using operations such as random fogging, night mode, grid occlusion and strong light interference simulation, the model simulates fog, low light and strong light interference scenarios to improve the generalization ability of the model.

[0257] S32. Use target detection data to perform trajectory tracking through a deep learning model to generate continuous trajectory data.

[0258] Furthermore, step S32 also includes the following sub-steps:

[0259] S321. An adaptive IOU matching mechanism is adopted, which combines target detection data and the Hungarian algorithm to associate the target location box with the historical trajectory and generate associated data.

[0260] S322. Extract the multimodal features corresponding to the associated data to generate initial visible light texture features and initial infrared thermal radiation features;

[0261] S323. Extract features from the initial visible light texture features and initial infrared thermal radiation features through two independent ResNet-34 branches to generate target visible light texture features and target infrared thermal radiation features.

[0262] S324. Use a Transformer encoder to perform cross-modal feature alignment between the target's visible light texture features and the target's infrared thermal radiation features, and calculate the cosine similarity between the features;

[0263] S325. Compare the cosine similarity with the preset similarity threshold to generate multimodal feature matching data;

[0264] S326. Using the Kalman filter algorithm, multimodal feature matching data and corresponding bounding box coordinate data are used to predict the position at the next moment, generating position prediction data;

[0265] S327. Use the actual coordinate data at the current moment to correct the position prediction data and update the trajectory status data to generate continuous trajectory data.

[0266] In this embodiment of the invention, the DeepSORT tracking algorithm optimization provides an adaptive IOU matching mechanism. Addressing the trajectory association errors that may occur in traditional DeepSORT when the target is moving at high speed, a dynamic IOU threshold adjustment strategy based on target speed is proposed. The core idea of ​​this mechanism is to appropriately relax the IOU matching threshold when the target's speed exceeds a set threshold (e.g., 20 m / s) to enhance the tracking robustness for high-speed targets. Specifically, the algorithm dynamically adjusts the matching parameters according to the real-time speed—if the target is at high speed, the IOU threshold is increased from the default 0.5 to a value calculated using a speed function (e.g., the faster the speed, the higher the threshold), thereby avoiding mismatches caused by excessive target displacement. Experiments show that this strategy can improve the trajectory association accuracy for high-speed targets by 12.3%, significantly improving tracking stability in complex scenarios.

[0267] Multimodal ReID Feature Fusion: To address the performance degradation of single-modal features in low visibility conditions (such as fog or nighttime), a dual-modal feature fusion network was constructed. This network contains two independent ResNet-34 branches:

[0268] Visible light branch: Extracting texture and color features from visible light images;

[0269] Infrared branch: Captures the thermal radiation characteristics of targets, enhancing adaptability to low-light conditions.

[0270] The feature vectors of the two branches are aligned across modalities using a TransformerEncoder, and complementary information is fused using an attention mechanism to generate more discriminative global features. This design enables the system to maintain 92.7% target ID consistency and achieve an intrusion prediction accuracy of 98.7% even under limited visible light conditions (such as visibility below 100 meters), effectively improving tracking continuity in complex environments.

[0271] The intrusion detection and prediction module includes an LSTM trajectory prediction model, which establishes an intrusion probability prediction model based on historical trajectories. Its state transition equation is as follows:

[0272] h t =tanh(W hx x t +W hh h t-1 );

[0273] p t =σ(W ph h t );

[0274] Among them, h t The hidden state vector at the current time step t stores the fusion encoding of temporal features and environmental information; tanh is the hyperbolic tangent activation function with an output range of [-1, 1], introducing a nonlinear transformation; W hx Input feature x t to hidden state h t The weight matrix; x t W is the multimodal input feature vector at the current time step t; hh The hidden state h from the previous moment t-1 To the current hidden state h t The weight matrix; h t-1 p is the hidden state vector of the previous time step t-1; t σ is the output probability value at the current time step t, ranging from [0, 1], representing "intrusion risk" or "trajectory prediction confidence"; σ is the Sigmoid activation function, with an output range of [0, 1], mapping the hidden state to probability values; W ph For the hidden state h t To output probability p t The weight matrix.

[0275] The system employs a sliding window mechanism (retaining the most recent 50 frames of trajectory data) and a CUDA-accelerated parallel computing architecture to achieve real-time prediction with a 30ms cycle. An electronic fence is triggered when the prediction probability is ≥0.9.

[0276] It should be noted that the dynamic trajectory warning system includes: (1) DeepSORT improved tracking algorithm; (2) motion model: Kalman filter to predict position (x,y,z); (3) data association: Hungarian algorithm + adaptive IOU threshold (0.3~0.7); (4) feature matching: ReID network to extract appearance features (cosine similarity>0.85).

[0277] By dynamically adjusting the IOU matching threshold (e.g., automatically widening or tightening based on the target's motion state) and combining it with the Hungarian algorithm, the optimal association between the current bounding box and historical trajectories is achieved, resulting in associated data and resolving the mismatch problem of traditional algorithms in high-speed motion scenarios. Visible light texture features and infrared thermal radiation features are extracted from the associated data to obtain initial visible light texture features and initial infrared thermal radiation features. Then, deep feature mining is performed using independent ResNet-34 networks to generate more discriminative target feature vectors, i.e., target visible light texture features and target infrared thermal radiation features. The attention mechanism of the Transformer encoder is used to semantically align the visible light and infrared features, and feature matching verification is achieved through cosine similarity calculation. The preset similarity threshold, as a key decision parameter, needs to be determined through extensive experimental data during the system training phase. Its role is to balance the accuracy and fault tolerance of multimodal feature matching—too high a threshold will lead to missed detections, while too low a threshold will introduce incorrect associations. The multimodal feature matching data and bounding box coordinates are input into a Kalman filter, combined with the target motion model (e.g., velocity, acceleration) to predict the position at the next moment, forming a preliminary trajectory estimate and obtaining position prediction data. The predicted position is corrected using the actual detected coordinates at the current moment, and trajectory parameters (such as position and velocity) are dynamically adjusted through a feedback mechanism to ensure trajectory continuity. This process is iterated cyclically, enabling the system to maintain stable tracking even when the target's motion state changes abruptly (such as acceleration or occlusion), thus obtaining continuous trajectory data.

[0278] S33. Calculate the intrusion probability corresponding to continuous trajectory data by combining the LSTM trajectory prediction model with the environmental compensation algorithm.

[0279] In this embodiment of the invention, the environmental compensation algorithm introduces a wind speed and visibility correction model:

[0280]

[0281] Where ΔP is the intrusion probability correction / risk increment, a "dynamic adjustment value" of intrusion risk based on speed-distance coordination, used to correct the base intrusion probability; Δ is the weight of wind speed, α = 0.15; β is the weight of visibility, β = 0.2; V ω Wind speed; The maximum wind speed threshold, D v Visibility; The minimum visibility threshold, This compensation mechanism reduces forecast errors by 18.6% under complex weather conditions.

[0282] The formula for calculating the intrusion probability is:

[0283] InvasionProb=σ(W*[v x ,vy ,v z ,Δt]+b);

[0284] Where InvasionProb is the drone intrusion probability, which is the risk probability of a drone entering a no-fly zone output by the model, and its value ranges from [0, 1]; σ is the Sigmoid activation function; W is the weight matrix; Δt is the time interval, which is the time difference between two adjacent frames of trajectory data; b is the bias term, which is the trainable bias parameter of the LSTM model.

[0285] Input: the velocity v of the trajectory over the past 50 frames x ,v y ,v z , direction angle θ, height z.

[0286] Output: Intrusion probability P∈[0,1], (if P>0.9 and the intrusion enters the electronic fence, trigger countermeasures).

[0287] Technical performance: End-to-end latency ≤200ms (Jetson AGX test), supports continuous tracking of drone targets with speed ≥30m / s.

[0288] S34. When the intrusion probability is greater than the preset probability threshold and the target in the continuous trajectory data enters the electronic fence, the warning signal and countermeasure command corresponding to the intrusion probability are encapsulated into a preset format to generate warning data.

[0289] The preset probability threshold is a critical value for intrusion risk set in advance by the system. It is used to quantify "what probability of an intrusion trend should trigger an early warning". Its core function is to balance "false alarms" and "false negatives".

[0290] In this embodiment of the invention, when the intrusion probability calculated by the LSTM trajectory prediction model combined with the environmental compensation algorithm exceeds a preset threshold, and continuous trajectory data shows that the target has entered the electronic fence (preset geographical boundary), the system will encapsulate the warning signal (associated with the drone model and threat level) and countermeasure instructions corresponding to the intrusion probability in a preset format (such as JSON) to generate warning data, forming a complete identification and warning closed loop.

[0291] Step 4012: Transmit the target fusion data and the corresponding abnormal event logs to the cloud server, and update the deep learning model using the target fusion data and abnormal event logs.

[0292] In this embodiment of the invention, if a hardware failure, data anomaly, or business logic error is detected, the anomaly event log (including timestamp, error code, and context information) will be uploaded to the cloud platform, i.e., the cloud server, in real time. This provides the cloud platform with a basis for fault diagnosis, supporting root cause analysis and system optimization. Based on historical data and real-time feedback, the cloud platform dynamically generates new business strategies (such as threshold adjustments and algorithm parameter optimizations) and distributes them to edge nodes via configuration files. The deep learning model is updated using target fusion data and anomaly event logs, achieving dynamic adaptation of edge-side strategies and enhancing the system's responsiveness to environmental changes or business needs. Through the aforementioned bidirectional data flow, the system can reduce redundant data transmission at the edge, centrally process complex tasks in the cloud, and optimize resource allocation; the model and strategies can be rapidly iterated and updated, ensuring real-time performance.

[0293] In embodiments of the present invention, such as Figure 13 As shown, this invention constructs a real-time intrusion detection and early warning system based on the YOLOv8 detector, the improved DeepSORT tracking algorithm, the LSTM trajectory prediction model, and the hierarchical response mechanism.

[0294] The system adopts a layered architecture design, achieving closed-loop control from target identification to threat handling through multi-module collaboration. The YOLOv8 detector takes image sequences as input and outputs detection results including target bounding box coordinates (x, y, w, h) and confidence (conf), providing basic data for subsequent tracking. The improved DeepSORT algorithm introduces an adaptive IOU matching mechanism, dynamically adjusting the trajectory association threshold for high-speed targets (the threshold is increased from 0.5 to 0.7 when the speed is >20m / s), and fuses visible light and infrared features through a multimodal ReID network, using the Transformer attention mechanism to align cross-modal information, effectively improving tracking robustness in complex scenarios. The LSTM trajectory prediction model takes historical trajectory data as input, calculates the target intrusion probability (P) through a sliding window mechanism (retaining 50 frames of trajectory), and corrects prediction errors by combining wind speed and visibility compensation models. When the target intrusion probability P≥0.9, an electronic fence is triggered, initiating countermeasures such as GNSS jamming or UAV capture nets.

[0295] In this embodiment of the invention, an intelligent sensing and countermeasure system for multi-source heterogeneous data is constructed. Its core technical architecture covers the entire chain optimization from hardware synchronization to cloud decision-making. At the hardware layer, by combining the IEEE 1588v2 precise time protocol with FPGA hardware timestamps, microsecond-level time synchronization (error ≤1ms) of multimodal sensors such as visible light, infrared, and multispectral sensors is achieved. Combined with a coaxial mounting bracket and the OpenCV checkerboard calibration algorithm, the spatial coordinate alignment error is controlled within 0.5 pixels. To address the transmission requirements of massive heterogeneous data, the system uses the Zstandard algorithm to achieve a 4:1 compression ratio and combines it with a Kafka message queue to construct an edge-cloud asynchronous transmission channel, ensuring a data loss rate of less than 0.03% while meeting real-time requirements.

[0296] At the data fusion level, the system constructs a three-level adaptive fusion architecture: at the data level, an LSTM network is used to dynamically generate sensor weight allocation strategies, and environmental adaptive adjustment is achieved through a physical rule base (e.g., automatically increasing the infrared weight to 0.6 in hazy scenarios); at the feature level, a Transformer self-attention mechanism is used to select discriminative features, eliminating 43% redundant dimensions while retaining 98.7% information fidelity; at the decision level, the DS evidence theory framework is improved, and an entropy weight method is introduced to dynamically adjust the conflict factor, effectively solving the misjudgment problem caused by sensor conflicts. In the deep learning-driven identification and early warning closed loop, a multi-head detection architecture based on YOLOv8 achieves collaborative optimization of target detection, model recognition, and threat classification, supporting continuous tracking of high-speed targets at 30m / s; the improved DeepSORT algorithm improves trajectory association accuracy by 12.3% by matching ReID features with an adaptive IOU threshold (0.3-0.7); combined with an LSTM intrusion probability model and an environmental compensation algorithm, the prediction error under complex meteorological conditions is reduced by 18.6%.

[0297] It is worth mentioning the data-level dynamic weighted fusion: sensor weights (α+β+γ=1) are dynamically generated through an LSTM network, supporting adaptive enhancement for abnormal scenarios (e.g., increasing the infrared weight to 0.6 in hazy weather, and increasing the multispectral weight to 0.5 in night mode). Experiments show that this mechanism can improve feature matching accuracy by 32% and reduce recall decay by 40% in complex lighting scenarios.

[0298] Feature-level attention fusion: Based on the Transformer self-attention mechanism, discriminative features are selected, and redundant dimensions are eliminated by 43% through the Top-50% threshold screening, while maintaining 98.7% of the original information fidelity.

[0299] Decision-level evidence fusion optimization: An improved DS evidence theory framework is introduced, and the conflict factor is dynamically adjusted by the entropy weight method to effectively solve the misjudgment problem caused by sensor conflict (such as misidentifying sunlight reflection as a drone), and significantly improve the reliability of decision-making.

[0300] The system possesses environmental adaptability and fault tolerance capabilities, responding in real-time to changes in environmental parameters such as light intensity, temperature, and humidity through a dynamic weight allocation mechanism (e.g., increasing the infrared weight to 0.7 in high-temperature scenarios and suppressing the visible light weight in strong light environments to ensure system performance stability under extreme conditions). In the event of hardware synchronization failure, it can degrade to software time synchronization (10ms accuracy), meeting the requirements of the IEC 61158 standard. In the edge-cloud collaborative architecture, edge nodes perform multimodal data fusion and local inference, uploading feature data to the cloud via the MQTT protocol; the cloud platform supports the dynamic distribution of TensorRTINT8 quantized models, enabling global trajectory prediction and threat assessment. The tiered degradation strategy includes redundancy solutions such as activating the infrared thermal imager when visible light fails and switching to RGB-thermal fusion when multispectral failure occurs, ensuring that the system can maintain basic functionality even when some modules fail. Ultimately, the intrusion response system driven by the LSTM prediction model can trigger countermeasures such as electronic fences and GNSS jamming when the target intrusion probability is ≥0.9, with a response latency controlled within 80ms, forming a complete closed loop of identification-early warning-countermeasure, improving response efficiency by 32% compared to traditional solutions. This achieves high precision, high robustness, and high real-time performance in the UAV identification and early warning system, providing reliable technical support for proactive defense in complex scenarios, demonstrating significant technological advancement and broad application prospects.

[0301] It is worth mentioning that this invention deeply integrates the high-resolution characteristics of visible light imaging with the all-weather detection advantages of infrared thermal imaging: the visible light camera, with its high frame rate and high color fidelity image acquisition capabilities, can quickly capture the outline and identification information of the drone in well-lit environments; the infrared thermal imager utilizes the thermal radiation generated by the drone's power system and electronic components to penetrate visual blind spots such as rain, fog, and darkness, achieving stable target tracking. When encountering extreme environments such as smoke and dust, the multispectral imaging module can effectively distinguish the drone from background clutter by analyzing the electromagnetic radiation characteristics of different wavebands.

[0302] After processing by intelligent fusion algorithms, multi-source data enables the system to construct a multi-dimensional information model that includes the target's spatial location, trajectory, and physical characteristics. Visible light images are used to accurately identify details such as the drone's model and paint scheme, infrared thermal images assist in locating the target's heat source, and multispectral data enhances the detection capability against complex backgrounds. This collaborative detection mode of "vision + thermal sensing + spectroscopy" improves detection range by more than 30% compared to other solutions, achieves an accuracy rate of up to 98%, and reduces the false alarm rate to below 5%. It not only overcomes the limitations of single-sensor applications but also achieves a comprehensive improvement in detection range, recognition accuracy, and environmental adaptability, providing reliable technical support for the intelligent management and control of drones in no-fly zones.

[0303] This solution boasts strong adaptability to various scenarios. Around airports, it can monitor drone intrusions in real time, ensuring the safety of civil aviation takeoffs and landings and preventing "black flight" incidents from interfering with flights. In important restricted areas, it can quickly identify suspicious drones, preventing the leakage of information in sensitive areas. At large-scale events, such as concerts and sporting events, it can promptly detect drones that illegally film or maliciously intrude, maintaining on-site order and protecting public privacy. At energy facilities such as nuclear power plants and oil and gas fields, it can effectively prevent safety accidents caused by drones.

[0304] The application of this multi-source fusion photoelectric identification solution has significantly improved the dynamic control capabilities of no-fly zones, effectively safeguarding the safety of critical facilities and the lives and property of the public. With the rapid development of the low-altitude economy, this technology system will demonstrate greater application value and technological potential in fields such as smart air traffic control and security monitoring, becoming an important technological foundation for the future refined management of low-altitude airspace.

[0305] Example 3

[0306] Please see Figure 14 , Figure 14 This is a structural block diagram of a drone identification and early warning system according to Embodiment 3 of the present invention.

[0307] A drone identification and early warning system includes:

[0308] The data acquisition module 1401 is used to acquire RGB image data, thermal radiation data and spectral data of the no-fly zone for drones through a visible light camera, an infrared thermal imager and a multispectral imager;

[0309] The data preprocessing module 1402 is used to transmit and preprocess RGB image data, thermal radiation data and spectral data to generate target RGB image data, target thermal radiation data and target spectral data.

[0310] The target data fusion generation module 1403 is used to take target RGB image data, target thermal radiation data and target spectral data, combine real-time environmental parameters to adaptively adjust the fusion weight, and perform multi-level data fusion to generate target fused data;

[0311] The early warning data generation module 1404 is used to perform drone identification and early warning based on target fusion data using a deep learning model and tracking prediction algorithm, and generate early warning data.

[0312] The cloud collaboration module 1405 is used to transmit the target fusion data and the corresponding abnormal event logs to the cloud server, and to update the deep learning model using the target fusion data and abnormal event logs.

[0313] In this embodiment of the invention, a drone monitoring system is provided that integrates synchronous acquisition of multi-source heterogeneous data, intelligent fusion, real-time identification, and dynamic early warning. This system can execute all the steps of Embodiments 1 and 2, and aims to achieve high-precision detection and threat assessment of drones. This system adopts an edge-cloud collaborative architecture, achieving high-precision detection, threat assessment, and dynamic early warning of drones through synchronous acquisition of multi-source heterogeneous data, intelligent fusion, and real-time decision-making.

[0314] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying and issuing early warnings for unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: RGB image data, thermal radiation data, and spectral data of no-fly zones for drones were collected using visible light cameras, infrared thermal imagers, and multispectral imagers. The RGB image data, thermal radiation data, and spectral data are transmitted and preprocessed to generate target RGB image data, target thermal radiation data, and target spectral data. Using the target RGB image data, the target thermal radiation data, and the target spectral data, the fusion weights are adaptively adjusted in conjunction with real-time environmental parameters, and multi-level data fusion is performed to generate target fused data; Based on deep learning models and tracking prediction algorithms, drone identification and early warning are performed on the target fusion data to generate early warning data. The target fusion data and the corresponding abnormal event logs are transmitted to the cloud server, and the deep learning model is updated using the target fusion data and the abnormal event logs.

2. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of transmitting and preprocessing the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data includes: The RGB image data, the thermal radiation data, and the spectral data are validated to generate a validation dataset. The data in the verification dataset is filtered for outliers to generate a filtered dataset; The filtered dataset is compressed to generate a compressed dataset, which is then asynchronously transmitted to the edge nodes via a Kafka message queue. Feature extraction is performed on the compressed dataset through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

3. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 2, characterized in that, The compressed dataset includes visible light data, infrared data, and multispectral data; The step of extracting features from the compressed dataset through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data includes: The visible light data is subjected to distortion correction and adaptive histogram equalization to generate initial RGB image data; The infrared data is converted into physical temperature values ​​and non-uniformity correction is performed to generate initial thermal radiation data; The multispectral data is radiometrically corrected, and the normalized vegetation index is calculated to generate initial spectral data. The initial RGB image data, the initial thermal radiation data, and the initial thermal radiation data are spatiotemporally aligned to generate intermediate RGB image data, intermediate thermal radiation data, and intermediate thermal radiation data. The intermediate RGB image data, the intermediate thermal radiation data, and the intermediate thermal radiation data are compressed respectively, and feature extraction is performed through the edge nodes to generate target RGB image data, target thermal radiation data, and target spectral data.

4. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 2, characterized in that, The step of transmitting and preprocessing the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data further includes: When abnormal features appear in the target RGB image data, the target thermal radiation data, and the target spectral data, pattern matching of the abnormal features is performed using a preset rule base to generate abnormal type data. When the abnormal data type is communication interruption, a preset retry processing operation is performed; When the abnormal data type is data anomaly, a preset redundancy processing operation is performed; When the abnormal data type is synchronous loss, a preset degradation mode strategy is executed.

5. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of generating target fused data by using the target RGB image data, the target thermal radiation data, and the target spectral data, adaptively adjusting the fusion weights in conjunction with real-time environmental parameters, and performing multi-level data fusion includes: By combining a long short-term memory network with a preset physical rule base and real-time environmental parameters, dynamic weights corresponding to the target RGB image data, the target thermal radiation data, and the target spectral data are dynamically generated respectively. The target RGB image data, the target thermal radiation data, and the target spectral data are weighted and fused according to the dynamic weights to generate initial fused data; The initial fused data is subjected to multimodal feature extraction, feature filtering and fusion to generate intermediate fused data; By combining the improved DS evidence theory with the preliminary decision data corresponding to the intermediate fused data and the environment adaptive voting weight, feature fusion is performed on the intermediate fused data to generate the target fused data.

6. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 5, characterized in that, The step of extracting multimodal features from the initial fused data, and then performing feature filtering and fusion to generate intermediate fused data includes: Visible light modal features were extracted from the initial fused data using a lightweight YOLOv7-Edge model. Infrared modal features in the initial fused data are extracted by combining a residual network with a convolutional block attention module. Multispectral modal features in the initial fused data are extracted by jointly modeling with a one-dimensional convolutional neural network and a support vector machine. The visible light modal features, the infrared modal features, and the multispectral modal features are concatenated to generate a high-dimensional vector; The importance score of each dimension in the high-dimensional vector is calculated using a self-attention mechanism; Select the dimension data corresponding to the importance scores that meet the preset score threshold, and construct discriminative features.

7. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of generating early warning data by performing UAV identification and early warning on the target fusion data based on a deep learning model and tracking prediction algorithm includes: Target detection data is generated by performing target detection on the target fusion data using a YOLOv8 detector. The target detection data is used to perform trajectory tracking through a deep learning model to generate continuous trajectory data. The intrusion probability corresponding to the continuous trajectory data is calculated by combining the LSTM trajectory prediction model with the environmental compensation algorithm. When the intrusion probability is greater than a preset probability threshold and a target in the continuous trajectory data enters the electronic fence, the warning signal and countermeasure command corresponding to the intrusion probability are encapsulated into a preset format to generate warning data.

8. The method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 7, characterized in that, The step of performing target detection on the target fusion data using a YOLOv8 detector to generate target detection data includes: Multi-scale features of the target fusion data are extracted using the YOLOv8 multi-head detection architecture; The detection head uses CIoU Loss to optimize the bounding box and combines it with the angle prediction branch to perform target detection on the multi-scale features and generate target localization boxes. The multi-scale features are used to perform model identification and threat level classification through a classification head to generate attribute data. Target detection data is constructed using the target bounding box and the attribute data.

9. A method for identifying and issuing early warning for unmanned aerial vehicles (UAVs) according to claim 7, characterized in that, The step of using the target detection data to perform trajectory tracking through a deep learning model and generate continuous trajectory data includes: An adaptive IOU matching mechanism is adopted, which combines the target detection data and the Hungarian algorithm to associate the target location box with the historical trajectory to generate associated data; Extract the multimodal features corresponding to the associated data to generate initial visible light texture features and initial infrared thermal radiation features; The initial visible light texture features and the initial infrared thermal radiation features are extracted by two independent ResNet-34 branches to generate target visible light texture features and target infrared thermal radiation features. The target visible light texture features and the target infrared thermal radiation features are aligned across modes using a Transformer encoder, and the cosine similarity between the features is calculated. The cosine similarity is compared with a preset similarity threshold to generate multimodal feature matching data; The Kalman filter algorithm is used to predict the position at the next moment by using the multimodal feature matching data and the corresponding bounding box coordinate data, thereby generating position prediction data. The position prediction data is corrected using the actual coordinate data at the current moment, and the trajectory status data is updated to generate continuous trajectory data.

10. A drone identification and early warning system, characterized in that, include: The data acquisition module is used to acquire RGB image data, thermal radiation data, and spectral data of no-fly zones for drones using a visible light camera, an infrared thermal imager, and a multispectral imager. The data preprocessing module is used to transmit and preprocess the RGB image data, the thermal radiation data, and the spectral data to generate target RGB image data, target thermal radiation data, and target spectral data; The target data fusion generation module is used to use the target RGB image data, the target thermal radiation data, and the target spectral data, and adaptively adjust the fusion weights in combination with real-time environmental parameters, and perform multi-level data fusion to generate target fused data; The early warning data generation module is used to perform UAV identification and early warning on the target fusion data based on a deep learning model and a tracking prediction algorithm, and generate early warning data. The cloud collaboration module is used to transmit the target fusion data and the corresponding abnormal event logs to the cloud server, and to update the deep learning model using the target fusion data and the abnormal event logs.

Images