An unmanned aerial vehicle intelligent patrol method and system based on edge-cloud cooperation

Abstract

This invention provides an intelligent UAV patrol method and system based on edge-cloud collaboration in the field of UAV perception and artificial intelligence. The method includes: Step S1, the UAV collects visible light video and infrared video and inputs them into a multimodal fusion perception model to obtain fusion perception results; Step S2, an edge-cloud collaboration mechanism is automatically triggered based on uncertainty scoring, or control commands are generated; Step S3, the ground station gathers the basic target data carrying temporal information transmitted back by the UAV and inputs it into a spatiotemporal attention model to obtain atomic event recognition results, which are then uploaded to the server; Step S4, the server inputs the atomic event recognition results and image slices uploaded by the edge-cloud collaboration mechanism into an anomaly event detection model to obtain anomaly event detection results. The advantages of this invention are: it greatly improves the UAV's accurate perception capability throughout the entire time in wide-area complex scenarios, the interpretability and understanding capability of long-term events, and the dynamic collaborative efficiency of edge-cloud resources.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of drone perception and artificial intelligence, and in particular to a drone intelligent patrol method and system based on edge-cloud collaboration. Background Technology

[0002] As a crucial component of the low-altitude economy, unmanned aerial vehicle (UAV) intelligent patrol technology is being widely applied in complex scenarios such as urban security patrols, disaster emergency rescue, and cross-border border monitoring. With the development of artificial intelligence and edge computing, UAV systems are gradually shifting from the traditional "remote-controlled flight and filming" paradigm to a "fully autonomous intelligent perception" paradigm, aiming to achieve all-weather, high-precision, and long-term automated operations. However, this transformation faces multiple challenges, including the dynamic nature of unstructured environments, the diversity of perceived targets, and limited onboard computing resources. Therefore, technological innovation is urgently needed to improve the system's environmental adaptability, event understanding capabilities, and resource scheduling efficiency.

[0003] Currently, intelligent drone patrol technology mainly relies on single-modal perception (such as visible light or infrared imaging) and isolated edge computing modes. At the perception level, while visible light (RGB) sensors can provide rich texture information, they are prone to failure under low light, smoke, or reflective conditions; infrared thermal imaging (IR), while able to compensate for some environmental deficiencies, is susceptible to interference from thermal cross-effects, making it difficult to achieve high-precision target detection independently. In terms of computing architecture, existing systems mostly adopt isolated edge or cloud processing modes: edge devices, due to limited computing resources, can only run lightweight models, resulting in insufficient perception accuracy in complex scenarios; while cloud-side processing, although powerful, suffers from high bandwidth consumption and latency in full data transmission, failing to meet real-time alarm requirements. Furthermore, for understanding long-term events (such as illegal sand mining and continuous pollution discharge), traditional methods rely on rule engines or deep learning black-box models, lacking interpretability and making it difficult to adapt to industry business rules.

[0004] Although existing technologies have achieved basic target detection, their core problems can be summarized into the following three aspects: 1. Poor environmental adaptability: Single-modal sensing technology cannot dynamically cope with unstructured environmental changes such as sudden changes in lighting and smoke obstruction in wide-area scenarios, leading to an increased false detection rate. For example, in water conservancy inspections, smoke and dust may cause visible light sensors to malfunction, while thermal cross-effects affect the accuracy of infrared sensors.

[0005] 2. Difficulty in understanding long-term events: Isolated edge computing models can only process short-term data and lack the ability to continuously reason about behavioral logic. Existing methods struggle to analyze the temporal relationships of complex events such as "loitering" and "violations of regulations," and the interpretability of deep learning models is insufficient, making it impossible to trace the causes of anomalies.

[0006] 3. Low edge-cloud collaboration efficiency: There is a lack of dynamic scheduling mechanism between resource-constrained edge devices and cloud computing power. Traditional offloading strategies based on fixed rules cannot adaptively balance computing power and accuracy, resulting in bandwidth waste or response delays, making it difficult to support collaborative operations of large-scale drone swarms.

[0007] The aforementioned problems highlight the shortcomings of existing UAV patrol technologies in terms of perception fusion, temporal reasoning, and system architecture. Therefore, how to provide a UAV intelligent patrol method and system based on edge-cloud collaboration to improve the UAV's accurate perception capabilities across all time periods in wide-area and complex scenarios, the interpretability and comprehensibility of long-term events, and the dynamic collaborative efficiency of edge-cloud resources has become an urgent technical problem to be solved. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method and system for intelligent inspection of UAVs based on edge-cloud collaboration, so as to improve the UAV's accurate perception capability in wide-area complex scenarios, the interpretability and understanding capability of long-term events, and the dynamic collaborative efficiency of edge-cloud resources.

[0009] In a first aspect, the present invention provides an intelligent patrol method for unmanned aerial vehicles (UAVs) based on edge-cloud collaboration, comprising the following steps: Step S1: Create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model; train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model; and compress the trained multimodal fusion perception model. Step S2: Deploy the compressed multimodal fusion perception model to the UAV on the edge, deploy the trained spatiotemporal attention model to the ground station on the edge, and deploy the trained abnormal event detection model to the server on the cloud. Step S3: During the patrol, the UAV collects visible light video and infrared video in real time, and inputs the visible light video and infrared video into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. Step S4: The UAV automatically triggers the end-to-cloud collaboration mechanism based on the uncertainty score, or generates corresponding control commands; Step S5: The ground station gathers the basic targets carrying time sequence information transmitted back by the UAV, inputs each basic target into the deployed spatiotemporal attention model, obtains the atomic event recognition results, and uploads the atomic event recognition results to the server; Step S6: The server inputs the received atomic event recognition result and the image slices uploaded by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain the abnormal event detection result; Step S7: The server constructs a difficult example dataset based on each of the image slices, and periodically triggers a retraining task based on the difficult example dataset to obtain the model weight file of the multimodal fusion perception model, and sends the model weight file to the UAV for updating.

[0010] Furthermore, in step S1, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , Mapping to the query space, key space, and value space yields the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, _t_ represents transpose; Softmax() represents the normalized exponential function; The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ; ; in, This represents the Sigmoid activation function; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication; The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target location and target type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Indicates the actual target location; Indicates the predicted target location; This represents the regularization term.

[0011] Furthermore, in step S1, the spatiotemporal attention model is constructed based on a temporal attention module, a spatial attention module, and an output classification head; The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention over the time dimension; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; in, Indicates the first Spatial augmentation features of layer output; This indicates multi-head self-attention in spatial dimensions; The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

[0012] Furthermore, in step S1, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module; The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; Cross-entropy loss is represented by λ; weight coefficients are represented by λ. Represents a robust semantic function; This represents the logical loss term.

[0013] Furthermore, step S2 specifically includes: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. Step S4 specifically involves: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

[0014] Secondly, the present invention provides an intelligent drone patrol system based on edge-cloud collaboration, comprising the following modules: The model training module is used to create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model, train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model, and compress the trained multimodal fusion perception model. The model deployment module is used to deploy the compressed multimodal fusion perception model to the edge-side drone, deploy the trained spatiotemporal attention model to the edge-side ground station, and deploy the trained abnormal event detection model to the cloud-side server. The fusion perception module is used by the UAV to collect visible light video and infrared video in real time during the patrol process, and input the visible light video and infrared video into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. The edge-cloud collaboration module is used by the UAV to automatically trigger the edge-cloud collaboration mechanism based on the uncertainty score, or to generate corresponding control commands. The atomic event recognition module is used by the ground station to collect the basic targets carrying time sequence information transmitted back by the UAV, input each of the basic targets into the deployed spatiotemporal attention model, obtain the atomic event recognition results, and upload the atomic event recognition results to the server; An abnormal event detection module is used by the server to input the received atomic event identification results and the image slices uploaded triggered by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain abnormal event detection results. The model update module is used by the server to construct a difficult example dataset based on each of the image slices, periodically trigger a retraining task based on the difficult example dataset, obtain the model weight file of the multimodal fusion perception model, and send the model weight file to the UAV for updating.

[0015] Furthermore, in the model training module, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , Mapping to the query space, key space, and value space yields the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, _t_ represents transpose; Softmax() represents the normalized exponential function; The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ; ; in, This represents the Sigmoid activation function; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication; The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target location and target type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Indicates the actual target location; Indicates the predicted target location; This represents the regularization term.

[0016] Furthermore, in the model training module, the spatiotemporal attention model is constructed based on the temporal attention module, the spatial attention module, and the output classification head; The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention over the time dimension; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; in, Indicates the first Spatial augmentation features of layer output; This indicates multi-head self-attention in spatial dimensions; The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

[0017] Furthermore, in the model training module, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module; The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; Cross-entropy loss is represented by λ; weight coefficients are represented by λ. Represents a robust semantic function; This represents the logical loss term.

[0018] Furthermore, the model deployment module is specifically used for: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. The endpoint-cloud collaboration module is specifically used for: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

[0019] The advantages of this invention are: 1. The multimodal fusion perception model, spatiotemporal attention model, and anomaly detection model are trained. The trained multimodal fusion perception model is compressed and deployed to the edge-side UAV, the trained spatiotemporal attention model is deployed to the edge-side ground station, and the trained anomaly detection model is deployed to the cloud-side server. During patrols, the UAV collects visible light and infrared video in real time and inputs it into the multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. Based on the uncertainty scores, the edge-cloud collaboration mechanism is automatically triggered, or corresponding control commands are generated. The ground station collects the basic targets carrying temporal information transmitted back by the UAV, inputs each basic target into the spatiotemporal attention model to obtain atomic event recognition results, and uploads them to the server. The server inputs the atomic event recognition results and image slices uploaded by the edge-cloud collaboration mechanism into the anomaly detection model to obtain anomaly event detection results. The server constructs a difficult example dataset based on each image slice and periodically triggers a retraining task based on the difficult example dataset to obtain the model weights of the multimodal fusion perception model. The document is distributed to the drone for updates; that is, through a multimodal fusion perception model deployed on the edge (combining cross-attention and adaptive gating fusion of visible light and infrared data), accurate target perception is ensured even in complex environments such as sudden changes in lighting and smoke obstruction, and an uncertainty score is generated as a trigger signal for intelligent collaboration. The edge uses a spatiotemporal attention model to analyze the time-series target data transmitted by the drone to identify atomic events, while the cloud uses an anomaly detection model that integrates neural networks and symbolic logic rules to perform interpretable reasoning on long-term atomic events, thereby transforming black-box prediction into logical judgments that conform to business rules. At the same time, the on-demand triggering mechanism based on uncertainty scores and the dynamic resource scheduling of the KubeEdge framework realize adaptive collaboration of edge and cloud resources, uploading key image slices only when the model uncertainty is high, which greatly optimizes bandwidth and computing power usage. In addition, by building a difficult example dataset on the server and periodically triggering model retraining, a closed loop of continuous optimization is formed, which ultimately greatly improves the drone's full-time accurate perception capability in wide-area complex scenarios, the interpretable understanding capability of long-term events, and the dynamic collaborative efficiency of edge and cloud resources.

[0020] 2. A distributed deployment approach with edge-cloud collaboration is adopted, deploying the multimodal fusion perception model on the edge (drone), the spatiotemporal attention model on the edge (ground station), and the abnormal event detection model on the cloud (server), achieving reasonable distribution of computing tasks. This architecture effectively reduces the computing burden on edge devices, avoids overloading of a single node, and improves the real-time performance and response speed of the overall system by utilizing the high-performance processing capabilities of the edge and cloud. For example, by dynamically managing resources through the KubeEdge framework, resource allocation can be adjusted according to parameters such as drone battery level and CPU temperature, further optimizing energy consumption and stability, making it suitable for large-scale or long-term patrol scenarios.

[0021] 3. The multimodal fusion perception model combines visible light video and infrared video, and achieves deep fusion of RGB and IR features through feature extraction, cross-attention calculation, and adaptive gating mechanism. This design can overcome the limitations of single modality (such as changes in lighting or severe weather) and improve the accuracy of target detection. In addition, the uncertainty perception detection head models uncertainty scoring through Gaussian distribution, enabling the system to quantify the reliability of perception results, provide a basis for subsequent decision-making, thereby reducing false alarms and false negatives and enhancing adaptability in complex environments.

[0022] 4. The UAV automatically triggers the end-to-cloud collaborative mechanism or generates control commands based on uncertainty scoring, which is an innovative adaptive strategy. When the uncertainty score is high, image slices are automatically uploaded to the cloud for in-depth analysis; otherwise, control commands are generated locally. This mechanism avoids unnecessary communication overhead while ensuring timely handling of critical events, improving system flexibility and resource utilization. For example, during patrols, if a blurry or abnormal target is detected, the system can proactively request cloud-side assistance to avoid erroneous decisions due to limited end-to-end capabilities.

[0023] 5. The spatiotemporal attention model captures the temporal dependencies of the basic target and the key regions within each frame through temporal attention and spatial attention modules, respectively, and can accurately identify atomic events (such as movement trajectories or behavioral patterns). This design is superior to traditional static detection methods because it considers the spatiotemporal context, improving the coherence and accuracy of event recognition. For example, in a monitored area, the model can identify continuous action sequences (such as crowds gathering or abnormal vehicle movement), providing more reliable input for abnormal event detection.

[0024] 6. The anomaly detection model not only uses neural networks to extract temporal features, but also integrates a symbolic logic rule base (such as MTL rules) to assist decision-making by calculating rule satisfaction. This hybrid approach combines the advantages of data-driven and knowledge-driven approaches, making the detection results more interpretable and robust. The introduction of the logistic loss function further optimizes the training process, ensuring that the model can maintain high accuracy in complex scenarios (such as rule conflicts or data sparsity) and reducing the risk of misjudgment.

[0025] 7. The server constructs a difficult example dataset based on image slices and periodically triggers retraining tasks to update the weight file of the multimodal fusion perception model. This enables the system to continuously improve itself and adapt to environmental changes or new abnormal patterns. For example, challenging samples encountered during the inspection process (such as occluded targets or new event types) will be included in the training to continuously improve model performance, extend the system's lifespan, and reduce the need for manual intervention.

[0026] 8. The multimodal fusion perception model is compressed before deployment to suit the resource constraints of the edge drone (such as limited computing power and storage space). This lightweight design ensures the efficient operation of the model on the edge without sacrificing core functionality. Combined with Kubernetes microservice management, the system can dynamically adjust model instances, further improving the flexibility and scalability of deployment, and making it suitable for diverse drone platforms. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a flowchart of an intelligent drone patrol method based on edge-cloud collaboration according to the present invention.

[0029] Figure 2 This is a schematic diagram of the structure of an intelligent drone patrol system based on edge-cloud collaboration according to the present invention. Detailed Implementation

[0030] The overall approach of the technical solution in this application is as follows: A multimodal fusion perception model deployed on the edge ensures accurate target perception even in complex environments such as sudden changes in lighting and smoke obstruction, and generates an uncertainty score as a trigger signal for intelligent collaboration. On the edge, a spatiotemporal attention model is used to analyze the time-series target data transmitted by the UAV to identify atomic events. On the cloud, an anomaly detection model that integrates neural networks and symbolic logic rules performs interpretable reasoning on long-term atomic events, thereby transforming black-box prediction into logical judgments that conform to business rules. Simultaneously, an on-demand triggering mechanism based on uncertainty scores and dynamic resource scheduling within the KubeEdge framework enable adaptive collaboration of edge and cloud resources, uploading key image slices only when model uncertainty is high, greatly optimizing bandwidth and computing power usage. Furthermore, a difficult example dataset is constructed on the server, and model retraining is triggered periodically, forming a continuously optimized closed loop to improve the UAV's full-time accurate perception capability in wide-area complex scenarios, its interpretable understanding of long-term events, and the dynamic collaborative efficiency of edge and cloud resources.

[0031] Please refer to Figures 1 to 2 As shown, a preferred embodiment of the present invention, a method for intelligent drone patrol based on edge-cloud collaboration, includes the following steps: Step S1: Create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model; train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model; and compress the trained multimodal fusion perception model. In practice, the INT8 quantization (PTQ) technique is used to compress the multimodal fusion perception model, and the dynamic range of activation values ​​is calculated using the calibration dataset. The FP32 weights are mapped to INT8, achieving a 4x inference speedup with an accuracy loss of <1%.

[0032] Step S2: Deploy the compressed multimodal fusion perception model to the UAV on the edge, deploy the trained spatiotemporal attention model to the ground station on the edge, and deploy the trained abnormal event detection model to the server on the cloud. Step S3: During the patrol, the UAV collects visible light video and infrared video in real time, and inputs the visible light video and infrared video into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets (such as people, vehicles, and ships) and uncertainty scores. Step S4: The UAV automatically triggers the end-to-cloud collaboration mechanism based on the uncertainty score, or generates corresponding control commands; Step S5: The ground station gathers the basic targets carrying time sequence information transmitted back by the UAV, inputs each basic target into the deployed spatiotemporal attention model, obtains atomic event recognition results (such as "someone bends over", "excavator arm moves", "person runs", "vehicle stops"), and uploads the atomic event recognition results to the server. Step S6: The server inputs the received atomic event identification result and the image slices uploaded by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain the abnormal event detection result (i.e., reasoning about complex behaviors with long cycles, such as "illegal sand mining" = "excavator action" + "lasting 20 minutes" + "prohibited mining area"). Step S7: The server constructs a difficult example dataset based on each image slice and manual annotation, and periodically triggers a retraining task based on the difficult example dataset to obtain the model weight file of the multimodal fusion perception model, and sends the model weight file to the UAV for updating.

[0033] In step S1, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , This is mapped to the query space, key space, and value space to obtain the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, represents transpose; Softmax() represents the normalization exponential function; that is, the infrared thermal features are "injected" into the RGB texture features through the attention mechanism; It describes the correlation between each pixel in the RGB feature map and all pixels in the IR feature map; This represents the scaling factor, used to prevent the softmax gradient from vanishing due to an excessively large dot product result. The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ; ; in, This represents the Sigmoid activation function, used to map the output to the (0, 1) interval; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication (Hadamard product). Used to automatically adjust weights in case of smoke (RGB failure) or thermal crossover (IR) failure. The value of each pixel represents whether that location is more RGB (close to 1) or IR (close to 0). The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target location and target type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Indicates the actual target location; Indicates the predicted target location (mean coordinates of the bounding box); The regularization term is used to prevent the model from increasing the uncertainty s indefinitely in order to reduce the first term; s represents the log-variance. .

[0034] To address the single-modal failure problem caused by abrupt changes in illumination, smoke obstruction, and thermal cross-effects in wide-area scenarios, this paper proposes a deep fusion of heterogeneous data from visible light (RGB) and infrared thermal imaging (IR). Based on a gated cross-attention feature fusion network, a learnable modality selection gating factor is constructed to dynamically suppress noise interference from low-quality modalities during the feature extraction stage, achieving complementary enhancement of RGB texture features and IR thermal radiation features. Furthermore, an uncertainty estimation mechanism is introduced, which not only improves detection accuracy in complex environments but also serves as a "confidence probe," automatically triggering a cloud-based collaborative mechanism when the uncertainty perceived at the edge is high.

[0035] In step S1, the spatiotemporal attention model is constructed based on a temporal attention module, a spatial attention module, and an output classification head. The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention in the time dimension, which calculates attention only in the T dimension, i.e., comparing features of the same spatial location in different time frames; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; in, Indicates the first Spatial augmentation features of layer output; This indicates multi-head self-attention in the spatial dimension. It only calculates attention in the N-dimensional dimension, that is, it compares the features of different spatial locations within the same time frame. The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

[0036] In step S1, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module. The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. This involves using Metric Temporal Logic (MTL) to mathematically represent business rules. For example, the "illegal dumping" event is defined as: a truck stops at the riverbank (Event A), and within the following 2 minutes, someone gets off the truck and lingers at the rear of the vehicle for more than 30 seconds (Event B). The corresponding MTL formula ∅ is: ; Where ∧ represents the logical AND operator; is the "eventually" operator in sequential logic. This indicates that it occurs at least once within the time interval [a, b]. This is a "global (Always)" operator in sequential logic, indicating that it continues to occur within the time interval [a, b]. , , Indicates atomic predicates; The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; λ represents the cross-entropy loss; λ represents the weighting coefficient, used to balance the weights of the data-driven loss term and the logical loss term. The robustness semantics function is defined as follows: when the input sequence x satisfies rule ∅, ρ > 0; when the rule is violated, ρ < 0. Its absolute value represents the degree of satisfaction or violation (e.g., time margin from the boundary). The logistic loss term is represented by -ρ > 0 if the model prediction violates the rule (i.e., ρ < 0), resulting in a positive loss value that forces the network to adjust its parameters through gradient descent to satisfy the rule; if the rule is satisfied (ρ > 0), the loss is 0.

[0037] For long-sequence video streams, metric timing logic (MTL) rules are introduced to enable the system to understand complex behaviors such as "wandering" and "illegal continuous operation".

[0038] Interpretable event detection integrating Neuro-Symbolic AI: Innovatively, Metric Temporal Logic (MTL) is introduced into the deep learning framework, transforming industry business rules into differentiable logical loss functions. This not only enables the model to identify anomalies but also allows it to trace back the causes of anomalies through logical formulas, solving the problem of the "black box" nature of traditional deep learning and its lack of interpretability.

[0039] Step S2 specifically involves: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. The microservices are divided as follows: Pod_Detection: A multimodal fusion sensing model that is encapsulated with TensorRT acceleration and runs on the edge.

[0040] Pod_Tracking: Runs the DeepSort tracking algorithm and maintains the target ID.

[0041] Pod_Logic: Encapsulates the MTL inference engine and runs on the cloud.

[0042] Resource scheduling: The K8s Master dynamically adjusts the number of Pod replicas or performs traffic shaping through Service Mesh based on the drone's current battery level, CPU temperature, and network bandwidth.

[0043] Step S4 specifically involves: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

[0044] Uncertainty-Driven Edge-Cloud Collaborative Triggering Mechanism: Unlike traditional offloading strategies based on fixed rules, this invention designs an uncertainty regression branch in the multimodal fusion perception model, outputting an uncertainty score (Aleatoric / Epistemic Uncertainty) for the basic target. When the uncertainty score exceeds the uncertainty threshold, it automatically determines that the current scenario exceeds the capabilities of the edge side, triggering the "edge-side slicing - cloud-side review" process, thus achieving an adaptive balance between computing power and accuracy.

[0045] A preferred embodiment of the present invention provides an intelligent drone patrol system based on edge-cloud collaboration, comprising the following modules: The model training module is used to create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model, train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model, and compress the trained multimodal fusion perception model. In practice, the INT8 quantization (PTQ) technique is used to compress the multimodal fusion perception model, and the dynamic range of activation values ​​is calculated using the calibration dataset. The FP32 weights are mapped to INT8, achieving a 4x inference speedup with an accuracy loss of <1%.

[0046] The model deployment module is used to deploy the compressed multimodal fusion perception model to the edge-side drone, deploy the trained spatiotemporal attention model to the edge-side ground station, and deploy the trained abnormal event detection model to the cloud-side server. The fusion perception module is used by the UAV to collect visible light video and infrared video in real time during the patrol process. The visible light video and infrared video are input into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets (such as people, vehicles, and ships) and uncertainty scores. The edge-cloud collaboration module is used by the UAV to automatically trigger the edge-cloud collaboration mechanism based on the uncertainty score, or to generate corresponding control commands. The atomic event recognition module is used by the ground station to collect the basic targets carrying time sequence information transmitted back by the UAV, input each of the basic targets into the deployed spatiotemporal attention model, obtain atomic event recognition results (such as "someone bends over", "excavator arm moves", "person runs", "vehicle stops"), and upload the atomic event recognition results to the server. The abnormal event detection module is used by the server to input the received atomic event identification results and the image slices uploaded triggered by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain the abnormal event detection results (i.e., reasoning about complex behaviors with long cycles, such as "illegal sand mining" = "excavator action" + "lasting 20 minutes" + "prohibited mining area"). The model update module is used by the server to construct a difficult example dataset based on each image slice and manual annotation, periodically trigger a retraining task based on the difficult example dataset, obtain the model weight file of the multimodal fusion perception model, and send the model weight file to the UAV for updating.

[0047] In the model training module, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , This is mapped to the query space, key space, and value space to obtain the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, represents transpose; Softmax() represents the normalization exponential function; that is, the infrared thermal features are "injected" into the RGB texture features through the attention mechanism; It describes the correlation between each pixel in the RGB feature map and all pixels in the IR feature map; This represents the scaling factor, used to prevent the softmax gradient from vanishing due to an excessively large dot product result. The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ; ; in, This represents the Sigmoid activation function, used to map the output to the (0, 1) interval; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication (Hadamard product). Used to automatically adjust weights in case of smoke (RGB failure) or thermal crossover (IR) failure. The value of each pixel represents whether that location is more RGB (close to 1) or IR (close to 0). The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target location and target type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Indicates the actual target location; Indicates the predicted target location (mean coordinates of the bounding box); The regularization term is used to prevent the model from increasing the uncertainty s indefinitely in order to reduce the first term; s represents the log-variance. .

[0048] To address the single-modal failure problem caused by abrupt changes in illumination, smoke obstruction, and thermal cross-effects in wide-area scenarios, this paper proposes a deep fusion of heterogeneous data from visible light (RGB) and infrared thermal imaging (IR). Based on a gated cross-attention feature fusion network, a learnable modality selection gating factor is constructed to dynamically suppress noise interference from low-quality modalities during the feature extraction stage, achieving complementary enhancement of RGB texture features and IR thermal radiation features. Furthermore, an uncertainty estimation mechanism is introduced, which not only improves detection accuracy in complex environments but also serves as a "confidence probe," automatically triggering a cloud-based collaborative mechanism when the uncertainty perceived at the edge is high.

[0049] In the model training module, the spatiotemporal attention model is constructed based on the temporal attention module, the spatial attention module, and the output classification head; The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention in the time dimension, which calculates attention only in the T dimension, i.e., comparing features of the same spatial location in different time frames; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

[0050] In the model training module, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module. The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. This involves using Metric Temporal Logic (MTL) to mathematically represent business rules. For example, the "illegal dumping" event is defined as: a truck stops at the riverbank (Event A), and within the following 2 minutes, someone gets off the truck and lingers at the rear of the vehicle for more than 30 seconds (Event B). The corresponding MTL formula ∅ is: ; Where ∧ represents the logical AND operator; is the "eventually" operator in sequential logic. This indicates that it occurs at least once within the time interval [a, b]. This is a "global (Always)" operator in sequential logic, indicating that it continues to occur within the time interval [a, b]. , , Indicates atomic predicates; The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; λ represents the cross-entropy loss; λ represents the weighting coefficient, used to balance the weights of the data-driven loss term and the logical loss term. The robustness semantics function is defined as follows: when the input sequence x satisfies rule ∅, ρ > 0; when the rule is violated, ρ < 0. Its absolute value represents the degree of satisfaction or violation (e.g., time margin from the boundary). The logistic loss term is represented by -ρ > 0 if the model prediction violates the rule (i.e., ρ < 0), resulting in a positive loss value that forces the network to adjust its parameters through gradient descent to satisfy the rule; if the rule is satisfied (ρ > 0), the loss is 0.

[0051] For long-sequence video streams, metric timing logic (MTL) rules are introduced to enable the system to understand complex behaviors such as "wandering" and "illegal continuous operation".

[0052] Interpretable event detection integrating Neuro-Symbolic AI: Innovatively, Metric Temporal Logic (MTL) is introduced into the deep learning framework, transforming industry business rules into differentiable logical loss functions. This not only enables the model to identify anomalies but also allows it to trace back the causes of anomalies through logical formulas, solving the problem of the "black box" nature of traditional deep learning and its lack of interpretability.

[0053] The model deployment module is specifically used for: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. The microservices are divided as follows: Pod_Detection: A multimodal fusion sensing model that is encapsulated with TensorRT acceleration and runs on the edge.

[0054] Pod_Tracking: Runs the DeepSort tracking algorithm and maintains the target ID.

[0055] Pod_Logic: Encapsulates the MTL inference engine and runs on the cloud.

[0056] Resource scheduling: The K8s Master dynamically adjusts the number of Pod replicas or performs traffic shaping through Service Mesh based on the drone's current battery level, CPU temperature, and network bandwidth.

[0057] The endpoint-cloud collaboration module is specifically used for: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

[0058] Uncertainty-Driven Edge-Cloud Collaborative Triggering Mechanism: Unlike traditional offloading strategies based on fixed rules, this invention designs an uncertainty regression branch in the multimodal fusion perception model, outputting an uncertainty score (Aleatoric / Epistemic Uncertainty) for the basic target. When the uncertainty score exceeds the uncertainty threshold, it automatically determines that the current scenario exceeds the capabilities of the edge side, triggering the "edge-side slicing - cloud-side review" process, thus achieving an adaptive balance between computing power and accuracy.

[0059] In summary, the advantages of this invention are: 1. The multimodal fusion perception model, spatiotemporal attention model, and anomaly detection model are trained. The trained multimodal fusion perception model is compressed and deployed to the edge-side UAV, the trained spatiotemporal attention model is deployed to the edge-side ground station, and the trained anomaly detection model is deployed to the cloud-side server. During patrols, the UAV collects visible light and infrared video in real time and inputs it into the multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. Based on the uncertainty scores, the edge-cloud collaboration mechanism is automatically triggered, or corresponding control commands are generated. The ground station collects the basic targets carrying temporal information transmitted back by the UAV, inputs each basic target into the spatiotemporal attention model to obtain atomic event recognition results, and uploads them to the server. The server inputs the atomic event recognition results and image slices uploaded by the edge-cloud collaboration mechanism into the anomaly detection model to obtain anomaly event detection results. The server constructs a difficult example dataset based on each image slice and periodically triggers a retraining task based on the difficult example dataset to obtain the model weights of the multimodal fusion perception model. The document is distributed to the drone for updates; that is, through a multimodal fusion perception model deployed on the edge (combining cross-attention and adaptive gating fusion of visible light and infrared data), accurate target perception is ensured even in complex environments such as sudden changes in lighting and smoke obstruction, and an uncertainty score is generated as a trigger signal for intelligent collaboration. The edge uses a spatiotemporal attention model to analyze the time-series target data transmitted by the drone to identify atomic events, while the cloud uses an anomaly detection model that integrates neural networks and symbolic logic rules to perform interpretable reasoning on long-term atomic events, thereby transforming black-box prediction into logical judgments that conform to business rules. At the same time, the on-demand triggering mechanism based on uncertainty scores and the dynamic resource scheduling of the KubeEdge framework realize adaptive collaboration of edge and cloud resources, uploading key image slices only when the model uncertainty is high, which greatly optimizes bandwidth and computing power usage. In addition, by building a difficult example dataset on the server and periodically triggering model retraining, a closed loop of continuous optimization is formed, which ultimately greatly improves the drone's full-time accurate perception capability in wide-area complex scenarios, the interpretable understanding capability of long-term events, and the dynamic collaborative efficiency of edge and cloud resources.

[0060] 2. A distributed deployment approach with edge-cloud collaboration is adopted, deploying the multimodal fusion perception model on the edge (drone), the spatiotemporal attention model on the edge (ground station), and the abnormal event detection model on the cloud (server), achieving reasonable distribution of computing tasks. This architecture effectively reduces the computing burden on edge devices, avoids overloading of a single node, and improves the real-time performance and response speed of the overall system by utilizing the high-performance processing capabilities of the edge and cloud. For example, by dynamically managing resources through the KubeEdge framework, resource allocation can be adjusted according to parameters such as drone battery level and CPU temperature, further optimizing energy consumption and stability, making it suitable for large-scale or long-term patrol scenarios.

[0061] 3. The multimodal fusion perception model combines visible light video and infrared video, and achieves deep fusion of RGB and IR features through feature extraction, cross-attention calculation, and adaptive gating mechanism. This design can overcome the limitations of single modality (such as changes in lighting or severe weather) and improve the accuracy of target detection. In addition, the uncertainty perception detection head models uncertainty scoring through Gaussian distribution, enabling the system to quantify the reliability of perception results, provide a basis for subsequent decision-making, thereby reducing false alarms and false negatives and enhancing adaptability in complex environments.

[0062] 4. The UAV automatically triggers the end-to-cloud collaborative mechanism or generates control commands based on uncertainty scoring, which is an innovative adaptive strategy. When the uncertainty score is high, image slices are automatically uploaded to the cloud for in-depth analysis; otherwise, control commands are generated locally. This mechanism avoids unnecessary communication overhead while ensuring timely handling of critical events, improving system flexibility and resource utilization. For example, during patrols, if a blurry or abnormal target is detected, the system can proactively request cloud-side assistance to avoid erroneous decisions due to limited end-to-end capabilities.

[0063] 5. The spatiotemporal attention model captures the temporal dependencies of the basic target and the key regions within each frame through temporal attention and spatial attention modules, respectively, and can accurately identify atomic events (such as movement trajectories or behavioral patterns). This design is superior to traditional static detection methods because it considers the spatiotemporal context, improving the coherence and accuracy of event recognition. For example, in a monitored area, the model can identify continuous action sequences (such as crowds gathering or abnormal vehicle movement), providing more reliable input for abnormal event detection.

[0064] 6. The anomaly detection model not only uses neural networks to extract temporal features, but also integrates a symbolic logic rule base (such as MTL rules) to assist decision-making by calculating rule satisfaction. This hybrid approach combines the advantages of data-driven and knowledge-driven approaches, making the detection results more interpretable and robust. The introduction of the logistic loss function further optimizes the training process, ensuring that the model can maintain high accuracy in complex scenarios (such as rule conflicts or data sparsity) and reducing the risk of misjudgment.

[0065] 7. The server constructs a difficult example dataset based on image slices and periodically triggers retraining tasks to update the weight file of the multimodal fusion perception model. This enables the system to continuously improve itself and adapt to environmental changes or new abnormal patterns. For example, challenging samples encountered during the inspection process (such as occluded targets or new event types) will be included in the training to continuously improve model performance, extend the system's lifespan, and reduce the need for manual intervention.

[0066] 8. The multimodal fusion perception model is compressed before deployment to suit the resource constraints of the edge drone (such as limited computing power and storage space). This lightweight design ensures the efficient operation of the model on the edge without sacrificing core functionality. Combined with Kubernetes microservice management, the system can dynamically adjust model instances, further improving the flexibility and scalability of deployment, and making it suitable for diverse drone platforms.

[0067] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for intelligent drone patrol based on edge-cloud collaboration, characterized in that: Includes the following steps: Step S1: Create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model; train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model; and compress the trained multimodal fusion perception model. Step S2: Deploy the compressed multimodal fusion perception model to the UAV on the edge, deploy the trained spatiotemporal attention model to the ground station on the edge, and deploy the trained abnormal event detection model to the server on the cloud. Step S3: During the patrol, the UAV collects visible light video and infrared video in real time, and inputs the visible light video and infrared video into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. Step S4: The UAV automatically triggers the end-to-cloud collaboration mechanism based on the uncertainty score, or generates corresponding control commands; Step S5: The ground station gathers the basic targets carrying time sequence information transmitted back by the UAV, inputs each basic target into the deployed spatiotemporal attention model, obtains the atomic event recognition results, and uploads the atomic event recognition results to the server; Step S6: The server inputs the received atomic event recognition result and the image slices uploaded by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain the abnormal event detection result; Step S7: The server constructs a difficult example dataset based on each of the image slices, and periodically triggers a retraining task based on the difficult example dataset to obtain the model weight file of the multimodal fusion perception model, and sends the model weight file to the UAV for updating.

2. The intelligent UAV patrol method based on edge-cloud collaboration as described in claim 1, characterized in that: In step S1, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , Mapping to the query space, key space, and value space yields the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, _t_ represents transpose; Softmax() represents the normalized exponential function; The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ； ； in, This represents the Sigmoid activation function; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication; The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target's location and type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Show the actual target location; Indicates the predicted target location; This represents the regularization term.

3. The intelligent UAV patrol method based on edge-cloud collaboration as described in claim 1, characterized in that: In step S1, the spatiotemporal attention model is constructed based on a temporal attention module, a spatial attention module, and an output classification head. The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention over the time dimension; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; in, Indicates the first Spatial augmentation features of layer output; This indicates multi-head self-attention in spatial dimensions; The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

4. The intelligent UAV patrol method based on edge-cloud collaboration as described in claim 1, characterized in that: In step S1, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module. The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; Cross-entropy loss is represented by λ; weight coefficients are represented by λ. Represents a robust semantic function; This represents the logical loss term.

5. The intelligent UAV patrol method based on edge-cloud collaboration as described in claim 1, characterized in that: Step S2 specifically involves: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. Step S4 specifically involves: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

6. A drone intelligent patrol system based on edge-cloud collaboration, characterized in that: Includes the following modules: The model training module is used to create a multimodal fusion perception model, a spatiotemporal attention model, and an abnormal event detection model, train the multimodal fusion perception model, the spatiotemporal attention model, and the abnormal event detection model, and compress the trained multimodal fusion perception model. The model deployment module is used to deploy the compressed multimodal fusion perception model to the edge-side drone, deploy the trained spatiotemporal attention model to the edge-side ground station, and deploy the trained abnormal event detection model to the cloud-side server. The fusion perception module is used by the UAV to collect visible light video and infrared video in real time during the patrol process, and input the visible light video and infrared video into the deployed multimodal fusion perception model to obtain fusion perception results including basic targets and uncertainty scores. The edge-cloud collaboration module is used by the UAV to automatically trigger the edge-cloud collaboration mechanism based on the uncertainty score, or to generate corresponding control commands. The atomic event recognition module is used by the ground station to collect the basic targets carrying time sequence information transmitted back by the UAV, input each of the basic targets into the deployed spatiotemporal attention model, obtain the atomic event recognition results, and upload the atomic event recognition results to the server; An abnormal event detection module is used by the server to input the received atomic event identification results and the image slices uploaded triggered by the end-cloud collaboration mechanism into the deployed abnormal event detection model to obtain abnormal event detection results. The model update module is used by the server to construct a difficult example dataset based on each of the image slices, periodically trigger a retraining task based on the difficult example dataset, obtain the model weight file of the multimodal fusion perception model, and send the model weight file to the UAV for updating.

7. The intelligent drone patrol system based on edge-cloud collaboration as described in claim 6, characterized in that: In the model training module, the multimodal fusion perception model is constructed based on a feature extraction module, a feature fusion module, and an uncertainty perception detection head; the feature fusion module is constructed based on a feature mapping and projection unit, a cross-attention calculation unit, and an adaptive gating mechanism unit; the uncertainty perception detection head is constructed based on a detection regression unit and an uncertainty regression unit. The feature extraction module is used to extract RGB feature maps and IR feature maps from the input visible light video and infrared video; The feature mapping and projection unit is used to receive the first feature output from the feature extraction module. RGB feature map of the layer and the IR feature map of the layer Through learnable weight matrix , , Mapping to the query space, key space, and value space yields the query tensor. Key tensors and value tensors : ; ; ; in, , , , Let H represent the feature map height, W represent the feature map width, and C represent the number of channels. Indicates the feature dimension after projection; The cross-attention calculation unit is used to query tensors. Key tensors Calculate the attention weight matrix for the input. Then with the value tensor Weighted summation and comparison with the query tensor Residual connection, outputting enhanced RGB feature map : ; ; in, _t_ represents transpose; Softmax() represents the normalized exponential function; The adaptive gating mechanism unit is used to enhance RGB feature maps by using a concatenation and lightweight convolutional network. and IR feature map Generate gating coefficients Based on the gating coefficient Weighted output fusion features : ; ; in, This represents the Sigmoid activation function; This represents the convolutional kernel weights of the gated network; This indicates a splicing operation along the channel dimension; Indicates the bias term; This indicates element-wise multiplication; The detection and regression unit is used to process the fused features through a fully connected layer or a convolutional layer. Perform reasoning and output a basic objective, including the target's location and type. The uncertainty regression unit is used to model the uncertainty of target position regression through Gaussian distribution, obtain uncertainty score, and output fusion perception result carrying the basic target and uncertainty score; The modal sensing loss function of the multimodal fusion sensing model is: ; in, This represents the loss value of the modality-sensing loss function; The value is , This represents the variance of the prediction; the larger the variance, the smaller the penalty weight on the regression error. Indicates the actual target location; Indicates the predicted target location; This represents the regularization term.

8. The intelligent drone patrol system based on edge-cloud collaboration as described in claim 6, characterized in that: In the model training module, the spatiotemporal attention model is constructed based on the temporal attention module, the spatial attention module, and the output classification head; The temporal attention module is used to apply layer normalization to the input base target, then extract temporal dependencies through multi-head self-attention in the temporal dimension, and finally connect it with the input base target residual to output intermediate features: ; in, Indicates intermediate features; This indicates multi-head self-attention over the time dimension; Representation layer normalization; Indicates the first The basic goal of layer input, Let B represent a real number, T represent the number of time frames, N represent the number of spatial patches per frame, and D represent the feature dimension. The spatial attention module is used to perform layer normalization on intermediate features, then focus on key regions within each frame through spatial dimension multi-head self-attention, and finally connect the residuals to the intermediate features to output spatially enhanced features: ; in, Indicates the first Spatial augmentation features of layer output; This indicates multi-head self-attention in spatial dimensions; The output classification head is used to pool along the spatiotemporal dimension into a global feature vector, and then the global feature vector is mapped to atomic action category probabilities through a fully connected layer. Based on the atomic action category probabilities, the atomic event recognition result is output.

9. The intelligent drone patrol system based on edge-cloud collaboration as described in claim 6, characterized in that: In the model training module, the abnormal event detection model is constructed based on a temporal feature encoding module, a symbolic logic integration module, and an event decision output module. The temporal feature encoding module is used to extract long temporal dependent features from atomic event recognition results and image slices through a neural network, and output an enhanced temporal representation; The symbolic logic integration module is used to calculate the rule satisfaction degree by taking time-series features as input and combining them with the MTL rule base. The event decision output module integrates temporal representation and rule satisfaction to output abnormal event detection results; The logistic loss function of the abnormal event detection model is: ; in, This represents the loss value of the logical loss function; Cross-entropy loss is represented by λ; weight coefficients are represented by λ. Represents a robust semantic function; This represents the logical loss term.

10. The intelligent drone patrol system based on edge-cloud collaboration as described in claim 6, characterized in that: The model deployment module is specifically used for: Based on the KubeEdge edge computing framework, the compressed multimodal fusion perception model is deployed to the drone on the edge, the trained spatiotemporal attention model is deployed to the ground station on the edge, and the trained abnormal event detection model is deployed to the server on the cloud. The endpoint and edge are abstracted as Kubernetes edge nodes. The multimodal fusion perception model, spatiotemporal attention model and abnormal event detection model are dynamically managed through Pod microservices, and resource allocation is adjusted according to the drone's battery level, CPU temperature and network bandwidth. The endpoint-cloud collaboration module is specifically used for: The drone determines whether the uncertainty score is greater than a preset uncertainty threshold. If so, it automatically triggers the end-to-cloud collaboration mechanism to upload the image slices corresponding to the uncertainty score in the visible light video and infrared video to the server. If not, it generates corresponding control commands based on the basic target.

Images