Abnormal behavior analysis and prediction system and method based on unmanned aerial vehicle monitoring system

Abstract

The application belongs to the technical field of public safety monitoring, and specifically discloses an abnormal behavior analysis and prediction system and method based on a UAV monitoring system, which comprises a feature extraction module, extracts scene global and object visual and text semantic features through panoramic segmentation and CLIP, an interaction module models space-time correlation through space-time pooling and attention mechanism, a judgment planner splits abnormal related features and adversarial features and calibrates them, and a prediction decoder outputs future abnormal probability by guiding double-path attention with a query vector. The application realizes a full-process closed loop from video stream input to abnormal probability output, breaks through the limitation of traditional methods that can only detect behaviors that have occurred, improves the early warning timeliness through end-to-end analysis, and is suitable for high dynamic monitoring scenes.

Description

Technical Field

[0001] This invention belongs to the field of public safety monitoring technology, and specifically relates to an abnormal behavior analysis and prediction system and method based on UAV monitoring system. Background Technology

[0002] With the upgrading of public safety and security requirements, traditional fixed cameras are unable to cope with the monitoring of abnormal crowd behavior in open areas such as squares and stations due to their limited field of view and insufficient coverage of dynamic scenes. Drones have become a supplementary means with their flexible patrol capabilities, but real-time analysis of video streams still faces core technical bottlenecks: on the one hand, crowd behavior has temporal dynamism, spatial interactivity and semantic multimodality, and existing methods rely on single visual features or rule-based models, which cannot accurately depict complex behavioral patterns; on the other hand, monitoring data has observational uncertainty, and existing algorithms lack effective calibration for this type of information, resulting in delayed anomaly prediction, high false alarm and false negative rates, and difficulty in supporting the practical needs of "proactive early warning".

[0003] In high-density scenarios such as airports and commercial districts, the aforementioned shortcomings directly restrict the effectiveness of security response: traditional systems often miss the intervention window because they "only identify anomalies that have occurred," such as only alerting when fighting escalates, or misjudging normal behavior (such as identifying normal queuing as gathering) due to "insufficient feature correlation." Existing technologies cannot adapt to the highly dynamic and multi-interference monitoring environment of drones, and are also difficult to overcome the limitations of single-modal and single-dimensional analysis. There is an urgent need to achieve real-time perception and proactive early warning of abnormal behavior through systematic model design. Summary of the Invention

[0004] The purpose of this invention is to overcome the deficiencies in the existing technology and provide an abnormal behavior analysis and prediction system and method based on UAV monitoring system.

[0005] A first aspect of the present invention provides an abnormal behavior analysis and prediction system based on an unmanned aerial vehicle (UAV) monitoring system, comprising:

[0006] The feature extraction module is used to extract multimodal features from the original video frame sequence to obtain scene semantic information and object semantic information;

[0007] The interaction module, based on scene semantic information and object semantic information, uses a dual-path parallel model to model the global temporal dynamics and local spatial semantic relationships, thus obtaining the temporal interaction code. Spatial interactive coding ;

[0008] Determining the planner: used for the time-interactive encoding Uncertainty estimation is performed, weights are generated to separate anomaly-related and adversarial features, and spatial interaction encoding is achieved through cross-attention. Correlate and output the calibrated features;

[0009] Predictive decoder: Used to map calibrated features to probabilistic predictions of future anomalous behavior.

[0010] A further approach is to use global features for the scene semantic information. The semantic information of the object includes visual features. Semantic text features ;

[0011] The feature extraction module includes an object encoder and a scene encoder;

[0012] The object encoder generates a panoramic segmentation mask based on a panoramic segmentation model, extracts object bounding boxes and category labels, and extracts object visual features through a CLIP visual encoder. Semantic text features are generated using the CLIP text encoder. ;

[0013] The scene encoder extracts global features from the entire frame image based on the CLIP visual encoder. And stacked along the time dimension.

[0014] A further embodiment is that the interaction module includes a time interaction module and an object interaction module;

[0015] The time interaction module is configured to analyze the visual features of objects. Spatial pooling is performed, and global features are associated with cross-temporal attention mechanisms. Generate time-interactive codes based on object motion patterns. ;

[0016] The object interaction module is configured to respond to visual features. Semantic text features Temporal pooling is performed, and visual features are fused using cross-modal attention. Semantic text features Generate spatial interactive coding .

[0017] A further embodiment is that the decision planner includes:

[0018] Uncertainty estimator: Based on a first-level multilayer perceptron and a sigmoid function, relevant and adversarial weights are generated, and time-interactive encoding is performed based on these weights. It is broken down into anomaly-related features and adversarial features;

[0019] Feature association unit: spatial interactive encoding of anomaly-related features and adversarial features respectively. Perform cross-attention calculation and output the calibrated features.

[0020] A further embodiment is that the prediction decoder includes:

[0021] Dual-path cross-attention layer: Guided by the query vector, cross-attention calculations are performed on the calibrated anomaly-related features and adversarial features respectively to generate two-path attention results;

[0022] Classifier: After concatenating the two attention results, the abnormal behavior category probability is output through a second multilayer perceptron and a Softmax activation function.

[0023] A second aspect of the present invention provides a method for abnormal behavior analysis and prediction based on an unmanned aerial vehicle (UAV) monitoring system. The method, using the aforementioned system, includes the following steps:

[0024] S1. Obtain the video frame sequence captured by the drone, perform multimodal feature extraction on the original video frame sequence, and obtain scene semantic information and object semantic information;

[0025] S2. Based on scene semantic information and object semantic information, a dual-path parallel model is used to model the global temporal dynamics and local spatial semantic associations to obtain the temporal interaction code. Spatial interactive coding ;

[0026] S3. Encode the time interaction. Uncertainty estimation is performed, weights are generated to separate anomaly-related and adversarial features, and spatial interaction encoding is achieved through cross-attention. Correlate and output the calibrated features;

[0027] S4. Map the calibrated features to probability predictions of future anomalous behavior.

[0028] A further solution is that S1 includes: processing a single frame image using a panoramic segmentation model to generate an object-level panoramic segmentation mask, and outputting the bounding box coordinates and category label of each object;

[0029] The CLIP model's visual encoder extracts features from the bounding box region of each object, generating visual features. ;

[0030] The CLIP model's text encoder converts object category labels into semantic text features. ;

[0031] The CLIP visual encoder is used to extract global features from the entire frame image, generating global features. ;

[0032] The global features Visual features Semantic text features After being aligned by time step, the data is passed to the interaction module to provide multimodal data support for spatiotemporal interaction analysis.

[0033] A further solution is that, in step S2, time-interactive coding... The acquisition process is as follows:

[0034] Along the object dimension Average pooling is performed to compress the features of multiple objects into a single vector, generating aggregated motion features. The aggregation motion characteristics The generation process is as follows:

[0035] ;

[0036] in, For time steps, For feature dimension, Representing the visual features of objects, these are feature vectors extracted by a visual encoder for each detected object in each frame of the video. Represents average pooling, subscript Indicates the first Average pooling is performed across each dimension. Represents the set of real numbers;

[0037] With global features For querying, with Using a cross-temporal attention mechanism to capture global temporal dependencies and generate temporal interaction codes, the system uses a key and a value set as keys. :

[0038] ;in, These are the query, key, and value, respectively. Represents the cross-attention mechanism;

[0039] The spatial interactive coding The acquisition process is as follows:

[0040] Visual features of objects and semantic text features Temporal pooling is performed along the time dimension t to preserve the object appearance and semantic features of keyframes, generating... and The and These are visual features and semantic text features The result of time pooling along the time dimension t;

[0041] by For querying, with Using a cross-modal attention mechanism to fuse visual and textual semantics as keys and values, spatial interaction codes are generated. :

[0042] ;in, Represents cross-modal attention mechanisms and indicates visual features of objects. Query semantic text to achieve cross-modal fusion of visual appearance and text labels.

[0043] A further embodiment is that S3 includes:

[0044] Uncertainty weight generation: Constructing the first multilayer perceptron to encode time interactions. The input to the first multilayer perceptron is transformed nonlinearly, and the probability weight P is output through the Sigmoid function, thus obtaining the adversarial weight 1-P.

[0045] Feature decomposition: Encoding time interaction By performing element-wise multiplication with the probability weight P and the adversarial weight 1-P respectively, the abnormal correlation features and adversarial features are obtained.

[0046] Cross-attention association: using anomaly-related features and adversarial features as queries, respectively, and spatial interaction encoding. Using the key and value as keys, the cross-attention mechanism is used to calculate and output calibrated anomaly-related features and adversarial features.

[0047] A further embodiment is that S4 includes:

[0048] Construct query vectors: Initialize dimensions as follows query vector Where M is the preset number of future time windows, and d1 is the feature dimension;

[0049] Dual-path cross-attention calculation:

[0050] by For the query, the calibrated anomaly-related features are the key and value, and the first-path attention features are generated through a cross-attention mechanism. ;

[0051] by For the query, the calibrated adversarial features are the key and value, and a second attention feature is generated through a cross-attention mechanism. ;

[0052] Multimodal feature fusion: and By concatenating the features along the feature dimensions, we obtain the fused features. ;

[0053] Probability distribution generation: through a second multilayer perceptron Mapped to unnormalized class scores Where C is the total number of abnormal behavior categories; after processing with the Softmax activation function, the probability distribution of abnormal behaviors in each future time window is output. .

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] This invention extracts semantic information of scene and object through a feature extraction module and combines it with the probability prediction mechanism of a predictive decoder to achieve end-to-end analysis from environmental perception to behavior prediction. This solves the problem that traditional methods can only detect behaviors that have already occurred, improves the timeliness of early warning, and realizes closed-loop processing of the entire process from drone video stream input to abnormal probability output, adapting to the real-time and system requirements of high dynamic monitoring scenarios.

[0056] This invention achieves precise object-level segmentation through panoramic segmentation. Combined with CLIP's visual-textual bimodal feature alignment, it constructs a bimodal object representation with visual and textual semantics, providing a high-resolution semantic data foundation for subsequent analysis and solving the problems of ambiguous object localization and semantic loss in traditional methods.

[0057] This invention employs temporal path spatial pooling of object visual features (suppressing appearance differences), cross-temporal attention to associate scene features with aggregated motion features (such as crowd movement trends) to capture temporal anomalies (such as sudden acceleration), spatial path temporal pooling of object visual and textual features (preserving key static appearances), and cross-modal attention to fuse textual semantics with visual features (such as "knife" labels guiding dangerous behavior recognition) to enhance local interactive reasoning. The temporal path focuses on global dynamics, while the spatial path strengthens object-level semantic associations, jointly addressing the challenge of interference between motion and appearance features in complex environments.

[0058] This invention presents a feature calibration mechanism based on uncertainty weights: Dynamic probability weights are generated using MLP and Sigmoid, decomposing temporal interaction features into anomaly-related features and adversarial features. A dual-path cross-attention approach is employed to associate the anomaly-related and adversarial features with spatial interaction codes, outputting calibrated features. Through dynamic weight allocation and feature decoupling design, anomaly clues and potential risks in normal observations are explicitly distinguished, improving sensitivity to hidden anomalies and avoiding the shortcomings of traditional methods in uncertainty modeling.

[0059] This invention employs a prediction architecture combining dual-path cross-attention and multi-view fusion. Guided by a query vector, it performs cross-attention on calibration anomaly-related features and adversarial features to generate two attention results. The dual-path results are then concatenated and activated using an MLP classifier and Softmax to output the probability distribution of future anomalous behavior. By utilizing a dual-view feature complementarity mechanism and combining dynamic queries to align future time windows, it achieves high-precision prediction of sudden and disguised anomalous behavior, overcoming the limitations of single-feature prediction. Attached Figure Description

[0060] The following figures are for illustrative purposes only and are not intended to limit the scope of the invention, wherein:

[0061] Figure 1 : Overall technical framework diagram of the system of this invention;

[0062] Figure 2 : Block diagram of feature extraction module;

[0063] Figure 3 Interactive module principle block diagram;

[0064] Figure 4 : Block diagram of the planner principle;

[0065] Figure 5 : Block diagram of the predictive decoder principle. Detailed Implementation

[0066] To make the objectives, technical solutions, design methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0067] The demand for video surveillance using drones in public safety management is increasing, such as real-time monitoring of pedestrian and vehicle behavior in scenarios like large-scale events and traffic management. Based on this, this embodiment provides an abnormal behavior analysis and prediction system based on a drone monitoring system. This system offers an innovative solution to these needs, combining the latest deep learning technology to make drones not just data collection tools, but also a crucial component of intelligent decision support. Through in-depth analysis of human behavior, the system can classify behavior and detect anomalies based on real-time video data, thereby predicting future behavior. This capability will undoubtedly greatly enhance the digitalization and intelligence level of public safety management. Its structure is as follows... Figure 1As shown, it includes: a feature extraction module, an interaction module, a decision planner, and a prediction decoder; the feature extraction module performs object-level segmentation on the input video frames using a panoramic segmentation model (such as Mask2Former), generates bounding boxes and category labels (such as "person", "bag", etc.), and uses the CLIP visual encoder to extract the visual features of the objects. Simultaneously, the category labels are transformed into semantic text features using the CLIP text encoder. This provides multimodal data support for subsequent analysis by extracting global features from the entire frame image through a scene encoder. It captures environmental dynamics (such as changes in lighting and crowd distribution). The fused scene, object visual, and text features are then passed to the interaction module. The temporal interaction module compresses object dimensions through spatial pooling and combines a cross-temporal attention mechanism to associate scene features with object motion patterns (such as crowd movement trends) to generate temporal interaction codes. The object interaction module retains key appearance information through temporal pooling and utilizes cross-modal attention to fuse object visual features with textual semantics (such as the association between "knife" and "theft") to generate spatial interaction codes. The planner estimates the uncertainty of the time-interactive encoding and generates weights using an MLP and a sigmoid function. Separate features related to the anomaly and antagonistic characteristics Then, through cross-attention and spatial interaction coding, the calibrated features are dynamically associated and output. and The predictive decoder uses the query vector. The calibration features are subjected to dual-path cross-attention computation to capture anomaly-related cues (such as crowd gathering) and adversarial patterns (such as sudden stillness) respectively. After concatenation, the features are passed through an MLP classifier and a Softmax activation function to output the class probability of future abnormal behavior. The entire system achieves efficient identification and proactive early warning of abnormal behavior in complex scenarios through multimodal fusion, uncertainty modeling, and dual-path prediction.

[0068] like Figure 2 As shown, the feature extraction module includes an object encoder and a scene encoder, and the input is the original RGB video frame sequence captured by the UAV.

[0069] The object encoder uses the Mask2Former panoramic segmentation model to process single-frame images, outputting object bounding box coordinates and category labels (such as "person" and "bag"). The CLIP visual encoder extracts the visual features of each bounding box region. in, For time steps, For the number of objects, For feature dimensions;

[0070] Category labels are transformed into semantic text features using the CLIP text encoder. .

[0071] Scene encoder: Uses CLIP visual encoder to extract global features from the entire frame image. It captures environmental information such as lighting and population distribution.

[0072] Output global features Visual features Semantic text features Pass it to the interaction module.

[0073] like Figure 3 As shown, the interaction module includes a time interaction module and an object interaction module;

[0074] Time-based interaction module: Visual features of objects Along the object dimension Spatial pooling (average pooling):

[0075] ;

[0076] With global features For the Query, after pooling For Key / Value pairs, generate temporal interaction codes through cross-temporal attention:

[0077] ;

[0078] Through attention mechanisms, global features Dynamic focus is placed on temporal patterns related to motion (such as a sudden acceleration of a crowd). It's important to note that in attention mechanisms, the " / " in Key / Value means "and," indicating that the same set of data is used as both a key and a value.

[0079] Secondly, "Key," "Value," and "Query" together constitute the three core elements of the attention mechanism, and their roles and functions can be compared to an information retrieval system:

[0080] Query: Represents the party that initiates the query, specifying what kind of information is needed.

[0081] Key: Represents the index of the database being retrieved, used to match the query and calculate the relevance score.

[0082] Value: Represents "the specific information content retrieved from the database", which is the actual data that is ultimately extracted and aggregated.

[0083] Object interaction module: Input visual features and semantic text features Time pooling (max pooling) along the time dimension t:

[0084] ;

[0085] by For Query, For Key / Value pairs, spatial interaction codes are generated through cross-modal attention:

[0086] ;

[0087] Enhance semantic associations, such as associating "knife" with "theft".

[0088] like Figure 4 As shown, the decision planner includes an uncertainty estimator, a feature association unit, and a loss constraint unit. The uncertainty estimator uses time-interactive encoding. The first multilayer perceptron and the sigmoid function are used to generate the relevant weights P and the adversarial weights 1-P; the formula is as follows: , The sigmoid function compresses the output to [0,1]. This represents the probability of correlation between the observation at each time step and future anomalies;

[0089] Time-interactive coding is performed based on relevant weight P and adversarial weight 1-P. Decomposed into anomaly-related features and antagonistic characteristics :

[0090] ;

[0091] Feature association unit:

[0092] by , For Query, For Key / Value pairs, calibrated features are output via dual-path cross-attention:

[0093] ;

[0094] ;

[0095] in, The interaction of objects related to association and anomalies (such as the spatial position of a "knife" in a fight). Capture confrontational patterns that may be overlooked in normal contexts (e.g., "sudden stillness in the crowd" may foreshadow riots);

[0096] and The acquisition process is as follows: let the extracted anomaly-related features (May contain genuine anomalous signals) Despatial interactive coding The semantics of the associated objects are searched to obtain calibrated anomaly-related features. At the same time, let the adversarial features be separated. (Potentially contains interfering or spoofed signals) also undergoes spatial interactive coding. The correlation is performed to obtain the calibrated adversarial features. .

[0097] The loss constraint unit constructs loss function constraints. and This ensures the model clearly distinguishes between anomaly-related features and adversarial features; the loss function is:

[0098] ;

[0099] in, To predict the decoder's prediction results, force One of them is related to the prediction result Alignment, and suppression, ensure that the model clearly distinguishes between anomaly-related features and adversarial features.

[0100] like Figure 5 As shown, the prediction decoder includes a dual-path cross-attention layer and a classifier;

[0101] Features after uncertainty calibration and query vector ( Perform bipath cross-attention for the number of future time windows:

[0102] = CrossAttention ；

[0103] C = CrossAttention ；

[0104] Classification prediction:

[0105] splicing characteristics: ;

[0106] The anomaly probability distribution is output through a second multilayer perceptron and Softmax:

[0107] ， ( (This refers to the number of abnormal categories, such as "theft" or "fighting").

[0108] For each query The category scores are normalized to generate a probability distribution. :

[0109] ;

[0110] Based on this, this embodiment also discloses a method for abnormal behavior analysis and prediction based on a drone monitoring system, including the following steps:

[0111] S1. Obtain the video frame sequence captured by the drone, perform multimodal feature extraction on the original video frame sequence, and obtain scene semantic information and object semantic information;

[0112] S2. Based on scene semantic information and object semantic information, a dual-path parallel model is used to model the global temporal dynamics and local spatial semantic associations to obtain the temporal interaction code. Spatial interactive coding ;

[0113] S3. Encode the time interaction. Uncertainty estimation is performed, weights are generated to separate anomaly-related and adversarial features, and spatial interaction encoding is achieved through cross-attention. Correlate and output the calibrated features;

[0114] S4. Map the calibrated features to probability predictions of future anomalous behavior, and output the probability distribution. .

[0115] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An abnormal behavior analysis and prediction system based on a drone monitoring system, characterized in that, include: The feature extraction module is used to extract multimodal features from the original video frame sequence to obtain scene semantic information and object semantic information; The interaction module, based on scene semantic information and object semantic information, uses a dual-path parallel model to model the global temporal dynamics and local spatial semantic relationships, thus obtaining the temporal interaction code. Spatial interactive coding ; Determining the planner: used for the time-interactive encoding Uncertainty estimation is performed, weights are generated to separate anomaly-related and adversarial features, and spatial interaction encoding is achieved through cross-attention. Correlate and output the calibrated features; Predictive decoder: used to map calibrated features to probabilistic predictions of future anomalous behavior; The interaction module includes a time interaction module and an object interaction module; The time interaction module is configured to analyze the visual features of objects. Spatial pooling is performed, and global features are associated with cross-temporal attention mechanisms. Generate time-interactive codes based on object motion patterns. ; The object interaction module is configured to respond to visual features. Temporal pooling is performed, and visual features are fused using cross-modal attention. Generate spatial interactive coding .

2. The abnormal behavior analysis and prediction system based on the UAV monitoring system according to claim 1, characterized in that, The scene semantic information is a global feature. The semantic information of the object includes visual features. ; The feature extraction module includes an object encoder and a scene encoder; The object encoder generates a panoramic segmentation mask based on a panoramic segmentation model, extracts object bounding boxes and category labels, and extracts object visual features through a CLIP visual encoder. Generated via CLIP text encoder ; The scene encoder extracts global features from the entire frame image based on the CLIP visual encoder. And stacked along the time dimension.

3. The abnormal behavior analysis and prediction system based on the UAV monitoring system according to claim 1, characterized in that, The decision planner includes: Uncertainty estimator: Based on a first-level multilayer perceptron and a sigmoid function, relevant and adversarial weights are generated, and time-interactive encoding is performed based on these weights. It is broken down into anomaly-related features and adversarial features; Feature association unit: spatial interactive encoding of anomaly-related features and adversarial features respectively. Perform cross-attention calculation and output the calibrated features.

4. The abnormal behavior analysis and prediction system based on the UAV monitoring system according to claim 1, characterized in that, The prediction decoder includes: Dual-path cross-attention layer: Guided by the query vector, cross-attention calculations are performed on the calibrated anomaly-related features and adversarial features respectively to generate two-path attention results; Classifier: After concatenating the two attention results, the abnormal behavior category probability is output through a second multilayer perceptron and a Softmax activation function.

5. A method for analyzing and predicting abnormal behavior based on an unmanned aerial vehicle (UAV) monitoring system, characterized in that, The system according to any one of claims 1-4 is characterized by the following steps: S1. Obtain the video frame sequence captured by the drone, perform multimodal feature extraction on the original video frame sequence, and obtain scene semantic information and object semantic information; S2. Based on scene semantic information and object semantic information, a dual-path parallel model is used to model the global temporal dynamics and local spatial semantic associations to obtain the temporal interaction code. Spatial interactive coding ; S3. Encode the time interaction. Uncertainty estimation is performed, weights are generated to separate anomaly-related and adversarial features, and spatial interaction encoding is achieved through cross-attention. Correlate and output the calibrated features; S4. Map the calibrated features to a probability prediction of future abnormal behavior; In S2, time-interactive coding The acquisition process is as follows: Along the object dimension Average pooling is performed to compress the features of multiple objects into a single vector, generating aggregated motion features. The aggregation motion characteristics The generation process is as follows: ； in, For time steps, For feature dimension, Representing the visual features of objects, these are feature vectors extracted by a visual encoder for each detected object in each frame of the video. Represents average pooling, subscript Indicates the first Average pooling is performed across each dimension. Represents the set of real numbers; With global features For querying, with Using a cross-temporal attention mechanism to capture global temporal dependencies and generate temporal interaction codes, the system uses a key and a value set as keys. : ;in, These are the query, key, and value, respectively. Represents the cross-attention mechanism; The spatial interactive coding The acquisition process is as follows: Visual features of objects and Temporal pooling is performed along the time dimension t to preserve the object appearance and semantic features of keyframes, generating... and The and These are visual features and The result of time pooling along the time dimension t; by For querying, with Using a cross-modal attention mechanism to fuse visual and textual semantics as keys and values, spatial interaction codes are generated. : ;in, Represents cross-modal attention mechanisms and indicates visual features of objects. Query semantic text to achieve cross-modal fusion of visual appearance and text labels.

6. The abnormal behavior analysis and prediction method based on an unmanned aerial vehicle (UAV) monitoring system according to claim 5, characterized in that, S1 includes: processing a single frame image using a panoramic segmentation model to generate an object-level panoramic segmentation mask, and outputting the bounding box coordinates and category label of each object; The CLIP model's visual encoder extracts features from the bounding box region of each object, generating visual features. ; The text encoder of the CLIP model converts object category labels into... ; The CLIP visual encoder is used to extract global features from the entire frame image, generating global features. ; The global features Visual features After being aligned by time step, the data is passed to the interaction module to provide multimodal data support for spatiotemporal interaction analysis.

7. The method for abnormal behavior analysis and prediction based on an unmanned aerial vehicle (UAV) monitoring system according to claim 5, characterized in that, S3 includes: Uncertainty weight generation: Constructing the first multilayer perceptron to encode time interactions. The input to the first multilayer perceptron is transformed nonlinearly, and the probability weight P is output through the Sigmoid function, thus obtaining the adversarial weight 1-P. Feature decomposition: Encoding time interaction By performing element-wise multiplication with the probability weight P and the adversarial weight 1-P respectively, the abnormal correlation features and adversarial features are obtained. Cross-attention association: using anomaly-related features and adversarial features as queries, respectively, and spatial interaction encoding. Using the key and value as keys, the cross-attention mechanism is used to calculate and output calibrated anomaly-related features and adversarial features.

8. The method for abnormal behavior analysis and prediction based on an unmanned aerial vehicle (UAV) monitoring system according to claim 5, characterized in that, S4 includes: Construct query vectors: Initialize dimensions as follows query vector Where M is the preset number of future time windows, and d1 is the feature dimension; Dual-path cross-attention calculation: by For the query, the calibrated anomaly-related features are the key and value, and the first-path attention features are generated through a cross-attention mechanism. ; by For the query, the calibrated adversarial features are the key and value, and a second attention feature is generated through a cross-attention mechanism. ; Multimodal feature fusion: and By concatenating the features along the feature dimensions, we obtain the fused features. ; Probability distribution generation: through a second multilayer perceptron Mapped to unnormalized class scores Where C represents the total number of abnormal behavior categories; after processing with the Softmax activation function, the probability distribution of abnormal behaviors in each future time window is output. .

Images